How to use Papermill in Google Colab

Happy holidays and welcome to my first story on Medium! This story will be a simple tutorial on how to programmatically modify the source code of a Colab Notebook to make it compatible with Papermill. If you are already familiar with Colab and Papermill, feel free to skip down to the notebook section.

Note to reader: this story assumes you already have familiarity with IPython or Jupyter Notebooks.

Naturally, Papermill has steadily become a tool for many data scientists, both amateur and professional, looking to scale their current notebooks.

Google Colaboratory is an earlier but equally notable addition to the data science toolkit, especially for amateurs. In October of 2017, Google open-sourced their internal notebook platform for data science, specifically oriented towards machine learning. Not only did the platform include Google Drive and Github integrations, but also a cloud environment built for data science with free GPU instances. This was a major game changer for anyone looking to get into data science but lacking the necessary income or resources to support it. I’ve personally been using Colab for some time now as it has relieved me of much of the package management overhead required in small data science projects. Unfortunately, when embarking on my journey to learn Papermill, I discovered that Colab did not natively support Papermill. After an evening of reading and debugging, I was able to figure out the compatibility issues and address them with only a few lines of code.

Using Papermill in a standard Jupyter notebook is a seamless experience.

Note to reader: alternatively, you could use the Github integration but you will need to click ‘Copy to Drive’ and rename them (removing ‘Copy of’ from the title).

Now you will have both notebooks stored inside your Google Drive, inside the path /My Drive/Colab Notebooks. At this point, you are more than welcome to run all the cells in the Colab-Papermill-Driver notebook and infer the changes made to the Sample.ipynb, skipping the rest of this section. If you are interested in learning more, please read on!

Bring the tab containing Colab-Papermill-Driver.ipynb into focus and run the cells below the Install and Import. In the next section, Connect, two connections are created to your Google Drive. This seems redundant but the upDrive object and downDrive object both have unique functions we will need to use in order to parameterize the Colab notebook.

In the navigate section, we change the current directory to the folder where Colab-Papermill-Driver.ipynb and Sample.ipynb should be located. The assert statement verifies you are in the correct folder.

The next section, Select and Modify, is where we begin making changes.

In this section, we will be uploading the parameterized file to Google Drive using PyDrive.

I hope this story was clear and you have not only added a new notebook to your toolkit, but also gained a deeper understanding of notebook development. I wouldn’t call this patch perfected by any means and I encourage anyone reading this to try to create a package such that the target notebook modifies its own JSON source code. To me, Papermill is really just the start of what I like to refer to as the Notebook Oriented Programming movement. It became apparent to most developers very quickly that Notebooks on their own aren’t particularly scalable. Most notably, they lack the abstraction capabilities of traditional Object Oriented Programming (inheritance, polymorphism, etc). The Netflix team seems to be doing a fantastic job with exploring this scalability and I’m excited to see how Metaflow evolves overtime. Again, I hope you had a good read and have happy holidays!

Related posts:

My blogging has gone off the boil this year. With the disappearance of the third place during the pandemic, work and home life have become intermingled. The space for reflection has diminished to the…

be it formal or easygoing, blockchain training is turning into a customary thing in the Philippines in spite of the fact that its utilization in various segments, instruction in itself, still can’t…

What is coral bleaching? Coral bleaching is when water is too warm; corals expel the algae (zooxanthellae) living in their tissues causing the coral to turn completely white. When a coral bleaches…