Specify requirements in a Layer project
This page describes how to manage requirements in Layer projects. It also provides a guide for how a data scientist can use a requirements file to manage dependencies in a Layer project with Python code.
You can indicate requirements two ways:
- Define requirements when initializing a project with layer.init.
- Define requirements with the pip_requirements around a function.
What is a Python requirements.txt file?
A requirements.txt file in Python is a file that contains all the library information needed to run your project.
New library versions might ship with breaking changes that break your application. Managing dependencies ensures that your data science project doesn’t break when a new version of a library is released.
It is unwise to automatically update your application to use the latest version of a package. Instead, test your application with the new library in a virtual environment. Then you can update your app when you're certain that the new library version won’t break anything.
A Python requirements.txt file might look like this:
Technically, you do not need to include the version numbers. However, we strongly recommend that you include version numbers in all your requirement files. This protects your Layer project from breaking when there are new library versions.
Here's how to create your own requirements.txt file.
List all Python packages
Do not use
pip freeze for generating the
requirements.txt in Google Colab notebooks. Google Colab comes with some pre-installed libraries that are directly loaded from the disk. If you try to run your code remotely in the Layer backend, you are likely to get an error as such libraries won't be found. Instead, we recommend you to specify manually whichever libraries your code require to run.
You can list all installed packages by using the
pip freeze command. This command the libraries installed in the active environment. When you run it, it prints a handy list that you can copy and paste into the requirements.txt file.
Alternatively, you can write the output of
pip freeze directly to a requirements.txt file.
pip freeze > requirements.txt
Make sure that you run
pip freeze while your virtual environment is active. This is important to ensure that you only get the packages used for the current project. Otherwise, you will get a list of the packages in your global environment.
Install packages from a requirements.txt
The requirements file simplifies installing the libraries needed to run a project. This comes in handy when you want to test your application in a different environment or push it to a production environment.
pip install command while referencing the requirements.txt file to install all the packages at once.
pip install -r requirements.txt