Skip to main content

Run a Layer project

Once you add decorators to your functions, you can run them using Layer infrastructure. This makes sense once you:

  • Process large amounts of data.
  • When your training code requires compute resources (See fabrics) greater than what you have locally.

How to run a project

This process uses the layer.run to pass a list of decorated functions that you want to execute to the Layer backend. For more information, refer to layer.run in the SDK library.

  1. Log in to Layer in your notebook.

  2. Initialize a project. For more information, refer to Create a Layer project.

  3. Pass a list of decorated functions to layer.run and then run the cell.

    This triggers following:

    • Layer executes the functions remotely and stores the generated entities in Layer so that you or others can retrieve them later.
    • Any entities cited in a resources decorator are uploaded to Layer.
    • All unit tests with assertions are run.

Example

For example, if we want to generate our dataset in the Layer backend, we would create a dataset, initialize a project, and then run it as shown below.

import layer
import pandas as pd
from layer.decorators import dataset

@dataset("my_products")
def create_product_dataset():
data = [[1, "product1", 15], [2, "product2", 20], [3, "product3", 10]]
dataframe = pd.DataFrame(data, columns=["Id", "Product", "Price"])
return dataframe

# Initialize a project so the entities we create become associated with this project
layer.login()
layer.init("my_example_project")

# Tell layer to execute our decorated dataset function remotely and
# register the returning dataframe as a dataset under Layer
layer.run([create_product_dataset])