https://kedro.org/ logo
#questions
Title
# questions
c

Camilo López

06/16/2023, 12:18 PM
Hi Team, I'm deploying Kedro with Databricks Workflows. We have a way to breakdown each node of the kedro pipeline in to a task of Databricks workflows Job. The issues is that each task takes ~10 seconds to create the Kedro session which generates a lot of overhead for the pipeline. Is a way to create the Kedro session faster or a recommendation to avoid this 10 additional seconds for each node?
n

Nok Lam Chan

06/16/2023, 1:25 PM
Can you do a quick profiling to check where is the bottleneck from?
Copy code
pip install pyinstrument
The easiest way to do is open a IPython/notebook session and run this in a cell
Copy code
%load_ext pyinstrument
Copy code
%%pyinstrument

from kedro.framework.session import KedroSession
session = KedroSession.create("../")
3 Views