Hi Team, I'm deploying Kedro with Databricks Workflows. We have a way to breakdown each node of the kedro pipeline in to a task of Databricks workflows Job. The issues is that each task takes ~10 seconds to create the Kedro session which generates a lot of overhead for the pipeline. Is a way to create the Kedro session faster or a recommendation to avoid this 10 additional seconds for each node?

Can you do a quick profiling to check where is the bottleneck from?

pip install pyinstrument

The easiest way to do is open a IPython/notebook session and run this in a cell

%load_ext pyinstrument

%%pyinstrument

from kedro.framework.session import KedroSession
session = KedroSession.create("../")