I do work with snowpark and I'm writing a snowpark DataSet. I dont like that I have to share the snowparksession via the DataSet and not per Node. I do have multiple possible situations as fasr as i know. 1-n Input datasets and 1-n Output datasets. I'm now implementing it with a singelton which forces me to use the same session over all nodes. I would prefer to have an individual session per node which would have the advantage of • Isolate workflows • allow parallel processing • Add different configuration for each session

Hi @Ralf Kowatsch Kedro’s

Dataset

API is designed for pure data I/O: it loads/returns data objects and writes them back, and it isn’t aware of per-node or per-run execution context. That means: • The

DataCatalog

is instantiated once per pipeline run, not per node. • Datasets are shared across nodes. • Any state stored inside a dataset (like a Snowpark session) is therefore global to that catalog instance. If you want one Snowpark session per node, not per pipeline, you need to move session management out of the dataset and into the node level or a hook. Possible options: • Create the Snowpark session inside each node - instead of letting the dataset manage it, you can open a session explicitly in the node • Use a hook to create a session per node - https://docs.kedro.org/en/unreleased/extend/hooks/introduction/ • If you really want to keep session logic in the dataset layer, the clean way is to make the dataset open and close its own Snowpark session each time it’s called