Hi, I'm recently creating a dataset for snowflake since the implemented one is outdated and does not contain any scd1,2 and generic tests. I'm asking myself what the cleanest implementation is. I do have two options in mind. • Session handling in the dataset ◦ Description: I handle the session in the dataset and Implement it as an singelton. The Advantages are that It's easy, creates little coupling and is pretty explicite. • Session handling in the Snowparkhook ◦ Description: I handle the snowpark session in an explicite hook which contains before_pipeline_run and after_pipeline_run. It is a nice, more explicite session handling. Does anyone of you have expirience in creating datasets and any input? Thanks in advance

Are you looking for upsert support (via merge/into)? If so,

ibis.TableDataset

now supports upserts, including on Snowflake, if that may help. 🙂 Re your question, Spark is the closest analog here, and Spark handles sessions creation through a hook.

Thanks for the hint, yes were looking for upsert but didnt want another dependencie. Thank's for your amazing support

Hey Ralf! I'd go with the hook approach. A Dataset's job theoretically is only loading and saving data; session lifecycle management is more closely related to infrastructure. Mixing both in the same class, in my view, blurs the boundary that makes Kedro so useful in the first place, which is modularization and division of responsibilities. It also makes the Dataset harder to test in isolation (ex. mocking or swapping the session in unit and integration tests). Kedro's

before_pipeline_run

hooks exist precisely for this kind of shared resource management. I'd initialize the Snowpark session once there, inject it via credentials or a shared config mechanism, and close it cleanly at the end. The singleton in the Dataset does have the advantage of being faster to get running. But if you're building a standard for a team, the hook will hold up much better in the long term; especially if you ever run nodes in parallel or need to reuse the Dataset outside a pipeline context. Happy to dig into specifics if you're running into constraints that make the hook harder to wire up! Best, Gustavo

I implemented the session with the before_pipeline_run,after_pipeline_run and on_pipeline_run which works fine for now. I have Kedro nodes that are expected to run about 2-3 hours and I would love to run them in parallel. Is there a chance to assign the snowpark_session to the data set before_dataset_loaded and release it after_dataset_save? The Idea ist to create a snowpark_session pool and use it to multithread

Hey! Nice, glad the session hooks are working. The idea makes sense. You can hook into

before_dataset_loaded

and

after_dataset_saved

to borrow and release sessions from a pool. Those hooks don't give you the dataset instance directly, so you'd have to inject the session at runtime, either by patching

catalog._datasets[dataset_name]

or by wrapping your datasets in a class that accepts a session at load/save time. For the pool,

queue.Queue

is the natural fit for me. Create your sessions at

before_pipeline_run

, push them into the queue, then

get()

before load/save and

put()

right after. One thing worth knowing beforehand is taht Snowpark Python now supports thread-safe session objects, so multiple threads can share the same session. That said, thread safety is controlled by the

PYTHON_SNOWPARK_ENABLE_THREAD_SAFE_SESSION

parameter, and the key protected operations include configuration changes, package and import management, and plan caching. So if your nodes touch any of that, a pool still makes more sense than a single shared session. Also, the runner choice matters a lot, since

ParallelRunner

uses multiprocessing while

ThreadRunner

uses multithreading. For your case,

ParallelRunner

is probably a no-go because all datasets need to be serialisable for multiprocessing to work, and any dataset that isn't will block the whole run. A Snowpark session almost certainly won't survive pickling. So

ThreadRunner

is the realistic path here, and it actually plays nicer with the session pool approach you're describing.