one question (maybe dumb) When using polars (idea...
# questions
e
one question (maybe dumb) When using polars (ideally all in LazyAPI). Does make sense to use other runner than Sequential? If polars will try to do query optimization, should kedro run nodes in parallel? How is gonna behave the execution?
👀 1
m
Interesting question. Do a trial run, check and share the results - we'll all benefit from it here 👍
🙌 1
d
It depends on your workflow. Are you using intermediate
MemoryDataset
, or are you writing to file? If you're using
MemoryDataset
(or just not explicitly defining the catalog entries in between),
SequentialRunner
probably makes sense, for the reason you mention (and Polars already leverages parallel execution). If you're writing to files after each node for whatever reason, you can benefit from parallelization.
thankyou 1