Erwin
09/05/2024, 2:32 PMmarrrcin
09/05/2024, 3:28 PMDeepyaman Datta
09/05/2024, 3:54 PMMemoryDataset
, or are you writing to file?
If you're using MemoryDataset
(or just not explicitly defining the catalog entries in between), SequentialRunner
probably makes sense, for the reason you mention (and Polars already leverages parallel execution).
If you're writing to files after each node for whatever reason, you can benefit from parallelization.