Hi 👋! I have been wondering for a long time if it was possible to process partition datasets one at a time (or in chunks) in parallel (e.g. when running with ParallelRunner). Today as I was wandering around I found this: https://github.com/kedro-org/kedro/issues/1413 I understand the code that is provided, but it only demonstrates how to write a node that both input and output a partionned dataset. The issue description itselfs talks about creating multiple nodes that would be ran in parallel and I don't see much on this and most of the links are dead. Could someone shed some light on this? Is this currently possible? Thanks :)

Hi @Guillaume Tauzin Kedro can process partitioned datasets in parallel, but not automatically at the partition level. The

ParallelRunner

parallelizes nodes, not partitions within a single dataset. Kedro’s execution model works like this: • Each node is a unit of computation. • The

ParallelRunner

executes multiple nodes simultaneously if their input/output dependencies allow it. • A

PartitionedDataSet

is a single dataset (even though it’s composed of multiple partitions). When a node reads or writes it, Kedro treats that as one I/O operation - it doesn’t spawn tasks for each partition automatically.

Hi @Elena Khaustova and thank you for your reply! I was hoping that I had missed something as the issue description clearly shows a fan-out of a partitioned node and this is what I hoped for. For some context, in kedro-dagster, I map Dagster partitions (which is a fanned-out version of Kedro partitions) by providing a

DagsterPartitionedDataset

and mapping Kedro nodes that have this type of dataset as an input and/or outputs to a Dagster

op

per partition that can run in parallel (and wait for all partition to be done to go to the next node). It is working in some cases, but there are a lot of edge cases that needs to be worked out. May I ask if there has been a discussion on this topic previously that I could read? Is there any plan on supporting easily parallelizing processing of partition of data?

I’m not aware of such discussions, but let’s see if other people are. And we can definitely start it 🙂