Hello guys, I have a problem with kedro node definition. In order to preprocess my data, I use Dask Cluster in one of my nodes. My problem : for each parallel processing, I need the output path witch is not accessible in the function of a node. Has anyone solved the problem ?

it’s less of a problem and more that kedro intentionally separates business logic from IO logic

Ok ok but how to you handle distributed computing with huge clusters ?

are you using the

dask.ParquetDataSet

?

I'm using custom webdataset for I/O efficiency but this type of dataset is not supported by kedro 😞

Okay but I’m still not sure why the filepaths need to be part of the node logic

I was not aware of dask deployment tutorial, thanks for sharing ! I'll read it and comeback to you