Hi kedro team ! Is it possible to run with asynchr...
# questionsa
Adrien
11/23/2023, 6:18 PMHi kedro team ! Is it possible to run with asynchrone and/or with multithread the save operation in lazy saving of a partitioned dataset ?
d
datajoely
11/23/2023, 6:19 PMDoes this work?
kedro run --asyncLoad and save node inputs and outputs asynchronously with threads
a
Adrien
11/23/2023, 6:20 PMNo, from what i read and test, it's only to load/write multiple dataset asynchronuously
Adrien
11/23/2023, 6:20 PMNot for each partition of a dataset
d
datajoely
11/23/2023, 6:20 PMOh yes gotcha
datajoely
11/23/2023, 6:20 PMI think the lazy saving is just a for loop
datajoely
11/23/2023, 6:20 PMyou need to subclass the partitioned dataset class to add that
a
Adrien
11/23/2023, 6:21 PMMmmh okok, but when i return the dict it's processed in the partition class or in the hook after_node_run ?
Adrien
11/23/2023, 6:22 PMI think i read on the doc than there is a trigger on the hook
d
datajoely
11/23/2023, 6:23 PMpartition class
a
Adrien
11/23/2023, 6:23 PMOk thanks !
Adrien
11/23/2023, 6:23 PMMaybe I'll make a PR to include it in the plugin
datajoely
11/23/2023, 6:24 PMit’s a for loop
datajoely
11/23/2023, 6:24 PMso you could maybe (1) chunk it and use multi-processing (2) try something like joblib
datajoely
11/23/2023, 6:24 PMJust know if you use
ParallelRunnera
Adrien
11/23/2023, 6:25 PMOh yes because all thread will already be alocated ? Right ?
d
datajoely
11/23/2023, 6:25 PMmore that Python and concurrency is just painful
a
Adrien
11/23/2023, 6:26 PMOk thanks !