Tomás Rojas03/06/2023, 11:32 PM
. I noticed they return a dictionary with bounded methods for loading each dataset. My question is: Is there a way to write the nodes simply as a function of the object returned by the bounded method or should I write the nodes thinking about the dictionary?
Tom C03/07/2023, 1:06 AM
and have the processing of the Partitions be concurrent natively within Kedro. I appreciate that it's likely going to be something that should be tailored to each problem. Perhaps I'll utilise asyncio.gather and at least minimise time spent waiting during each data loading operation.
datajoely03/07/2023, 10:11 AM
it will load datasets concurrently IIRC but due to python weirdness you can’t use async and ParallelRunner at the same time
kedro run --async
Tom C03/07/2023, 10:16 PM
load catalogue entries AND PartitionedDataSet partitions in the async event loop? Or will it async across the catalogue but sync within each catalogue entry?
Tomás Rojas03/07/2023, 10:22 PM
Tom C03/07/2023, 10:25 PM
loop loading the partitions. I assume this is to allow for a memory-safe method to iterate through the partitions and process each one.
Tomás Rojas03/07/2023, 10:26 PM