Tomás Rojas
03/06/2023, 11:32 PMPartitionedDataSets
. I noticed they return a dictionary with bounded methods for loading each dataset. My question is: Is there a way to write the nodes simply as a function of the object returned by the bounded method or should I write the nodes thinking about the dictionary?Tom C
03/07/2023, 1:06 AM--runner=ParallelRunner
and have the processing of the Partitions be concurrent natively within Kedro. I appreciate that it's likely going to be something that should be tailored to each problem.
Perhaps I'll utilise asyncio.gather and at least minimise time spent waiting during each data loading operation.datajoely
03/07/2023, 10:11 AMkedro run --async
it will load datasets concurrently IIRC but due to python weirdness you can’t use async and ParallelRunner at the same timeTom C
03/07/2023, 10:16 PM--async
load catalogue entries AND PartitionedDataSet partitions in the async event loop? Or will it async across the catalogue but sync within each catalogue entry?Tomás Rojas
03/07/2023, 10:22 PMTom C
03/07/2023, 10:25 PMfor
loop loading the partitions. I assume this is to allow for a memory-safe method to iterate through the partitions and process each one.Tomás Rojas
03/07/2023, 10:26 PM