Hello, Is there a way to prevent Kedro from loading a data set that's used as input to multiple nodes every time it's being used? (using ThreadRunner)

Upload the data and keep it in as MemoryDataset

There’s also a

CachedDataSet

wrapper for this

@Nelson Zambrano If it's memory dataset - the data is keep in memory and get garbage collected when it's not needed so there isn't penalty there

Is using

CachedDataSet

a good choice for a scenario where a dataset isn't a

MemoryDataSet

but is accessed by multiple nodes? @Nok Lam Chan

I think so