https://kedro.org/ logo
#questions
Title
# questions
n

Nelson Zambrano

02/16/2024, 5:40 AM
Hello, Is there a way to prevent Kedro from loading a data set that's used as input to multiple nodes every time it's being used? (using ThreadRunner)
s

Shubham Gupta

02/16/2024, 6:01 AM
Upload the data and keep it in as MemoryDataset
d

datajoely

02/16/2024, 10:00 AM
There’s also a
CachedDataSet
wrapper for this
n

Nok Lam Chan

02/16/2024, 4:49 PM
@Nelson Zambrano If it's memory dataset - the data is keep in memory and get garbage collected when it's not needed so there isn't penalty there
🥳 1
n

Nelson Zambrano

02/19/2024, 7:56 PM
Is using
CachedDataSet
a good choice for a scenario where a dataset isn't a
MemoryDataSet
but is accessed by multiple nodes? @Nok Lam Chan
👍🏼 1
n

Nok Lam Chan

02/19/2024, 8:12 PM
I think so