https://kedro.org/ logo
#questions
Title
# questions
p

Pedro Arthur

10/31/2022, 7:52 PM
Hi! Is there a way to cache Node outputs and avoid running it if the cached result exists?
1
n

Nok Lam Chan

11/01/2022, 11:16 AM
https://kedro.readthedocs.io/en/stable/kedro.io.CachedDataSet.html Maybe this is useful? In general, you should think about what needs to be saved, keeping all nodes’ output will also increase memory consumption.
p

Pedro Arthur

11/01/2022, 1:29 PM
Thanks for the reply. I'm already using the CachedDataSet, yet when calling
SequentialRunner().run(pipeline)
it executes all nodes. The solution I found was to use
SequentialRunner().run_only_missing(pipeline)
👍 1
8 Views