Hi! Is there a way to cache Node outputs and avoid...
# questions
p
Hi! Is there a way to cache Node outputs and avoid running it if the cached result exists?
1
n
https://kedro.readthedocs.io/en/stable/kedro.io.CachedDataSet.html Maybe this is useful? In general, you should think about what needs to be saved, keeping all nodes’ output will also increase memory consumption.
p
Thanks for the reply. I'm already using the CachedDataSet, yet when calling
SequentialRunner().run(pipeline)
it executes all nodes. The solution I found was to use
SequentialRunner().run_only_missing(pipeline)
👍 1