Is it possible to load a "versioned" pipeline run ...
# questions
f
Is it possible to load a "versioned" pipeline run (versioned with experiment tracking)? I have a couple of
MemoryDataSet
flowing around a pipeline, and I want to inspect them for individual tracked pipeline runs after the run (e.g. load them again like with
session.run(to_outputs...)
but for a specific experiment run from the past.)
n
Are you thinking some sort of time-travel feature?
Memory data are ephermeral, if you want to inspect them after run you need to persist it to disk and saving them in the catalog.
f
Well, more of a way inspect parts of an already run pipeline captured by experiment tracking. That is already possible with versioned datasets, but not for MemoryDatasets, which - I think?! - needs some sort of re-running that tracked pipeline.
n
You have to persist it instead of using MemoryDataSet, once you finish the run everything is gone it’s impossible to extract them.
f
Thank you @Nok Lam Chan for making that clear. One kind of related follow-up question to me is then: Is there a flag or another supported way to exclude a single
kedro run
from the experiment tracking?
t
Not at the moment, no. If you want to exclude a run you'd have to delete it from the session.db file
👍 1