I first tried by creating a custom_resolver that would generate a temp file, but cleaning it up afterwards turned out to be quite complicated (basically i would need to create a hook for it), so the cleanest options is to include it in the kedro-mlflow dataset

This would increase consistency between interactive and CLI workflow

And maybe we can tweak it to accept a temp folder manger and pass it through a resolver

Need to prototype a little to find out the best developer experience

i was thinking btw, that the reverse of this idea would also be interesting, so an option to only save the reference to the artifact to mlflow but not the artefact itself. such than when you load it, it loads the locations of the artefact (given the experiment id for example) and then loads the data using the underlying dataset. basically an mlflow artefact dataset where you can choose whether you want to save only to mlflow, only to the filepath (but version with mlflow), or save to both. for as far as i understood the current implementation saves to both right?

I am not sure we want to do this, this is quite confusing

If you don't want to save in mlflow, just keep the underlying dataset

You can have 2 envs, one with mlflow and one without

unfortunately kedro’s build in tools do not allow one to pass the mlflow_run_id from runtime params to the factories where you need them, and so i had to do some ugly hacks in a custom cli implementation and an environment variable

anyway, im rambling off a bit. ill get back to you with a proposal for saving only to mlflow like discussed earlier

Yes I understand for the incompatibility runtime_params+ factory but the resolver should still be the way to go