Hey all we re currently running MLflow experiments using a K

Question

Hey all we re currently running MLflow experiments using a Kedro pipeline The pipeline produces intermediate datasets I d like to run multiple experiments concurrently while avoiding file collisions What is the best approach for doing this in Kedro Does someone know whether we can refer to `params` in the `catalog yml` in order to make the paths dynamic

Erwin · Answer

you could use Jinja + namespaces

Emilio Gagliardi · Answer

following to learn slightly smiling face not sure how jinja + namespaces work

Cory Maklin · Answer

Hey < Erwin> do you have an example of using Jinja in the catalog by chance

datajoely · Answer

Are you using the `kedro mlflow` plug in

datajoely · Answer

and I think hooks are the best way to do this if you re not

Lodewic van Twillert · Answer

< datajoely> small typo but surely you meant `kedro mlflow` plugin wink

Lodewic van Twillert · Answer

Regarding referring to `params` to make paths dynamic using namespaces few days ago a similar question was asked and in the reply thread we might have a possible implementation for you Without any additional work no you cannot refer to `params` in the `catalog yml` at the moment However assuming you are indeed using the `kedro mlflow` plugin and logging mlflow artifacts using the ` artifacts MlflowArtifactDataSet` then I wouldn t think you need to refer to your params yml to make the filepath dynamic Since the dataset would be logged as an artifacts to the mlflow run with the params you need thinking face If you share some more about how you are setting up your concurrent runs I suspect you could get away with just using `namespaces` by following the example in the docs here

datajoely · Answer

yes kedro mlflow facepalming