hello 🙂 Goal: Use the pipeline name given through CLI to dynamically name the experiment in mlflow (example of experiment name:

local_experiment_test_pipeline

the

local_experiment

is set by the developper working on the project and the

test_pipeline

should be added dynamically through the conf/env variables). Context: I have several pipelines defined in a

pipeline_registry

. For naming of artifacts/mlflow runs, I would like to make the name of the pipeline being run an environment variable that I can reference in the kedro project conf. How would you recommend making the pipeline name an environment variable? thanks for your input ! ☀️ Explored strategies: • I looked into the before_run Hook, however this gets called after the conf is set • I looked into rewriting a run.py but I think maybe I need to touch on the cli.py to get the cli args. Exploring this atm • Could a kedro plugin be the way to go? Update: I can get the pipeline name as an environment variable now by saving it in cli.py following the template provided here (https://docs.kedro.org/en/0.18.14/development/commands_reference.html). This is an environment variable so I am trying to add it to the globals_dict in settings.py to be able to use it in conf. However, it seems cli.py gets called after the conf is initialized in settings.py. How can we dynamically set this variable then?? 🥲

how are you using mlflow

I'm using kedro-mlflow and in the mlflow.yaml I want to set the tracking/experiment/name variable. I am letting local developpers set the name of the experiment as they see fit (with a variable in globals.yaml) I want to add the pipeline name to whatever they choose. (the pipeline is being triggered by a kedro run --pipeline=pipeline_name) I know by default that mlflow.yaml sets the run name to the pipeline name, just trying to find how to get access to that info from within the conf and it's not clear 😅

I’m not sure then @Yolan Honoré-Rougé would you have an ideas?

I'm seeing kedro-mlflow specific hooks such as

mlflow_node_hook.after_context_created
        mlflow_node_hook.before_pipeline_run

here: https://github.com/Galileo-Galilei/kedro-mlflow/blob/64b8e94e1dafa02d979e7753dab9b9dfd4d7341c/tests/framework/hooks/test_run_name.py#L19 so I will look into them 😊 but happy for any pointers too

you can do some stuff with a custom CLI

yes apologies, I forgot a 1: we are at 0.18.13 my understanding is that it uses the OmegaConfigLoader so it's more in the realms of the 0.19.0 😕 seeing with my dev team if we can upgrade

yeah - we have a migration guide here https://docs.kedro.org/en/latest/resources/migration.html

I think you can use the OmegaConfigloader in 0.18.13, just update your settings.py. On the other topic, it's indeed partially solved in

kedro

and

kedro-mlflow

with the

runtime_params

resolver but it's still frustrating because the resolver does not add by default the

pipeline_name

and the

env

value, you have to specify it manually through the CLI which is not ideal. I've though about opening a PR for a while.