hello :slightly_smiling_face: Goal: Use the pipeli...
# questions
hello 🙂 Goal: Use the pipeline name given through CLI to dynamically name the experiment in mlflow (example of experiment name:
is set by the developper working on the project and the
should be added dynamically through the conf/env variables). Context: I have several pipelines defined in a
. For naming of artifacts/mlflow runs, I would like to make the name of the pipeline being run an environment variable that I can reference in the kedro project conf. How would you recommend making the pipeline name an environment variable? thanks for your input ! ☀️ Explored strategies: • I looked into the before_run Hook, however this gets called after the conf is set • I looked into rewriting a run.py but I think maybe I need to touch on the cli.py to get the cli args. Exploring this atm • Could a kedro plugin be the way to go? Update: I can get the pipeline name as an environment variable now by saving it in cli.py following the template provided here (https://docs.kedro.org/en/0.18.14/development/commands_reference.html). This is an environment variable so I am trying to add it to the globals_dict in settings.py to be able to use it in conf. However, it seems cli.py gets called after the conf is initialized in settings.py. How can we dynamically set this variable then?? 🥲
how are you using mlflow
this should be possible with kedro hooks
I'm using kedro-mlflow and in the mlflow.yaml I want to set the tracking/experiment/name variable. I am letting local developpers set the name of the experiment as they see fit (with a variable in globals.yaml) I want to add the pipeline name to whatever they choose. (the pipeline is being triggered by a kedro run --pipeline=pipeline_name) I know by default that mlflow.yaml sets the run name to the pipeline name, just trying to find how to get access to that info from within the conf and it's not clear 😅
I’m not sure then @Yolan Honoré-Rougé would you have an ideas?
👍 1
I'm seeing kedro-mlflow specific hooks such as
Copy code
here: https://github.com/Galileo-Galilei/kedro-mlflow/blob/64b8e94e1dafa02d979e7753dab9b9dfd4d7341c/tests/framework/hooks/test_run_name.py#L19 so I will look into them 😊 but happy for any pointers too
and I see that this has actually been implemented with the run_time params (it's great!) 😄 👏 but the downside is we are using kedro 0.18.3 so we don't have that option yet, any ideas how to do so without the run_time_params? 😕
👍 1
you can do some stuff with a custom CLI
😵‍💫 1
but it’s quite hard IIRC
you should be able to upgrade to at least
without any breaking changes, so I’m not sure if it was added there or in 0.19.x
😊 1
yes apologies, I forgot a 1: we are at 0.18.13 my understanding is that it uses the OmegaConfigLoader so it's more in the realms of the 0.19.0 😕 seeing with my dev team if we can upgrade
yeah - we have a migration guide here https://docs.kedro.org/en/latest/resources/migration.html
🥳 1
I think you can use the OmegaConfigloader in 0.18.13, just update your settings.py. On the other topic, it's indeed partially solved in
with the
resolver but it's still frustrating because the resolver does not add by default the
and the
value, you have to specify it manually through the CLI which is not ideal. I've though about opening a PR for a while.
👍 1