https://kedro.org/ logo
#questions
Title
# questions
f

fmfreeze

11/23/2023, 11:44 AM
A question to all the kedro-mlflow users out there: is it possible to set the experiment name and the run name dynamically for each run? More generalized, the question may also be: How can one change yaml config parameters at dynamically during runtime? Maybe with a custom hook? If yes how, and how to access mlflow.yml config? Anyways, thank you @Yolan Honoré-Rougé for your awesome plugin and the effort you put into the docs.
m

marrrcin

11/23/2023, 12:12 PM
You can disable plugin’s default hook, then write your own by inheriting from the original one and put your custom behaviour / generation logic there 🙂
+
Copy code
# src/<package_name>/settings.py

DISABLE_HOOKS_FOR_PLUGINS = ("kedro-mlflow",)
HOOKS = (YourCustomMLflowHook(), )
https://docs.kedro.org/en/stable/hooks/introduction.html#disable-auto-registered-plugins-hooks
y

Yolan Honoré-Rougé

11/23/2023, 12:49 PM
This is a very frequently required feature, and there is certainly a lot of improvements to do there. I have a couple of issues opened about it, but I wonder if it is possible to use the runtime_params resolver introduced recently to pass args through extra_params
f

fmfreeze

11/23/2023, 1:14 PM
Thx @marrrcin, that might be a proper plan B. @Yolan Honoré-Rougé, I don't know what the
runtime_params
introduced beside renaming the
extra_params
argument. Some thoughts: I assume a hook has access to that
runtime_params
dictionary. But from a plugin/hook perspective, a plugin does only require to know whether a
runtime_params
entry addresses the plugin itself or not, to be able to properly process it further resp. override its own default setting. Do you agree? So some kind of naming convention or similar mechanism from kedro side is required for
runtime_params
to address a specific plugin/hook I guess 🤷‍♂️
y

Yolan Honoré-Rougé

11/23/2023, 11:12 PM
I've never tested it, but mlflow.yml is just "normal" configuration so this:
Copy code
#mlflow.yml
server:
    tracking:
        run:
            name: "${runtime_params:mlflow_run_name, null}"
and then
kedro run --params mlflow_run_name=foo
may "just" work out of the box
I assume a hook has access to that
runtime_params
dictionary. But from a plugin/hook perspective, a plugin does only require to know whether a
runtime_params
entry addresses the plugin itself or not, to be able to properly process it further resp. override its own default setting. Do you agree?
Reading the source code (https://github.com/kedro-org/kedro/blob/e8f1bfd72992336ec12591b49a5fa2654217472f/kedro/config/omegaconf_config.py#L387-L398), the
_runtime__params
are stored in the
OmegaConfigLoader
, so if the plugin uses the project's configloader, I think it can benefit from all the resolvers automatically. Feel free to test and tell me :)
👍 1
f

fmfreeze

11/24/2023, 9:45 AM
yeeees 🙂 >
Copy code
#mlflow.yml
> server:
>     tracking:
>         run:
>             name: "${runtime_params:mlflow_run_name, null}"
> and then
kedro run --params mlflow_run_name=foo
That does exactly what I asked for. Only: instead of
null
i prefer a proper default experiment name 😄 Remains the question, how such runtime_param can be set when starting a run programmatically in python. But I am sure this is covered in the docs somewhere. Thank you, awesome.
🥳 2
m

marrrcin

11/27/2023, 8:16 AM
That’s a nice one, adding to my tool belt @Yolan Honoré-Rougé 💪
2 Views