Hey Everyone! I am trying to load the configuration dict in the code (specifically in pipelines.py to build a dynamic pipeline). I see this suggestion on how to do this: https://docs.kedro.org/en/stable/configuration/parameters.html#how-to-load-parameters-in-code However, this seems not to load the runtime CLI parameters at all. When looking at the OmegaConfigLoader I can see that it has by default _*runtime_params=None*_ so naturally the snippet from that link will ignore the runtime params. Is there a way so that I could keep the runtime_params alive while loading the OmegaConfigLoader directly? Like in the example? I know that the Kedro context holds the runtime_params passed in the CLI but this is not available in the pipeline.py module, right? Any suggestions?

What use case are you trying to achieve by employing dynamic pipelines?

We are using Kedro for large sandbox where we mix and match subpipelines for running a machine learning study. This means that we would like to have some pipelines parametrised by a set of configs which defines which nodes to use when building the pipeline, e.g.: pipeline name: load_graphs usage: kedro run --pipeline=load_graph --params graphs:"g1-g2-g3" or usage: kedro run --pipeline=load_graph --params graphs:"g1-g3" This would run a loading pipeline which creates various combinations of graphs for our runs.

Have you explored using

--tags

for that?

Hmm not yet, will have a look now!

Please let us know if it satisfies your use case, if not, then we’ll think more 😄

I think it can, I didnt manage to re-implement that pipeline using tags yet because of time. However, it would be great if that config loaded "in code" (https://docs.kedro.org/en/stable/configuration/parameters.html#how-to-load-parameters-in-code) could contain all the parameters needed, including the runtime parameters. It's probably not trivial because it would require a way to pass

runtime_params

declared from CLI which is not possible I think?

Kedro does not support using config loaders during pipeline creation and it’s by design. A lot of use cases for dynamic pipelines can be achieved using other features (such as tags) - generating pipelines on the fly via some parameters (although tempting and “easy”), increases the error surface and makes the pipelines less (or not) reproducible. You can also check out https://getindata.com/blog/kedro-dynamic-pipelines/ - it’s one of the approaches that’s fully compatible with Kedro and doesn’t do any hacking.

Thanks 🙂

For jupyter mostly

ah right, ok 🙂