Hey team, Is it possible (or is there any workaround) to use a parameter in the catalog.yml using

OmegaConfigLoader

? My use case is that i want to select a parameter in Databricks Workflows and have it override a kedro param at runtime. I was trying to use a global (in globals.yml), as those can be used in the catalog.yml, but unfortunately they can not be overriden at runtime, according to the docs

Someone will reply to you shortly. In the meantime, this might help:

You can use the oc.env omega conf revolver here I think

wdym? to use a parameter in the catalog.yml? oc.env reads from environment variables

Do you want to override a Kedro parameter or read a parameter defines in catalog.yml? How did you select a parameter in workflow?

i want to: 1. do a run with a parameter (prinscreen 1) 2. (then this is passed as a task run in Parameters - "--run_folder","*{{job.parameters.run_folder}}*") 3. now i want this "run_folder" param to also be used in my catalog - e.g.:

mytable:
  catalog: mycatalog
  write_mode: overwrite
  database: gold_dm_${params:run_folder}
  table: mytable

You're likely looking for the

runtime_params

resolver, identical to

globals

but specifically for overriding params at runtime

Ah nice, I didn't know about

runtime_params

. Very useful, especially for changing output folder. But does it also try to find it in globals, or do we have to provide it everytime we run? Although I am using a custom loader based on

TemplatedConfigLoader

in 1 project. Another simple solution could be the following: In your main entrypoint script, dump the global parameters needed to be overriden in

conf/local/globals.yml

main.py

import os
import yaml

os.makedirs("conf/local", exist_ok=True)

with open("conf/local/globals.yml", "w") as f:
    yaml.dump(global_params_to_override)

# Now run your kedro pipeline (assure that local env is default else specify it)

Ah this is great, didn't know about

runtime_params

either. That solves for my use case entirely! Thank you!

Great! how could we have laid out the docs so it was easier to for you to find?

well, to be fair, the section on

runtime_params

was right above the one i've linked to in my question, even though I could swear I had read the entire parametrization docs 😅 so i was helpless, the docs are great!

It's complex stuff and there's lots of content there, any feedback is still helpful!

well maybe the configuration docs could be organized by use cases / usage patterns - or at least provided a summary of use cases and solutions at the end, something like: • I don't need to use parameters in

catalog.yml

-> regular

parameters.yml

and

globals.yml

with no custom resolvers • I need to use parameters in

catalog.yml

but will not override them at runtime -> use

globals

and resolve them in

catalog.yml

(and

parameters.yml

, if needed) • I need to use parameters in

catalog.yml

AND will override them at runtime -> use

runtime_params

and resolve them in

catalog.yml

(and

parameters.yml

, if needed) But to be fair my patterns might be niche and the usage patterns of 90% of kedro users be completely different.

it's still helpful, we could maybe structure a tutorial around that flow