Hi short and simple question I can t find easy answer to how Kedro #questions

Hi - short and simple question I can't find easy a...

Artur Dobrogowski

02/15/2024, 4:55 PM

Hi - short and simple question I can't find easy answer to - how to verify which env I am running kedro in? I'm specifying some env but some values in catalog parameters seem to not be set correctly and I want to debug this. I see there is

context.config_loader.env

, but can I actually read params of catalog values like

_foo_bar

K 1

datajoely

02/15/2024, 4:56 PM

You can print it from a very simple

before_pipeline_run

hook https://docs.kedro.org/en/stable/kedro.framework.hooks.specs.PipelineSpecs.html#kedro.framework.hooks.specs.PipelineSpecs.before_pipeline_run

datajoely

02/15/2024, 4:56 PM

you could also set it with the

KEDRO_ENV

var

Artur Dobrogowski

02/15/2024, 4:56 PM

yes maybe I worded the question wrong, I wanted to make sure I'm using correct values of catalog params - how can I read those?

Artur Dobrogowski

02/15/2024, 4:56 PM

To expand on this issue I'm relying on

catalog_xxx.yml

that's put in local env to replace some of the params in base env - and they seem to be set incorrectly

Artur Dobrogowski

02/15/2024, 4:57 PM

values for omegaconf like

_foo_bar

Artur Dobrogowski

02/15/2024, 4:57 PM

that used to be globals

datajoely

02/15/2024, 4:58 PM

so it’s done by folder not filename

datajoely

02/15/2024, 4:58 PM

all folders within the folder are scooped into one

datajoely

02/15/2024, 4:58 PM

and then keys present in the custom env override base

Artur Dobrogowski

02/15/2024, 4:58 PM

so I need to put it to

local/catalog/xxx.yml

for it to work?

datajoely

02/15/2024, 4:58 PM

instead of local you’d do

prod

datajoely

02/15/2024, 4:59 PM

IIRC local may take precedence in every case

Ankita Katiyar

02/15/2024, 4:59 PM

So the thing is all the interpolated values are resolved within the environment first and then merged - so

_foo_bar

from the run env is not being resolved into the

base

catalog entry

thankyou 1

Ankita Katiyar

02/15/2024, 4:59 PM

Use the

globals:

resolver for this

Artur Dobrogowski

02/15/2024, 5:00 PM

oh my

Artur Dobrogowski

02/15/2024, 5:00 PM

but I want to use OmegaConf

Artur Dobrogowski

02/15/2024, 5:00 PM

is it a setting of omegaconf or its a separate config loader?

Artur Dobrogowski

02/15/2024, 5:02 PM

how should I handle for example when the catalog is the same for all envs only some part of the paths changes? I've used to do it by changing the

_variables

in each env that are part of the path - you're saying that I can't replace those with omegaconf?

Artur Dobrogowski

02/15/2024, 5:07 PM

ok I see you mention this https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-use-global-variables-with-the-omegaconfigloader

Artur Dobrogowski

02/15/2024, 5:07 PM

thanks for this insight @Ankita Katiyar 🙂

Ankita Katiyar

02/15/2024, 5:09 PM

Woops sorry, I was about to send the docs link too. Let me know if

globals

resolver works for you!

Artur Dobrogowski

02/15/2024, 5:10 PM

well if it resolves before catalog does it will do

Ankita Katiyar

02/15/2024, 5:10 PM

It does, the globals are loaded and merged from both environments first and then resolved into the catalog/parameters where they are mentioned

Artur Dobrogowski

02/15/2024, 5:20 PM

and to answer my own question for others: you can check what current catalog resolves to with

context.config_loader['catalog']

K 1

Open in Slack

Previous Next