Hello everyone, I have a question regarding the usage of environments in combination with the OmegaConfigLoader.

I have a file called

catalog_globals.yml

in my

base/

config folder, and also in my

prod/

config folder. When I execute

kedro run --env=prod

, the settings from the file in

base/

are still used.

cc @Gerrit Schoettler

Thanks a lot @Nok Lam Chan! I still have the question on how to configure my catalog based on the env. It seems like the

OmegaConfigLoader

is not using the

catalog_globals.yml

file in the folder of the env (it's still using the file in the base folder).

Could you share a minimal example to demonstrate your problem? maybe shows the relevant entries in

base/catalog.yml

base/catalog_globals.yml

prod/catalog_globals.yml

prod/catalog.yml

base/catalog_globals.yml

_bucket_name: <s3://ABC>

prod/catalog_globals.yml

_bucket_name: local_path_1

base/catalog.yml

catalog_entry_1:
  type: pandas.ParquetDataSet
  filepath: ${_bucket_name}/file_name_1.parquet

catalog_entry_2:
  type: pandas.ParquetDataSet
  filepath: ${_bucket_name}/file_name_2.parquet

prod/catalog.yml

catalog_entry_2:
  type: pandas.ParquetDataSet
  filepath: local_path_2/file_name_2.parquet

Indeed - what you observed is an expected behavior. Variable interpolation (templating) is expected to work for the same environment, you can’t interpolate values from other environment. The reason for that is because we think it is error-prone to interpolate cross environment because it is hard to know where is that value coming from.

Thanks a lot for the clarification and the pointer to the feature in development. @Ankita Katiyar I'd be very happy about a workaround, if you have any hints! 🙂

I would say that creating a separate entry in

prod/catalog.yml

which will overwrite the one in

base/catalog.yml

is the only thing that will work

Thanks a lot @Ankita Katiyar! I didn't know that values cannot be propagated between environments.

I’m not sure if namespacing might be useful in this particular case. We are also working on this feature https://github.com/kedro-org/kedro/issues/2531 After which you might be able to pass

kedro run --params: __my_bucket_ = "local__path_1"

to override “_ my_ bucket” for that particular run

Thanks a lot @Ankita Katiyar!! Looking forward to that feature 🙂

Hi @Gerrit Schoettler, So the globals functionality with

OmegaConfigLoader

is out already. Docs - https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-use-global-variables-with-the-omegaconfigloader

yes, saw the update in 0.18.13 🙂 And I understand that the variable name can't start with "_" in globals.

What we ended up doing was enabling people to use “globals” with a

globals:

resolver

okay, does that prohibit the use of --params? With my new globals.yml file everything works (setting the bucket_name for my catalog). I just can't modify it with --params

Yep, you can’t manipulate globals with --params

is there any way to manipulate globals from the outside?

With Kedro 0.18.14, you will be able to specify in your catalog

catalog_entry_1:
  type: pandas.ParquetDataSet
  filepath: "${runtime_params:bucket_name. default_value}"/file_name_1.parquet

@Ankita Katiyar shouldn't it use the runtime_param resolver instead?

But runtime_params wouldn’t work in

globals.yml

you would have to do it directly in the catalog

Any ideas on how I can modify catalog parameters from the outside?

This is the docs for the runtime_params functionality but it isn’t released yet - https://docs.kedro.org/en/latest/configuration/advanced_configuration.html#how-to-override-configuration-with-[…]rameters-with-the-omegaconfigloader

Any thoughts on when it will be released?

This week, ideally! 🙂

okay! 🙂 I felt like I have been waiting for 0.18.13 and then I was confused when I still couldn't achieve my task of separating prod/test environments from the outside

You can do the same thing with globals (assuming the premise is the same as what was in the original question) where you can define

base/globals.yml

with

bucket_name:<s3_path>

and a

prod/globals.yml

with

bucket_name:<local_path>

where the

path

variable will be overwritten by the

prod

version And your

catalog.yml

will look like:

catalog_entry_1:
  type: pandas.ParquetDataSet
  filepath: "${globals:bucket_name. default_value}"/file_name_1.parquet

ah interesting, so the globals work across environments?

Yep yep, I think I mentioned it wouldn’t before it was implemented but we implemented it with

globals

resolver to work across envs

premise is still the same as in the original question 🙂

catalog_globals.yml

is read as the “catalog” but not as “globals” if that makes sense, so variables in

catalog_globals

are only available to catalog entries and have to begin with underscores so as to not be read by the config_loader as separate catalog entries

Cool, thanks a lot for the super fast reply @Ankita Katiyar!!

Yes this is working already

Can confim that it's working as I expected, I just didn't know that I need to nest the ${ }

Cool! let us know if you find any issues, feedbacks are welcomed!

Yes, will let you know if anything doesn't work. Thanks for the very quick reply!!