I noticed there is a new

OmegaConfLoader

class available in the main branch of kedro. When is it expected to be included in a release (v0.18.5 maybe)?

@Merel should know the answer to this 🙂

@Matthias Roels You can track the progress of this work on our configuration overhaul milestone here, and yes, we plan to phase in

OmegaConf

as our primary configuration loader from 0.18.5. We're using

OmegaConf

because it solves a ton of problems that users have had with our existing support for configuration. Do you have any thoughts on this?

Yes the idea is to deprecate the old config loaders and use

OmegaConfLoader

from the

0.19.0

series onwards. In

0.18.5

we’re introducing the new config loader, but it will be a work in progress and you’ll see new features for it coming in the following releases.

Nice! I am a big fan of this idea and it would definitely solve some of the challenges we currently have! One issue that we have that this solution doesn’t solve (others won’t solve this either) is the following scenario: suppose you have a model (and corresponding pipelines) that are slight variations (encoded in config) for different countries and products, with a significant amount of configuration overlapping for all products in a certain country. Then you would need to introduce hierarchy (base -> country -> product -> local). Currently, there is no simple way to achieve this (other then creating a custom config loader that overwrites

get_config_paths

). It’s not hard to do that, but it would be nicer if we could somehow just inject this logic…

Could you elaborate a bit on that? What do you want to achieve? Being able to run pipelines with slight variations and not having to duplicate config?

Well… given 3 config envs

country_a x product_a

,

country_b x product_a

and

country_b x product_c

. Let’s say 10% is shared between all envs, so that goes in base. 25% of the configuration is overlapping between the envs related to

country_b

, So we made an intermediate env related to country-level config (as this is a significant amount that you would otherwise duplicate). It can’t be used on it’s own (there no pipelines compatible with e.g

env=country_b

) but there is no alternative if you want to use the same hierarchical config loading.

Kedro 0.18.5 is out! Here are the release notes for it. Please try it out and let us know what you think.