https://kedro.org/ logo
#questions
Title
# questions
m

Matthias Roels

01/11/2023, 7:17 PM
I noticed there is a new
OmegaConfLoader
class available in the main branch of kedro. When is it expected to be included in a release (v0.18.5 maybe)?
And perhaps a follow-up question: did I read that correctly in the issue tracker that this config loader will eventually replace all others?
a

Antony Milne

01/12/2023, 9:20 AM
@Merel should know the answer to this 🙂
y

Yetunde

01/12/2023, 9:38 AM
@Matthias Roels You can track the progress of this work on our configuration overhaul milestone here, and yes, we plan to phase in
OmegaConf
as our primary configuration loader from 0.18.5. We're using
OmegaConf
because it solves a ton of problems that users have had with our existing support for configuration. Do you have any thoughts on this?
đź‘Ť 3
m

Merel

01/12/2023, 10:00 AM
Yes the idea is to deprecate the old config loaders and use
OmegaConfLoader
from the
0.19.0
series onwards. In
0.18.5
we’re introducing the new config loader, but it will be a work in progress and you’ll see new features for it coming in the following releases.
🚀 1
m

Matthias Roels

01/12/2023, 11:23 AM
Nice! I am a big fan of this idea and it would definitely solve some of the challenges we currently have! One issue that we have that this solution doesn’t solve (others won’t solve this either) is the following scenario: suppose you have a model (and corresponding pipelines) that are slight variations (encoded in config) for different countries and products, with a significant amount of configuration overlapping for all products in a certain country. Then you would need to introduce hierarchy (base -> country -> product -> local). Currently, there is no simple way to achieve this (other then creating a custom config loader that overwrites
get_config_paths
). It’s not hard to do that, but it would be nicer if we could somehow just inject this logic…
m

Merel

01/12/2023, 12:42 PM
Could you elaborate a bit on that? What do you want to achieve? Being able to run pipelines with slight variations and not having to duplicate config?
m

Matthias Roels

01/12/2023, 5:32 PM
Well… given 3 config envs
country_a x product_a
,
country_b x product_a
and
country_b x product_c
. Let’s say 10% is shared between all envs, so that goes in base. 25% of the configuration is overlapping between the envs related to
country_b
, So we made an intermediate env related to country-level config (as this is a significant amount that you would otherwise duplicate). It can’t be used on it’s own (there no pipelines compatible with e.g
env=country_b
) but there is no alternative if you want to use the same hierarchical config loading.
y

Yetunde

02/24/2023, 3:05 PM
Kedro 0.18.5 is out! Here are the release notes for it. Please try it out and let us know what you think.
đź‘Ť 1
5 Views