I noticed there is a new `OmegaConfLoader` class a...
# questions
I noticed there is a new
class available in the main branch of kedro. When is it expected to be included in a release (v0.18.5 maybe)?
And perhaps a follow-up question: did I read that correctly in the issue tracker that this config loader will eventually replace all others?
@Merel should know the answer to this 🙂
@Matthias Roels You can track the progress of this work on our configuration overhaul milestone here, and yes, we plan to phase in
as our primary configuration loader from 0.18.5. We're using
because it solves a ton of problems that users have had with our existing support for configuration. Do you have any thoughts on this?
đź‘Ť 3
Yes the idea is to deprecate the old config loaders and use
from the
series onwards. In
we’re introducing the new config loader, but it will be a work in progress and you’ll see new features for it coming in the following releases.
🚀 1
Nice! I am a big fan of this idea and it would definitely solve some of the challenges we currently have! One issue that we have that this solution doesn’t solve (others won’t solve this either) is the following scenario: suppose you have a model (and corresponding pipelines) that are slight variations (encoded in config) for different countries and products, with a significant amount of configuration overlapping for all products in a certain country. Then you would need to introduce hierarchy (base -> country -> product -> local). Currently, there is no simple way to achieve this (other then creating a custom config loader that overwrites
). It’s not hard to do that, but it would be nicer if we could somehow just inject this logic…
Could you elaborate a bit on that? What do you want to achieve? Being able to run pipelines with slight variations and not having to duplicate config?
Well… given 3 config envs
country_a x product_a
country_b x product_a
country_b x product_c
. Let’s say 10% is shared between all envs, so that goes in base. 25% of the configuration is overlapping between the envs related to
, So we made an intermediate env related to country-level config (as this is a significant amount that you would otherwise duplicate). It can’t be used on it’s own (there no pipelines compatible with e.g
) but there is no alternative if you want to use the same hierarchical config loading.
Kedro 0.18.5 is out! Here are the release notes for it. Please try it out and let us know what you think.
đź‘Ť 1