Hello everyone. I have a question with regard to environments, since I'm seemingly misunderstanding them. I searched a bit in this channel, but (unless I grossly misread something) didn't see this specific question. I have a

dask.yml

in my

conf/base

which contains the following (real config is much larger, but this gets the point across):

dask_cloudprovider:
  region: eu-central-1
  instance_type: t3.xlarge
  n_workers: 36

And a

dask.yml

in another environment, e.g.

conf/low

with the following:

dask_cloudprovider:
  instance_type: t3.small
  n_workers: 8

Which I activate using

kedro run --env=low

. Now, I would have expected the

config_loader

(

TemplatedConfigLoader

) to contain something like

{'dask_cloudprovider': {'region: 'eu-central-1', 'instance_type': 't3.small', 'n_workers': 8}}

. However, it overrides the entire entry, resulting in the

config_loader

containing:

{'dask_cloudprovider': {'instance_type': 't3.small', 'n_workers': 8}}

. Is there any way to get what I was expecting out of the box? I don't really want to copy my entire configuration N-times for each environment, especially since only a few of the keys change. Is the intended use case for environments different to what I'm trying to use it for (say, only for top-level entries)?

So today this works with a destructive merge on keys, we’re in the process of switching to an OmegaConf backed config loader so supporting this sort of behaviour with a custom resolver will be possible in the future

Ah, I see. Thanks for the info. Can you tell me if this is also on the 0.18.5 roadmap? I found https://github.com/kedro-org/kedro/milestone/9 which seems like it's the relevant milestone. Or am I misinterpreting? : )

so it’s coming incrementally

That's completely understandable. Is there any potential for contributions on that end, or would you prefer to keep that part to the current maintainers, since they likely have the most experience with it?

Oooh good question - I’m going to have to point that to @Merel 🙂

Hi @Filip Panovski! Milestone https://github.com/kedro-org/kedro/milestone/9 contains all non breaking changes regarding configuration. For Kedro

0.19.0

we’ll be introducing some more “radical” changes captured in https://github.com/kedro-org/kedro/milestone/16 (we still need to add some tasks here, but the other milestone is our priority at the moment) Contributions are welcome, but we’d like to first release our new

OmegaConfLoader

and get feedback before committing to implementations for the

0.19.0

release.

This is exactly the task you’re talking about @Filip Panovski https://github.com/kedro-org/kedro/issues/2122 Thanks @Merel

Cool! Thanks for sharing. I'm not as well-versed with the topic as the participants there on #2122, but I've subscribed to the issue and would like to help out if I can when

0.19.0

development begins (even if it just ends up being writing some extra tests). Have a great weekend.

Thanks for the mention! You can count me in to contribute as well. In fact, it’s one of my professional milestones this year to contribute to an open source project 😅

Amazing! We really appreciate contributions 😄 You can always have a look at open issues, and especially the ones marked with “Community” are issues we’d like help with: https://github.com/kedro-org/kedro/issues?q=is%3Aopen+is%3Aissue+label%3ACommunity