Filip Panovski
02/02/2023, 5:01 PMdask.yml
in my conf/base
which contains the following (real config is much larger, but this gets the point across):
dask_cloudprovider:
region: eu-central-1
instance_type: t3.xlarge
n_workers: 36
And a dask.yml
in another environment, e.g. conf/low
with the following:
dask_cloudprovider:
instance_type: t3.small
n_workers: 8
Which I activate using kedro run --env=low
.
Now, I would have expected the config_loader
(TemplatedConfigLoader
) to contain something like {'dask_cloudprovider': {'region: 'eu-central-1', 'instance_type': 't3.small', 'n_workers': 8}}
.
However, it overrides the entire entry, resulting in the config_loader
containing: {'dask_cloudprovider': {'instance_type': 't3.small', 'n_workers': 8}}
.
Is there any way to get what I was expecting out of the box? I don't really want to copy my entire configuration N-times for each environment, especially since only a few of the keys change. Is the intended use case for environments different to what I'm trying to use it for (say, only for top-level entries)?datajoely
02/02/2023, 5:13 PMFilip Panovski
02/03/2023, 10:02 AMdatajoely
02/03/2023, 10:49 AMFilip Panovski
02/03/2023, 11:07 AMdatajoely
02/03/2023, 11:08 AMMerel
02/03/2023, 11:23 AM0.19.0
we’ll be introducing some more “radical” changes captured in https://github.com/kedro-org/kedro/milestone/16 (we still need to add some tasks here, but the other milestone is our priority at the moment)
Contributions are welcome, but we’d like to first release our new OmegaConfLoader
and get feedback before committing to implementations for the 0.19.0
release.datajoely
02/03/2023, 11:24 AMFilip Panovski
02/03/2023, 12:00 PM0.19.0
development begins (even if it just ends up being writing some extra tests).
Have a great weekend.Matthias Roels
02/03/2023, 3:32 PMMerel
02/03/2023, 3:45 PM