Feature highlight! (from Top 10 features added to the Kedro ecosystem in 2024) Powerful configuration management apple settings 512 In Kedro 0.18.5, we introduced the

OmegaConfigLoader

, replacing the legacy

ConfigLoader

and

TemplatedConfigLoader

. This new configuration loader, powered by OmegaConf, brings a range of features that were previously unavailable. Here are two notable enhancements: 1. Config file merging strategy: with

OmegaConfigLoader

, you can specify a custom merging strategy for your config files, moving beyond the default destructive merge. This flexibility allows you to define different strategies for specific configurations directly in your project's

settings.py

file. For example:

from kedro.config import OmegaConfigLoader

CONFIG_LOADER_CLASS = OmegaConfigLoader

CONFIG_LOADER_ARGS = {
    "merge_strategy": {
        "parameters": "soft",
        "spark": "destructive",
        "mlflow": "soft",
    }
}

In this case, "soft" merging will preserve existing values unless explicitly overridden, while "destructive" merging will completely replace them. 2. Dynamic value resolvers: instead of hard-coding values in your configuration files,

OmegaConfigLoader

enables dynamic computation of values using OmegaConf’s resolver functionality. Resolvers allow you to define custom logic to calculate values for parameters or catalog entries, or even inject values from external sources—making your configurations more flexible and adaptable. meow code Brought to you by: @Ankita Katiyar @Merel K Available from: Kedro

0.19.0

📖 Read more: https://docs.kedro.org/en/stable/configuration/index.html

Any examples of how the soft merge would look like? IF i have this defined in the base:

dataset:
    filepath: s3://....
    type: CSVDataset

and then defining the following in another environment:

dataset:
    filepath: tmp/....

would preserve the type field?I couldn't find any example in the docs as well.

Hey @Fazil Topal, yeah, the

type

field would be preserved when

merge_strategy

for

catalog

is set to

soft

! Docs link: https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-change-the-merge-strategy-used-by-omegaconfigloader Maybe we should add an example there!

Thanks @Ankita Katiyar adding an example would be super nice. It wasn't 100% clear what the soft would do exactly. Specially with nested keys etc.