👋 Hello, team! New to Kedro community. Excited to be part of this! I have a question - we were earlier using Kedro 0.18.4 with Kedro Configloader as part of the GithubActions CI checks. Ever since upgrading pipeline to Kedro 0.19.2, I was getting a CI failure.

/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/tests/test_run.py:16: in <module>
    from kedro.config import ConfigLoader
ImportError: cannot import name 'ConfigLoader' from 'kedro.config' (/home/runner/.cache/pypoetry/virtualenvs/audiences-NIUknnYt-py3.9/lib/python3.9/site-packages/kedro/config/__init__.py)

Upon checking the Kedro Docs, https://docs.kedro.org/en/stable/configuration/config_loader_migration.html#configloader-to-omegaconfigloader, I've updated my settings.py file to have Kedro OmegaConfigLoader. However, the pipeline which used to run fine earlier, gives the following errors when executed locally using base environment. so after changing it to omega config loader, it is giving errors by saying that duplicate keys exist for base and mlops environments. Although they are different.

ValueError: Duplicate keys found in

../../../conf/base/mlflow.yml and

../../../conf/mlops/mlflow.yml: server, tracking

If I remove one of the mlflow.yml files, it then gives the duplicate error for all the catalog and parameters between the two environments.

ValueError: Duplicate keys found in

../../../conf/mlops/catalog.yml and

../../../conf/base/catalog.yml:

constraint_performance, target_profile, targeting_dataset,...

Problem is, the catalog definitely has to be same between the two environments as we are dealing with exactly same datasets/datatypes/output paths etc. Not sure how to fix this.

Hello, could you share the

CONFIG_LOADER_ARGS

that you pass in

settings.py

?

That’s what I think it is too 🤔

yup thats correct - default args. # Migrate to OmegaConfigLoader, as TemplatedConfigLoader/ ConfigLoader will be deprecated soon. CONFIG_LOADER_CLASS = OmegaConfigLoader CONFIG_LOADER_ARGS = { "config_patterns": { "spark": ["spark*/", "spark*/**"], } }

You just need to change it to add

base_env

and

default_run_env

too -

CONFIG_LOADER_ARGS = {
      "base_env": "base",
      "default_run_env": "local",
          "config_patterns": {
              "spark": ["spark*", "spark*/**"],
          }
}

@Shubham Jha But you are using passing spark as a config pattern? So if you don't provide anything for

CONFIG_LOADER_ARGS

, it should work as expected, if you added new one you need to provide the arugments.

Thanks @Ankita Katiyar - your suggestion works. Somehow I missed to add this base_env. @Nok Lam Chan - actually no arguments were needed to be passed earlier. So what you're saying makes sense. I will test again after removing the

CONFIG_LOADER_ARGS