:wave: Hello, team! New to Kedro community. Excit...
# questions
👋 Hello, team! New to Kedro community. Excited to be part of this! I have a question - we were earlier using Kedro 0.18.4 with Kedro Configloader as part of the GithubActions CI checks. Ever since upgrading pipeline to Kedro 0.19.2, I was getting a CI failure.
Copy code
/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/tests/test_run.py:16: in <module>
    from kedro.config import ConfigLoader
ImportError: cannot import name 'ConfigLoader' from 'kedro.config' (/home/runner/.cache/pypoetry/virtualenvs/audiences-NIUknnYt-py3.9/lib/python3.9/site-packages/kedro/config/__init__.py)
Upon checking the Kedro Docs, https://docs.kedro.org/en/stable/configuration/config_loader_migration.html#configloader-to-omegaconfigloader, I've updated my settings.py file to have Kedro OmegaConfigLoader. However, the pipeline which used to run fine earlier, gives the following errors when executed locally using base environment. so after changing it to omega config loader, it is giving errors by saying that duplicate keys exist for base and mlops environments. Although they are different.
ValueError: Duplicate keys found in
../../../conf/base/mlflow.yml and
../../../conf/mlops/mlflow.yml: server, tracking
If I remove one of the mlflow.yml files, it then gives the duplicate error for all the catalog and parameters between the two environments.
ValueError: Duplicate keys found in
../../../conf/mlops/catalog.yml and
constraint_performance, target_profile, targeting_dataset,...
Problem is, the catalog definitely has to be same between the two environments as we are dealing with exactly same datasets/datatypes/output paths etc. Not sure how to fix this.
Hello, could you share the
that you pass in
my intuition is this ^
That’s what I think it is too 🤔
yup thats correct - default args. # Migrate to OmegaConfigLoader, as TemplatedConfigLoader/ ConfigLoader will be deprecated soon. CONFIG_LOADER_CLASS = OmegaConfigLoader CONFIG_LOADER_ARGS = { "config_patterns": { "spark": ["spark*/", "spark*/**"], } }
the original Kedroconfig loader was not modified to pass any arguments though, at least not in the pipeline. it was a commented out settings.py with no arguments being passed.
You just need to change it to add
too -
Copy code
      "base_env": "base",
      "default_run_env": "local",
          "config_patterns": {
              "spark": ["spark*", "spark*/**"],
@Shubham Jha But you are using passing spark as a config pattern? So if you don't provide anything for
, it should work as expected, if you added new one you need to provide the arugments.
Thanks @Ankita Katiyar - your suggestion works. Somehow I missed to add this base_env. @Nok Lam Chan - actually no arguments were needed to be passed earlier. So what you're saying makes sense. I will test again after removing the
👍🏼 1
❤️ 1