Hi, require some clarification on OmegaConfigLoader(). I have duplicated keys across base and local conf environment. How do retrieve the configs without

ValueError: Duplicate keys found in ...

? In my code, have a function that runs before the actual kedro pipeline. I wish to retrieve the config in the function and prioritize the config attributes defined in local env sample code

###### main.py #####
if __name__ == "__main__":
    # Bootstrap the project to make the config loader available
    project_path = Path.cwd()
    bootstrap_project(project_path)

    # Create a Kedro session
    with KedroSession.create(project_path=project_path) as session:
        # You can now access the catalog, pipeline, etc. from the session
        # For example, to run the pipeline:
        conf_eda() # <------------- function
        session.run()
        pass

##### myfunc.py #####

def conf_eda():
    project_path = Path.cwd()
    conf_path = str(project_path/"conf")
    conf_loader = OmegaConfigLoader(
        conf_source=conf_path,
        )  
    parameters = conf_loader["parameters"] # <----------- error

    print(parameters["model_options"])

##### conf/base/parameters_data_science.yml #####
model_options:
  test_size: 100
  random_state: 3

##### conf/local/parameters_data_science.yml #####
model_options:
  test_size: 300
  random_state: 3

Hi @Chee Ming Siow, Kedro's merging rules do allow duplicate keys across different environments (like

base

and

local

), with values in

local

overriding those in

base

. https://docs.kedro.org/en/stable/configuration/configuration_basics.html#configuration-loading However, if the same key appears more than once within a single environment (e.g., defined in multiple files under

conf/base/

), you'll get a

ValueError

for duplicate keys. Could you check if that's the case in your setup?

Yup, checked that it is defined in different environment. I appended the screenshot of the error. If I used

session.load_context().config_loader

it is working as expected.

with KedroSession.create(project_path=project_path) as session:
        # You can now access the catalog, pipeline, etc. from the session
        # For example, to run the pipeline:
        # Retrieve the config loader from the session
        config_loader = session.load_context().config_loader
        # Access configurations
        parameters = config_loader.get("parameters")
        print(parameters) #<------ OK, result is expected

        conf_eda() # <------- Not OK, using OmegaConfigLoader()
        session.run()

I believe that I passed in the wrong params in my

OmegaConfigLoader()

in my function that resulted in

ValueError

.

thanks, you right, I think you should pass config loader from Kedro session to your function and use it there, to manage environments properly, like that:

config_loader = session.load_context().config_loader

conf_eda(config_loader)

Ok thanks