Hi everyone, I am trying to run

kedro run --conf-source=/path/to/another/conf

to specify another configuration source as stated in the docs. I also have another function that programmatically reads in the configs:

from kedro.config import OmegaConfigLoader
from kedro.framework.project import settings

def read_omegaconfig(
    source: str = "catalog",
    conf_source: str = settings.CONF_SOURCE,
) -> Dict[str, Any]:
    """Extracts catalog configuration from the conf folder

    Args:
        source: str
            Source of the configuration. e.g. 'catalog' or 'parameters'
        conf_source: str
            Source of the configuration files. e.g. 'conf'

    Returns:
        Dict[str, Any]: Configuration dictionary
    """
    conf_loader = OmegaConfigLoader(
        conf_source=conf_source, base_env="base", default_run_env="local"
    )
    conf_catalog = conf_loader[source]
    return conf_catalog

However, this seems to still be reading the default

conf

directory that is defined in

settings.CONF_SOURCE

, instead of the new

--conf-source

directory. I don't really understand how the CLI's

--conf-source

flag works. I thought the CLI flag would have replaced

CONF_SOURCE

in the settings somehow. Is there another way to solve the problem that I'm facing? Appreciate any help! Thanks :)

Someone will reply to you shortly. In the meantime, this might help:

Hi, how is your above function called in your actual code? You are not supposed to interact directly with the config loader unless you're doing very specific stuff. I think what you are looking for is likely the after_context_created hook which gives you access to the OmegaConfigLoader defined by kedro itself. In case you relly need to access it which is doubt)… I haven't look at the code but conf_source is likely overriden in the framework.cli run command, retrieving the CLI parameters to override settings.

the above function is called in almost everywhere in my code. for example inside my

nodes.py

and

pipeline.py

i run the training loop, but the

ModelTrainer

class is defined in another script which reads in a big

parameters_model_config.yml

file (which i will then read in via the above function). i am doing this because i want to avoid passing in a lot of parameters via my

pipeline.py

file. if it's not possible, then is there a way i can find out (in my Python script) what CLI flags were being passed to during

kedro run

? thanks.

Yes you can use

before_pipeline_run

hooks to do this

sorry but does anyone have an example code on how to use hooks to read in CLI params? so for example when i run

kedro run --conf-source=/path/to/another/conf

, the whole pipeline will have information on the conf source path. thanks for the replies so far!

The real question is more : can you describe what you are trying to do? We cannot help because we don't understand your goal, hence it's really hard to help. I don't think using conf_source directly will ever be the solution to your problem. For instance, you can just ok nstantiate the ModelTrainer in a node , pass all the parameters of your dict config file to the node as a single parameter (the dict of parameters) and launch your training. Can you show some code please?