Dear team, I have several configurations (e.g.,

conf

&

dev

for local and dev deployments, respectively). I'm trying to switch between these using the CLI flag (e.g.,

kedro run --conf-source dev

and

kedro run --conf-source conf

but kedro seems to always default to using

conf

unless I specify

CONF_SOURCE = "dev"

in the

settings.py

file. Am I doing something wrong?

Someone will reply to you shortly. In the meantime, this might help:

sorry the correct command for this is :

kedro run --env dev

So there's no CLI command that does the same as

CONF_SOURCE

in

settings.py

?

well are we talking about the same thing:

├── conf
│   ├── base
│   │   └── parameters
│   ├── dev
│   ├── local
│   ├── prod
│   ├── stg
│   ├── test1
│   └── test2

those are called

configuration environments

so if you run

kedro run --env dev

it will take the

conf/dev

folder as the place to get config from The

CONF_SOURCE

directory in

settings.py

is designed to tell kedro to look for the

conf/

folder at the top.

kedro run --conf-source path/to/folder or path/to/archive.zip

does something slightly different and is designed for packaging

@datajoely that's not entirely correct, you can also provide a path to where your config lives. So it does work the same as changing

settings.py

or at least it should.

Thanks both! Kedro is still defaulting to the

conf/

configuration even if I pass a full path (e.g.,

kedro run --conf-source /Users/lino_fernandes/<path-to-kedro-project>/dev

What version of Kedro are you using? And can you maybe check what happens if you point to a directory where there's no valid config setup? Just to verify the path is read properly.

I'm on Kedro v0.19.11. If I point to a different path, that doesn't have a valid config (e.g., the data/ folder) I get the following error:

MissingConfigException: Given configuration path either does not exist or is not a valid directory: /Users/lino_fernandes/<path-to-kedro-project>/data/base

Hi @Lino Fernandes, Have you seen the changes to default env settings here - https://docs.kedro.org/en/stable/resources/migration.html#changes-to-the-default-environments ? I tried replicating the issue but for me the

kedro run

picks the conf-source specified. I created a

custom_conf

directory at my kedro project root and did

kedro run --conf-source custom_conf

. And my settings.py only has -

from kedro.config import OmegaConfigLoader

CONFIG_LOADER_CLASS = OmegaConfigLoader

So that error message does suggest that it's reading the paths correctly. I've tried to replicate your issue, but so far haven't been able to. Can you maybe add some log statements to print the conf path that's being used so we can be sure which one is being used?

Yes, I think it's working as intended, but I'm accessing the conf_loader from inside

nodes.py

to get some credentials. When I create a new instance of the conf_loader it always defaults to

conf/

. I guess this was just not the cleanest implementation from my side. Anyway, it works fine if I specify

CONF_SOURCE

in

settings.py

.

Ahhh yes if you're accessing the config loader directly, you'll have to ensure the conf_source is overwritten. See the docs https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#read-configuration that describe how to use the config loader directly.