https://kedro.org/ logo
#questions
Title
# questions
m

Massinissa Saïdi

02/08/2023, 1:59 PM
Hello kedroids ! I have a question about the read priority of credentials files. Suppose I have a
conf/base
and
conf/prod
environment and my
credentials.yml
file in
conf/local
. If I run
kedro run
, will
conf/local/credentials.yml
overwrite
conf/base/credentials.yml
? And if I run
kedro run --env prod
which credentials file will be used? I have the impression that it is the local file that is always used? Thank you
d

datajoely

02/08/2023, 2:00 PM
hey so the answer is on a per key basis, if the same key exists in both. Also a regular kedro run can be thought of as
kedro run --env base
I think local still wins in every case
m

Massinissa Saïdi

02/08/2023, 2:03 PM
"Local wins !" Thank you 🙂
d

datajoely

02/08/2023, 2:04 PM
you can mess around with your own config loader if you want to change this
m

Massinissa Saïdi

02/08/2023, 2:05 PM
But it seems if I have set nothing in
local/credentials.yml
and smthg in
prod/credentials.yml
I will get None value
d

datajoely

02/08/2023, 2:11 PM
hmm I wouldn’t expect that
i.e. the key only exists in prod?
m

Massinissa Saïdi

02/08/2023, 2:19 PM
yes, the credentials in local was empty
and I remove
local/credentials.yml
i got this error:
Copy code
No files found in ['/Users/.../conf/base', '/Users/.../conf/local'] matching the glob pattern(s): ('credentials*', 'credentials*/**', '**/credentials*')
  warn(f"Credentials not found in your Kedro project config.\n{str(exc)}")
Ok its maybe because I read credentials from
KedroSession
as
Copy code
KedreSession.create().load_context()._get_config_credentials()
d

datajoely

02/08/2023, 2:41 PM
ah you need the file there
but it can be empty
or more specifically you need a file under the base or local folder to be present, but no content is needed.
a

Ankita Katiyar

02/08/2023, 2:43 PM
If you do
kedro run --env prod
, the config will be read from
conf/prod
and
conf/local
will not be read.
👍 1
Essentially config is always read from the
base
directory and then the run time env config which is
local
by default but if you specify
--env prod
it will load config only in
prod
👍 1
m

Massinissa Saïdi

02/08/2023, 3:26 PM
yes I think my mistake was my checking with this
KedreSession.create().load_context()._get_config_credentials()
d

datajoely

02/08/2023, 3:38 PM
sorry my mistake @Massinissa Saïdi - thanks @Ankita Katiyar
👍 2
m

Massinissa Saïdi

02/08/2023, 3:58 PM
do you think its possible to get credentials infos depending of environment with a function inside code ? Becase this
KedreSession.create().load_context()._get_config_credentials()
returns only what in local/credentials.yml
d

datajoely

02/08/2023, 3:59 PM
so all of this doesn’t happen in the context
the resolution is in the ConfigLoader class
m

Massinissa Saïdi

02/08/2023, 4:07 PM
with ConfigLoader class when I read the credentials with env variables it doesnt convert them
d

datajoely

02/08/2023, 5:12 PM
🙏 1
5 Views