Hello! I have a question about using environment variables in catalog.yml file. In that file, I have a data wherein I did an SQL query from my database and in the credentials part, I explicitly wrote the credential of that db. Now, I already stored that credential in an environment variable since I used it when connecting to my db in one of my nodes in my project. What I wanna do now is to use that environment variable in the catalog.yml file instead of explicitly writing the complete details of my db for security purposes since it is a sensitive information. Any thoughts? Thanks for the help!

You can use the``oc.env`` resolver as described here https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-load-credentials-through-environment-variables

Thanks for this. I've tried this now. However, I still got the same error. Here are the modifications I made on my project:

credentials.yml

postgres_db:
   con: ${oc.env:POSTGRES_DB_URI}

catalog.yml

spot_prices_ohlc:
  type: CachedDataset
  dataset:
    type: pandas.SQLQueryDataset
    sql: [SQL Query here]
    credentials: 
      con: postgres_db

And then I also set up the OmegaConfigLoader at

settings.py

from kedro.config import OmegaConfigLoader

CONFIG_LOADER_CLASS = OmegaConfigLoader

Can you check if I missed something or if I did a mistake on these modifications? Thanks a lot.

Your issue is that if you load credentials from an env var, you don't need to fill the credentials.yml. the correct syntax is :

credentials.yml

# empty

catalog.yml

spot_prices_ohlc:
  type: CachedDataset
  dataset:
    type: pandas.SQLQueryDataset
    sql: [SQL Query here]
    credentials: 
       con: ${oc.env: POSTGRES_DB_URI}

(Btw, What version of kedro are you using? You don't need to declare

OmegaConfigLoader

in

settings.py

if you use

kedro>=0.19

)

Thanks. Will try this. I am using kedro version 0.18.14.

Also add this to your

settings.py

-

from omegaconf.resolvers import oc

CONFIG_LOADER_ARGS = {
    "base_env": "base",
    "default_run_env": "local",
    "custom_resolvers": {
        "oc.env": oc.env,
    }
}

I tried it now and got this error:

TypeError: ConfigLoader.__init__() got an unexpected keyword argument 'custom_resolvers'

hmm you also need to have these lines in the

settings.py

-

from kedro.config import OmegaConfigLoader

CONFIG_LOADER_CLASS = OmegaConfigLoader

From 0.19.x

OmegaConfigLoader

is the default config loader but it has to be set explicitly in the 0.18.x series

thanks! finally got rid of that error, your solution seemed to have worked. but, i got a new error and i am not sure if it's on my implementation or is it a related error due to the integration of the omegaconfigloader because I didn't have this issue previously when I haven't used environment variables yet? here it is:

UnsupportedValueType: Value 'date' is not a supported primitive type full_key: start_date object_type=dict

Can you show the parameter or catalog entry this error is coming from?

The error seems to be coming from my streamlit app because in my Kedro project, I also implemented a streamlit app to display the generated outputs from my pipelines and nodes. In that app, I have several inputs that prompt users to enter values to it and two of those are the start_date and end_date inputs. They worked previously when omegaconfigloader wasn't integrated yet which is why I am also curious whether the error is something related to the omegaconfigloader or not.

omegaconf

, the underlying library for

OmegaConfigLoader

doesn’t support non primitive data types like date 🤔 Would it be possible to get these values from streamlit as strings? and then convert them to date format using a custom resolver

I see. Sources stated the same. I was able to overcome that error since date is not really supported by the library 😅 Even data frames are not supported but I already did some possible fixes on those and I am currently running the app again to check if the errors will still occur. Will provide updates here for the results, whether it is a success or not 😅

What could kedro have done to improve your experience?

Through Kedro, I was able to expand my knowledge in terms of handling environment variables and working on credentials in general, particularly with a data pipeline like Kedro.