https://kedro.org/ logo
Powered by
#questions
Title
# questions
f

Fazil B. Topal

08/16/2023, 6:25 PM
hey all, we have the following use case regardings configs. We want to do the following: `conf/base`:
Copy code
_pandas:
  type: pandas.CSVDataSet
  someconfhere: ....

companies:
  type: ${_pandas.type}
  filepath: data/01_raw/companies.csv
conf/staging
:
Copy code
_pandas:
  type: pandas.CSVDataSet
  stagingConfHere: ....
We wanna define the generic catalog in the base and other related ones in the following confs. However this doesn't work at the moment, we have the to define the whole 
companies
thing again. I believe it's due to this logic here: https://github.com/kedro-org/kedro/blob/main/kedro/config/omegaconf_config.py#L297 When we load the base catalog, we resolve the values and then we read the next env and resolve it again. Our example above only works if resolving is done last. I am not sure if it was a design choice or not. I thought about overwriting some functions but load_and_merge is a big one and i'd have to overwrite the 
__getitem__
as well but that can done easily i think. Thanks in advance!
👍🏼 1
👍 1
y

Yolan Honoré-Rougé

08/16/2023, 8:26 PM
Hi, I am facing the same issue and I think this is arguably a bug (it changes the current behaviour of ``(Templated)ConfigLoader`` and does not seems to be done on purpose). If someone if the teams see this, is there a workaround for this ?
👍🏼 1
👍 1
a

Ankita Katiyar

08/17/2023, 9:25 AM
Hi Fazil and Yolan, I believe this is the intended behaviour for 
OmegaConfigLoader
. We don’t necessarily allow for variable interpolation across environments.
We are working on “globals” feature currently which might help in this case. Essentially, you would have variables in 
conf/base/globals.yml
or 
conf/staging/globals.yml
which would be loaded and resolved first (duplicate keys in staging would overwrite base in globals) . And then you could inject this into your catalog like -
Copy code
companies:
  type: "${globals:_pandas.type}"
  filepath: data/01_raw/companies.csv
🥳 1
There is another one in the works which would allow you to override interpolation keys through the CLI
f

Fazil B. Topal

08/17/2023, 9:32 AM
Thanks Ankita, the globals feature would exactly solve this case I believe. I find that syntax a bit more clear, I see it's already in PR. Do we expect to get this out in the next release? I can get by for 1-2 weeks as is if we'll have that soon 🙂
a

Ankita Katiyar

08/17/2023, 9:37 AM
We are aiming for a release sometime next week, I’ll keep you posted.
🥳 2
thankyou 3
A workaround for now could be a custom 
OmegaConfigLoader
with overridden 
__getitem__
and 
load_and_merge_dir
where the resolution for variable happens outside the 
load_and_merge_dir
method. I can help if you’d like.
f

Fazil B. Topal

08/17/2023, 9:40 AM
Yes, that's was i was gonna do but since this release is coming soon, i can wait till then 🙂 thanks so much!
🙌 1
but your PR seems easier than overriding, couple of functions and it works? Who knew 😄
😄 1
y

Yolan Honoré-Rougé

08/17/2023, 10:09 AM
Thank you @Ankita Katiyar for the explanation. I don't really the rationale to prevent cross environnement interpolation, but I am quite happy with the globals solution. I also had the same solution of overriding ``OmegaConfigLoader`` in mind but I'll wait for the release too ;) Thank you very much!
Powered by