Hi all I am happy to learn that `0 18 13` now lets us use gl

Question

Hi all I am happy to learn that `0 18 13` now lets us use globals and templating variables with `OmegaConfigLoader` party wizard But I have trouble getting it to work with configuration for multiple environments Would you expect this simplified example to work I have a catalog yml using templating variables and I want to change the template variables for different environments Hopefully then I don t need to re define entries ``` base catalog yml primary input 1 type pandas CSVDataSet filepath $ storage prefix data primary input 1 csv credentials $ storage credentials ``` Base template for ` storage` ``` base catalog templating yml storage prefix base credentials null``` Local template for ` storage` ``` local catalog templating yml storage prefix data lt different here credentials null``` Template for ` aws` as example ``` local catalog templating yml storage prefix <s3 my data bucket data> credentials dev s3``` When testing the ` storage` variable does not get overwritten when changing environments I expect that the dataset in the `base` environment has a different filepath than in the `local` environment But nothing changes in the `local` env disappointed gt `base` ``` reload kedro env=base catalog datasets primary input 1 describe gt gt gt filepath PurePosixPath test omega templating base data primary input 1 csv protocol file load args save args index False version None ``` gt `local` ``` reload kedro env=local catalog datasets primary input 1 describe gt gt gt filepath PurePosixPath test omega templating base data primary input 1 csv protocol file load args save args index False version None ``` ^ I expected ` filepath test omega templating data data primary input 1 csv`

Ankita Katiyar · Answer

Try using globals with the globals resolver for templating values across environments <https docs kedro org en stable configuration advanced configuration html how to use global variables with the omegaconfigloader>

Ankita Katiyar · Answer

The global variables should be in `globals yml`

Ankita Katiyar · Answer

Regular variable interpolation does not work across environments by design the catalog entries with variable interpolation are resolved per environment But for globals all the globals are loaded first the keys in local will overwrite the base and then resolved in the catalog parameters etc

Lodewic van Twillert · Answer

Oh ok yeah I considered that first but then opted against `globals` because I wouldn t actually re use these values for both params and catalog The docs say `The benefit of using globals over regular variable interpolation is that the global variables are shared across different configuration types such as catalog and parameters ` and I didn t want that benefit but it does suggest that regular variable interpolation would do the same thing And it would make my template variables a little shorter in syntax nerd face Makes sense to use globals though given how they are loaded I will try that next thanks for the explanation +1

Lodewic van Twillert · Answer

To follow up your suggestion worked and I also managed to reduce the syntax the way I wanted Here s what I end up with hope to hear your thoughts `globals yml` The only file that changes between each environment `catalog yml` I don t wan t to re define this in any environment only 1 in `base` `catalog templating yml` to capture the globals and turn them into composite parameters used throughout the catalog And reduce syntax length in the `catalog yml` gt `base` ``` globals yml storage prefix base credentials null folders raw 01 raw intermediate 02 intermediate primary 03 primary feature 04 feature model input 05 model input models 06 models model output 07 model output reporting 08 reporting``` ``` catalog yml primary input 1 type pandas CSVDataSet filepath $ folders primary input 1 csv credentials $ credentials ``` ``` catalog templating yml folders raw $ globals storage prefix $ globals folders raw intermediate $ globals storage prefix $ globals folders intermediate primary $ globals storage prefix $ globals folders primary feature $ globals storage prefix $ globals folders feature model input $ globals storage prefix $ globals folders model input models $ globals storage prefix $ globals folders models model output $ globals storage prefix $ globals folders model output reporting $ globals storage prefix $ globals folders reporting credentials $ globals storage credentials ``` gt `local` ``` globals yml storage prefix data credentials null``` gt `aws` ``` globals yml storage prefix <s3 my data bucket data> credentials dev s3```

Ankita Katiyar · Answer

Perhaps we should add the part about cross environment interpolation also being a benefit of using the globals feature to the docs smile