Nok Lam Chan
08/08/2023, 10:40 AMHello everyone, I have a question regarding the usage of environments in combination with the OmegaConfigLoader.
I have a file calledcc @Gerrit Schoettlerin mycatalog_globals.yml
config folder, and also in mybase/
config folder. When I executeprod/
, the settings from the file inkedro run --env=prod
are still used.base/
base
+ local
as default, when you specific prod
, it means base
+ prod
. If the same entry exist in both environment, prod
takes priority but otherwise they will be merged instead.Gerrit Schoettler
08/08/2023, 10:46 AMOmegaConfigLoader
is not using the catalog_globals.yml
file in the folder of the env (it's still using the file in the base folder).Nok Lam Chan
08/08/2023, 10:50 AMbase/catalog.yml
base/catalog_globals.yml
prod/catalog_globals.yml
prod/catalog.yml
Gerrit Schoettler
08/08/2023, 11:28 AM_bucket_name: <s3://ABC>
prod/catalog_globals.yml
_bucket_name: local_path_1
base/catalog.yml
catalog_entry_1:
type: pandas.ParquetDataSet
filepath: ${_bucket_name}/file_name_1.parquet
catalog_entry_2:
type: pandas.ParquetDataSet
filepath: ${_bucket_name}/file_name_2.parquet
prod/catalog.yml
catalog_entry_2:
type: pandas.ParquetDataSet
filepath: local_path_2/file_name_2.parquet
kedro run --env=prod,
, it still uses the _bucket_name
from the base/catalog_globals.yml
, although I was intending to overwrite itNok Lam Chan
08/08/2023, 12:05 PMGerrit Schoettler
08/08/2023, 12:09 PMAnkita Katiyar
08/08/2023, 12:52 PMprod/catalog.yml
which will overwrite the one in base/catalog.yml
is the only thing that will workbase
first and then load and resolve the config in local
(or prod
in this case). Even with the globals resolver that is still in development, we never propagate any values from run time environments to base environment.Gerrit Schoettler
08/08/2023, 2:18 PMAnkita Katiyar
08/08/2023, 2:59 PMkedro run --params: __my_bucket_ = "local__path_1"
to override “_ my_ bucket” for that particular runGerrit Schoettler
08/10/2023, 10:03 AMkedro run --params: __my_bucket_ = "local__path_1"
should look like?Ankita Katiyar
10/16/2023, 8:46 AMOmegaConfigLoader
is out already. Docs - https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-use-global-variables-with-the-omegaconfigloaderGerrit Schoettler
10/16/2023, 8:47 AMAnkita Katiyar
10/16/2023, 8:47 AMglobals:
resolverGerrit Schoettler
10/16/2023, 8:49 AMAnkita Katiyar
10/16/2023, 8:50 AMGerrit Schoettler
10/16/2023, 8:50 AMAnkita Katiyar
10/16/2023, 8:51 AMcatalog_entry_1:
type: pandas.ParquetDataSet
filepath: "${runtime_params:bucket_name. default_value}"/file_name_1.parquet
Nok Lam Chan
10/16/2023, 8:51 AMAnkita Katiyar
10/16/2023, 8:51 AMglobals.yml
you would have to do it directly in the catalogGerrit Schoettler
10/16/2023, 8:52 AMAnkita Katiyar
10/16/2023, 8:53 AMGerrit Schoettler
10/16/2023, 8:53 AMAnkita Katiyar
10/16/2023, 8:53 AMGerrit Schoettler
10/16/2023, 8:54 AMAnkita Katiyar
10/16/2023, 8:58 AMbase/globals.yml
with bucket_name:<s3_path>
and a prod/globals.yml
with bucket_name:<local_path>
where the path
variable will be overwritten by the prod
version
And your catalog.yml
will look like:
catalog_entry_1:
type: pandas.ParquetDataSet
filepath: "${globals:bucket_name. default_value}"/file_name_1.parquet
Gerrit Schoettler
10/16/2023, 8:59 AMAnkita Katiyar
10/16/2023, 9:00 AMglobals
resolver to work across envsGerrit Schoettler
10/16/2023, 9:00 AM--env=prod
to control the environment from the deployment pipelineAnkita Katiyar
10/16/2023, 9:01 AMcatalog_globals.yml
is read as the “catalog” but not as “globals” if that makes sense, so variables in catalog_globals
are only available to catalog entries and have to begin with underscores so as to not be read by the config_loader as separate catalog entries--env=prod
or you can just define a default value to be used when a global value is not definedGerrit Schoettler
10/16/2023, 9:04 AM${runtime_params:bucket_name, <*default_value*>}
Is there any way to take the default value from the globals.yml, instead of hardcoding it in the catalog?${runtime_params:bucket_name, ${globals:bucket_name}}
Nok Lam Chan
10/23/2023, 2:51 PM${globals: bucket_name, ${runime_params: bucket_name}}
won’t work and this is by design.Gerrit Schoettler
10/23/2023, 2:54 PMNok Lam Chan
10/23/2023, 2:55 PMGerrit Schoettler
10/23/2023, 2:55 PM