Iñigo Hidalgo
10/19/2023, 2:33 PMmy_notebook.ipynb
conf/
credentials.yml
catalog.yml
#credentials.yml
azure_blob:
storage_options:
account_name: ${oc.env:STORAGE_ACCOUNT_ENV_KEY}
#catalog.yml
input_data:
type: polars.GenericDataset
filepath: az://${oc.env:CONTAINER_NAME_ENV_KEY}
credentials: azure_blob
# notebook
from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog
conf_loader = OmegaConfigLoader("conf/", base_env="", default_run_env="")
conf_loader["catalog"]
UnsupportedInterpolationType: Unsupported interpolation type oc.env
full_key: filtered_allocation.filepath
object_type=dict
I seem to be missing this step to enable resolvers https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-use-resolvers-in-the-omegaconfigloader
How would I do this in the interactive "standalone" way?
Thanks!CONFIG_LOADER_CLASS = OmegaConfigLoader
CONFIG_LOADER_ARGS = {
"custom_resolvers": {
"oc.env": oc.env,
}
}
before trying to load the config but it didn't make a difference, not surprising since it's meant to go inside settings.pyconf_loader = OmegaConfigLoader("conf/", base_env="", default_run_env="", **CONFIG_LOADER_ARGS)
Should've expected that hahadatajoely
10/19/2023, 2:51 PMIñigo Hidalgo
10/19/2023, 2:59 PMcredentials=conf_loader["credentials"]
to the DataCatalog.from_config methoddatajoely
10/19/2023, 3:09 PMYolan Honoré-Rougé
10/19/2023, 3:18 PMIñigo Hidalgo
10/19/2023, 3:35 PMdatajoely
10/19/2023, 3:42 PMJo Stichbury
10/19/2023, 4:31 PMIñigo Hidalgo
10/19/2023, 4:41 PMJo Stichbury
10/19/2023, 4:58 PMYolan Honoré-Rougé
10/19/2023, 5:13 PMJo Stichbury
10/19/2023, 5:14 PMYolan Honoré-Rougé
10/19/2023, 5:15 PMIñigo Hidalgo
10/23/2023, 5:08 PMglobals_dict=os.environ
from the old TemplatedConfigLoader but to avoid breaking existing code it would be nice to be able to preserve the existing syntax to interpolate ${environment_variable} without needing to specify oc.env
at the startdatajoely
10/23/2023, 5:09 PMafter_context_created
hook?Iñigo Hidalgo
10/23/2023, 5:13 PMdatajoely
10/23/2023, 5:13 PMglobals_dict
Iñigo Hidalgo
10/23/2023, 5:13 PMdatajoely
10/23/2023, 5:13 PMos.environ
to extra_params
when you create a context either via the CLI or via custom contextIñigo Hidalgo
10/23/2023, 5:18 PMself._globals
and a reference to a globals resolver, do you think that could be a way to rescue old functionality?datajoely
10/23/2023, 5:18 PMIñigo Hidalgo
10/23/2023, 5:20 PM_get_globals_value
could already be doing that? Basically I would just have to include the environment variables in self._globals?globals_oc = OmegaConf.create(self._globals)
interpolated_value = OmegaConf.select(
globals_oc, variable, default=default_value
)
Yolan Honoré-Rougé
10/23/2023, 5:21 PMafter_context_hook
you can do something like context.config_loader._globals.update(os.environ)
?Iñigo Hidalgo
10/23/2023, 5:21 PMYolan Honoré-Rougé
10/23/2023, 5:22 PMIñigo Hidalgo
10/23/2023, 5:22 PMYolan Honoré-Rougé
10/23/2023, 5:23 PMIñigo Hidalgo
10/23/2023, 5:24 PMYolan Honoré-Rougé
10/23/2023, 5:25 PMNok Lam Chan
10/23/2023, 5:58 PMYolan Honoré-Rougé
10/23/2023, 6:37 PM__get__
method of the config loaders does a lot of thing under the hood (on my phone too, I can't check either)Nok Lam Chan
10/24/2023, 12:35 PM# after_context_created
conf["globals"] = conf["globals"].update(my_conf) # some dict
Iñigo Hidalgo
10/24/2023, 2:35 PMNok Lam Chan
10/24/2023, 3:15 PM