Juan Luis
03/09/2023, 12:37 PMdtypes
to the upcoming polars.CSVDataSet
, not sure if there's a way to specify non-primitive types in the catalog YAML? https://github.com/kedro-org/kedro-plugins/issues/124datajoely
03/09/2023, 12:38 PMJuan Luis
03/09/2023, 12:38 PMdatajoely
03/09/2023, 12:39 PMJuan Luis
03/09/2023, 12:40 PMdatajoely
03/09/2023, 12:41 PMpl.xxxx
is resolved in a safe waymarrrcin
03/09/2023, 12:42 PMdatajoely
03/09/2023, 12:43 PMconverters
element of pd.read_excel
which take full functions as arguments and there’s no way to expose that as YAML todayJuan Luis
03/09/2023, 12:44 PM!!python
YAML stuff, I tried but could not determine a constructor for the tag 'tag:<http://yaml.org|yaml.org>,2002:python/name:polars.Float64'
datajoely
03/09/2023, 12:45 PMJuan Luis
03/09/2023, 12:45 PMTemplatedConfigLoader
trick worked, thanks @marrrcin!datajoely
03/09/2023, 12:50 PMOmegaConf
resolvers part of settings.py a bit like @marrrcin has done with his workaroundmarrrcin
03/09/2023, 1:40 PMMerel
03/09/2023, 1:42 PMmarrrcin
03/09/2023, 1:45 PMclass PatchedOmegaConfigLoader(OmegaConfigLoader):
"""
This class is a patch for the OmegaConfigLoader class, it enables the use of oc.env interpolation
for all config files.
"""
def load_and_merge_dir_config(
self,
conf_path: str,
patterns: Iterable[str],
read_environment_variables: Optional[bool] = False,
) -> Dict[str, Any]:
return super().load_and_merge_dir_config(
conf_path, patterns, read_environment_variables=True
)
And in the settings.py
CONFIG_LOADER_CLASS = PatchedOmegaConfigLoader
# Keyword arguments to pass to the `CONFIG_LOADER_CLASS` constructor.
CONFIG_LOADER_ARGS = {"config_patterns": {"azureml": ["azureml*"]}}
I wanted to have some config entires for kedro-azureml plugin be settable from CI/CD at runtime, so I’ve used the env interpolation thereMerel
03/09/2023, 1:49 PMmarrrcin
03/09/2023, 1:50 PMMerel
03/09/2023, 2:19 PMOmegaConfigLoader
? I’m trying out how to make this work nicely. I can get it working when subclassing OmegaConfigLoader
but I’m wondering if there’s an easier waymarrrcin
03/09/2023, 2:20 PMMerel
03/09/2023, 2:23 PMdatajoely
03/09/2023, 3:24 PM