Ana Man
03/10/2023, 1:22 PMOmegaConfigLoader
. Apart from adding CONFIG_LOADER_CLASS = OmegaConfigLoader
to the settings.py, what other minimum changes are needed in your project to use this loader? having issues with running it 'out the box' (btw relatively new to kedro ecosystem)Juan Luis
03/10/2023, 1:53 PMAna Man
03/10/2023, 1:55 PMconf/base/parameters
) are not found in the Dataloged johnson
03/10/2023, 5:48 PMpipeline_1:
my_param: 10
Ana Man
03/13/2023, 11:44 AMed johnson
03/13/2023, 1:30 PMdatajoely
03/13/2023, 1:38 PMOmegaConfigLoader
support an empty catalog, that is possibly an oversight, not sure. @Merel do you have view here?
2. Re the second point @Ed Henry - it’s actually not going to be a 1:1 for replacement, but will over time provide more and more functionality, I think the plan is once we have at least feature parity with TemplatedConfigLoader
we will deprecate that and in 1.0.0 we have discussed just renaming OmegaConfigLoader
to just the ConfigLoader
Merel
03/13/2023, 1:44 PMOmegaConfigLoader
doesn’t inherit from the TemplatedConfigLoader
but from the base AbstractConfigLoader
and so it has completely separate functionality.datajoely
03/13/2023, 1:59 PMAna Man
03/13/2023, 2:11 PM> _ = session.run()
test_project.py:285:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../../../venv/lib/python3.9/site-packages/kedro/framework/session/session.py:408: in run
catalog = context._get_catalog(
../../../../../venv/lib/python3.9/site-packages/kedro/framework/context/context.py:280: in _get_catalog
conf_catalog = self.config_loader["catalog"]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = OmegaConfigLoader(conf_source=/private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytes...tials': ['credentials*', 'credentials*/**', '**/credentials*'], 'logging': ['logging*', 'logging*/**', '**/logging*']})
key = 'catalog'
def __getitem__(self, key) -> Dict[str, Any]:
"""Get configuration files by key, load and merge them, and
return them in the form of a config dictionary.
Args:
key: Key of the configuration type to fetch.
Raises:
KeyError: If key provided isn't present in the config_patterns of this
``OmegaConfigLoader`` instance.
MissingConfigException: If no configuration files exist matching the patterns
mapped to the provided key.
Returns:
Dict[str, Any]: A Python dictionary with the combined
configuration from all configuration files.
"""
# Allow bypassing of loading config from patterns if a key and value have been set
# explicitly on the ``OmegaConfigLoader`` instance.
if key in self:
return super().__getitem__(key)
if key not in self.config_patterns:
raise KeyError(
f"No config patterns were found for '{key}' in your config loader"
)
patterns = [*self.config_patterns[key]]
read_environment_variables = key == "credentials"
# Load base env config
base_path = str(Path(self.conf_source) / self.base_env)
base_config = self.load_and_merge_dir_config(
base_path, patterns, read_environment_variables
)
config = base_config
# Load chosen env config
run_env = self.env or self.default_run_env
env_path = str(Path(self.conf_source) / run_env)
env_config = self.load_and_merge_dir_config(
env_path, patterns, read_environment_variables
)
# Destructively merge the two env dirs. The chosen env will override base.
common_keys = config.keys() & env_config.keys()
if common_keys:
sorted_keys = ", ".join(sorted(common_keys))
msg = (
"Config from path '%s' will override the following "
"existing top-level config keys: %s"
)
_config_logger.debug(msg, env_path, sorted_keys)
config.update(env_config)
if not config:
> raise MissingConfigException(
f"No files of YAML or JSON format found in {base_path} or {env_path} matching"
f" the glob pattern(s): {[*self.config_patterns[key]]}"
)
E kedro.config.abstract_config.MissingConfigException: No files of YAML or JSON format found in /private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytest-210/test_other_kedro_loader0/test_project/kedro_test_project/conf/base or /private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytest-210/test_other_kedro_loader0/test_project/kedro_test_project/conf/local matching the glob pattern(s): ['catalog*', 'catalog*/**', '**/catalog*']
../../../../../venv/lib/python3.9/site-packages/kedro/config/omegaconf_config.py:177: MissingConfigException