https://kedro.org/ logo
#questions
Title
# questions
a

Ana Man

03/10/2023, 1:22 PM
Hi again! i have a quick questions about
OmegaConfigLoader
. Apart from adding
CONFIG_LOADER_CLASS = OmegaConfigLoader
to the settings.py, what other minimum changes are needed in your project to use this loader? having issues with running it 'out the box' (btw relatively new to kedro ecosystem)
j

Juan Luis

03/10/2023, 1:53 PM
hello @Ana Man! are you seeing an error message? or what kind of issues are you having?
a

Ana Man

03/10/2023, 1:55 PM
Hi! its stating that all my parameters for my project (that i have defined in my parameter.yml file located in
conf/base/parameters
) are not found in the Datalog
e

ed johnson

03/10/2023, 5:48 PM
are you using nested parameters in your parameters config(s) e.g.
Copy code
pipeline_1:
    my_param: 10
a

Ana Man

03/13/2023, 11:44 AM
I managed to fix this issue by upgrading to 0.18.6
But i also have another question, if i create a simple pipeline that doesn't have any inputs, and i have an empty catalog / parameter.yml , it seems that i can't use OmegaConfigLoader as it expected a populated catalog (although this is possible with TemplateConfigLoader). Just want to double check that this was intentional
e

ed johnson

03/13/2023, 1:30 PM
I would naively expect that OmegaConfigLoader would be backwards compatible with TemplateConfigLoader, but I'm not sure. Maybe @datajoely can comment on this functionality.
d

datajoely

03/13/2023, 1:38 PM
so there are two things here
1. I think we should make
OmegaConfigLoader
support an empty catalog, that is possibly an oversight, not sure. @Merel do you have view here? 2. Re the second point @Ed Henry - it’s actually not going to be a 1:1 for replacement, but will over time provide more and more functionality, I think the plan is once we have at least feature parity with
TemplatedConfigLoader
we will deprecate that and in 1.0.0 we have discussed just renaming
OmegaConfigLoader
to just the
ConfigLoader
👍 1
m

Merel

03/13/2023, 1:44 PM
Point 1 is indeed an oversight. Feel free to create an issue for it and we’ll address it asap. On point 2, @datajoely is correct. The
OmegaConfigLoader
doesn’t inherit from the
TemplatedConfigLoader
but from the base
AbstractConfigLoader
and so it has completely separate functionality.
👍 1
d

datajoely

03/13/2023, 1:59 PM
@Ana Man would you mind sharing the stack trace thrown with an empty catalog?
a

Ana Man

03/13/2023, 2:11 PM
Copy code
>           _ = session.run()

test_project.py:285: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../../venv/lib/python3.9/site-packages/kedro/framework/session/session.py:408: in run
    catalog = context._get_catalog(
../../../../../venv/lib/python3.9/site-packages/kedro/framework/context/context.py:280: in _get_catalog
    conf_catalog = self.config_loader["catalog"]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = OmegaConfigLoader(conf_source=/private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytes...tials': ['credentials*', 'credentials*/**', '**/credentials*'], 'logging': ['logging*', 'logging*/**', '**/logging*']})
key = 'catalog'

    def __getitem__(self, key) -> Dict[str, Any]:
        """Get configuration files by key, load and merge them, and
        return them in the form of a config dictionary.
    
        Args:
            key: Key of the configuration type to fetch.
    
        Raises:
            KeyError: If key provided isn't present in the config_patterns of this
               ``OmegaConfigLoader`` instance.
            MissingConfigException: If no configuration files exist matching the patterns
                mapped to the provided key.
    
        Returns:
            Dict[str, Any]:  A Python dictionary with the combined
               configuration from all configuration files.
        """
    
        # Allow bypassing of loading config from patterns if a key and value have been set
        # explicitly on the ``OmegaConfigLoader`` instance.
        if key in self:
            return super().__getitem__(key)
    
        if key not in self.config_patterns:
            raise KeyError(
                f"No config patterns were found for '{key}' in your config loader"
            )
        patterns = [*self.config_patterns[key]]
    
        read_environment_variables = key == "credentials"
    
        # Load base env config
        base_path = str(Path(self.conf_source) / self.base_env)
        base_config = self.load_and_merge_dir_config(
            base_path, patterns, read_environment_variables
        )
        config = base_config
    
        # Load chosen env config
        run_env = self.env or self.default_run_env
        env_path = str(Path(self.conf_source) / run_env)
        env_config = self.load_and_merge_dir_config(
            env_path, patterns, read_environment_variables
        )
    
        # Destructively merge the two env dirs. The chosen env will override base.
        common_keys = config.keys() & env_config.keys()
        if common_keys:
            sorted_keys = ", ".join(sorted(common_keys))
            msg = (
                "Config from path '%s' will override the following "
                "existing top-level config keys: %s"
            )
            _config_logger.debug(msg, env_path, sorted_keys)
    
        config.update(env_config)
    
        if not config:
>           raise MissingConfigException(
                f"No files of YAML or JSON format found in {base_path} or {env_path} matching"
                f" the glob pattern(s): {[*self.config_patterns[key]]}"
            )
E           kedro.config.abstract_config.MissingConfigException: No files of YAML or JSON format found in /private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytest-210/test_other_kedro_loader0/test_project/kedro_test_project/conf/base or /private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytest-210/test_other_kedro_loader0/test_project/kedro_test_project/conf/local matching the glob pattern(s): ['catalog*', 'catalog*/**', '**/catalog*']

../../../../../venv/lib/python3.9/site-packages/kedro/config/omegaconf_config.py:177: MissingConfigException
4 Views