https://kedro.org/ logo
Join Slack
Powered by
Hi again! i have a quick questions about `OmegaCon...
# questions
a
Hi again! i have a quick questions about 
OmegaConfigLoader
. Apart from adding 
CONFIG_LOADER_CLASS = OmegaConfigLoader
to the settings.py, what other minimum changes are needed in your project to use this loader? having issues with running it 'out the box' (btw relatively new to kedro ecosystem)
j
hello @Ana Man! are you seeing an error message? or what kind of issues are you having?
a
Hi! its stating that all my parameters for my project (that i have defined in my parameter.yml file located in 
conf/base/parameters
) are not found in the Datalog
e
are you using nested parameters in your parameters config(s) e.g.
Copy code
pipeline_1:
    my_param: 10
a
I managed to fix this issue by upgrading to 0.18.6
But i also have another question, if i create a simple pipeline that doesn't have any inputs, and i have an empty catalog / parameter.yml , it seems that i can't use OmegaConfigLoader as it expected a populated catalog (although this is possible with TemplateConfigLoader). Just want to double check that this was intentional
e
I would naively expect that OmegaConfigLoader would be backwards compatible with TemplateConfigLoader, but I'm not sure. Maybe @datajoely can comment on this functionality.
d
so there are two things here
1. I think we should make 
OmegaConfigLoader
support an empty catalog, that is possibly an oversight, not sure. @Merel do you have view here? 2. Re the second point @Ed Henry - it’s actually not going to be a 1:1 for replacement, but will over time provide more and more functionality, I think the plan is once we have at least feature parity with 
TemplatedConfigLoader
we will deprecate that and in 1.0.0 we have discussed just renaming 
OmegaConfigLoader
to just the 
ConfigLoader
👍 1
m
Point 1 is indeed an oversight. Feel free to create an issue for it and we’ll address it asap. On point 2, @datajoely is correct. The 
OmegaConfigLoader
doesn’t inherit from the 
TemplatedConfigLoader
but from the base 
AbstractConfigLoader
and so it has completely separate functionality.
👍 1
d
@Ana Man would you mind sharing the stack trace thrown with an empty catalog?
a
Copy code
>           _ = session.run()

test_project.py:285: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../../venv/lib/python3.9/site-packages/kedro/framework/session/session.py:408: in run
    catalog = context._get_catalog(
../../../../../venv/lib/python3.9/site-packages/kedro/framework/context/context.py:280: in _get_catalog
    conf_catalog = self.config_loader["catalog"]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = OmegaConfigLoader(conf_source=/private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytes...tials': ['credentials*', 'credentials*/**', '**/credentials*'], 'logging': ['logging*', 'logging*/**', '**/logging*']})
key = 'catalog'

    def __getitem__(self, key) -> Dict[str, Any]:
        """Get configuration files by key, load and merge them, and
        return them in the form of a config dictionary.
    
        Args:
            key: Key of the configuration type to fetch.
    
        Raises:
            KeyError: If key provided isn't present in the config_patterns of this
               ``OmegaConfigLoader`` instance.
            MissingConfigException: If no configuration files exist matching the patterns
                mapped to the provided key.
    
        Returns:
            Dict[str, Any]:  A Python dictionary with the combined
               configuration from all configuration files.
        """
    
        # Allow bypassing of loading config from patterns if a key and value have been set
        # explicitly on the ``OmegaConfigLoader`` instance.
        if key in self:
            return super().__getitem__(key)
    
        if key not in self.config_patterns:
            raise KeyError(
                f"No config patterns were found for '{key}' in your config loader"
            )
        patterns = [*self.config_patterns[key]]
    
        read_environment_variables = key == "credentials"
    
        # Load base env config
        base_path = str(Path(self.conf_source) / self.base_env)
        base_config = self.load_and_merge_dir_config(
            base_path, patterns, read_environment_variables
        )
        config = base_config
    
        # Load chosen env config
        run_env = self.env or self.default_run_env
        env_path = str(Path(self.conf_source) / run_env)
        env_config = self.load_and_merge_dir_config(
            env_path, patterns, read_environment_variables
        )
    
        # Destructively merge the two env dirs. The chosen env will override base.
        common_keys = config.keys() & env_config.keys()
        if common_keys:
            sorted_keys = ", ".join(sorted(common_keys))
            msg = (
                "Config from path '%s' will override the following "
                "existing top-level config keys: %s"
            )
            _config_logger.debug(msg, env_path, sorted_keys)
    
        config.update(env_config)
    
        if not config:
>           raise MissingConfigException(
                f"No files of YAML or JSON format found in {base_path} or {env_path} matching"
                f" the glob pattern(s): {[*self.config_patterns[key]]}"
            )
E           kedro.config.abstract_config.MissingConfigException: No files of YAML or JSON format found in /private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytest-210/test_other_kedro_loader0/test_project/kedro_test_project/conf/base or /private/var/folders/15/rn_1jn813x31f7f9b9mssm480000gn/T/pytest-of-anasthasia_manu/pytest-210/test_other_kedro_loader0/test_project/kedro_test_project/conf/local matching the glob pattern(s): ['catalog*', 'catalog*/**', '**/catalog*']

../../../../../venv/lib/python3.9/site-packages/kedro/config/omegaconf_config.py:177: MissingConfigException
7 Views
Open in Slack
PreviousNext
Powered by