https://kedro.org/ logo
#questions
Title
# questions
f

Felipe Monroy

02/02/2024, 1:31 PM
Hi, I know is still not possible to use something similar to the dataset factory patterns with the parameters. Do you know if there's any reason for this? Maybe I am missing something (like is already possible with OmegaConf) but I believe it would be a nice feature. At the moment I am using a custom DataCatalog to achieve it.
d

datajoely

02/02/2024, 1:35 PM
ooh I’d love to learn more about how this works? Is it a bit like Hydra to generate different permutation?
f

Felipe Monroy

02/02/2024, 1:40 PM
Not really, it is a bit limited. I am only reusing the catalog resolver with the parameters.
Copy code
class DataCatalogParamsFactory(DataCatalog):
    """DataCatalog that also resolves factories in the parameters."""

    @overrides
    def _get_dataset(
        self,
        dataset_name: str,
        version: Version | None = None,
        suggest: bool = True,
    ) -> AbstractDataset:
        params_pattern = {
            k: v
            for k, v in self._datasets.items()
            if k.startswith("params:") and self._is_pattern(k)
        }

        matched_pattern = self._match_pattern(params_pattern, dataset_name)
        if dataset_name not in self._datasets and matched_pattern:
            # If the dataset is a patterned dataset, materialise it and add it to
            # the catalog
            config_copy = copy.deepcopy(params_pattern[matched_pattern])
            dataset_config = self._resolve_config(
                dataset_name, matched_pattern, config_copy
            )

            # This only works with MemoryDataset, all the params corresponds to this.
            if isinstance(dataset_config, MemoryDataset):
                self.add(dataset_name, dataset_config)
                return dataset_config

        return super()._get_dataset(dataset_name, version, suggest)
a

Ankita Katiyar

02/02/2024, 1:40 PM
There’s an issue open for something like this - would you mind adding your use case as a comment? https://github.com/kedro-org/kedro/issues/3086
👍🏼 1
ty 1
j

Julian Nowak

03/06/2024, 9:17 AM
I added mine as well
❤️ 2
d

datajoely

03/06/2024, 9:18 AM
Thanks for contributing!
😎 1