in other news, I'm having trouble passing

dtypes

to the upcoming

polars.CSVDataSet

, not sure if there's a way to specify non-primitive types in the catalog YAML? https://github.com/kedro-org/kedro-plugins/issues/124

Unfortunately no, we can now introduce a OmegaConf resolver for this

yeap, that's what I imagined

it’s a engineering question, in truth we’re not sure how to approach this yet as it touches wider questions

yeah I meant "quick & dirty workaround" 😄 found https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#resolvers

quick and dirty is you subclassing the polars class and handling it yourself

Maybe globals for quick & dirty? https://kedro-org.slack.com/archives/C03RKP2LW64/p1676881692806879?thread_ts=1676863743.414139&cid=C03RKP2LW64

we also have users who want to use the

converters

element of

pd.read_excel

which take full functions as arguments and there’s no way to expose that as YAML today

about the

!!python

YAML stuff, I tried but

could not determine a constructor for the tag 'tag:<http://yaml.org|yaml.org>,2002:python/name:polars.Float64'

yeah sorry I was saying we haven’t enabled it for security reasons

oh, gotcha

@Merel tagging for visibility as longer term we should have an OmegaConf solution here 🙂

Yeah, it would be great to have OmegaConf resolvers configurable, right now I had to create a subclass of OmegaConfigLoader to run environment resolvers for non-credentials configs

@marrrcin Would you be able to share that class? I’d love to see it! And another question is, what are you using environment variables for outside of credentials?

It’s nasty, don’t look 🙈

class PatchedOmegaConfigLoader(OmegaConfigLoader):
    """
    This class is a patch for the OmegaConfigLoader class, it enables the use of oc.env interpolation
    for all config files.
    """

    def load_and_merge_dir_config(
        self,
        conf_path: str,
        patterns: Iterable[str],
        read_environment_variables: Optional[bool] = False,
    ) -> Dict[str, Any]:
        return super().load_and_merge_dir_config(
            conf_path, patterns, read_environment_variables=True
        )

And in the

settings.py

CONFIG_LOADER_CLASS = PatchedOmegaConfigLoader
# Keyword arguments to pass to the `CONFIG_LOADER_CLASS` constructor.
CONFIG_LOADER_ARGS = {"config_patterns": {"azureml": ["azureml*"]}}

I wanted to have some config entires for kedro-azureml plugin be settable from CI/CD at runtime, so I’ve used the env interpolation there

Thank you! That’s very insightful

In a way, the settings in our plugins are similar to settings of credentials, as they are more environment-specific and have practically no influence on the reproducibility of the pipelines

@marrrcin Have you by any chance played around with using custom resolvers with the

OmegaConfigLoader

? I’m trying out how to make this work nicely. I can get it working when subclassing

OmegaConfigLoader

but I’m wondering if there’s an easier way

Not yet

Let me know if you do! 🙂

feedback welcome 😄 https://github.com/kedro-org/kedro/issues/2407