Hi fellows, I would like to parse parameters like

1D

as ISO8601 duration. After a bit of digging, I thought of using a custom resolver for

OmegaConf

as follows

from kedro.config import OmegaConfigLoader

CONFIG_LOADER_CLASS = OmegaConfigLoader

CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "parse-duration": parse_iso_8601_duration
    }
}

When using the catalog with an entry like

duration: ${parse-duration:1d}

it works really well. Now, I would like to do the same while passing the parameter through the CLI. I tried

kedro run --pipeline=test_parsing --params duration:\${parse-duration:1d}

escaping the second part of the command. It seems that the loader goes indeed through the parser but the value is not associated with the variable

duration

. The test pipeline is the following

from kedro.pipeline import Pipeline, pipeline, node


def create_pipeline(**kwargs) -> Pipeline:
    return pipeline([node(lambda d: print(d), inputs=["params:duration"], outputs=None)])

Is custom resolver the correct way to do so? If so, any hint on what may go wrong with the CLI?

I think it would be easier to leave it as string on YAML / config level and parse it in the node. But I would like to see what others think, because it’s an interesting case 🤔

Actually, this is what we are doing right now but it means to do it in all of the pipelines. I would have prefer to get the right type after parsing the catalog. I was also thinking of using a hook.

I also initially though about a hook, but then the hook would need to know what to parse and what not to parse 🤔

Indeed.

Summon @Nok Lam Chan / @Ankita Katiyar 🧙

Taking a look! We don’t resolve the CLI params really, but this could be something we can add for 0.19 (cc @Sajid Alam since he’s already working on a related ticket) I’ll look more into this unless @Nok Lam Chan has some more insights in the meantime. Once the catalog is created in the context for a session run, the parameters are added to the catalog as `MemoryDataset`s and that’s what is used by the pipelines as inputs, so manipulating parameters after that point doesn’t work - I tried with some hooks

As @Ankita Katiyar said, CLI expect simple parameters, I don’t like the idea of trying to resolve

--params

as this is what we try to avoid from the lesson of

TemplatedConfigLoader

and is why we introduce

runtime_params

resolver. Ideally you want this to work

duration: "${foo: ${runtime_params: duration, 'placeholder'}}"

Unfortunately it doesn’t, you may work around this with

duration: "${foo: ${runtime_params: duration_, 'placeholder'}}"

and using

kedro run --pipeline=test_parsing --params duration_=<something>

Thanks for all the feedback. I missed the runtime parameters in the documentation. The combination of it with the custom resolver solve my question. Thanks!