Hi fellows, I would like to parse parameters like ...
# questions
Hi fellows, I would like to parse parameters like
as ISO8601 duration. After a bit of digging, I thought of using a custom resolver for
as follows
Copy code
from kedro.config import OmegaConfigLoader


    "custom_resolvers": {
        "parse-duration": parse_iso_8601_duration
When using the catalog with an entry like
Copy code
duration: ${parse-duration:1d}
it works really well. Now, I would like to do the same while passing the parameter through the CLI. I tried
Copy code
kedro run --pipeline=test_parsing --params duration:\${parse-duration:1d}
escaping the second part of the command. It seems that the loader goes indeed through the parser but the value is not associated with the variable
. The test pipeline is the following
Copy code
from kedro.pipeline import Pipeline, pipeline, node

def create_pipeline(**kwargs) -> Pipeline:
    return pipeline([node(lambda d: print(d), inputs=["params:duration"], outputs=None)])
Is custom resolver the correct way to do so? If so, any hint on what may go wrong with the CLI?
🤔 1
I think it would be easier to leave it as string on YAML / config level and parse it in the node. But I would like to see what others think, because it’s an interesting case 🤔
Actually, this is what we are doing right now but it means to do it in all of the pipelines. I would have prefer to get the right type after parsing the catalog. I was also thinking of using a hook.
I also initially though about a hook, but then the hook would need to know what to parse and what not to parse 🤔
Summon @Nok Lam Chan / @Ankita Katiyar 🧙
Taking a look! We don’t resolve the CLI params really, but this could be something we can add for 0.19 (cc @Sajid Alam since he’s already working on a related ticket) I’ll look more into this unless @Nok Lam Chan has some more insights in the meantime. Once the catalog is created in the context for a session run, the parameters are added to the catalog as `MemoryDataset`s and that’s what is used by the pipelines as inputs, so manipulating parameters after that point doesn’t work - I tried with some hooks
As @Ankita Katiyar said, CLI expect simple parameters, I don’t like the idea of trying to resolve
as this is what we try to avoid from the lesson of
and is why we introduce
resolver. Ideally you want this to work
Copy code
duration: "${foo: ${runtime_params: duration, 'placeholder'}}"
Unfortunately it doesn’t, you may work around this with
Copy code
duration: "${foo: ${runtime_params: duration_, 'placeholder'}}"
and using
Copy code
kedro run --pipeline=test_parsing --params duration_=<something>
👍 1


The parameters is actually interpolated correctly as you expected until it get override at the end with a
call. Fundamentally, the issue is that there could be a recursive definition of “runtime_params”, and right now config loader take any
value as is. what I mean by
Copy code
key: $runtime_params: {"key"} + 1
If I do
kedro run --params key=1
, what should be the value? Should it be
Thanks for all the feedback. I missed the runtime parameters in the documentation. The combination of it with the custom resolver solve my question. Thanks!
👍🏼 1
👍 1