https://kedro.org/ logo
#questions
Title
# questions
f

Flavien

11/16/2023, 3:24 PM
Hi fellows, I would like to parse parameters like
1D
as ISO8601 duration. After a bit of digging, I thought of using a custom resolver for
OmegaConf
as follows
Copy code
from kedro.config import OmegaConfigLoader

CONFIG_LOADER_CLASS = OmegaConfigLoader

CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "parse-duration": parse_iso_8601_duration
    }
}
When using the catalog with an entry like
Copy code
duration: ${parse-duration:1d}
it works really well. Now, I would like to do the same while passing the parameter through the CLI. I tried
Copy code
kedro run --pipeline=test_parsing --params duration:\${parse-duration:1d}
escaping the second part of the command. It seems that the loader goes indeed through the parser but the value is not associated with the variable
duration
. The test pipeline is the following
Copy code
from kedro.pipeline import Pipeline, pipeline, node


def create_pipeline(**kwargs) -> Pipeline:
    return pipeline([node(lambda d: print(d), inputs=["params:duration"], outputs=None)])
Is custom resolver the correct way to do so? If so, any hint on what may go wrong with the CLI?
🤔 1
m

marrrcin

11/16/2023, 3:41 PM
I think it would be easier to leave it as string on YAML / config level and parse it in the node. But I would like to see what others think, because it’s an interesting case 🤔
f

Flavien

11/16/2023, 3:42 PM
Actually, this is what we are doing right now but it means to do it in all of the pipelines. I would have prefer to get the right type after parsing the catalog. I was also thinking of using a hook.
m

marrrcin

11/16/2023, 3:43 PM
I also initially though about a hook, but then the hook would need to know what to parse and what not to parse 🤔
f

Flavien

11/16/2023, 3:43 PM
Indeed.
m

marrrcin

11/16/2023, 3:43 PM
Summon @Nok Lam Chan / @Ankita Katiyar 🧙
a

Ankita Katiyar

11/16/2023, 5:30 PM
Taking a look! We don’t resolve the CLI params really, but this could be something we can add for 0.19 (cc @Sajid Alam since he’s already working on a related ticket) I’ll look more into this unless @Nok Lam Chan has some more insights in the meantime. Once the catalog is created in the context for a session run, the parameters are added to the catalog as `MemoryDataset`s and that’s what is used by the pipelines as inputs, so manipulating parameters after that point doesn’t work - I tried with some hooks
n

Nok Lam Chan

11/17/2023, 4:02 AM
As @Ankita Katiyar said, CLI expect simple parameters, I don’t like the idea of trying to resolve
--params
as this is what we try to avoid from the lesson of
TemplatedConfigLoader
and is why we introduce
runtime_params
resolver. Ideally you want this to work
Copy code
duration: "${foo: ${runtime_params: duration, 'placeholder'}}"
Unfortunately it doesn’t, you may work around this with
Copy code
duration: "${foo: ${runtime_params: duration_, 'placeholder'}}"
and using
Copy code
kedro run --pipeline=test_parsing --params duration_=<something>
👍 1

https://noklam.github.io/blog/posts/kedro_config_loader/2023-11-16-kedro-config-loader-dive-deep_files/figure-html/13754d50-1-image.png

The parameters is actually interpolated correctly as you expected until it get override at the end with a
_nested_dict_update
call. Fundamentally, the issue is that there could be a recursive definition of “runtime_params”, and right now config loader take any
--params
value as is. what I mean by
recursive
is
Copy code
key: $runtime_params: {"key"} + 1
If I do
kedro run --params key=1
, what should be the value? Should it be
1
or
1+1
or
1+1+1+1+1+1…
f

Flavien

11/17/2023, 8:59 AM
Thanks for all the feedback. I missed the runtime parameters in the documentation. The combination of it with the custom resolver solve my question. Thanks!
👍🏼 1
👍 1