Flavien
11/16/2023, 3:24 PM1D
as ISO8601 duration.
After a bit of digging, I thought of using a custom resolver for OmegaConf
as follows
from kedro.config import OmegaConfigLoader
CONFIG_LOADER_CLASS = OmegaConfigLoader
CONFIG_LOADER_ARGS = {
"custom_resolvers": {
"parse-duration": parse_iso_8601_duration
}
}
When using the catalog with an entry like
duration: ${parse-duration:1d}
it works really well. Now, I would like to do the same while passing the parameter through the CLI. I tried
kedro run --pipeline=test_parsing --params duration:\${parse-duration:1d}
escaping the second part of the command. It seems that the loader goes indeed through the parser but the value is not associated with the variable duration
.
The test pipeline is the following
from kedro.pipeline import Pipeline, pipeline, node
def create_pipeline(**kwargs) -> Pipeline:
return pipeline([node(lambda d: print(d), inputs=["params:duration"], outputs=None)])
Is custom resolver the correct way to do so? If so, any hint on what may go wrong with the CLI?marrrcin
11/16/2023, 3:41 PMFlavien
11/16/2023, 3:42 PMmarrrcin
11/16/2023, 3:43 PMFlavien
11/16/2023, 3:43 PMmarrrcin
11/16/2023, 3:43 PMAnkita Katiyar
11/16/2023, 5:30 PMNok Lam Chan
11/17/2023, 4:02 AM--params
as this is what we try to avoid from the lesson of TemplatedConfigLoader
and is why we introduce runtime_params
resolver.
Ideally you want this to work
duration: "${foo: ${runtime_params: duration, 'placeholder'}}"
Unfortunately it doesn’t, you may work around this with
duration: "${foo: ${runtime_params: duration_, 'placeholder'}}"
and using
kedro run --pipeline=test_parsing --params duration_=<something>
https://noklam.github.io/blog/posts/kedro_config_loader/2023-11-16-kedro-config-loader-dive-deep_files/figure-html/13754d50-1-image.pngâ–ľ
_nested_dict_update
call.
Fundamentally, the issue is that there could be a recursive definition of “runtime_params”, and right now config loader take any --params
value as is.
what I mean by recursive
is
key: $runtime_params: {"key"} + 1
If I do kedro run --params key=1
, what should be the value? Should it be 1
or 1+1
or 1+1+1+1+1+1…
Flavien
11/17/2023, 8:59 AM