https://kedro.org/ logo
#questions
Title
# questions
f

Flavien

02/13/2024, 8:06 PM
Hi folks, I encountered an unexpected behavior regarding the parsing of parameters in version 0.19.2. I implemented a small function to parse date-time from the catalog
Copy code
def parse_timezone_aware_datetime_string(datetime_string: str) -> datetime
which I registered as a custom resolver
Copy code
CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "parse_timezone_aware_datetime_string": parse_timezone_aware_datetime_string
    },
    "base_env": "base",
    "default_run_env": "local",
}
In the catalog, I added a static parameter
Copy code
start: ${parse_timezone_aware_datetime_string:2023-12-13T23:00:00+01:00}
which is correctly parsed
Copy code
In [1]: catalog.load("params:start")
[02/13/24 20:56:47] INFO     Loading data from params:start (MemoryDataset)...                                                                                             data_catalog.py:483
Out[1]: datetime.datetime(2023, 12, 13, 22, 0, tzinfo=datetime.timezone.utc)
Now, if I add a parameter to be evaluated at run time
Copy code
test: ${runtime_params:the_test,2}
and I run a pipeline with
--params="the_test=1"
then
omegaconf
raises an error
Copy code
omegaconf.errors.UnsupportedValueType: Value 'datetime' is not a supported primitive type
    full_key: start
    object_type=dict
If I add the parameter
start
to
--params
then I don't have the problem. Let me know if I am not using the configuration loader properly. Thanks in advance!
👀 1
a

Ankita Katiyar

02/14/2024, 9:15 AM
Hey Flavien, are you trying to nest the resolvers? I’m not sure why having a separate runtime parameter would cause the custom resolver to not work. 🤔 Would it be possible to post the full stacktrace?
f

Flavien

02/14/2024, 9:37 AM
@Ankita Katiyar I did not intend to nest the resolvers. If I did, it was not intentional. 😅 I created a new project from scratch with only the two parameters mentioned above and a single pipeline with a single node
lambda s: s
Here is the full traceback.
a

Ankita Katiyar

02/14/2024, 10:05 AM
Could I see the project so I can reproduce and debug this please? 😅
f

Flavien

02/14/2024, 10:31 AM
Copy code
kedro run --params=the_test=1
a

Ankita Katiyar

02/14/2024, 10:41 AM
Thank you, I’ll take a look and get back to you
The error’s coming from
omegaconf
, seems like they do not support non primitive types -
Copy code
from pandas import Timestamp
from omegaconf import OmegaConf
x = {'start': Timestamp('2023-12-13 22:00:00+0000', tz='UTC'), 'test': 1, 'the_test': 1}
y = {'the_test': 1 }
conf = OmegaConf.merge(x,y)
This gives the same error. Also -
Copy code
conf = OmegaConf.create(x)
It’s working when there’s no runtime params because the config is loaded with
omegaconf
and then the parameter with the custom resolver is resolved after. But when there’s run time parameters, there’s an extra
OmegaConf.merge(parameters, runtime_parameters)
step performed by Kedro and this is after the parameter with the custom resolver was already resolved into
Timestamp
type
👀 1
I think this we could address this on Kedro side with delaying the resolution of the resolvers. I’ll open an issue for this. Thanks for bringing this up, I think this is one of the edge cases we hadn’t come across before! 😅
In your case, in the meanwhile, I would say converting to
Timestamp
can be a helper function in the pipeline and loading the parameter as a string would work if you’re also intending to use runtime parameters.
f

Flavien

02/14/2024, 12:13 PM
Thanks @Ankita Katiyar. I had found another workaround in the meantime. 👍
4 Views