Hi folks I encountered an unexpected behavior regarding the Kedro #questions

Hi folks, I encountered an unexpected behavior reg...

Flavien

02/13/2024, 8:06 PM

Hi folks, I encountered an unexpected behavior regarding the parsing of parameters in version 0.19.2. I implemented a small function to parse date-time from the catalog

Copy code

def parse_timezone_aware_datetime_string(datetime_string: str) -> datetime

which I registered as a custom resolver

Copy code

CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "parse_timezone_aware_datetime_string": parse_timezone_aware_datetime_string
    },
    "base_env": "base",
    "default_run_env": "local",
}

In the catalog, I added a static parameter

Copy code

start: ${parse_timezone_aware_datetime_string:2023-12-13T23:00:00+01:00}

which is correctly parsed

Copy code

In [1]: catalog.load("params:start")
[02/13/24 20:56:47] INFO     Loading data from params:start (MemoryDataset)...                                                                                             data_catalog.py:483
Out[1]: datetime.datetime(2023, 12, 13, 22, 0, tzinfo=datetime.timezone.utc)

Now, if I add a parameter to be evaluated at run time

Copy code

test: ${runtime_params:the_test,2}

and I run a pipeline with

--params="the_test=1"

then

omegaconf

raises an error

Copy code

omegaconf.errors.UnsupportedValueType: Value 'datetime' is not a supported primitive type
    full_key: start
    object_type=dict

If I add the parameter

start

--params

then I don't have the problem. Let me know if I am not using the configuration loader properly. Thanks in advance!

👀 1

Ankita Katiyar

02/14/2024, 9:15 AM

Hey Flavien, are you trying to nest the resolvers? I’m not sure why having a separate runtime parameter would cause the custom resolver to not work. 🤔 Would it be possible to post the full stacktrace?

Flavien

02/14/2024, 9:37 AM

@Ankita Katiyar I did not intend to nest the resolvers. If I did, it was not intentional. 😅 I created a new project from scratch with only the two parameters mentioned above and a single pipeline with a single node

lambda s: s

Here is the full traceback.

Untitled

Ankita Katiyar

02/14/2024, 10:05 AM

Could I see the project so I can reproduce and debug this please? 😅

Flavien

02/14/2024, 10:31 AM

I put it here : https://github.com/PetitLepton/kedro-failing-params

Flavien

02/14/2024, 10:31 AM

Copy code

kedro run --params=the_test=1

Ankita Katiyar

02/14/2024, 10:41 AM

Thank you, I’ll take a look and get back to you

Ankita Katiyar

02/14/2024, 11:03 AM

The error’s coming from

omegaconf

, seems like they do not support non primitive types -

Copy code

from pandas import Timestamp
from omegaconf import OmegaConf
x = {'start': Timestamp('2023-12-13 22:00:00+0000', tz='UTC'), 'test': 1, 'the_test': 1}
y = {'the_test': 1 }
conf = OmegaConf.merge(x,y)

This gives the same error. Also -

Copy code

conf = OmegaConf.create(x)

Ankita Katiyar

02/14/2024, 11:05 AM

It’s working when there’s no runtime params because the config is loaded with

omegaconf

and then the parameter with the custom resolver is resolved after. But when there’s run time parameters, there’s an extra

OmegaConf.merge(parameters, runtime_parameters)

step performed by Kedro and this is after the parameter with the custom resolver was already resolved into

Timestamp

type

👀 1

Ankita Katiyar

02/14/2024, 11:09 AM

I think this we could address this on Kedro side with delaying the resolution of the resolvers. I’ll open an issue for this. Thanks for bringing this up, I think this is one of the edge cases we hadn’t come across before! 😅

Ankita Katiyar

02/14/2024, 11:12 AM

In your case, in the meanwhile, I would say converting to

Timestamp

can be a helper function in the pipeline and loading the parameter as a string would work if you’re also intending to use runtime parameters.

Flavien

02/14/2024, 12:13 PM

Thanks @Ankita Katiyar. I had found another workaround in the meantime. 👍

10 Views

Open in Slack

Previous Next