Flavien
02/13/2024, 8:06 PMdef parse_timezone_aware_datetime_string(datetime_string: str) -> datetime
which I registered as a custom resolver
CONFIG_LOADER_ARGS = {
"custom_resolvers": {
"parse_timezone_aware_datetime_string": parse_timezone_aware_datetime_string
},
"base_env": "base",
"default_run_env": "local",
}
In the catalog, I added a static parameter
start: ${parse_timezone_aware_datetime_string:2023-12-13T23:00:00+01:00}
which is correctly parsed
In [1]: catalog.load("params:start")
[02/13/24 20:56:47] INFO Loading data from params:start (MemoryDataset)... data_catalog.py:483
Out[1]: datetime.datetime(2023, 12, 13, 22, 0, tzinfo=datetime.timezone.utc)
Now, if I add a parameter to be evaluated at run time
test: ${runtime_params:the_test,2}
and I run a pipeline with --params="the_test=1"
then omegaconf
raises an error
omegaconf.errors.UnsupportedValueType: Value 'datetime' is not a supported primitive type
full_key: start
object_type=dict
If I add the parameter start
to --params
then I don't have the problem.
Let me know if I am not using the configuration loader properly. Thanks in advance!Ankita Katiyar
02/14/2024, 9:15 AMFlavien
02/14/2024, 9:37 AMlambda s: s
Here is the full traceback.Ankita Katiyar
02/14/2024, 10:05 AMFlavien
02/14/2024, 10:31 AMFlavien
02/14/2024, 10:31 AMkedro run --params=the_test=1
Ankita Katiyar
02/14/2024, 10:41 AMAnkita Katiyar
02/14/2024, 11:03 AMomegaconf
, seems like they do not support non primitive types -
from pandas import Timestamp
from omegaconf import OmegaConf
x = {'start': Timestamp('2023-12-13 22:00:00+0000', tz='UTC'), 'test': 1, 'the_test': 1}
y = {'the_test': 1 }
conf = OmegaConf.merge(x,y)
This gives the same error. Also -
conf = OmegaConf.create(x)
Ankita Katiyar
02/14/2024, 11:05 AMomegaconf
and then the parameter with the custom resolver is resolved after. But when there’s run time parameters, there’s an extra OmegaConf.merge(parameters, runtime_parameters)
step performed by Kedro and this is after the parameter with the custom resolver was already resolved into Timestamp
typeAnkita Katiyar
02/14/2024, 11:09 AMAnkita Katiyar
02/14/2024, 11:12 AMTimestamp
can be a helper function in the pipeline and loading the parameter as a string would work if you’re also intending to use runtime parameters.Flavien
02/14/2024, 12:13 PM