https://kedro.org/ logo
#questions
Title
# questions
r

Richard Purvis

10/27/2023, 2:40 AM
Has anyone successfully saved a
SQLTableDataset
while specifying data types? From what I’m seeing, the pandas API requires a dictionary of SQL alchemy type objects. Would this require a custom resolver to pass the types through the catalog? FYI I’m attempting to save tables to a SQL Server/Azure DWH.
j

Juan Luis

10/27/2023, 8:36 AM
Would this require a custom resolver to pass the types through the catalog?
yes! here's an example for Polars:
r

Richard Purvis

10/27/2023, 4:27 PM
Thanks. What would be a good way to add arguments for these? For instance,
sqlalchemy.types.String(32)
j

Juan Luis

10/27/2023, 4:29 PM
maybe
"${sqlalchemy:String:32}"
and you split the value by
:
?
r

Richard Purvis

10/27/2023, 4:30 PM
Copy code
CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "sql_types": lambda x: getattr(sqlalchemy.types, x),
    }
}
I'm not sure how the nested keys would work with that, though
Alternately,
Copy code
CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "sql_string": lambda x: sqlalchemy.types.String(x),
    }
}
j

Juan Luis

10/30/2023, 9:17 AM
I was thinking
Copy code
CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "sql_types": lambda value: getattr(sqlalchemy.types, value.split(":")[0])(value.split(":")[1]),
    }
}
or rather
Copy code
def get_sql_type(value):
    type_cls_name, arg = value.split(":")
    type_cls = getattr(sqlalchemy.types, type_cls_name)
    return type_cls_name(arg)
does it make sense @Richard Purvis?
r

Richard Purvis

10/30/2023, 1:44 PM
Yes, thanks.
👍🏼 1