https://kedro.org/ logo
#questions
Title
# questions
c

Chandan Malla

11/21/2023, 10:59 AM
Something like this
m

marrrcin

11/21/2023, 11:00 AM
see my reply above
c

Chandan Malla

11/21/2023, 11:04 AM
I saw dataset factories, it seems like a regex, would not help me as table names are very long and i could pass them in my pipelines, would be very inefficient. I want to parameterize the table name in globals.yml or parameters.yml and then namespace should pick it on its own
I used templateconfigloader, it helps me to parametrize table names but I still can't do it according to namespace
m

marrrcin

11/21/2023, 11:32 AM
You can combine both namespaces (from dataset factories) and globals (with the help of OmegaConfigLoader)
c

Chandan Malla

11/21/2023, 12:32 PM
I am using this {{namespace}.${preprocessing_table}} And I get the error , no config value found for preprocessing_table. My config is stored as chandan.preprocessing_table where Chandan is the namespace
c

Chandan Malla

11/21/2023, 12:35 PM
IF I use it like this: ${{namespace}.{preprocessing.ballbyball}} then error: ValueError: Single '}' encountered in format string
m

marrrcin

11/21/2023, 1:29 PM
Please read about how the OmegaConf interpolation works first, especially using
${}
and relative interpolations
${..<name>}
c

Chandan Malla

11/21/2023, 1:58 PM
I read it and I rewrote the code: this is my
catalog_globals.yml
_odi_men:
preprocessing:
ballbyball: TABLE_NAME
this is catalog.yml:
"{namespace}.raw_ballbyball":
type: pandas.SQLQueryDataSet
sql: select * from schema.${{_{namespace}.preprocessing.ballbyball}}
credentials: con
This is not working and query gets resolved as: select * from schema.{_odi_men.preprocessing.ballbyball}
sql: select * from schema.${_odi_men.preprocessing.ballbyball}
This works, but that's not what I want
hey @marrrcin, does kedro does not support using data factory and omegaConf simultaneously ?
m

marrrcin

11/22/2023, 4:18 PM
@Ankita Katiyar will most likely know ☝️
👀 1
a

Ankita Katiyar

11/22/2023, 4:34 PM
My suspicion is that the
{namespace}
which should come from the dataset factory thing is resolved much later in the session and not when the config is initially loaded - so
omegaconf
doesn’t know what
schema.${{_{namespace}.preprocessing.ballbyball}}
is referring to
🤔 1
2 Views