Something like this

see my reply above

I saw dataset factories, it seems like a regex, would not help me as table names are very long and i could pass them in my pipelines, would be very inefficient. I want to parameterize the table name in globals.yml or parameters.yml and then namespace should pick it on its own

You can combine both namespaces (from dataset factories) and globals (with the help of OmegaConfigLoader)

I am using this {{namespace}.${preprocessing_table}} And I get the error , no config value found for preprocessing_table. My config is stored as chandan.preprocessing_table where Chandan is the namespace

IF I use it like this: ${{namespace}.{preprocessing.ballbyball}} then error: ValueError: Single '}' encountered in format string

Please read about how the OmegaConf interpolation works first, especially using

${}

and relative interpolations

${..<name>}

I read it and I rewrote the code: this is my

catalog_globals.yml

_odi_men:

preprocessing:

ballbyball: TABLE_NAME

this is catalog.yml:

"{namespace}.raw_ballbyball":

type: pandas.SQLQueryDataSet

sql: select * from schema.${{_{namespace}.preprocessing.ballbyball}}

credentials: con

This is not working and query gets resolved as: select * from schema.{_odi_men.preprocessing.ballbyball}

@Ankita Katiyar will most likely know ☝️

My suspicion is that the

{namespace}

which should come from the dataset factory thing is resolved much later in the session and not when the config is initially loaded - so

omegaconf

doesn’t know what

schema.${{_{namespace}.preprocessing.ballbyball}}

is referring to