Jordan Barlow
09/11/2023, 12:59 PMdatajoely
09/11/2023, 1:02 PMextra_params
attr of the before_pipeline_run
hookJordan Barlow
09/11/2023, 1:25 PM${<http://globals.site|globals.site>}_${globals.service}_usage:
type: pandas.SQLQueryDataSet
filepath: data/01_raw/${globals.service}.sql
load_args:
index_col: "my_index_col"
params:
start_date: ${globals.start_date}
end_date: ${globals.end_date}
credentials: ${globals.site.db_creds}
layer: Raw
But as I understand, this isn't currently possible with catalog templating.
Dynamically generating the catalog at runtime via hooks seems like the next best option.datajoely
09/11/2023, 1:29 PMJordan Barlow
09/11/2023, 1:53 PM0.18.13
) be done right from settings.py
? Something like:
CONFIG_LOADER_ARGS = {
"custom_resolvers": {
"runtime_param": lambda x: self.runtime_params[x],
}
}
Though I don't understand the OmegaConf implementation well enough to know the correct syntax here, surely self
is not rightdatajoely
09/11/2023, 2:28 PMJordan Barlow
09/11/2023, 2:29 PMdatajoely
09/11/2023, 2:29 PMJordan Barlow
09/11/2023, 2:45 PMExtendedOmegaConfigLoader
was implemented (within settings.py
?) or what the contents of register_custom_resolver
was supposed to be!datajoely
09/11/2023, 3:19 PMCONFIG_LOADER_CLASS
Jordan Barlow
09/11/2023, 3:35 PMdatajoely
09/11/2023, 4:01 PMFazil B. Topal
09/11/2023, 4:11 PMJordan Barlow
09/11/2023, 4:27 PM