Andreas Tsangarides
11/02/2023, 10:25 AM{start_time_utc}
in the query replaced by the parameter
# catalog.yml
raw-query-dataset:
type: <my-package>.io.TrinoSQLDataSet
query_param_names:
- start_time_utc
What the dataset does with query_param_names
is to look for each of those keys in the kedro parameters and inject them into the query, so essentially this is a parameterised query
# parameters.yml
start_time_utc: 2023-01-01 00:00:00
In the dataset class itself, this is how i load the kedro parameters:
def _load_kedro_params() -> Dict[str, Any]:
conf_path = str(settings.CONF_SOURCE)
# get kedro run env to load the params from the correct conf dir
env = os.getenv("KEDRO_ENV", "local")
conf_loader = ConfigLoader(conf_source=conf_path, default_run_env=env)
return conf_loader.get("parameters", "parameters*/**")
This means the yml file is read, but if I run the pipeline using
kedro run --params:"start_time_utc: 2023-03-01 13:00:00"
The dataset still loads the parameter from the file…
Any way to access the merged dictionary of the parameters overwritten by the cli extra/runtime params? I cannot find a way to access the current session/context anymoredatajoely
11/02/2023, 10:27 AMbefore_pipeline_run
hook and pass the parameter to your catalog entry that wayAndreas Tsangarides
11/02/2023, 10:29 AMdatajoely
11/02/2023, 10:29 AMAndreas Tsangarides
11/02/2023, 10:32 AMthe right way to do all of this is to use aok, that would work 🙏 any example for adding onto the catalog?hook and pass the parameter to your catalog entry that waybefore_pipeline_run
Nok Lam Chan
11/02/2023, 11:56 AMYolan Honoré-Rougé
11/02/2023, 12:46 PMAndreas Tsangarides
11/02/2023, 6:29 PMtrino
)
3. the runtime_params
require the OmegaConfigLoader
?Nok Lam Chan
11/02/2023, 10:52 PMOmegaConfigLoader
is required and it will be the default in 0.19 seriesread_sql_template
In your catalog, you will define it something like this.
# catalog.yml
MyCustomSqlDataset:
...
sql: ${read_sql_template: <file_path>, <dict_of_variable>}
Andreas Tsangarides
11/03/2023, 8:00 AMNok Lam Chan
11/03/2023, 10:15 AMkedro-neptune
? Do they come with a custom config loader?Andreas Tsangarides
11/08/2023, 12:20 PMConfigLoader
(and TemplatedConfigLoader
) and have asked them to make it work with omega..
they will do it, but have not provided a time estimateYolan Honoré-Rougé
11/10/2023, 8:03 AMSQLQueryDataset
, you just need to install pip install trino[sqlalchemy]
and set the connection in the credentials, so you can likely avoid the custom dataset.