Ziren Lin
03/28/2023, 3:44 PM#globals.yml
order_number: 'abc'
#catalog.yml
sql:
type: pandas.SQLQueryDataset
sql: "SELECT * FROM table WHERE column = ${order_number}"
Nok Lam Chan
03/28/2023, 3:57 PMZiren Lin
03/28/2023, 4:03 PMconf/base/
but some other subfolders e.g. conf/base/parameters
.sql
file and pass in the filepath
instead of sql queries directly in the catalog.yml. Is there any way to pass the parameters as well?Nok Lam Chan
03/28/2023, 4:09 PM0.18.5
"parameters": ["params*", "params*/**", "**/params*"]
conf/base/parameters/<pipeline_name>.yml
kedro pipeline create <pipeline_name>
filepath
option, there aren’t native way of injecting runtime parameters.Ziren Lin
03/28/2023, 4:14 PMCONFIG_LOADER_ARGS = {
"globals_pattern": "*pipeline_XXX.yml",
}
But it seems that it only looks for yml file in conf/base or conf/local directory? I put this yml file in conf/base/parametersNok Lam Chan
03/28/2023, 4:39 PM"parameters": ["params*", "params*/**", "**/params*"]
This is the default patternZiren Lin
03/28/2023, 4:42 PM.sql
file and pass as filepath
. Is there anyway i could do that? Should I use Jinja or somethingNok Lam Chan
03/28/2023, 4:47 PMSQLQueryDataSet
, and pass in the “templated variable” into the dataset - compile the sql query in the _load
methodZiren Lin
03/28/2023, 4:51 PM_load
method, can I still put the query in a seperate .sql
file? The context is that we want to put all sql queries in a separate folder to better organize.Nok Lam Chan
03/28/2023, 4:53 PMsql
field, which works with parameters already
2. use filepath
which load the sql file directly and execute the queryZiren Lin
03/28/2023, 4:55 PMNok Lam Chan
03/28/2023, 4:55 PMdef _load(self) -> pd.DataFrame:
load_args = copy.deepcopy(self._load_args)
engine = self.engines[self._connection_str].execution_options(
**self._execution_options
) # type: ignore
if self._filepath:
load_path = get_filepath_str(PurePosixPath(self._filepath), self._protocol)
with self._fs.open(load_path, mode="r") as fs_file:
load_args["sql"] = fs_file.read()
## Add your Jinja stuff here
return pd.read_sql_query(con=engine, **load_args)
Ziren Lin
03/28/2023, 4:56 PMNok Lam Chan
03/28/2023, 5:00 PMZiren Lin
03/28/2023, 5:04 PM.yml
file? Really appreciate your time and efforts in helping me with this problem! Thanks a lot