Eugene P
12/02/2022, 4:41 PMsample_sql_query_data:
type: pandas.SQLQueryDataSet
credentials: postgres_re_db
sql: SELECT * FROM rr_norm.sample_gov_torgi
Unfortunately, the amount of queries grows fast and catalog.yaml starts bloating with long query strings. Also, it looks like not a good idea to keep sql queries strings within the catalog.yaml itself for reproducibility.
What would be the most kedroic/pythonic approach to extract queries from the catalog.yaml to a separate folder/module? AFAIK (or understood from googling) yaml doesn’t natively has include/import features?Ben Levy
12/02/2022, 5:16 PMfilepath
argument instead of sql
(see the docs https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.pandas.SQLQueryDataSet.html)data
folder, or in any other folder in the project that you create (e.g., sql/
)Eugene P
12/02/2022, 5:37 PM