Eugene P12/02/2022, 4:41 PM
Unfortunately, the amount of queries grows fast and catalog.yaml starts bloating with long query strings. Also, it looks like not a good idea to keep sql queries strings within the catalog.yaml itself for reproducibility. What would be the most kedroic/pythonic approach to extract queries from the catalog.yaml to a separate folder/module? AFAIK (or understood from googling) yaml doesn’t natively has include/import features?
sample_sql_query_data: type: pandas.SQLQueryDataSet credentials: postgres_re_db sql: SELECT * FROM rr_norm.sample_gov_torgi
Ben Levy12/02/2022, 5:16 PM
argument instead of
(see the docs https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.pandas.SQLQueryDataSet.html)
folder, or in any other folder in the project that you create (e.g.,
Eugene P12/02/2022, 5:37 PM