https://kedro.org/ logo
#questions
Title
# questions
o

Oscar Villa

02/08/2023, 9:42 PM
Hi, guys. Maybe somebody knows what it is the pattern when you have very long queries? I'm getting data from BigQuery trought pandas.GBQQueryDataSet but the queries are so long, making the catalog.yaml looks dirty. Is that the right way or should I store the queries in files and call from them, or store the queries as views in Bigquery? What do you use to do? Any suggestion is appreciated. Thanks in advance.
1
n

Nok Lam Chan

02/09/2023, 5:37 AM
For SQLDataSet you can specify a file path with the sql, not sure why this is not included for the GBQDataset. I will have a closer look later. https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.pandas.SQLQueryDataSet.html
K 1
o

Oscar Villa

02/09/2023, 1:07 PM
Thank you so much, @Nok Lam Chan. My bad: I checked again the pandas.GBQQueryDataSet documentation and I found the filepath parameter that you mention. So, I created a file.sql with the query and then passed in its filepath as parameter: it works smoothly
n

Nok Lam Chan

02/09/2023, 4:56 PM
Awesome 🙂 Great to hear it works well
👍 1
4 Views