Hey y'all, I've been working on a kedro dataset to enable loading data from SQL tables using Ibis. This allows for filtering tables using python code without needing to load the entire table like

pd.read_sql

would do. The dataset implementation is here, and I wrote a tiny example pipeline here. I would appreciate any feedback you could give, as an issue on the project or a response through here :)

I love this - @Iñigo Hidalgo, @Jo Stichbury blog post and maybe whole section of the docs?

This is awesome contribution : ) 🔥

I'm looking forward to testing this @Iñigo Hidalgo! 🔥 for future readers, more context in this thread https://github.com/kedro-org/kedro/issues/2374

From the thread:

> I think the more solid path forward would be to say "all things SQL in Kedro go through Ibis" and make it a "first level" optional dependency. No need to add more levels of indirection I'd say. To confidently commit to that plan, I propose we put this initial dataset in the hands of users and play around with it a bit more.

>

@Juan Luis

Would definitely be cautious about that, if other users are like us they probably have a lot of written SQL queries I don’t think they’d want to migrate to Ibis directly 😅 Though promoting it as a more robust and flexible alternative would probably do the job

good point. yeah maybe we could keep SQLQueryDataSet as a escape hatch, but then favor the Ibis approach as much as possible

This looks really cool ! Will definitely look into & use Thx 👍

hello! I'm working on Ibis, just a quick comment on this:

Would definitely be cautious about that, if other users are like us they probably have a lot of written SQL queries I don’t think they’d want to migrate to Ibis directly

we agree! Ibis has a `.sql()` method on tables to act as an escape hatch and enable going back and forth between Ibis and SQL easy (with a complementary

<http://ibis.to|ibis.to>_sql()

method). it also accepts a SQL dialect and uses sqlglot under the hood

this is great to know, thanks @Cody Peterson for chiming in! welcome to the Kedro Slack 🙌🏼K