They’re cooking up some magic at DuckDB - I think I like it - but this isn’t your ISO compliant SQL you learnt at school https://duckdb.org/2023/08/23/even-friendlier-sql.html

We’re back in 90s then - everything will be fragmented in SQL area, every company will have it’s own “better SQL” and after 5 years someone will decide to have another version of ISO SQL. IT is going circles 😄

Yeah I’m in two minds, part of the reason I like DataFrames so much is that the syntax for doing things like pivots, lambdas and comprehensions is much more intuitive. You can see exactly why DuckDB trying to address those shortcomings of SQL’s rigidity, for those really invested in that ecosystem specifically it makes sense - but it’s not transferable to a BigQuery, Snowflake etc. I’d love to hear what @Cody Peterson thinks about this sort of innovation / deviation - because there is definitely user value with a lot of changes happening at pace!

we (the Voltron Data we) are pushing standards, particularly Substrait as an intermediary representation, to help solve the SQL dialect problem. then Ibis (created by Wes McKinney, supported by Voltron Data and other companies) as a standard Python API for dataframes that's backend-agnostic a lot of it is written in our Codex: The Composable Codex | Voltron Data and this blog by Wes is a great look back: Wes McKinney - The Road to Composable Data Systems: Thoughts on the Last 15 Years and the Future TLDR; we're working on it, it's hard 🙂

how do you feel DuckDBs deviation from SQL standards will affect things?

eh...we love DuckDB, they claim to follow the Postgres standard, but everyone who claims that never does. they have been an early adopter of Substrait, ADBC, and other standards personally (and I think on behalf of VoDa) we don't see SQL or DSLs going away, but as long as they all speak the same IR we can compose systems together. so it's fine, it's expected, it'd be nice if there were just a SQL standard

phenomenal answer - super interesting

yeah, and they do deviate for good (usually customer-requested) reasons similar issue in the Python space -- people claim to make pandas or PySpark-compatible APIs, but they never truly are. I can't find the diagram but we have a nice slide, right now Ibis "transpiles" down to SQL generally in whatever syntax the backend uses; ideally in a few years, Ibis can just produce Substrait plans and all backends can consume those

oh man