As counterpoint <@U03RJ2PH79D> experimental pyspar...
# randomd
datajoely
09/26/2023, 5:05 PMAs counterpoint @Nok Lam Chan experimental pyspark support for duckdb is bonkers! Basically replace your connection string…
https://duckdb.org/2023/09/26/announcing-duckdb-090.html
👀 4
m
marrrcin
09/27/2023, 7:05 AM Copy code
from duckdb.experimental.spark.sql import SparkSession as session
from duckdb.experimental.spark.sql.functions import lit, cold
datajoely
09/27/2023, 7:57 AMyeah it’s actually really interesting
datajoely
09/27/2023, 7:57 AMbasically the way Snowpark was made, but this one isn’t even pretending
🫠 1
lolsob 2
datajoely
09/28/2023, 2:56 PMimage.png
datajoely
09/28/2023, 2:56 PMit’s experimental
👀 1
n
Nok Lam Chan
09/28/2023, 2:57 PMThe DataFrame protocol seems to solve these problem more fundamentally. There has been so many library trying to copy pandas/Spark API
d
datajoely
09/28/2023, 2:58 PMor ibis
2 Views