https://kedro.org/ logo
#random
Title
# random
d

datajoely

09/26/2023, 5:05 PM
As counterpoint @Nok Lam Chan experimental pyspark support for duckdb is bonkers! Basically replace your connection string… https://duckdb.org/2023/09/26/announcing-duckdb-090.html
👀 4
m

marrrcin

09/27/2023, 7:05 AM
Copy code
from duckdb.experimental.spark.sql import SparkSession as session
from duckdb.experimental.spark.sql.functions import lit, col
This one is sick 😮
d

datajoely

09/27/2023, 7:57 AM
yeah it’s actually really interesting
basically the way Snowpark was made, but this one isn’t even pretending
🫠 1
lolsob 2
image.png
it’s experimental
👀 1
n

Nok Lam Chan

09/28/2023, 2:57 PM
The DataFrame protocol seems to solve these problem more fundamentally. There has been so many library trying to copy pandas/Spark API
d

datajoely

09/28/2023, 2:58 PM
or ibis
2 Views