https://kedro.org/ logo
#resources
Title
# resources
i

Iñigo Hidalgo

12/20/2023, 10:24 PM
j

Juan Luis

12/21/2023, 8:11 AM
Polars + Delta + MinIO is my go-to architecture for small projects 💯
👍🏼 1
n

Nok Lam Chan

12/21/2023, 12:39 PM
Can we make Kedro using delta native versioning instead of the vanilla one when it's available?
i

Iñigo Hidalgo

12/21/2023, 12:48 PM
I'm trying to find the thread but can't, I remember discussing this with @datajoely at some point
@Juan Luis within the context of polars + delta, do you use the TableMerger functionality to perform upserts? I am working on an example medallion architecture ELT where I will try to use polars, delta, minio, prefect (and maybe kedro if i get that far) https://github.com/inigohidalgo/prefect-polygon-etl and I am at the point where I want to start implementing upserts
j

Juan Luis

02/12/2024, 11:42 AM
@Iñigo Hidalgo I'm not using TableMerger. I had a quick look and I just saw that Polars
write_delta
exposes it:
If ‘merge’, return a
TableMerger
object to merge data from the DataFrame with the existing data.
https://docs.pola.rs/py-polars/html/reference/api/polars.DataFrame.write_delta.html I think at this point you're ahead of myself. keep us posted!
🚀 1
2 Views