Kedro is an open-sourced Python framework for creating maintainable and modular data science code.

Kedro

Polars, Delta Tables and SCDs <https://www.edgarbahilo.com/poor-mans-data-lake-with-polars-deltalake/>

Polars + Delta + MinIO is my go-to architecture for small projects :100: 

Can we make Kedro using delta native versioning instead of the vanilla one when it's available?

I'm trying to find the thread but can't, I remember discussing this with <@U03R8FW4HUZ> at some point

<@U046V0QC124> within the context of polars + delta, do you use the TableMerger functionality to perform upserts?

I am working on an example medallion architecture ELT where I will try to use polars, delta, minio, prefect (and maybe kedro if i get that far) <https://github.com/inigohidalgo/prefect-polygon-etl> and I am at the point where I want to start implementing upserts

<@U0495B76UG1> I'm not using TableMerger. I had a quick look and I just saw that Polars `write_delta` exposes it:

&gt; If ‘merge’, return a `TableMerger` object to merge data from the DataFrame with the existing data.
<https://docs.pola.rs/py-polars/html/reference/api/polars.DataFrame.write_delta.html>

I think at this point you're ahead of myself. keep us posted!