Kedro is an open-sourced Python framework for creating maintainable and modular data science code.

Kedro

Hey, does IncrementalDataSet support the SCD/merge statement to update/insert/delete specific rows from its (previous) partitions?

so this is fully available only with the deltalake connector? <@U03R8FW4HUZ>

no it was more a point on the conceptual pieces around this

image.png

if you’re going to do an “out of dag” operation in Kedro you need to start doing fake datasets to preserve the right execution order

it’s not super elegant and speaks to how the paradigm doesn’t really work for `UD`

hmm, okay. Do you have any examples with these out of dag operations and fake datasets?

but for SQL we don’t have the same dataframe like abstraction

so not sure we have anything to go with SQL

longer term Ibis may be a solution for this