04/03/2023, 11:12 AM
Hey, does IncrementalDataSet support the SCD/merge statement to update/insert/delete specific rows from its (previous) partitions?
so this is fully available only with the deltalake connector?
04/03/2023, 11:44 AM
no it was more a point on the conceptual pieces around this
if you’re going to do an “out of dag” operation in Kedro you need to start doing fake datasets to preserve the right execution order
it’s not super elegant and speaks to how the paradigm doesn’t really work for
04/03/2023, 2:47 PM
hmm, okay. Do you have any examples with these out of dag operations and fake datasets?
04/03/2023, 2:51 PM
in the doc i shared
but for SQL we don’t have the same dataframe like abstraction
so not sure we have anything to go with SQL
longer term Ibis may be a solution for this
04/03/2023, 3:02 PM