https://kedro.org/ logo
#questions
Title
# questions
d

Damian Fiłonowicz

04/03/2023, 11:11 AM
Hey, is there any way to use an IncrementalDataSet with a SQL query to incrementally fetch partitions from RDBMS tables, or it works only with object storage partitioned data?
n

Nok Lam Chan

04/03/2023, 11:19 AM
I don’t think this is supported natively - @datajoely may have better idea. It was designed to work with files -

https://www.youtube.com/watch?v=v7JSSiYgqpg

is an example of how it can be used.
d

datajoely

04/03/2023, 11:36 AM
no we don’t natively support this - if you look at our DeltaLake docs we talk about how to do the
UD
part of
CRUD
in Kedro, but it’s not a fantastic fit for our load/save pattern https://docs.kedro.org/en/stable/tools_integration/pyspark.html#spark-and-delta-lake-interaction
👍 1
d

Damian Fiłonowicz

04/03/2023, 2:43 PM
ok thanks for the information :)
2 Views