https://kedro.org/ logo
#questions
Title
# questions
a

Alex Ferrero

02/16/2023, 10:48 AM
Hey team, is there anyway I can write to a delta table using the catalog making an upsert like in SQL? I have seen in kedro's code that the only supported modes are append, overwrite, error, errorifexists and ignore.
d

datajoely

02/16/2023, 10:52 AM
https://kedro.readthedocs.io/en/stable/tools_integration/pyspark.html#spark-and-delta-lake-interaction If you look at the dataset docs the UD part of CRUD doesn’t really fit into the kedro flow, so we would push you to use the Python API like this example https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.spark.DeltaTableDataSet.html And then return a dummy value so that we can coerce correct DAG execution order
a

Alex Ferrero

02/16/2023, 10:59 AM
I will use the Python API then, thank you very much! 🙌
d

datajoely

02/16/2023, 11:00 AM
👍 1
this diagram is important though
it’s the least worst way we could think of doing this in Kedro
❤️ 1
4 Views