Channels
announcements
introduce-yourself
resources
random
plugins-integrations
user-research
questions
questions-so
job-posting
help-
circleci-status
dependencies-releases
Powered by
#questions
Title
# questions
a
Alex Ferrero
02/16/2023, 10:48 AM
Hey team, is there anyway I can write to a delta table using the catalog making an upsert like in SQL? I have seen in kedro's code that the only supported modes are append, overwrite, error, errorifexists and ignore.
d
datajoely
02/16/2023, 10:52 AM
https://kedro.readthedocs.io/en/stable/tools_integration/pyspark.html#spark-and-delta-lake-interaction
If you look at the dataset docs the UD part of CRUD doesn’t really fit into the kedro flow, so we would push you to use the Python API like this example
https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.spark.DeltaTableDataSet.html
And then return a dummy value so that we can coerce correct DAG execution order
a
Alex Ferrero
02/16/2023, 10:59 AM
I will use the Python API then, thank you very much! 🙌
d
datajoely
02/16/2023, 11:00 AM
👍 1
this diagram is important though
it’s the least worst way we could think of doing this in Kedro
❤️ 1
4 Views
Post