Hello Team, I would like to know the recommended approach for implementing schema evolution in Delta tables within Databricks. Currently, I am encountering an issue with the dataset kedro_datasets.databricks.managed_table_dataset Whenever I attempt to add new columns using the upsert mode, an Exception is raised (there is check in the dataset implementation) Fortunately, I have control over the schema before performing the upsert operation. Hence, once I approve the schema changes, I expect to be able to utilize schema evolution during the upsert. In this context, I strongly believe that the exception raised during schema changes should be made optional, allowing for a smoother schema evolution process. For me, who should accept/deny schema evolution must be spark session

# Enable automatic schema evolution

spark.sql("SET spark.databricks.delta.schema.autoMerge.enabled = true")

Hi @Erwin Thanks for your question. I've just directed it to one of the team that worked on the dataset and will ask them to get back to you when they can, or point me to someone who is able to help.