Hi everyone, I'm working with Kedro and using databricks.ManagedTableDataset to manage Delta tables in Databricks. I need to parameterize the catalog so that it not only defines the schema (column names and types) but also allows storing metadata such as: Column Type Comment (description of the column) Tags (custom metadata like domain, etc.) Here’s an example of my current catalog.yml:

test_save:
  type: databricks.ManagedTableDataset
  catalog: blabla
  database: blabla
  table: blabla
  dataframe_type: spark
  write_mode: "overwrite"
  schema:
   fields:
     - name: "column_name"
       type: "column_type"
       nullable: false
       comment: "the description of column_name"
       tags:
          first_tag: "first_tag value"
          second_tag: "second_tag value"

hey @Vinicius Albert maybe you can look into custom dataset to suit your needs. Something based on this might work managed_table_dataset.py