Camilo López
06/23/2023, 1:56 AMManagedTableDataSet
with Databricks Unity Catalog and I didn't find a way to store tables on a external location (ABFS Azure). There's a way of storing a external table with pure-spark : df.write.mode(mode).option("path", table_path).saveAsTable(f"{catalog_name}.{schema_name}.{table_name}")
, where table_path
its the path for the external location like <abfss://container@storage_account.dfs.core.windows.net/raw>
.
There's a way to pass this path to the ManagedTableDataSet
when saving the data? Or should I go and create a CustomManagedTableDataSet
with this capability?ManagedExternalTableDataSet
that overrides the save functions like data.write.format("delta").option("path", self.table_path)
Nok Lam Chan
06/23/2023, 1:21 PMCamilo López
06/23/2023, 4:01 PM