Erwin
08/17/2023, 7:28 PMbase
with a catalog_globals.yml: _env: dev
2. databricks_qas
with a catalog_globals.yml: _env: qas
In my base folder I have a catalog.yml
pidata_silver:
type: projetc.io.databricks.ManagedTableDataSet
database: xx_aa_${_env}_silver
table: operations_concentrator_pidata_tpidata
write_mode: upsert
primary_key: timestamp_str
dataframe_type: spark
partition_columns: year_month
layer: silver
However when I use the conf databricks_qas
the variable in my catalog xx_aa_${_env}_silver,
is not changed to xx_aa_qas_silver
how I was expecting. It keeps xx_aa_dev_silver
, it seems that in the catalog there is no override.
Is this true? any workaround?
@Kremlin Huamanpidata_silver,
I can override the one in base. Not ideal, because I need to duplicate catalog. It does not make a lot of sense to be able to use _env
as “global” inside the same enviroment, since at the end of the day I need to duplicate the catalog anyways.Fazil B. Topal
08/21/2023, 11:16 AM