[OmegaConf x Catalog] Hi Team, I’ve read the docu...
# questions
[OmegaConf x Catalog] Hi Team, I’ve read the documentation regarding advanced configuration, very helpful! I’m clear on how we can use templates for catalogs. I’m wondering if there is a way to replace the below anchor as well created for a specific dataset type, e.g.,
with csv? Or would you refrain from using these anchors as it makes the config less readable/explicit which against your philosophy?
Copy code
_spark_csv: &spark_csv
  type: spark.SparkDataSet
  file_format: csv
    sep: ","
    header: True
    inferSchema: True

  <<: *spark_csv
  filepath: "${_base_path}/${_folders.raw}/name_of_dataset"
The reason I’m asking is that there is a very little chance I would change config e.g.,
for a specific dataset type, so pretty much I need • pandas csv, • pandas excel, • pandas parquet, • spark csv, • spark parquet • and pickle so, I’m wondering whether there is a way to define these in one place, and in the catalog I would just the template. What do you think? Thank you in advance!
Hi Ankita, Oh no, I haven’t heard of this before, how could I miss it? Thanks, let me look into it.
Ok, it’s been released 8 days ago, that’s not a long time 🙂
Yeah, it was quite recent! Let me know if this solves your problem! 😄