hello everyone! does anyone have an example using ...
# questionsa
Andreas_Kokolantonakis
05/25/2023, 1:23 PMhello everyone! does anyone have an example using partition by when saving parquet files using kedro catalog? thank you very much in advance
n
Nok Lam Chan
05/25/2023, 1:25 PMAny chance this may help? https://github.com/kedro-org/kedro/issues/1567
a
Andreas_Kokolantonakis
05/25/2023, 1:28 PMhmm, I can see this issue is still open, I would like to specify the partition columns to save in the save_args.
n
Nok Lam Chan
05/25/2023, 3:39 PMWhat’s your existing config looks like? If you just need partition by columns you should be able to use SparkDataSet itself but not PartitionedDataSet
save_args: Save args passed to Spark DataFrame write options.
Similar to load_args this is dependent on the selected file
format. You can pass ``mode`` and ``partitionBy`` to specify
your overwrite mode and partitioning respectively. You can find
a list of options for each format in Spark DataFramehttps://docs.kedro.org/en/stable/kedro.extras.datasets.spark.SparkDataSet.html