https://kedro.org/ logo
#questions
Title
# questions
t

tom kurian

11/30/2023, 11:08 AM
Hi Team, a small doubt in kedro spark dataset:
```weather_cleaned:
type: spark.SparkDataSet
filepath: data/02_intermediate/data.parquet
file_format: parquet```
Is it possible to pass multiple fikepaths in the filepath arguement.
j

Juan Luis

11/30/2023, 12:40 PM
hi @tom kurian, how would you expect that to work? in the specific case of
SparkDataSet
like, concatenating the different paths and aggregating that in a single df?
t

tom kurian

11/30/2023, 12:41 PM
in normal spark code, we can pass it as like this, spark.read.parquet([path1, path2])
j

Juan Luis

11/30/2023, 8:58 PM
I'm not very familiar with Spark but looking at the code I don't think this is possible yet: https://github.com/kedro-org/kedro-plugins/blob/a360bfdb2eae3cf95d7217a7f1f46e1baf79acbb/kedro-datasets/kedro_datasets/spark/spark_dataset.py#L427 would you like to open an issue about it?
👍 1