Hi Team, a small doubt in kedro spark dataset: &g...
# questions
t
Hi Team, a small doubt in kedro spark dataset:
```weather_cleaned:
type: spark.SparkDataSet
filepath: data/02_intermediate/data.parquet
file_format: parquet```
Is it possible to pass multiple fikepaths in the filepath arguement.
j
hi @tom kurian, how would you expect that to work? in the specific case of
SparkDataSet
like, concatenating the different paths and aggregating that in a single df?
t
in normal spark code, we can pass it as like this, spark.read.parquet([path1, path2])
j
I'm not very familiar with Spark but looking at the code I don't think this is possible yet: https://github.com/kedro-org/kedro-plugins/blob/a360bfdb2eae3cf95d7217a7f1f46e1baf79acbb/kedro-datasets/kedro_datasets/spark/spark_dataset.py#L427 would you like to open an issue about it?
👍 1