Hi Team, for PartitionedDataSets or any other way ...
# questions
b
Hi Team, for PartitionedDataSets or any other way of reading in multiple files at the same time, how can we specify regex-style notations? Eg. I have files called
data_type_a_1.csv
, d`ata_type_a_2.csv`… and I want to read those in together - tried to simply put
*
in the filename_suffix, eg.
"data_type_a_*.csv"
, but that’s not working, so I’m definitely missing something simple here 😅
Aha, my mistake was sticking to PartitionedDataSet, I believe I can do
*
notation in SparkDataSet