Has anyone tried to customize Spark Dataset to read multiple folders in HDFS?

Maybe the

PartitionedDataset

is useful? https://docs.kedro.org/en/stable/data/partitioned_and_incremental_datasets.html#partitioned-dataset-load

https://sites.google.com/site/hellobenchen/home/wiki/big-data/spark/read-data-files-from-multiple-sub-folders does spark natively support reading subfolders already?