Kevin Riera
11/10/2023, 12:32 PMkedro, version 0.18.12
and reading files from an S3 bucket.
I am trying to read and write a kedro.io.PartitionedDataSet
containing parquet files.
I am successfully writing my parquet1
, parquet2
till parquetN
files in a folder. But when reading, instead of of loading my N parquets I am loading each chunk of these separately, one folder level deeper. This happens despite playing with the suffix
load option and it seems that this issue has already been observed in the past : github issue with more details.
Has anyone experienced this and did you managed to find a workaround ?
Thank you so much for your help
@Ben LevyJuan Luis
11/10/2023, 12:41 PM