Sid Shetty
07/26/2023, 5:24 PMnulls
. Is there any workaround here that avoids me having to add another node to put these partitions together and ideally just read as a pandas.ParquetDataSet? Perhaps passing the schema of the original dataframe or even specifying it explicitly?Juan Luis
07/26/2023, 5:29 PMload_args
to your dataset to control how to pd.read_parquet
will be used, these will get passed directlySid Shetty
07/26/2023, 5:41 PMuse_pandas_metadata
seems like just what I was looking for!