Hello, is there a reason why

pandas.ParquetDataSet

does not use pandas all the time? I would like to use it for partitioned data, and I want to use the

filters

that

pandas.read_parquet

provides, but it is not available for

pyarrow.parquet.ParquetDataset.read

. Doing a quick test and using

pd.read_parquet

every time seems to work ok, even though it does not behave exactly the same.

It actually predates parquet support in Pandas

OK, I do not have the time tonight, but I will open an issue and try to make a PR afterwards. Thanks for pointing the

GenericDataSet

, I was not aware of it!

Chiming in here, pyarrow does support this, I think you're just using the wrong portion of the api. Below will return instance of pyarrow table which has a method for casting to pandas https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html

I think using pandas everywhere would make the

loard_args

consistent and not depend on whether we try to read a folder or a file