Daniel Lee07/18/2023, 8:42 AM
, I would like to
to partition by the date in the dataset and save into different folders by date in parquet like how we can do it for
. Is there a way we could partition using pandas?
Nok Lam Chan07/18/2023, 10:22 AM
, I advise using whatever Parquet offer because it’s a native implementation and often you get better predicate pushdown for performance. Regarding to pandas, any Dataset that not offer partitioning can be partitioned with