Quick question, I noticed the implementation of

_load

for the pandas parquet_dataset is different in

kedro.extras.datasets

vs the

kedro-datasets

plugin! The difference is significant as the one in kedro extras can be extremely slow (2hours compared to 10sec to load a dataset)! In our case, we had a dataset on S3 generated by a Spark job (hence a “directory” of (snappy) parquet files with a

_SUCCESS

file) with 137808 rows and 6410 columns. With that dataset, I could validate that

pq.ParquetDataset(load_path, filesystem=self._fs).read(**self._load_args)

took indeed longer that 15mins (after that, I ran out of patience since

pd.read_parquet()

on the same dataset was loading within 10sec’s). So the question is: should we already switch from kedro extras datasets to the new kedro-datasets plugin to solve this issue? Is this plugin already ready to use with the current kedro version (v0.18.x)? And can we then simply remove the pandas extras from our

requirements

?

Yes

kedro-datasets

is ready for use! The earlier you can start using it instead of the datasets in

kedro.extras

the better, because those will be completely removed in Kedro 0.19.0.

FYI Our original Parquet dataset in extras predates Pandas implementing it on their end, so the newer one defers to the pandas implementation

Thanks for the quick replies! I will plan the change ASAP then 😄. And yes, I am aware that at some point in time, pandas wasn’t able to do this. But I was surprised by the difference in speed 😱. Especially given that they all use pyarrow under the hood.

Yeah IIRC we didn’t even use the pyarrow library at first

On a side note, is there a reason why delta-spark is set a lot more strict in kedro-datasets compared to kedro? https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/setup.py#L79 https://github.com/kedro-org/kedro/blob/main/setup.py#L76

https://github.com/kedro-org/kedro-plugins/issues/159 Just created the issue to keep track of it