Kedro is an open-sourced Python framework for creating maintainable and modular data science code.

Kedro

Hi,
Is it possible to raise / remove the ceiling for pyarrow, it is currently pinned to &lt;7.0 and we wanted to use kedro with some libraries which needs pyarrow 8. Thank you!

also we only use pyarrow for `pandas.ParquetDataset` so you should be able to just upgrade the library with that being the only risk of it breaking

We use <https://pypi.org/project/optbinning/|optbinning> which for some reason gives a segmentation error on pyarrow &lt; 8 in mac but works fine on other OS. We use `pandas.parquertDataset` in our pipeline, I tried with pyarrow == 8 nothing breaks

Do you have a good way to upgrade pyarrow without pip’s dependency resolver complaining of the conflict ? Only way I can think of is installing in 2 steps.

<@U044Z3ZMGHE> feel free to open a PR on <https://github.com/kedro-org/kedro-plugins/tree/main/kedro-datasets> to bump up the version of pyarrow

Also which version of Kedro are you using?

I’ve just raised the PR <https://github.com/kedro-org/kedro-plugins/pull/100>

I’m not sure the best way to force pip aside from the two steps

Is it possible to do this for for 0.17 as well, as we haven’t migrated to 0.18 yet ?

Thanks for the PR I think this will be very helpful :slightly_smiling_face:

it’s not possible to retroactively change releases like that

yes of course not retroactively changing the release. I was thinking it its a candidate for a patch version, but haven’t seen patch version in kedro yet.

Patching would only really happen for fixing bug issues. Bumping of dependencies doesn’t really warrant a patch unfortunately.

nw - thanks a lot for the help Merel and Joel! We can live with two step install to solve this until we upgrade to 0.18

You’ll also note our move to `kedro-datasets` being  a standalone package will make this way way easier in the future as we can bump / change / push releases way faster than corep