https://kedro.org/ logo
#questions
Title
# questions
v

Vaibhav

01/18/2023, 11:15 AM
Hi, Is it possible to raise / remove the ceiling for pyarrow, it is currently pinned to <7.0 and we wanted to use kedro with some libraries which needs pyarrow 8. Thank you!
d

datajoely

01/18/2023, 11:17 AM
which libraries?
also we only use pyarrow for
pandas.ParquetDataset
so you should be able to just upgrade the library with that being the only risk of it breaking
v

Vaibhav

01/18/2023, 12:56 PM
We use optbinning which for some reason gives a segmentation error on pyarrow < 8 in mac but works fine on other OS. We use
pandas.parquertDataset
in our pipeline, I tried with pyarrow == 8 nothing breaks
Do you have a good way to upgrade pyarrow without pip’s dependency resolver complaining of the conflict ? Only way I can think of is installing in 2 steps.
m

Merel

01/18/2023, 1:20 PM
@Vaibhav feel free to open a PR on https://github.com/kedro-org/kedro-plugins/tree/main/kedro-datasets to bump up the version of pyarrow
Also which version of Kedro are you using?
d

datajoely

01/18/2023, 1:21 PM
I’m not sure the best way to force pip aside from the two steps
v

Vaibhav

01/18/2023, 1:36 PM
Is it possible to do this for for 0.17 as well, as we haven’t migrated to 0.18 yet ?
Thanks for the PR I think this will be very helpful 🙂
d

datajoely

01/18/2023, 1:36 PM
it’s not possible to retroactively change releases like that
v

Vaibhav

01/18/2023, 1:44 PM
yes of course not retroactively changing the release. I was thinking it its a candidate for a patch version, but haven’t seen patch version in kedro yet.
m

Merel

01/18/2023, 1:45 PM
Patching would only really happen for fixing bug issues. Bumping of dependencies doesn’t really warrant a patch unfortunately.
v

Vaibhav

01/18/2023, 1:48 PM
nw - thanks a lot for the help Merel and Joel! We can live with two step install to solve this until we upgrade to 0.18
d

datajoely

01/18/2023, 1:50 PM
You’ll also note our move to
kedro-datasets
being a standalone package will make this way way easier in the future as we can bump / change / push releases way faster than corep