https://kedro.org/ logo
Join Slack
Powered by
Hi, Is it possible to raise / remove the ceiling f...
# questions
v
Hi, Is it possible to raise / remove the ceiling for pyarrow, it is currently pinned to <7.0 and we wanted to use kedro with some libraries which needs pyarrow 8. Thank you!
d
which libraries?
also we only use pyarrow for 
pandas.ParquetDataset
so you should be able to just upgrade the library with that being the only risk of it breaking
v
We use optbinning which for some reason gives a segmentation error on pyarrow < 8 in mac but works fine on other OS. We use 
pandas.parquertDataset
in our pipeline, I tried with pyarrow == 8 nothing breaks
Do you have a good way to upgrade pyarrow without pip’s dependency resolver complaining of the conflict ? Only way I can think of is installing in 2 steps.
m
@Vaibhav feel free to open a PR on https://github.com/kedro-org/kedro-plugins/tree/main/kedro-datasets to bump up the version of pyarrow
Also which version of Kedro are you using?
d
I’m not sure the best way to force pip aside from the two steps
v
Is it possible to do this for for 0.17 as well, as we haven’t migrated to 0.18 yet ?
Thanks for the PR I think this will be very helpful 🙂
d
it’s not possible to retroactively change releases like that
v
yes of course not retroactively changing the release. I was thinking it its a candidate for a patch version, but haven’t seen patch version in kedro yet.
m
Patching would only really happen for fixing bug issues. Bumping of dependencies doesn’t really warrant a patch unfortunately.
v
nw - thanks a lot for the help Merel and Joel! We can live with two step install to solve this until we upgrade to 0.18
d
You’ll also note our move to 
kedro-datasets
being a standalone package will make this way way easier in the future as we can bump / change / push releases way faster than corep
Open in Slack
PreviousNext
Powered by