Quick question why is kedro datasets no longer compatible wi

Question

Quick question why is kedro datasets no longer compatible with Spark 3 4

datajoely · Answer

looking into this

marrrcin · Answer

Just FYI have no problems with `pyspark==3 4 1` and SparkDataSet from kedro extras <https github com kedro org kedro blob 0293dc15812b27330bba31a01c7b332b3165af2a kedro extras datasets spark spark dataset py> Kedro 0 18 12 haven t tested with `kedro datasets` though

datajoely · Answer

we deffo made the change here <https github com kedro org kedro plugins commit 062d570662e00267f5f8f88c9cb013937e7879e1>

datajoely · Answer

but trying to work out why

datajoely · Answer

current hypothesis is that maybe 3 11 support was the issue

datajoely · Answer

equally the dates are confusing here 3 11 support was added to PySpark in 3 4 0

Matthias Roels · Answer

Yeah the dates are because Spark provides patch updates for several versions

Matthias Roels · Answer

My guess is that it has something to do with the exception handling in the SparkDataset There is a deprecation comment in that file But that should be no reason to remove support for Spark 3 4 In fact this is something that needs to be fixed in SparkDataset

datajoely · Answer

Yes agreed

datajoely · Answer

the person who made this change is at the tooth dentist will get an answer to you shortly

Sajid Alam · Answer

Hi I believe this was an oversight as we moved between `pyproject toml` and `setup py` midway through the ticket

datajoely · Answer

thanks < Sajid Alam>

Matthias Roels · Answer

So it should be ok with 3 4

Sajid Alam · Answer

Yes

datajoely · Answer

if you want to force it it will work

Matthias Roels · Answer

But it will be fixed with a 1 6 1 release

Sajid Alam · Answer

Yes there is a PR out for it now <https github com kedro org kedro plugins pull 323>