Quick question, why is kedro-datasets no longer co...
# questionsm
Matthias Roels
08/31/2023, 12:03 PMQuick question, why is kedro-datasets no longer compatible with Spark 3.4?
d
datajoely
08/31/2023, 12:16 PMlooking into this
👍 1
m
marrrcin
08/31/2023, 12:33 PMJust FYI, have no problems with
pyspark==3.4.1kedro-datasetsd
datajoely
08/31/2023, 12:37 PMwe deffo made the change here
https://github.com/kedro-org/kedro-plugins/commit/062d570662e00267f5f8f88c9cb013937e7879e1
datajoely
08/31/2023, 12:37 PMbut trying to work out why
datajoely
08/31/2023, 12:38 PMcurrent hypothesis is that maybe 3.11 support was the issue
🤔 1
datajoely
08/31/2023, 12:39 PMequally the dates are confusing here, 3.11 support was added to PySpark in 3.4.0
m
Matthias Roels
08/31/2023, 12:40 PMYeah the dates are because Spark provides patch updates for several versions
👍 2
Matthias Roels
08/31/2023, 12:42 PMMy guess is that it has something to do with the exception handling in the SparkDataset. There is a deprecation comment in that file. But that should be no reason to remove support for Spark 3.4. In fact, this is something that needs to be fixed in SparkDataset…
👍🏼 1
d
datajoely
08/31/2023, 12:42 PMYes agreed
datajoely
08/31/2023, 12:42 PMthe person who made this change is at the 🦷 dentist, will get an answer to you shortly
🙃 1
😂 1
🤪 1
😅 1
s
Sajid Alam
08/31/2023, 1:23 PMHi, I believe this was an oversight as we moved between
pyproject.tomlsetup.py❤️ 1
d
datajoely
08/31/2023, 1:30 PMthanks @Sajid Alam
m
Matthias Roels
08/31/2023, 2:11 PMSo it should be ok with 3.4?
s
Sajid Alam
08/31/2023, 2:12 PMYes!
d
datajoely
08/31/2023, 2:33 PMif you want to force it, it will work
m
Matthias Roels
08/31/2023, 3:31 PMBut it will be fixed with a 1.6.1 release?
s
Sajid Alam
08/31/2023, 3:33 PMYes, there is a PR out for it now: https://github.com/kedro-org/kedro-plugins/pull/323
👍 1
2 Views