https://kedro.org/ logo
#questions
Title
# questions
f

Flavien

08/17/2023, 4:26 PM
Hi, I am a bit confused with the inclusion of optional dependencies (maybe related to this). If I understand the working of optional dependencies in
pyproject.toml
, I should be able to install the group
spark
in
kedro-datasets
with
pip install kedro-datasets[spark]
(I tested the syntax with
pip install httpx[socks]
). But it does not seem to work as
pyspark
is not installed. It works with
pip install kedro-datasets[spark.SparkDataSet]
. Thanks in advance for pointing me to my mistake. :)
j

Juan Luis

08/17/2023, 4:30 PM
hi @Flavien, in kedro-datasets 1.5.0 we messed up the optional dependencies a bit, we're working on a solution that will see the light soon (hopefully tomorrow or Monday)
f

Flavien

08/17/2023, 4:39 PM
Excellent! Thanks for the reply and lifting my confusion. 😅
j

Juan Luis

08/18/2023, 2:45 PM
hi @Flavien, kedro-datasets 1.5.3 fixes this issue, please try again!
f

Flavien

08/18/2023, 3:04 PM
Let me do that now. 👍
It works like a charm. 👍
j

Juan Luis

08/18/2023, 3:13 PM
fantastic, thanks for checking!
2 Views