Hi team, After yesterdays release of

kedro-datasets==1.5.0

, our CI started failing during system tests which do a

kedro run

for a pipeline with spark (see the screenshot). As far as i can see,

SparkDataSet

is still defined with the same name as before. When we used

kedro-datasets==1.4.2

the same tests were running smoothly. I also couldn't find anything specific in the release notes - do we have to update our code (mb some import statements or how it is specified within the requirements)?

You're not pointing to

kedro_datasets

in the apparent broken import

this is how it looks like in the catalog:

_csv: &csv
  type: spark.SparkDataSet
  file_format: csv
  load_args:
    sep: ","
    header: True
    inferSchema: True
  save_args:
    header: True
    mode: overwrite

prm_observation_time_frame:
  <<: *csv
  filepath: data/03_primary/prm_observation_time_frame.csv
  layer: primary

what confused me the most was that in 1.4.2 it worked

Hmm...

Hi!

How was it working in 1.4.2 without the

spark-sparkdataset

extra I wonder?

image.png

Ah, yes,

spark.SparkDataSet

extra on

kedro-datasets

will do nothing. 🙂

Flag https://github.com/kedro-org/kedro-plugins/pull/263, I don’t recalled we changed this behavior, likely a bug

yeah, i ended up pinning kedro-datasets to an older version that worked as well, i just was hoping for a more robust fix 🤓 @Deepyaman Datta, we have all the relevant extras mentioned in

setup.cfg

of the starter, exactly the same as it was before, so i'd assume that correct extras are installed (however, can't confirm 100%, cause our CI commands only list full packages through

pip freeze

)

@Elena Mironova Can you open an issue on https://github.com/kedro-org/kedro-plugins/issues/new/choose? Thanks for flagging this, ideally if we are gonna fix it we should fix it in the 1.5.x series

Sure! done: https://github.com/kedro-org/kedro-plugins/issues/290

@Elena Mironova 1.5.2 is out and should have fixed this issue.

Thank you!! So should we just have

kedro-datasets

in requirements, without optional extras?

If you need spark then

pip install kedro-datasets[spark.SparkDataSet]