Hey team, I'm getting the following exception:

An exception occurred when parsing config for dataset 'raw_web@delta':
Object 'DeltaTableDataSet' cannot be loaded from 'kedro.extras.datasets.spark'. Please see the documentation on how to install relevant dependencies for kedro.extras.datasets.spark.DeltaTableDataSet:
<https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html>

If I go to the link, it doesn't tell me what I should be installing. I already have the following in the requirements.txt:

kedro-datasets[spark,delta]

There are two delta datasets, you need to declare them explicitly

pandas.DeltaTableDataSet

spark.DeltaTableDataSet

hey @Cory Maklin, not sure if it's related but we're having some issues with the additional dependencies in kedro-datasets these days.

for future reference all of the requirements are here https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/pyproject.toml

can you try

$ pip install "kedro-datasets[spark-base,hdfs-base,s3fs-base]" "delta-spark>=1.0, <3.0"

and try again?

Thanks

also @Cory Maklin where did you see the

pip install kedro-datasets[spark,delta]

instruction? we have a

[spark]

extra but I don't see we ever had a

[delta]

one, I could be wrong thought

I just added

[delta]

since it wasn't working

fair enough, thanks 👍🏼

pip install "kedro-datasets[spark-base,hdfs-base,s3fs-base]"

yeah it's broken atm. kedro-datasets next release (today or Friday) will fix it