Hi fellows, I am cleaning dependencies in our

kedro

code and, upon scrutiny, I am a bit confused by the dependencies for

databricks.ManagedTableDataset

. In

pyproject.toml

, https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/pyproject.toml, is stated

hdfs-base = ["hdfs>=2.5.8, <3.0"]
s3fs-base = ["s3fs>=2021.4"]
...
databricks-managedtabledataset = ["kedro-datasets[hdfs-base,s3fs-base]"]
databricks = ["kedro-datasets[databricks-managedtabledataset]"]

But in the implementation, I don't see any reference to those two packages while the dataset requires

pyspark

which is not stated as a dependency if I am not mistaken. Could you tell me if my interpretation is incorrect?

This is possibly true and no one has ever reported the issue until now because you’re so likely to have Pyspark in a databricks env I don’t think it’s come up! Please submit a PR or a GitHub issue it would be much appreciated

I will do that. 👍

there are several issues related to this, see https://github.com/kedro-org/kedro-plugins/issues/135 and linked issues

@Juan Luis, while I see the connection between the two groups of datasets, they are nevertheless independent if I am not mistaken. By mentioning the issues, do you mean that we shouldn't fix the Databricks part without fixing the more generic Spark ones?

sorry, just seeing this. I was just highlighting that the state of the spark and databricks datasets is messy, in case folks want to solve those issues in a holistic manner. @Flavien did you end up opening an issue about your original problem?

No, I didn't. I can do that tomorrow. I can make a small PR for the Databricks part too but, as I am off starting next week, the Spark part would be too much work.