Hello, I have just started a fresh kedro project (1.0). The nodes and pipeline are all set. I can list the pipeline successfully. It works on a single .csv dataset. Not using pyspark. However, while trying to run, it first threw error:

LinkageError occurred while loading main class org.apache.spark.launcher.Main java.lang.UnsupportedClassVersionError:

A little google search told me it's not finding the java installation. To resolve, I installed the latest java (jdk25). Now, the error has changed to:

Py4JJavaError: An error occurred while calling <http://None.org|None.org>.apache.spark.api.java.JavaSparkContext. : java.lang.UnsupportedOperationException: getSubject is not supported

I have checked the java path, and it's pointing to

/usr/lib/jvm/java-11-openjdk-amd64/

despite explicitly mentioning

/usr/lib/jvm/jdk-25.0.1-oracle-x64/bin

in the environment. But I think the main issue is, it seems, with pyspark which is not launching, throwing the same error. Since I do not need pyspark in this project, is there a way to disable it for time being, just to test my pipeline? Or else, how else could I fix this? Thanks!

Hi @Shah, I think the easiest solution will be not to install PySpark at all Typical reasons this happens: 1. You installed a Kedro extra that pulls in Spark dependencies:

pip install kedro-datasets[spark]

or

pip install kedro[spark]

2. Your

pyproject.toml

or

requirements.txt

includes

pyspark

. 3. Your

settings.py

or

hooks.py

imports Spark or defines a Spark session. 4. Your

catalog.yml

includes any dataset whose type starts with

spark.

(e.g.

spark.SparkDataSet

).

Hi Elena, No. 3 😒 Yes, I may have mistakenly added that extra (sometimes 'free' extras come with their own baggage). Is there an easy way out of this? Or should I delete and create a new kedro project?