Hi team, trying to run kedro with spark.SparkDataSets on databricks. When running in a notebook, looks like there is an issue between the spark session of the notebook and the one the project is trying to create. Can someone assist with resolving this conflict?

hi @Ezekiel Day, running Kedro on Databricks requires some special care, and notably the CLI doesn't work. you can make it run by using this Python code:

from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project

bootstrap_project(project_root)

with KedroSession.create(project_path=project_root, env="databricks") as session:
    session.run()

please have a look at these docs for more info: https://docs.kedro.org/en/stable/integrations/databricks_workspace.html#run-your-kedro-project-from-the-databricks-notebook