Hi team, trying to run kedro with spark.SparkDataS...
# questions
Hi team, trying to run kedro with spark.SparkDataSets on databricks. When running in a notebook, looks like there is an issue between the spark session of the notebook and the one the project is trying to create. Can someone assist with resolving this conflict?
Seems I cannot invoke project pipeline run with %sh kedro run ... If someone can help explain why, that would be awesome!
hi @Ezekiel Day, running Kedro on Databricks requires some special care, and notably the CLI doesn't work. you can make it run by using this Python code:
Copy code
from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project


with KedroSession.create(project_path=project_root, env="databricks") as session:
please have a look at these docs for more info: https://docs.kedro.org/en/stable/integrations/databricks_workspace.html#run-your-kedro-project-from-the-databricks-notebook
👍 1