Anu Arora
11/25/2022, 1:45 PMdbx execute <workflow-name> --cluster-id=<cluster-id>
; kedro is failing on the below error;
/local_disk0/.ephemeral_nfs/envs/pythonEnv-f0037269-19cc-4c81-9dc2-43bcd22cd8ff/lib/python3.8/site-packages/kedro/framework/startup.py in _get_project_metadata(project_path)
64
65 if not pyproject_toml.is_file():
---> 66 raise RuntimeError(
67 f"Could not find the project configuration file '{_PYPROJECT}' in {project_path}. "
68 f"If you have created your project with Kedro "
RuntimeError: Could not find the project configuration file 'pyproject.toml' in /databricks/driver.
I can see that the file was never packaged but I am not sure if it was supposed to be packaged or not. Plus it is pointing to working directory as /databricks/driver somehow. Below is the python file I am running: as spark_python_task
from kedro.framework.project import configure_project
from kedro.framework.session import KedroSession
package_name = "project_comm"
configure_project(package_name)
with KedroSession.create(package_name,env="base") as session:
session.run()
Any help would be great!!
PS: I have tried with dbx deploy and launch as well and is still facing the same issueYetunde
11/25/2022, 1:50 PMdbx
.Jannic Holzer
11/25/2022, 1:50 PMAnu Arora
11/25/2022, 1:52 PM