Hi Team, I am trying to make dbx work with kedro 0.18 using wheel file. I have resolved majority of the issues but i am stuck on one issue(hopefully the last); while executing the

dbx execute <workflow-name> --cluster-id=<cluster-id>

; kedro is failing on the below error;

/local_disk0/.ephemeral_nfs/envs/pythonEnv-f0037269-19cc-4c81-9dc2-43bcd22cd8ff/lib/python3.8/site-packages/kedro/framework/startup.py in _get_project_metadata(project_path)
     64 
     65     if not pyproject_toml.is_file():
---> 66         raise RuntimeError(
     67             f"Could not find the project configuration file '{_PYPROJECT}' in {project_path}. "
     68             f"If you have created your project with Kedro "

RuntimeError: Could not find the project configuration file 'pyproject.toml' in /databricks/driver.

I can see that the file was never packaged but I am not sure if it was supposed to be packaged or not. Plus it is pointing to working directory as /databricks/driver somehow. Below is the python file I am running: as spark_python_task

from kedro.framework.project import configure_project
from kedro.framework.session import KedroSession

package_name = "project_comm"

configure_project(package_name)


with KedroSession.create(package_name,env="base") as session:
    session.run()

Any help would be great!! PS: I have tried with dbx deploy and launch as well and is still facing the same issue

I'm also going to tag @Pedro Abreu or potentially @poornima p. They have worked with

dbx

.

Hey Anu, I'm taking a look into this now too

Thanks Yetu, I have spoken to Pedro already and Di and him helped in resolving majority of the issues but got stuck on this last one.