One thing that I'm having some trouble is with the conf folder, when I run in databricks those dir are missing and execution fails, i've made it work by packaging the configs and fixing the conf dir at settings.py

but it doesnt seem very ellegant

right now i'm using dbx execute for the development workflow

it works almost nicely, it takes sometime to install the package at databricks and it does the installation every run

if I remove the dependecies after the first run, it goes alot faster

thinking on maybe writing some cache-like check for the setup.py install

for developing I went back to notebooks, I've created a notebook 'kedro_runner' which you run specifying your pipeline or any kedro run cli argument

I was using 'dbx run' for trying to keep everyhing at vscode, but it is a bit slow as we have to pacakge everything and install everytime at databricks cluster. Between when I tested this out and now databricks launched a vscode extension, but I haven't had the time to test it out