Can I use an existing Databricks cluster with kedro-databricks? By default, kedro-databricks tries to create new resources, which my account doesn’t have permission to do. Is there a way to specify an existing cluster ID instead? I tried editing

conf/dev/databricks.yml

with the following code:

default:
  tasks:
  - existing_cluster_id: 0924-121047-3jcdtqh1

But running

kedro databricks bundle --overwrite

raises an error:

AssertionError: lookup_key task_key not found in updates: [{'existing_cluster_id': '0924-121047-3jcdtqh1'}]

Thank you, Nok! As always helpful. Now I don't get this error. But it still tries to create a cluster and I still get Permission denied error. My current

databricks.yml

looks like this:

default:
  tasks:
  - task_key: default
    existing_cluster_id: 0924-121047-3jcdtqh1

So the kedro databricks bundle --overwrite command work and you already run the deploy command?

yes

Check if you may still have some of the bundle creating a cluster instead of use the existing one. It may be helpful if you can reduce this to a single task to debug first then basically do a binary search and see which part of the config is problematic

From the first look it seems that it is not using new bundle definitions, but uses some old ones, but I'm not sure from where it gets them

If you have deploy databricksassets bundle, I think you can see those job definitions from databricks UI directly

Okay, this was my mistake that I didn't deploy them after changes

Sorry I hadn’t seen this. Thank you @Nok Lam Chan for helping out 😊