Hello, anyone have a guide / example for running Kedro pipelines on DataBricks with dbx?

the docs are being revised currently after full research round https://github.com/kedro-org/kedro/issues/2105

Very interesting! We are facing same challenges on Databricks. By the way, have you tackle how to input secrets from Azure KeyVault or DBs secrets?

environment variables are often the right solution to this

Hey Nikos and Toni, we're defining a best-practices way of working with Kedro on Databricks (with a focus on using

dbx

) under this ticket 😃 we'll have a guide very soon after that

Thanks all - having a look through docs 🙏

thanks a lot! sharing this with my team 🙂

Hi @datajoely, I am also interested in the current ways existing to input Azure KeyVault or DB secrets, would it be possible to share the examples ?

Same as @Mathilde Lavacquery, we still dont have a simple way to connect through credentials from catalog when running on Databricks cluster. Environment variables do not work if you read from credentials.yml (unless you edit your data type I guess). Could you clarify how do you recommend to deal with credentials in a cluster? This is the weakest point we found for kedro so far and we haven’t fix this yet 😞 (Thanks a lot in advance!)

Hey Toni, thanks very much for bringing this up, this is interesting. I don't think we have official advice for this yet. I'll open a ticket to investigate / create a solution for this, though it would be good to know: Why doesn't setting the environment variables for the cluster through the DB UI (like this) work in this case? My understanding is that the workers would be able to read these env variables

so this is the best way to do this via env vars today

thanks a lot, let me share this with the team 🙂