Hi team 👋 Question about multi-environment deployments in Azure Databricks stack with `kedro-databricks`: We deploy to three tiers, each needing a different workspace path (

/Workspace/shared/testprj/dev

,

/acc

,

/prd

). Currently we hardcode a single

workspace_path

in

databricks.yml

. 1. What's the recommended pattern — one

databricks.yml

with per-target

workspace_path

overrides, or separate files per environment? 2. If one file: is variable substitution supported (bundle

${var.*}

, env vars)? 3. How does

kedro databricks deploy

pick the target — CLI flag, env var, or config? And how does that relate to Kedro's

--env

? 4. Can the path be injected at deploy time from our GitHub Actions workflow instead of being committed in the repo? A reference example of a dev/acc/prd setup with GitHub Actions would be super helpful 🙏

@Jens Peder Meldgaard do you have any thoughts on the above?

Hi @sarthak.updated It depends.. do you have one shared workspace for all environments or separate for each? If you have one, then I'd recommend you use different workspace_paths for each environment. If it's separate workspaces then it doesn't matter as much as they will not interfere with each other. You can use vars and environment variables. Kedro-databricks simply generates the bundle resources for you - so when you deploy, it is an actual Databricks Automation Bundle. In regards to env/target. If no --target is provided, kedro-databricks will add whatever is set in the --env flag. So these are synonymous. The reason why you would use both, would be when you want to deploy the same kedro env to multiple targets. If that is not the case, you can just use --env. If you use vars, then you can inject values into these in your ci/cd using the

kedro databricks deploy -- --var="my_var=value"

. Note anything after

--

is passed directly to the databricks cli.

Hi @Jens Peder Meldgaard Thank you for looking into this and providing a clarity of thoughts the way it needs to be handled. We indeed have separate workspaces for each environment. Also, we didn't get approval to use the kedro-databricks plugin at my org. Hence, now have to re-invent the wheel for multi-env deployment using DAB + Github Actions. Will circle back once we start to build and face challenges in that later this month. For now your explanation helps to start building in a certain direction. Thanks again!

If I may ask, did you get a reason for not being allowed? Just curious if there is anything I can do to help adoption :)

I didn't get specific answer but were cited with vulnerabilities risk for an external plugin being used. CISO is critical on this. As we have recent instances with GitHub vulnerabilities which was a costly affair. I am trying to follow up with them. I am not sure how to provide a concrete explanation that the plugin is safe to use and is well maintained. As I don't find any artifacts about it.

Very fair. I have done my best to limit the number of external dependencies specifically to limit the risk of supply chain attacks. If you get a more specific reason, please let me know and I'll see what I can do to address it. Thank you for asking! :)

if you would be so kind just to give me some pointers about what steps you have taken to limit the occurrence of and mitigate the risk of supply-chain attacks here. I can then prepare a more compelling usecase for using the plugin and reducing our efforts and timelines

This has mostly just been limiting the number of external dependencies - and making sure that I only use well-known dependencies. On the list of required dependencies, the only "weird" one is

fuso

which is used for the merging ligic. However, this is my own library that has no external dependencies.

fuso

was originally part of

kedro-databricks

but I split it out to allow for more specific testing and to be able to use it in other projects.

Thank you @Jens Peder Meldgaard! Will look into it 🙂

I'm someone who is in a similar boat with azure databricks and trying to utilize this. Are there plans to make this kedro native?

@Brian Parbhu, I'm not entirely sure what you mean.. AFAIK, this is the "native" way to extend kedro functionality. :)

Ah sorry for the misunderstanding. Is there a possibility for your work to be integrated within the main repository of kedro and be supported in the same way Airflow and Mlflow are supported? That's what I meant by native again sorry for the confusion

I'm not part of the Kedro core team, and I think that would require them taking over maintenance. :)

Understood

Hi @Brian Parbhu, just to clarify a bit more.

kedro-airflow

is a separate plugin owned and maintained by the core Kedro team.

kedro-mlflow

is just like

kedro-databricks

a plugin owned and maintained by external contributors. That said, both the mlflow and databricks plugin follow best practices and are recommended plugins by the core team. Feature and/or structure wise there isn't a difference between the core team maintaining versus a trusted Kedro contributor.