In addition to the last one: Question about dependency management when deploying Kedro to Databricks without the

kedro-databricks

plugin (we use our own custom deployment via Databricks Asset Bundles + GitHub Actions): Our project has both

pyproject.toml

(Kedro project config) and

requirements.txt

. Our GitHub Actions workflow currently runs

uv pip install -r requirements.txt

, then packages the project and deploys the bundle ourselves. 1. Is there anything in Kedro itself (e.g.,

kedro package

, session/config loading) that depends on

requirements.txt

existing, or is it purely a pip convention we can drop in favor of

pyproject.toml

? 2. For running on Databricks clusters, what's the recommended way to declare runtime dependencies — bake them into the wheel's

pyproject.toml

dependencies, or install from a requirements file on the cluster/job definition? 3. Best practice for splitting dev/test dependencies —

pyproject.toml

optional-dependencies (e.g.,

[project.optional-dependencies] dev

) vs. a separate

requirements-dev.txt

?

I don't recall anything specific to

requirements.txt

, it may be becauase kedro use

cookiecutters

to generate template so it is easier to manage this as separate file. For users it's totally fine to merge it into pyproject.toml 2. What do you mean by runtime dependencies? Can you explains more and give an example 3. I personally just use

pyproject.toml

these days, they are almost equivalent

Hi @Nok Lam Chan, By runtime dependencies, I mean 1. Where and how should we specify Databricks cluster details (compute configuration) for running the Kedro pipeline jobs after deployment? 2. Where to store cluster configuration (in databricks.yml? OR in conf parameters? and How the jobs are deployed from the bundle so that it will use the cluster?

I may be outdated with

kedor-databricks

, from my understanding the focus of its it packaging kedro pipeline and setting up databricks bundle assets quickly. For the things that you mention, I think this is covered by DBA itself, i.e. the

resource

file where you can define different job running the same kedro pipeline with a different compute profile.