Very specific question, but would be happy to talk more to/learn from anybody who's facing these challenges: • Are there people who are primarily developing transformation code outside of Databricks (i.e. using Kedro, I assume not too many people using alternatives in here 😉), and trying to better orchestrate that? • More specifically, are you trying to do this not in Databricks (i.e. not using Databricks Workflows), but using Dagster or Airflow?

We are using Argo Workflows on kubernetes. So no Databricks in our stack, but neither Airflow nor Dagster… For now, we built all our yaml files “by hand” using the knowledge of the kedro project’s pipeline structure. Still haven’t figured out a good way to automate this. I can talk more about the particular challenges if you want.

I'm specifically trying to find people orchestrating code that runs on Databricks in this case. But happy to talk more about your issue, too! I will DM you to make sure I understand.

Oh then I misunderstood your question 🙈