Hi everyone 👋 I am really excited to finally share a plugin I’ve been working on for over a year: kedro-dagster which allows user to orchestrate Kedro pipelines using dagster, a modern asset-oriented Python orchestrator. It also comes with a companion example repo kedro-dagster-example. It has been an amazing learning journey for me and I am grateful to the Kedro community for creating such a positive environment where all questions are answered with kindness and curiosity. A special thanks to @Deepyaman Datta for the early guidance :) Main features - Lets you map your existing Kedro pipelines into the Dagster orchestrator: auto-generate a

definitions.py

and use a

dagster.yml

in your config to specify jobs, schedules, executors etc. - Preserves your Kedro hooks intact. In particular, it works seamlessly with kedro-mlflow. - Experimentally supports Dagster partitions by fanning-out Kedro nodes acting on partitioned datasets. - The example repo is a full-blown small project showing how it can be wired up. It makes use of dynamic pipelines and also showcases distributed hyperparameter tuning using a new `optuna.StudyDataset` experimental dataset. Get started - Docs: https://kedro-dagster.readthedocs.io/ - Plugin repo: https://github.com/gtauzin/kedro-dagster - Example repo: https://github.com/gtauzin/kedro-dagster-example - Kedro-Viz of the example repo: https://gtauzin.github.io/kedro-dagster-example/ I would love your help & feedback It would mean a lot if you could: * Try it out in one of your Kedro projects * Spot issues, missing bits or docs gaps * Share how you would use it, or ideas for features/improvements * Reach out if you would like to contribute! I am looking forward to hearing what you think and how you might use it! :)

That's fantastic @Guillaume Tauzin! Would you like to present and demo the plugin in one of our upcoming Coffee Chats? It's live streamed on YouTube/LinkedIn. 😄 (cc: @Elena Khaustova)

Thanks @Merel 🙂 I think it would be nice! Initially, I was thinking to stress test it a bit, gather some feedback and make a couple of improvements before writing a blog post, but a Kedro Coffee Chat would be amazing too :)

@Guillaume Tauzin Amazing work! It's really a shining example of how Kedro deployment should be done. I hope this can also inspire improvements across the board, e.g. in

kedro-airflow

.

Amazing! @Guillaume Tauzin let me know when you’re ready to do it 🙂

Thanks a lot for the kind words @Deepyaman Datta and for all the discussions :)

Amazing work @Guillaume Tauzin !!! 🤩

Thank you @Alice Cima :)