Hi everyone 👋 I am really excited to finally share a plugin I’ve been working on for over a year: kedro-dagster which allows user to orchestrate Kedro pipelines using dagster, a modern asset-oriented Python orchestrator. It also comes with a companion example repo kedro-dagster-example. It has been an amazing learning journey for me and I am grateful to the Kedro community for creating such a positive environment where all questions are answered with kindness and curiosity. A special thanks to @Deepyaman Datta for the early guidance :) Main features - Lets you map your existing Kedro pipelines into the Dagster orchestrator: auto-generate a

definitions.py

and use a

dagster.yml

in your config to specify jobs, schedules, executors etc. - Preserves your Kedro hooks intact. In particular, it works seamlessly with kedro-mlflow. - Experimentally supports Dagster partitions by fanning-out Kedro nodes acting on partitioned datasets. - The example repo is a full-blown small project showing how it can be wired up. It makes use of dynamic pipelines and also showcases distributed hyperparameter tuning using a new `optuna.StudyDataset` experimental dataset. Get started - Docs: https://kedro-dagster.readthedocs.io/ - Plugin repo: https://github.com/gtauzin/kedro-dagster - Example repo: https://github.com/gtauzin/kedro-dagster-example - Kedro-Viz of the example repo: https://gtauzin.github.io/kedro-dagster-example/ I would love your help & feedback It would mean a lot if you could: * Try it out in one of your Kedro projects * Spot issues, missing bits or docs gaps * Share how you would use it, or ideas for features/improvements * Reach out if you would like to contribute! I am looking forward to hearing what you think and how you might use it! :)

That's fantastic @Guillaume Tauzin! Would you like to present and demo the plugin in one of our upcoming Coffee Chats? It's live streamed on YouTube/LinkedIn. 😄 (cc: @Elena Khaustova)

Thanks @Merel 🙂 I think it would be nice! Initially, I was thinking to stress test it a bit, gather some feedback and make a couple of improvements before writing a blog post, but a Kedro Coffee Chat would be amazing too :)

@Guillaume Tauzin Amazing work! It's really a shining example of how Kedro deployment should be done. I hope this can also inspire improvements across the board, e.g. in

kedro-airflow

.

Amazing! @Guillaume Tauzin let me know when you’re ready to do it 🙂

Thanks a lot for the kind words @Deepyaman Datta and for all the discussions :)

Amazing work @Guillaume Tauzin !!! 🤩

Thank you @Alice Cima :)

Hi all, we're interested in using Guillaume's new plugin in a multi-hospital research project in Switzerland (as Dagster is one of the prominent schedulers in the existing infrastructures). We are curious what is the procedure / roadmap for this plugin to become an officially maintained kedro plugin (as those in https://github.com/kedro-org/kedro-plugins) ? Thank you for clarifying any procedures the kedro org has on this topic 😊

Hi @Laure Vancau, thanks for reaching out! We don't really have a procedure for that as such. The plugins in

kedro-plugins

are those owned by the core Kedro team and that's mostly based on capacity. We heavily rely on external contributors to maintain all the extra plugins like

kedro-mlflow

and

kedro-dagster

. We, and the community, do our best to support and update all the plugins inside and outside of

kedro-plugins

.

Thank you @Laure Vancau and nice to see you here 🙂 In relation to that, there was recently a sprint during which a couple of core Kedro team member tried the plugins and gave some feedback. 🎉 That being said, I must say that I perfectly understand that it can be worrying to decide to use a recent plugin for a long-term production project when a large portion of existing Kedro plugins are no longer maintained. I guess for

kedro-mlflow

, it is less of a problem as the plugin is relatively old and the main author is also in the Kedro TSC. At this stage, there is not much that I can do except for keeping the package maintained as well as I can. Note that I also welcome contributions to

kedro-dagster

at any level (one-off commit to maintenance or driving design/vision for the package). My interest is for the plugin to meet the need of the users and remain up-to-date as both Kedro and Dagster develop.

Thank you both for your input on the matter 😊

@Laure Vancau A bit late to the party, but that's great news! Don't hesitate to contact me directly if there's a need! 🥳