Hello Team! So it's been a few months since we started using kedro and it's time to deploy some of the pipelines we have created. We need to choose an orchestrator but this is not our field of expertise, so I wanted to ask for some help. We would like something simple to setup and use collaboratively. Also my company requires it is free (at least for now), our cloud provider is AWS and we already use mlflow. Here are the alternatives we found: • Prefect (open-source, seems nice to use, kedro support, but free tier imposes limitations) • Flyte (free?, open-source, seems nice to use, no kedro support) • MLRun (free and open-source, no kedro support? seems nice to use but a bit more than an orchestrator, requires python 3.9) • Kubeflow Pipelines (free and open-source, kedro plugin, @marrrcin and others seem to think it is complex to setup and maintain) • Airflow (free and open-source, kedro plugin) • Sagemaker (Amazon, kedro plugin, personally dislike its UI and how other AWS services are organized around it) What would you recommend? What should we consider to make such a decision? Thanks for your help :)

hello @Guillaume Tauzin! disclaimer: I'm not an expert in any of these tools (alternative way of saying "it depends" 🙂) when you talk about the limitations of the various free tiers, I guess those apply to the corresponding Cloud/Hosted option, right? taking the example of Prefect, to the best of my knowledge Prefect OSS doesn't have any limitations. the free tier of Prefect Cloud does, though (max of 5 000 runs/day). I guess something like that applies to Flyte vs Union, or Airflow vs Amazon MWAA (yes, AWS offers a managed Airflow service). if you intend to operate the orchestrator yourself, then you're free to choose from the different OSS options. what do you want out of an orchestrator? given that your business logic (hence CPU-bound tasks) will live in the Kedro pipelines themselves, probably you'll want to pick a simple orchestrator that dispatches tasks, centralizes logs, displays execution status (and in my personal opinion, you don't need Kubernetes for that). tl;dr: think carefully whether you want to operate your orchestrator yourself, or use some managed service. then, you need to think where your pipelines will run. considering that your cloud provider is AWS, you'll be looking at Amazon EC2, Amazon ECS, etc. definitely not the same hardware where your orchestrator lives, otherwise you risk taking it down accidentally! again taking Prefect as an example, looks like

prefect-aws

allows you to deploy your flows on ECS. the most tried and tested orchestrator out there is Airflow, and there's an official Kedro plugin for it. but building Kedro translators isn't really a terribly difficult task, just see the code snippet in our docs that translates pipelines to Prefect for example. I'll let others comment on their specific experiences 👂🏼

Hi @Juan Luis and thanks for your reply! > to the best of my knowledge Prefect OSS doesn't have any limitations. the free tier of Prefect Cloud does, though (max of 5 000 runs/day) I missed that, thanks for pointing it out! > if you intend to operate the orchestrator yourself, then you're free to choose from the different OSS options. what do you want out of an orchestrator? given that your business logic (hence CPU-bound tasks) will live in the Kedro pipelines themselves, probably you'll want to pick a simple orchestrator that dispatches tasks, centralizes logs, displays execution status (and in my personal opinion, you don't need Kubernetes for that). Indeed, I think this is what I need. Also, I wonder if it could allow me to: • Choose whether I run a set of node on a single machine or each node on a different machine (and if I am asking myself this question, does it mean I separated my pipelines/nodes wrongly?) • Choose the target machine for each node (i.e. some task I would like to run on a small/big EC2 instance or some others on a Dask cluster) For now, I'll have a deeper look at

prefect

and

airflow

documentations, thanks! :)

Dagster is another one not on your list. In general, building an orchestrator integration isn't that bad; they all look kinda similar. If you want to go with Flyte or something, it shouldn't be that hard. I'm kind of bought into the view Dagster pushes, that a data orchestrator should be asset-oriented (like Kedro is also asset-oriented), and not task-oriented (e.g. Airflow). You can read more here: https://dagster.io/blog/impedance-mismatch-in-data-orchestration (There are other asset-oriented orchestrators, but I would need to find a list of which is which to not make a mistake :P)

Hi @Deepyaman Datta! Thank you for your reply. I was looking at the bridge script between kedro and prefect in the docs as @Juan Luis suggested and it does seem quite straightforward. It's also a very transparent way to use an orchestrator from kedro. I had a look at

dagster

and watched the talk by Pete Hunt and it does make a lot of sense. I will have a better look, thanks for sharing!

this "Principal Components Analysis" of orchestrators also considers Kedro an asset-based micro-orchestrator https://www.run.house/blog/lean-data-automation-a-principal-components-approach in Kedro we run nodes yes, that many times end up in calling a dataset

.save()

method, so I definitely agree with the asset-oriented nature. I think the automatic arrangement of the DAG makes Kedro a poor task-based orchestrator (every time somebody tries that, they have to create "dummy datasets" to have control over said tasks)

Interesting read, thank you. I see your point. Kedro is structured around a data catalog, has inbuilt load/save logic at the node levels and generate automatically a DAG based on data lineage, therefore it is asset-driven. Even if we can't do

kedro materialize --assets model_input_table

and we have to do

kedro run --pipelines data_processing

@Guillaume Tauzin about Flyte - I have a working prototype of

kedro-flyte

- I hope we will be able to open source it in Nov/Dec this year.

@marrrcin Thanks, would be nice to be able to try it out. May I ask you what would make you choose

flyte

over say

prefect

for your

kedro

pipeline orchestration?

Main reason is in the execution layer: Prefect is mainly an orchestrator. Flyte is an orchestrator and an execution layer - it natively supports distributed execution. Prefect does not.

I see, if I undersdood propely

prefect

seems to have a task runner concept that can make use of external tools like

dask

or

ray

., while

flyte

is kubernetes-native so you get distributed execution natively (while still having access to

dask

,

ray

and

spark

throught OSS kubernetes operators). One painful point for me is Kubernetes, I have only a basic understanding of how it works and not much time to focus on learning it. Even though I would get some (limited) support from a DevOps in my company that has Kubernetes expertise, I am wondering if I should choose an orchestrator that relies on a technology that I don't understand. If any of you have some advice or experience being in this situation I would really appreciate it. 🙂

The question to answer is - what is your use case for the orchestrator? If it's for scheduling Kedro pipelines or whether it's for scaling up Kedro pipelines. For 1, Prefect is good option, for 2 IMHO no.

Got it, thanks @marrrcin for your help 🙂

thanks a lot for sharing @Guillaume Tauzin ! how was the experience of building the code that translates Kedro into Dagster assets?

@Juan Luis It was straightforward but I had to make some choice as dagster has different options to define a generic task. I replied with more details on the spike issue. I will keep looking into better integrating kedro with dagster, While I have a translation script, it is not sufficient to actually use dagster in production (e.g. I can't assign execution targets to nodes or pipelines atm). Also, I believe there are other concept in dagster that could be mapped to kedro or handled with kedro's omegaconf setup.

Very exciting! I haven't actually used Dagster myself, but this was something I was hoping to explore, so glad you beat me to it. I'll definitely take a look later. :)

@Deepyaman Datta Thanks for recommending it. :) I would really appreciate any feedback, thank you!

As I recently joined Dagster (full disclosure: I had barely started the process with Dagster and had no idea I was going to end up there, when I brought it up; truly did think it was an interesting possibility from what I'd seen from the outside), I'm definitely interested in helping contribute to this. I'll follow up on the issue at some point, but where I personally want to start with is understanding how the dbt integration works (as it seems that is a very successful and widely used Dagster integration), and trying to model the Kedro integration off of that as a starting point. I'm still very much onboarding, so you probably know a lot more in terms of Dagster concepts and things than I do at this point @Guillaume Tauzin—I still need to go through Dagster University! 😂

Wow amazing news @Deepyaman Datta! Congratulations 🎉 I am very happy to hear this. I have been working in the past month to improve the kedro-dagster integration and it has now taken the form of a plugin. It still requires quite a bit of work and it is still untested and not properly documented, but obviously it would be a real treat for me to work with you on this. Obviously, you are in a much better place than me to contribute to this effort! I was planning on writing proper doc before sharing it here and asking for feedback, but please do not hesitate to reach out if you want to discuss it whenever you are ready!

omg look at this logo!!

Oh, wow, that is an amazing logo!

@Juan Luis Ahaha thank you! It was mostly me procrastinating writing the documentation last weekend xD @Deepyaman Datta Thank you, really appreciate your kind words. I'll definitely keep you updated on this thread!

Awesome work @Guillaume Tauzin and congrats on the new job @Deepyaman Datta. Great to see this, I have added some thoughts on the spike issue from my brief time of thinking about kedro/dagster would love to see how that matches others perspective 🙂

Thanks @Florian d! Great to see you are also looking into this :)

Thanks for the comment! Very excited to see more people interested in this. 🙂 I have also put this on the radar internally in Dagster. That said, priority is obviously heavily influenced by amount of interest from the community; it's great to know at least you two are looking into this, and I hope will get a lot more interest as people see the initial integration demo! Also, on the off chance either of you (or anybody who sees this) is using Dagster commercially, or in a large/established deployment, do let me know! Feels like it can't hurt to share that back from the perspective of increasing priority from Dagster's perspective.

Great to hear this @Deepyaman Datta! kedro-dagster is for now a personal project but is being tested internally at my company. Once I am able to demonstrate how useful dagster can be, it will be most likely used within my organization. For the context, I am starting DS efforts in a small company (equipping electricity grids with sensors) < 30 people.