https://kedro.org/ logo
#questions
Title
# questions
n

Nan Dong

05/31/2023, 3:20 PM
Hello everyone! Long time Kedro user here, which completely revolutionized how our data team writes pipelines. We are trying to choose an orchestrator, can anyone provide some insights on the pros and cons of Airflow vs Prefect? We are considering Airflow because there’s experience in using that in the team, but some have found it cumbersome to repackage the Kedro project each time there’s code change (containerized with Docker). We also have experience with Prefect 1.0, using the script included in the Kedro deployment docs, but are not sure about how the migration to Prefect 2.0 will work with Kedro. Thank you all in advance!
🔥 4
j

Juan Luis

05/31/2023, 3:36 PM
hi @Nan Dong, welcome to the Kedro Slack and thanks for your kind words! we were discussing about Prefect 2.0 just today. Kedro on Prefect 2.0 is still uncharted territory, so we cannot comment on that (we're looking for contributors that want to update those docs). about the choice of orchestrators, I like this blog posts by our friends of Neptune.ai https://neptune.ai/blog/argo-vs-airflow-vs-prefect-differences
n

Nan Dong

05/31/2023, 3:38 PM
Thank you for the quick reply and link, @Juan Luis !
Would you also be able to point me to some resources on why Airflow is not a good framework for authoring data pipelines? We have some team members recommending doing that, but I feel pretty strongly (gut feeling) it’s a bad idea… 😐 Just wondering if there are resources to help me articulate why.
j

Juan Luis

05/31/2023, 3:42 PM
there's lots of articles criticising Airflow, it was one of the big topics in the data sphere last year 😄 this one in particular was making the rounds for months: https://blog.fal.ai/the-unbundling-of-airflow-2/
I think the main pain point of Airflow is passing data between tasks, but I'm by no means an expert
n

Nan Dong

05/31/2023, 3:44 PM
I’m familiar with the many pieces critical of Airflow in recent times 😆. Thank you for the resources, Juan!
🙌🏼 1