Hello everyone! Long time Kedro user here, which completely revolutionized how our data team writes pipelines. We are trying to choose an orchestrator, can anyone provide some insights on the pros and cons of Airflow vs Prefect? We are considering Airflow because there’s experience in using that in the team, but some have found it cumbersome to repackage the Kedro project each time there’s code change (containerized with Docker). We also have experience with Prefect 1.0, using the script included in the Kedro deployment docs, but are not sure about how the migration to Prefect 2.0 will work with Kedro. Thank you all in advance!

hi @Nan Dong, welcome to the Kedro Slack and thanks for your kind words! we were discussing about Prefect 2.0 just today. Kedro on Prefect 2.0 is still uncharted territory, so we cannot comment on that (we're looking for contributors that want to update those docs). about the choice of orchestrators, I like this blog posts by our friends of Neptune.ai https://neptune.ai/blog/argo-vs-airflow-vs-prefect-differences

Thank you for the quick reply and link, @Juan Luis !

there's lots of articles criticising Airflow, it was one of the big topics in the data sphere last year 😄 this one in particular was making the rounds for months: https://blog.fal.ai/the-unbundling-of-airflow-2/

I’m familiar with the many pieces critical of Airflow in recent times 😆. Thank you for the resources, Juan!