Long time Kedro user here, which completely revolutionized how our data team writes pipelines. We are trying to choose an orchestrator, can anyone provide some insights on the pros and cons of Airflow vs Prefect? We are considering Airflow because there’s experience in using that in the team, but some have found it cumbersome to repackage the Kedro project each time there’s code change (containerized with Docker). We also have experience with Prefect 1.0, using the script included in the Kedro deployment docs, but are not sure about how the migration to Prefect 2.0 will work with Kedro. Thank you all in advance!
05/31/2023, 3:36 PM
hi @Nan Dong, welcome to the Kedro Slack and thanks for your kind words!
we were discussing about Prefect 2.0 just today. Kedro on Prefect 2.0 is still uncharted territory, so we cannot comment on that (we're looking for contributors that want to update those docs).
about the choice of orchestrators, I like this blog posts by our friends of Neptune.aihttps://neptune.ai/blog/argo-vs-airflow-vs-prefect-differences
05/31/2023, 3:38 PM
Thank you for the quick reply and link, @Juan Luis !
Would you also be able to point me to some resources on why Airflow is not a good framework for authoring data pipelines? We have some team members recommending doing that, but I feel pretty strongly (gut feeling) it’s a bad idea… 😐 Just wondering if there are resources to help me articulate why.