Hello everyone Long time Kedro user here which completely re

Question

Hello everyone Long time Kedro user here which completely revolutionized how our data team writes pipelines We are trying to choose an orchestrator can anyone provide some insights on the pros and cons of Airflow vs Prefect We are considering Airflow because there s experience in using that in the team but some have found it cumbersome to repackage the Kedro project each time there s code change containerized with Docker We also have experience with Prefect 1 0 using the script included in the Kedro deployment docs but are not sure about how the migration to Prefect 2 0 will work with Kedro Thank you all in advance

Juan Luis · Answer

hi < Nan Dong> welcome to the Kedro Slack and thanks for your kind words we were discussing about Prefect 2 0 just today Kedro on Prefect 2 0 is still uncharted territory so we cannot comment on that we re looking for contributors that want to update those docs about the choice of orchestrators I like this blog posts by our friends of

Nan Dong · Answer

Thank you for the quick reply and link < Juan Luis>

Nan Dong · Answer

Would you also be able to point me to some resources on why Airflow is not a good framework for authoring data pipelines We have some team members recommending doing that but I feel pretty strongly gut feeling it s a bad idea neutral face Just wondering if there are resources to help me articulate why

Juan Luis · Answer

there s lots of articles criticising Airflow it was one of the big topics in the data sphere last year smile this one in particular was making the rounds for months <https blog fal ai the unbundling of airflow 2 >

Juan Luis · Answer

I think the main pain point of Airflow is passing data between tasks but I m by no means an expert

Nan Dong · Answer

I m familiar with the many pieces critical of Airflow in recent times laughing Thank you for the resources Juan