Kedro is an open-sourced Python framework for creating maintainable and modular data science code.

Kedro

Airflow is far from dead! <https://medium.com/apache-airflow/how-we-orchestrate-2000-dbt-models-in-apache-airflow-90901504032d>

_Personal Opinion_ 

I think Airflow is a wonderful piece of technology and has adapted to the K8s / cloud native world in a way that Luigi, Azkaban and others around the same time could not.

If I was starting a company tomorrow I would reach for Flyte (:second_place_medal:), Meatflow (:third_place_medal: )or Dagster (:first_place_medal:) because they are data-centric and optimised for ML workflows by design.

on that note: <https://www.linkedin.com/posts/greenbergdon_lean-data-automation-a-principal-components-activity-7206420802643091456-ULei?utm_source=share&amp;utm_medium=member_ios|https://www.linkedin.com/posts/greenbergdon_lean-data-automation-a-principal-components-activity-7206420802643091456-ULei?utm_source=share&amp;utm_medium=member_ios>

Or if you’re Airflow 2.0 you’re a new kid on the block
<https://www.youtube.com/watch?v=jXmRrChXUrI>

^ the part at 32:49 is super relevant to Kedro cc <@U03SB8YSP97>

yup, we got to chat a good amount with Jarek at PyCon US, great guy!

Yeah — fun one to write!  So Hamilton/Kedro + dlt/dbt (all good _asset layer_) options can work nicely with an orchestrator. Also using an orchestrator as a scheduler thus leveraging dagster/prefect/airflow/metaflow/flyte to run those/track larger jobs (personaly I’d use dagster/prefect for ease of iteration).

<https://medium.com/@hugolu87/how-to-get-started-with-apache-airflow-2024-updated-c388fb5433e4>

(conflict in interest, because it’s written by a competitor, but it’s amusing nonetheless)