I'm doing a bit of investigation into kedro, and wondered about the best way of running pipelines in a k8s cluster. I see there's a kubeflow plugin https://github.com/getindata/kedro-kubeflow, but the project seems little used and not very actively maintained. Similar observations for the airflow plugin https://github.com/getindata/kedro-airflow-k8s. There also argo workflows, but the docs showing how to do this are out of date https://docs.kedro.org/en/stable/deployment/argo.html, and there's a plugin, kedro-argo, but that seems out of date. So I guess my question is: what's the recommended way to run distributed pipelines in a k8s cluster using well maintained and production ready tooling?

The Getindata's plugin for airflow is no longer needed - everything can be now achieved with Kedro's official Airflow plugin - see https://getindata.com/blog/deploying-kedro-pipelines-gcp-composer-airflow-node-grouping-mlflow/

OK thanks, so in summary the best approach is to run airflow in the k8s cluster and to use the official airflow plugin.

It's not the best (there's never such thing 🙂 ), it's one of the ways of doing that. Depending on what you already have in your k8s cluster, you might take different approaches.

yeah, atm I have an ad hoc system of orchestrating a number of etl type tasks, mediated by passing messages in redis streams. I'm looking at ways to improve this...

You can go down the route of different tools that will help you with ETL too and re-purpose it for running Kedro. Once you package your Kedro project into Docker image, you can run it basically anywhere you want.

yeah, I need to do some experiments; I want to get a hello world example running e2e, but that's why I'm trying to figure out which orchestrator to start with.