Hi Everyone, does anyone know whether Kedro supports Prefect 2.0? The deployment configuration with Prefect in the official documentation seems to refer to Prefect 1.0.

In theory, yes. In practice however I would not recommend using kedro if you are already using Prefect (and vice versa).

@Matthias Roels why wouldn’t you recommend using them together? Could you elaborate?

In my opinion, apart from the orchestration component of Prefect, they are largely overlapping in functionality. With Prefect’s decorators for tasks and flows, it is easy enough to create pipelines definitions. The only additional advantage of kedro is its data catalog and datasets. Other than that, Prefect + hydra can cover most of your needs.

Thanks @Matthias Roels . It’s really a puzzle to us and I’d love to hear your thoughts on this

If you are already using kedro and are looking for an orchestrator, I would either opt for a managed Airflow (astronomer, AWS managed Airflow, GCP Cloud Composer) for the simple reason that this is still the number one most used orchestrator (and hence lot of support/troubleshooting available) OR, if you are heavily invested ik k8s, Argo Workflows as this is a k8s native workflow engine and slightly more flexible/modern. In my project, we went for kedro-Argo Workflows because we are indeed heavily invested in k8s!

Thanks @Matthias Roels! We are heavily invested. How do you run a lighter weight Kedro deployment? e.g. for local development work we use docker compose and for simple GitHub workflow (CI jobs) we don’t necessarily want to provision a cluster

Well that’s something else I forgot to mention. We also use Argo Events and combined with Argo Workflows, we have an event driven workflow orchestration framework. So general purpose and powerful that we use this to run our CI workloads too (with Kaniko for building container images) instead of our enterprise Jenkins framework. Hence, we can run a kedro pipeline (or a set of kedro pipelines) during CI with a simple argo submit to let it run in our dev cluster.

Thanks Matthias, that’s a lot of new terms for me but it looks very interesting and I’ll read about it. One thing to note is that we are deploying on-prem air-gapped on managed k8s envs (e.g. AKS / Rancher), so no access whatsoever to SaaS/cloud-based managed services

No worries, we are not using any SaaS service either. We have made our own implementations (too much of a hassle from a compliance pov) so it all runs in our own environment! It is actually not that hard. If creating an operator feels to difficult, you could start with a simple Helm chart as well

I’ll need 2 DevOps to maintain the setup you just described haha

Then who’s managing your clusters, docker images, …?

and someone who is very hands on the technology to do the initial bring up of the stack (or myself)

In our case, it’s a project with roughly 35 developers (data scientists and data engineers). We have a separate infra team to manage our cloud setup (terraform) and we have 1.6FTE on the project (incl. me) to manage the rest (cluster apps with Helm, …)

Yeah I guess there’s a lot of grunt work to be done to have a good infra and a working setup

It’s actually just 1 project running at massive scale (1 solution rolled out in for country x product combination across EMEA) and we are still scaling up in the region 😅. So lot of work to be done to onboard new country and product combinations (data integration, model building with input from business stakeholders,…). But indeed, we could only do the platform side with 1.6FTE thanks to solid processes that we have in place. In the early days, we where a 3FTE team to give room for extensive feature development.