From this talk (), Lim mentions that your deployed pipeline does not need to have the same granularity as your development pipeline. Is this something that is build in Kedro? Or how to achieve this?

something like this may or may not be in development…

haha

👀

@datajoely you beat me to the response.

Cool

Yes - you just have to be quite manual in terms of how you orchestrate stuff

like in an automated fashion?

I think there will need to be some human element defining a ‘compile step’ of sorts, but there may be some heuristics that we can rely on too

mmm.... ok. Seems rather vague at this moment. Probably better to wait to emerge within the docs/Kedro itself

Yes you have the option of using any of the 1:1 converter plugins, but we’re working on this space in earnest

just for completeness, there's a kedro-airflow plugin at the moment @Michel van den Berg

So, is Airflow the recommended way of going to production for a Kedro project?

Do you have any existing infrastructure? Kedro-docker could be good enough if you don’t need too much.

No existing infrastructure

If you're using k8s, then Argo (instead of Kubeflow pipelines) might be a great choice too. We have some fans of this. And please check out GetInData's range of plugins: https://github.com/getindata (CC: @marrrcin)

I know a different team in the company uses Argo

It is likely a bigger decision than just Kedro. For a company context you will need orchestrator more than just data pipeline, I.e. you may need to connect a Kedro pipeline to other services etc. So I would say if you have Argo already it may be good to stick with one Orchestrator.

+100 to what @Nok Lam Chan said. If anything, a selling point of Kedro is that it's (relatively) easy to deploy using your orchestrator of choice, and that Kedro doesn't force you to choose any specific orchestration tool/that decision is independent.