https://kedro.org/ logo
#questions
Title
# questions
m

Michel van den Berg

07/11/2023, 12:58 PM
From this talk (

https://www.youtube.com/watch?v=-FedSW2SN7A

), Lim mentions that your deployed pipeline does not need to have the same granularity as your development pipeline. Is this something that is build in Kedro? Or how to achieve this?
👍 1
d

datajoely

07/11/2023, 12:58 PM
something like this may or may not be in development…
m

Michel van den Berg

07/11/2023, 12:59 PM
haha
those are the full spectrum of options yes 🙂
d

datajoely

07/11/2023, 12:59 PM
👀
n

Nok Lam Chan

07/11/2023, 12:59 PM
@datajoely you beat me to the response.
m

Michel van den Berg

07/11/2023, 1:00 PM
Cool
just read the link
so, it currently is already possible?
Looking at the airflow runner, it does it 1:1 right?
d

datajoely

07/11/2023, 1:03 PM
Yes - you just have to be quite manual in terms of how you orchestrate stuff
M:1 and 1:1 don’t work in distributed execution world for different reasons
we’re trying to think through how best to provide the correct M:N granularity, isolate dependencies etc
m

Michel van den Berg

07/11/2023, 1:04 PM
like in an automated fashion?
instead of manual
d

datajoely

07/11/2023, 1:05 PM
I think there will need to be some human element defining a ‘compile step’ of sorts, but there may be some heuristics that we can rely on too
m

Michel van den Berg

07/11/2023, 1:07 PM
mmm.... ok. Seems rather vague at this moment. Probably better to wait to emerge within the docs/Kedro itself
d

datajoely

07/11/2023, 1:10 PM
Yes you have the option of using any of the 1:1 converter plugins, but we’re working on this space in earnest
j

Juan Luis

07/11/2023, 1:36 PM
just for completeness, there's a kedro-airflow plugin at the moment @Michel van den Berg
what @Nok Lam Chan and @datajoely meant is that we're working on improving this process 🙂
m

Michel van den Berg

07/11/2023, 1:38 PM
So, is Airflow the recommended way of going to production for a Kedro project?
n

Nok Lam Chan

07/11/2023, 1:44 PM
Do you have any existing infrastructure? Kedro-docker could be good enough if you don’t need too much.
m

Michel van den Berg

07/11/2023, 1:44 PM
No existing infrastructure
We are in the process of creating a k8s cluster
y

Yetunde

07/11/2023, 1:46 PM
If you're using k8s, then Argo (instead of Kubeflow pipelines) might be a great choice too. We have some fans of this. And please check out GetInData's range of plugins: https://github.com/getindata (CC: @marrrcin)
👍 1
👍🏼 2
m

Michel van den Berg

07/11/2023, 1:49 PM
I know a different team in the company uses Argo
So that might be a viable option
n

Nok Lam Chan

07/11/2023, 1:55 PM
It is likely a bigger decision than just Kedro. For a company context you will need orchestrator more than just data pipeline, I.e. you may need to connect a Kedro pipeline to other services etc. So I would say if you have Argo already it may be good to stick with one Orchestrator.
💯 2
d

Deepyaman Datta

07/11/2023, 5:29 PM
+100 to what @Nok Lam Chan said. If anything, a selling point of Kedro is that it's (relatively) easy to deploy using your orchestrator of choice, and that Kedro doesn't force you to choose any specific orchestration tool/that decision is independent.
2 Views