Has anyone played around with kedro and Dagster?
# plugins-integrationsf
Florian d
05/18/2024, 5:51 PMHas anyone played around with kedro and Dagster?
n
Nok Lam Chan
05/18/2024, 5:58 PM@Camilo López maybe? We briefly talk about writing a deployment guide for Dagster before
j
Juan Luis
05/20/2024, 6:27 AMnot yet but would like to give it a shot ahead of my EuroPython workshop :)
Juan Luis
05/20/2024, 6:27 AMwhat do you like about Dagster compared with the alternatives @Florian d ?
f
Florian d
05/22/2024, 9:46 AMHi @Juan Luis well some features look nice but mostly it’s because it’s being used right now in a project I’m looking at
✔️ 1
Florian d
05/22/2024, 9:47 AMI am contemplating the idea of a plugin similar to kedro-azureml. So always keen to see if there are some pocs floating around
Florian d
05/30/2024, 7:35 PM@Nok Lam Chan @Juan Luis any further thoughts on this? I looked a bit at dagster and the dbt plugin for it. As the idea in dagster is around assets (inputs, outputs and operation forms an asset) what they seem to do in the dbt case is to "render" the dbt pipeline and convert any nodes to dagster assets that then get used. This happens at load time of assets (can be reloaded). So I'd imagine something similar could be achieved with kedro as well.
Curious how you were thinking about it.
n
Nok Lam Chan
05/30/2024, 10:04 PMA topic that I would really want to come back, unfortunately I don’t think I can spend time this week or next week 🥲
🥲 1
j
Juan Luis
05/31/2024, 6:55 AMI want to look at this soon-ish (before July??) but I can't make any promises, lots on my plate 😭
f
Florian d
05/31/2024, 7:01 AMhaha I know the feeling so of course no worries. Just wanted to know if you had already initial thoughts but it seems like it might be more a "really want/need to look at this" situation just now. Keep me updated and I'll try to prototype some stuff when I have time as well 🙂
💯 1
j
Juan Luis
05/31/2024, 8:33 AMI mean, every blog post I see out there uses Dagster, so to me it's clear that they're sort of winning the "modern alternative to Airflow" battle. I look into their docs often and they have very good architectural ideas.
one day I devoted ~10 minutes to try to make it work for the first time and it was a weird experience though, so I want to sit a few hours and come up with better ideas.
f
Florian d
05/31/2024, 8:38 AMsounds great, more than happy to also jump on a call to have a brief chat about it down the line!
🥳 1
n
Nok Lam Chan
05/31/2024, 9:00 AMHow much is actually open source for dagster? I signed up for dagster+ but never get to play with it
f
Florian d
05/31/2024, 9:01 AMI think there is a fair amount, + feature are quite neat though (e.g. branch deployments for pipeline changes)
n
Nok Lam Chan
05/31/2024, 10:14 AMI see! For now I don't think I can devote enough time to do a PoC or experiment with it. If you do come up with something, happy to learn more about it or discuss. We also want to write a deployment guide for Dagster
c
Camilo López
05/31/2024, 2:51 PMHey keep me posted (and count me in), i was on vacations last 3 weeks
j
Juan Luis
05/31/2024, 2:57 PM@Camilo López you mentioned you worked with Kedro + Dagster + k8s right? any extra learnings (or code) you can share?
13 Views