https://kedro.org/ logo
#random
Title
# random
j

Juan Luis

02/02/2024, 6:10 PM
yay or nay? (created with https://mymlops.com/builder)
❤️ 5
i

Iñigo Hidalgo

02/05/2024, 9:18 AM
What about data versioning? 😛 I know you use deltatables, but I'm not sure if that actually fits the commonly-used definition of data versioning in the scope of MLOps, right?
j

Juan Luis

02/05/2024, 11:54 AM
good catch 🦅 my problem with data versioning is that I know what it means to version a CSV file, but I have no idea how folks are doing this in production. do you think Delta Tables are more for the data engineering side of the equation @Iñigo Hidalgo?
i

Iñigo Hidalgo

02/05/2024, 12:00 PM
in my (limited) experience and opinion, yeah, delta tables and the equivalent SCD SQL stuff is more on the data engineering E(LT/TL) side than the ML side. While loading data "as of" a certain date is important, I think what is more relevant is the transformations applied to that source data which is a mixture of "as of", and code versioning. I don't have any real experience on this side of things beyond knowing it's something we don't really do and maybe would want to do at some point in the future 😆 We work a lot with time series data, a lot of external forecasts etc, so the first side, SCDs, are a lot more important to us right now
💡 1
👍🏼 1
j

Juan Luis

02/05/2024, 12:01 PM
this prompted me to ask on MLOps.Community 👀 https://mlops-community.slack.com/archives/C015J2Y9RLM/p1707134303101179
In our case, we are using a delta lake approach (+ some parquet files). CSV is prohibited
🙈
n

Nok Lam Chan

02/08/2024, 2:43 PM
I think what is more relevant is the transformations applied to that source data which is a mixture of "as of", and code versioning.
It only truly reproducible when you have both, it sounds like a good thing but there doesn't seem to be a big demand for it? People seem to happy with the basic versioning
j

Juan Luis

02/08/2024, 2:49 PM
or, nobody uses it... I don't think we have a way to know at the moment
2 Views