Hey everyone, does the Kedro versioned dataset fea...
# questionsc
Cory Maklin
09/07/2023, 4:51 PMHey everyone, does the Kedro versioned dataset feature work like Delta tables? In other words, if a job is reading a dataset, and another job writes to that same dataset, the first job will continue to read the old version of the dataset?
d
datajoely
09/07/2023, 5:15 PMNo it’s much much more primitive, it just maintains a folder of different run results
datajoely
09/07/2023, 5:16 PMIt’s a minimal solution for those with limited infra
datajoely
09/07/2023, 5:16 PMif you want Delta use it!
i
Iñigo Hidalgo
09/08/2023, 8:35 AMIt should be possible to write a spark-less DeltaTable kedro dataset, though I'm not familiar enough with Delta Tables to know if the non-spark/non-databricks implementation also guarantees ACID.
Iñigo Hidalgo
09/08/2023, 8:38 AMWould be interesting to see how the
deltatable.history()d
datajoely
09/08/2023, 9:10 AMwe have a Pandas implementation too https://docs.kedro.org/en/stable/kedro_datasets.pandas.DeltaTableDataSet.html
It uses the rust library underneath with no JVM
datajoely
09/08/2023, 9:11 AMI also think you can get to delta via Polars and Ibis
datajoely
09/08/2023, 9:11 AMwhich use the same thing down the rabbit hole
i
Iñigo Hidalgo
09/08/2023, 9:11 AMOooh nice!!
Yeah, I was able to interact with it using polars and duckdb here 🙂
https://github.com/inigohidalgo/delta-python/blob/main/main.py
Iñigo Hidalgo
09/08/2023, 9:12 AMnot a kedro implementation though
Iñigo Hidalgo
09/08/2023, 9:13 AMHow does the pandas deltatable work with kedro versioning?
d
datajoely
09/08/2023, 9:24 AMso I don’t think we’ve enabled Kedro versioning, we just use the delta implementation
datajoely
09/08/2023, 9:24 AMKedro Versioning was always a minimal solution
i
Iñigo Hidalgo
09/08/2023, 9:31 AMmakes sense. thanks!
Iñigo Hidalgo
09/08/2023, 9:35 AMi guess anyways they aren't really equivalent, since kedro versions are different iterations of the same dataset, but they could be completely different results, for example different model pickles from different training runs. whereas the delta history is made up of diffs, so it's more of a way to follow how a certain table evolved over time, with new rows being added as time passes
d
datajoely
09/08/2023, 9:52 AMYes, but I do wonder if we’re able to inject any metadata into those diffs now you mentioni t
i
Iñigo Hidalgo
09/08/2023, 9:56 AMwhat sort of metadata would you be including? some reference to kedro versions?
d
datajoely
09/08/2023, 9:57 AMYeah the our version ID is actually just our session ID
🎉 1
datajoely
09/08/2023, 9:57 AMso you could marry the two worlds
9 Views