Debanjan Banerjee
11/16/2023, 12:31 PMSparkDataSet
? im trying to version a csv. Funny thing is that it fails by a VersionNotFoundError
but it still saves the new version. Can someone suggest any ideas ?datajoely
11/16/2023, 12:33 PMJuan Luis
11/16/2023, 1:10 PMDebanjan Banerjee
11/16/2023, 1:42 PMNok Lam Chan
11/17/2023, 8:55 AMsome_path/{version}/dataset.parquet
Moreover, I agree Delta offer native versioning (more efficient) and could be a better choice. Note that CLI argument such as --load-version
assume the Kedro versioning scheme so it won’t work with any native versioning. But again, you can use a templated value to get around with it.