Hi Team, has anyone tried using versioning on `Spa...
# questions
Hi Team, has anyone tried using versioning on
? im trying to version a csv. Funny thing is that it fails by a
but it still saves the new version. Can someone suggest any ideas ?
versioning and spark dont co-exist
use delta if you want that
how can i create versions of a file on HDFS ? I dont want the file to be appended / upserted etc but every run, a new file should be created with the latest timestamp etc
is that possible ?
I imagined that would be exactly the same as local filesystem. You can always instruct your own versioning schema by doing some templating value with directory. i.e.
Moreover, I agree Delta offer native versioning (more efficient) and could be a better choice. Note that CLI argument such as
assume the Kedro versioning scheme so it won’t work with any native versioning. But again, you can use a templated value to get around with it.