Hi there. I have a data set which is a custom bina...
# questions
c
Hi there. I have a data set which is a custom binary file. I mostly want to use kedro for this file because it can handle versions on S3 out of the box. Basically, I want to load two versions of those files in a node, do some super quick comparison, and output a data frame and whether the comparison was successful. If it was, then I want the next node to copy the files I read into another location on S3. Is there a "blob" or "raw file" kind of data set in kedro? Thanks
👀 1
m
Looking at the implementation of TextDataset, you should be able to override _fs_args_._open_args_load_ to set mode=rb and wb for write accordingly. But I haven't tested it. https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-2.1.0/api/kedro_datasets.text.TextDataset.html#kedro_datasets.text.TextDataset
👀 1