Hi everyone, sorry, quick question: What is the re...
# questions
b
Hi everyone, sorry, quick question: What is the recommended way to register numpy arrays in the data catalog? AI agents propose using a "numpy.NumpyDataset", but this seems to be a hallucination, as far as I can tell? Just store it as PickleDataset? Or write a custom NumpyDataset? Thanks!
👀 1
r
Hi @Bjarne Hiller, Thank you for using Kedro. Yes we do not have
numpy.NumpyDataset"
in kedro-datasets. Could you please let us know your use-case ? What are you trying to do with the numpy arrays ?
Just store it as PickleDataset
Yes this is possible
Write a custom NumpyDataset
Based on your needs, you can do this as well If persistency is not needed, you can also use MemoryDataset. Thank you
b
NumpyDataset.py
Hi Ravi and thanks for the reply. Basically, this is just a standard data science project, where I have a pipeline for preprocessing. At the end of this pipeline, I get numpy arrays x and y for the actual model training / cross validation, detached from the file format of the original input data. I thought this was a quite common approach, therefore I was surprised to not see something as basic as numpy included in kedro-datasets. While pickle works, .npy and .npz files would be the recommended way to store numpy arrays I guess. I implemented a very barebones custom dataset, but it does only support npy, not npz files (see snippet).
👍 1
r
Thank you for sharing the snippet. I am not 100% sure on why this was not supported until now. I will get back with any historic reason from the team. But in general, numpy arrays are generally computational intermediates or rarely used as a long-term storage artifacts. This might be the reason but I can confirm on this. Having said that, implementing a custom datasets is straight forward. If the team thinks this to be a good idea to support Numpy datasets, we would love to get community contributions.
❤️ 1
b
Thanks, that would be great!
👍 1