Hello all! When running my Kedro pipeline, I get this error:

Unable to allocate 533. KiB for an array with shape (136561,) and data type float32

. Indeed, I'm creating a float32 numpy array of dimension 136,561. 533 KB is nothing since I have 16 GB of RAM... Also, I can create an float32 numpy array of dimension (50000, 1310) without an error in a Jupyter notebook, which is 262 MB. I'm wondering if this error can be related to Kedro.

Hey Francis! What kind of environment are you running your pipeline in? I'm seeing here that it might be a linux kernel quirk.

Hi Laura! My pipeline runs on Windows 10, in a Conda environment with Python 3.10.

From what I'm seeing this is probably not a Kedro specific issue. Windows has some quirkiness when it comes to virtual memory allocation, though your array is very small. Some things you could try are debugging the pipeline itself to see if that's not the source of the problem, testing it in other environments, and maybe checking if the program is actually able to use the available system memory or if the OS is holding it back for some reason.

Thank you for your answer Laura! I reran my code and weirdly enough, I don't get this error anymore even though I create bigger arrays!