To keep the other thread focused : Is there a way ...
# questionsa
Alexandre Ouellet
02/07/2025, 2:58 PMTo keep the other thread focused : Is there a way to manage a dataset of about 1 million files in AzureML? The files are about 4k each of binary data, and are entirely independent from each other.
h
Hall
02/07/2025, 2:58 PMSomeone will reply to you shortly. In the meantime, this might help:
a
Alexandre Ouellet
02/07/2025, 3:00 PMWe are already using a partitionned dataset, but my main complaint is related to azureml more specifically (and maybe python) : It takes a long, long while to provision a compute that can handle that, and a partitionned dataset needs to list the files to map it to a callable that loads the data in memory
Alexandre Ouellet
02/07/2025, 3:01 PMmy main issue is that it takes about 15 minutes from "pipeline is started" to it actually getting inside my node's function (and that is true regardless of the size of the image, as I have to say our image is quite big (some specialized tools + nvidia drivers are quite huge))
Alexandre Ouellet
02/07/2025, 3:01 PMThe dataset does still fit in memory (for now, I suspect it won't at one point)
n
Nok Lam Chan
02/08/2025, 8:12 PMIs it on a blob storage? I suspect it takes a lot of time just to iterate the directory and open a million files...
p
Paul Weiss
02/09/2025, 3:00 PMIIRC kedro uses fsspec and that is much slower at enumerating large numbers of files than e.g. pyarrowfs-adlgen2 that uses a newer ADLS API. Maybe you want to look into that.
a
Alexandre Ouellet
02/10/2025, 3:18 PMYes, it is on a blob storage, and yes, just enumerating large files is an issue (even locally on an nvme)
Alexandre Ouellet
02/10/2025, 3:22 PMThere is a way to mount blob storage into AML's compute, but it seems to not be an option with Kedro-AzureML (and preliminary tests seems to suggests that they are as slow as fsspec). Downloading (as in, modify the command sent to AML to ask it to download the dataset for us) seems faster (I'm guessing the download is parallelized vs listing everything, then downloading everything piece by piece), but again, not supported by kedro-azureml (though it would seem doable, though not exactly trivial)
3 Views