Is there a way to use ImageDataGenerator and flow_from_directory functions in kedro? I would like to save the dataset in memory and then use it later for model training, but I got the error message:

DatasetError: Failed while saving data to data set MemoryDataset().

Someone will reply to you shortly. In the meantime, this might help:

Hi Shu-Chun Wu, Which Kedro and Python version are you using?

I would need the full stack trace, but I suspect this is due to deepcopying a non pickleable object with keras. in your catalog, use:

my_dataset:
    type: MemoryDataset
    copy_mode: assign

For your output dataset

@Jitendra Gundaniya kedro=0.19.8 and python=3.11