Hi everybody, What could be the issue when i try to run a kedro pipeline in AzureML here: I also tried PickleDatasets, but same result. Are there any known issues so far with kedro pipeline conversion / transferability to AzureML? Or am i missing sth else? Thanks in advance! Additional info here: i create the container like that:

docker build --progress=plain --build-arg BASE_IMAGE=python:3.10.16-slim -t ABCDE.azurecr.io/kedro:latest .

And submit the job like this:

kedro azureml run -p de -s FGHZUI --aml-env kedro_env

Hi Mattis, it seems like you need to use dataset types that persist data to a storage location accessible by all nodes in your AzureML pipeline. I suspect AzureML is not sharing the memory space between each node that is being run. Change

checkup_file_path

(and any other datasets passed between AzureML steps) from

kedro.io.MemoryDataset

or

kedro.io.PickleDataset

(without a filepath) to a persistent dataset type: Example: •

pandas.CSVDataset

•

pandas.ParquetDataset

•

pickle.PickleDataset

(but ensure you provide a

filepath

)