Thank you in advance !

Couple questions, • Is

pokemon.extras.datasets.image_dataset.ImageDataSet

your custom class? • Do you have the error message?:) My guesses so far is that in your node you say

dataset.load()

- but the input to your node will already be the loaded dataset. Whatever the output of

pokemon.extras.datasets.image_dataset.ImageDataSet._load()

is - a numpy array maybe? Never mind, PartitionedDataSet I see 🤓 Then my 2nd guess is that in your pipeline you say

outputs="pokemon_images"

, but your node returns 2 values. So maybe you should be catching both of them?

INFO Loading data from 'pokemon' (PartitionedDataset)... data_catalog.py:475 INFO Running node: load_pokemon_images([pokemon]) -> [pokemon_images] node.py:331 ERROR Node 'load_pokemon_images([pokemon]) -> [pokemon_images]' failed with error: node.py:356 'dict' object has no attribute 'load' WARNING No nodes ran. Repeat the previous command to attempt a new run. runner.py:213 ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /home/rosana.eljurdi/.local/bin/kedro:8 in <module> │ │ │ │ 5 from kedro.framework.cli import main │ │ 6 if name == '__main__': │ │ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │ │ ❱ 8 │ sys.exit(main()) │ │ 9 │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/framework/cli/cli.py:211 in main │ │ │ │ 208 │ """ │ │ 209 │ _init_plugins() │ │ 210 │ cli_collection = KedroCLI(project_path=Path.cwd()) │ │ ❱ 211 │ cli_collection() │ │ 212 │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/click/core.py:1157 in call │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/framework/cli/cli.py:139 in main │ │ │ │ 136 │ │ ) │ │ 137 │ │ │ │ 138 │ │ try: │ │ ❱ 139 │ │ │ super().main( │ │ 140 │ │ │ │ args=args, │ │ 141 │ │ │ │ prog_name=prog_name, │ │ 142 │ │ │ │ complete_var=complete_var, │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/click/core.py:1078 in main │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/click/core.py:1688 in invoke │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/click/core.py:1434 in invoke │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/click/core.py:783 in invoke │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/framework/cli/project.py:453 in │ │ run │ │ │ │ 450 │ with KedroSession.create( │ │ 451 │ │ env=env, conf_source=conf_source, extra_params=params │ │ 452 │ ) as session: │ │ ❱ 453 │ │ session.run( │ │ 454 │ │ │ tags=tag, │ │ 455 │ │ │ runner=runner(is_async=is_async), │ │ 456 │ │ │ node_names=node_names, │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/framework/session/session.py:435 │ │ in run │ │ │ │ 432 │ │ ) │ │ 433 │ │ │ │ 434 │ │ try: │ │ ❱ 435 │ │ │ run_result = runner.run( │ │ 436 │ │ │ │ filtered_pipeline, catalog, hook_manager, session_id │ │ 437 │ │ │ ) │ │ 438 │ │ │ self._run_called = True │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/runner/runner.py:103 in run │ │ │ │ 100 │ │ │ self._logger.info( │ │ 101 │ │ │ │ "Asynchronous mode is enabled for loading and saving data" │ │ 102 │ │ │ ) │ │ ❱ 103 │ │ self._run(pipeline, catalog, hook_manager, session_id) │ │ 104 │ │ │ │ 105 │ │ self._logger.info("Pipeline execution completed successfully.") │ │ 106 │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/runner/sequential_runner.py:70 in │ │ _run │ │ │ │ 67 │ │ │ │ 68 │ │ for exec_index, node in enumerate(nodes): │ │ 69 │ │ │ try: │ │ ❱ 70 │ │ │ │ run_node(node, catalog, hook_manager, self._is_async, session_id) │ │ 71 │ │ │ │ done_nodes.add(node) │ │ 72 │ │ │ except Exception: │ │ 73 │ │ │ │ self._suggest_resume_scenario(pipeline, done_nodes, catalog) │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/runner/runner.py:331 in run_node │ │ │ │ 328 │ if is_async: │ │ 329 │ │ node = _run_node_async(node, catalog, hook_manager, session_id) │ │ 330 │ else: │ │ ❱ 331 │ │ node = _run_node_sequential(node, catalog, hook_manager, session_id) │ │ 332 │ │ │ 333 │ for name in node.confirms: │ │ 334 │ │ catalog.confirm(name) │ │ │ │ /home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/runner/runner.py:426 in │ │ _run_node_sequential │ │ │ │ 423 │ ) │ │ 424 │ inputs.update(additional_inputs) │ │ 425 │ │ │ ❱ 426 │ outputs = _call_node_run( │ │ 427 │ │ node, catalog, inputs, is_async, hook_manager, session_id=session_id │ │ 428 │ ) │ │ 429 │ │ │

Yeah so my first guess was correct I think. The

PartitionedDataSet

will return a dictionary of datasets. So change your node function: • remove

dataset.load()

• change input variable name to something like

dict_of_images

to avoid confusion • Don't return your input dataset, seems like your node just needs to count the images

def load_pokemon_images(dict_of_images):
    # Optionally, you can process or manipulate the loaded data here
    # For example, if you want to return the number of images loaded:
    num_images = len(dict_of_images)
    
    return num_images

@Lodewic van Twillert "Yeah so my first guess was correct I think. The

PartitionedDataSet

will return a dictionary of datasets.": well I actually need to load the images but I can not access the data

Okay so in that case, following the example here: https://docs.kedro.org/en/stable/kedro.io.PartitionedDataset.html How about this,

def load_pokemon_images(dataset):  # `dataset` is a dictionary, one entry for each image I expect
    num_images = len(dataset) 

    # Load all the partitions
    for partition_id, partition_load_func in loaded.items():
       # The actual function that loads the data
       image_data = partition_load_func()    
   
    return num_images

Not sure what exactly what you want to do to each image, but you could add it in the for-loop

Thank you for your reply: when I run the above code, I get the following error: image_data = partition_load_func() Traceback (most recent call last): File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/io/core.py", line 210, in load return self._load() File "/home/rosana.eljurdi/Kedro_Project/pokemon/src/pokemon/extras/datasets/image_dataset.py", line 34, in _load load_path = get_filepath_str(self._get_load_path(), self._protocol) File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/io/core.py", line 596, in _get_load_path if not self._version: AttributeError: 'ImageDataSet' object has no attribute '_version' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3508, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-3-5eebafde8793>", line 1, in <module> image_data = partition_load_func() File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/io/core.py", line 631, in load return super().load() File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/io/core.py", line 219, in load raise DatasetError(message) from exc kedro.io.core.DatasetError: Failed while loading data from data set ImageDataSet(filepath=/home/rosana.eljurdi/Kedro_Project/pokemon/data/01_raw/pokemon-images-and-types/images/images/abomasnow.png, protocol=file). 'ImageDataSet' object has no attribute '_version'

Hmm not sure exactly without seeing more of your custom

ImageDataSet

. What is it's parent class? Is it

AbstractVersionedDataset

? Are you sure you need the Versioned dataset? Does your custom class overwrite the

__init__()

without assigning

self.version

perhaps? 🤔 Because this line, comes from the

AbstractVersionedDataset

and by default would set the

self.version

attribute so I'm surprised it says it doesn't have that attribute on this line in your error:

File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/io/core.py", line 596, in _get_load_path
    if not self._version:

for the ImageDataSet,

from pathlib import PurePosixPath
from kedro.io.core import (
AbstractVersionedDataSet,
get_filepath_str,
get_protocol_and_path,
)
import fsspec
import numpy as np
# PIL is the package from Pillow
from PIL import Image
from typing import Dict, Any

class ImageDataSet(AbstractVersionedDataSet):
        def __init__(self, filepath: str):
                """Creates a new instance of ImageDataSet to load / save image data for given
                    filepath.
                    Args:
                    filepath: The location of the image file to load / save data.
                """
                # parse the path and protocol (e.g. file, http, s3, etc.)
                protocol, path = get_protocol_and_path(filepath)
                self._protocol = protocol
                self._filepath = PurePosixPath(path)
                self._fs = fsspec.filesystem(self._protocol)
                
                    
        def _load(self) -> np.ndarray:
                """Loads data from the image file.
                Returns:
                Data from the image file as a numpy array
                """
                # using get_filepath_str ensures that the protocol and path are appended
                #˓→correctly for different filesystems
                load_path = get_filepath_str(self._get_load_path(), self._protocol)
                with self._fs.open(load_path) as f:
                        image = Image.open(f).convert("RGBA")
                        return np.asarray(image)
                    

        def _save(self, data: np.ndarray) -> None:
                """Saves image data to the specified filepath.
                # using get_filepath_str ensures that the protocol and path are appended
                #˓→correctly for different filesystems
                """
                save_path = get_filepath_str(self._get_save_path(), self._protocol)
                with self._fs.open(save_path, "wb") as f:
                        image = Image.fromarray(data)
                        image.save(f)
        
        def _describe(self) -> Dict[str, Any]:
                """Returns a dict that describes the attributes of the dataset."""
                return dict(filepath=self._filepath, protocol=self._protocol)

Change the parent class to

AbstractDataSet

instead, and let me know how that changes things

@Lodewic van Twillert now I get another error : 'ImageDataSet' object has no attribute '_get_load_path' Traceback (most recent call last): File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/io/core.py", line 210, in load return self._load() File "/home/rosana.eljurdi/Kedro_Project/pokemon/src/pokemon/extras/datasets/image_dataset.py", line 34, in _load load_path = get_filepath_str(self._get_load_path(), self._protocol) AttributeError: 'ImageDataSet' object has no attribute '_get_load_path' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3508, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-1-19522f69f156>", line 1, in <module> partition_load_func() File "/home/rosana.eljurdi/.local/lib/python3.10/site-packages/kedro/io/core.py", line 219, in load raise DatasetError(message) from exc kedro.io.core.DatasetError: Failed while loading data from data set ImageDataSet(filepath=/home/rosana.eljurdi/Kedro_Project/pokemon/data/01_raw/pokemon-images-and-types/images/images/abomasnow.png, protocol=file). 'ImageDataSet' object has no attribute '_get_load_path'

Ok, two questions again then: are you sure you need to use a Custom DataSet class? If so, you can change you

_load()

function to

def _load(self) -> np.ndarray:
                with self._fs.open(self._filepath) as f:
                        image = Image.open(f).convert("RGBA")
                        return np.asarray(image)

Second question, have you considered using https://docs.kedro.org/en/stable/kedro_datasets.pillow.ImageDataSet.html ?

It has worked 😄😄. Thank you so much !

For future reference and other people , I see now where you got the dataset from , and even better is to use the example in the tutorial from the docs: Non-versioned: https://docs.kedro.org/en/stable/data/how_to_create_a_custom_dataset.html#the-complete-example • Inherits from

AbstractDataSet

• More complete

_load()

method then the last example I sent 👍 And the Versioned dataset variant: https://docs.kedro.org/en/stable/data/how_to_create_a_custom_dataset.html#how-to-implement-versioning-in-your-dataset • Inherits

AbstractVersionedDataSet

• Be sure to include

version

as an argument to

__init__

• Call

super.__init__(..., version=version)

to make sure it's instantiated I think at some point you mixed those two classes and that's where it got broken

Mm. Yes that is right! Thank you alot for the help.

thanks @Lodewic van Twillert for chiming in! 🙌🏼