Thank you in advance

Question

Lodewic van Twillert · Answer

Couple questions Is `pokemon extras datasets image dataset ImageDataSet` your custom class Do you have the error message ~My guesses so far is that in your node you say `dataset load ` but the input to your node will already be the loaded dataset Whatever the output of `pokemon extras datasets image dataset ImageDataSet load ` is a numpy array maybe ~ Never mind PartitionedDataSet I see nerd face Then my 2nd guess is that in your pipeline you say `outputs= pokemon images ` but your node returns 2 values So maybe you should be catching both of them

Rosana EL-JURDI · Answer

INFO Loading data from pokemon PartitionedDataset data catalog py 475 INFO Running node load pokemon images pokemon gt pokemon images node py 331 ERROR Node load pokemon images pokemon gt pokemon images failed with error node py 356 dict object has no attribute load WARNING No nodes ran Repeat the previous command to attempt a new run runner py 213 Traceback most recent call last home rosana eljurdi local bin kedro 8 in lt module gt 5 from kedro framework cli import main 6 if name == main 7 sys argv 0 = re sub r script pyw| exe $ sys argv 0 8 sys exit main 9 home rosana eljurdi local lib python3 10 site packages kedro framework cli cli py 211 in main 208 209 init plugins 210 cli collection = KedroCLI project path=Path cwd 211 cli collection 212 home rosana eljurdi local lib python3 10 site packages click core py 1157 in call home rosana eljurdi local lib python3 10 site packages kedro framework cli cli py 139 in main 136 137 138 try 139 super main 140 args=args 141 prog name=prog name 142 complete var=complete var home rosana eljurdi local lib python3 10 site packages click core py 1078 in main home rosana eljurdi local lib python3 10 site packages click core py 1688 in invoke home rosana eljurdi local lib python3 10 site packages click core py 1434 in invoke home rosana eljurdi local lib python3 10 site packages click core py 783 in invoke home rosana eljurdi local lib python3 10 site packages kedro framework cli project py 453 in run 450 with KedroSession create 451 env=env conf source=conf source extra params=params 452 as session 453 session run 454 tags=tag 455 runner=runner is async=is async 456 node names=node names home rosana eljurdi local lib python3 10 site packages kedro framework session session py 435 in run 432 433 434 try 435 run result = runner run 436 filtered pipeline catalog hook manager session id 437 438 self run called = True home rosana eljurdi local lib python3 10 site packages kedro runner runner py 103 in run 100 self 101 Asynchronous mode is enabled for loading and saving data 102 103 self run pipeline catalog hook manager session id 104 105 self Pipeline execution completed successfully 106 home rosana eljurdi local lib python3 10 site packages kedro runner sequential runner py 70 in run 67 68 for exec index node in enumerate nodes 69 try 70 run node node catalog hook manager self is async session id 71 done nodes add node 72 except Exception 73 self suggest resume scenario pipeline done nodes catalog home rosana eljurdi local lib python3 10 site packages kedro runner runner py 331 in run node 328 if is async 329 node = run node async node catalog hook manager session id 330 else 331 node = run node sequential node catalog hook manager session id 332 333 for name in node confirms 334 catalog confirm name home rosana eljurdi local lib python3 10 site packages kedro runner runner py 426 in run node sequential 423 424 inputs update additional inputs 425 426 outputs = call node run 427 node catalog inputs is async hook manager session id=session id 428 429

Rosana EL-JURDI · Answer

home rosana eljurdi local lib python3 10 site packages kedro runner runner py 392 in call node run 389 is async=is async 390 session id=session id 391 392 raise exc 393 hook manager hook after node run 394 node=node 395 catalog=catalog home rosana eljurdi local lib python3 10 site packages kedro runner runner py 382 in call node run 379 gt dict str Any 380 381 try 382 outputs = node run inputs 383 except Exception as exc 384 hook manager hook on node error 385 error=exc home rosana eljurdi local lib python3 10 site packages kedro pipeline node py 357 in run 354 purposely catch all exceptions 355 except Exception as exc 356 self logger error Node s failed with error n s str self str exc 357 raise exc 358 359 def run with no inputs self inputs dict str Any 360 if inputs home rosana eljurdi local lib python3 10 site packages kedro pipeline node py 348 in run 345 elif isinstance self inputs str 346 outputs = self run with one input inputs self inputs 347 elif isinstance self inputs list 348 outputs = self run with list inputs self inputs 349 elif isinstance self inputs dict 350 outputs = self run with dict inputs self inputs 351 home rosana eljurdi local lib python3 10 site packages kedro pipeline node py 388 in run with list 385 f sorted inputs keys 386 387 Ensure the function gets the inputs in the correct order 388 return self func inputs item for item in node inputs 389 390 def run with dict self inputs dict str Any node inputs dict str str 391 Node inputs and provided run inputs should completely overlap home rosana eljurdi Kedro Project pokemon src pokemon pipelines loader nodes py 8 in load pokemon images 5 6 def load pokemon images dataset 7 Load the dataset using your custom ImageDataSet 8 images = dataset load 9 10 Optionally you can process or manipulate the loaded data here 11 For example if you want to return the number of images loaded AttributeError dict object has no attribute load

Rosana EL-JURDI · Answer

Yes this is my custum dataset Thank you for your help slightly smiling face

Lodewic van Twillert · Answer

Yeah so my first guess was correct I think The `PartitionedDataSet` will return a dictionary of datasets So change your node function remove `dataset load ` change input variable name to something like `dict of images` to avoid confusion Don t return your input dataset seems like your node just needs to count the images ```def load pokemon images dict of images Optionally you can process or manipulate the loaded data here For example if you want to return the number of images loaded num images = len dict of images return num images```

Rosana EL-JURDI · Answer

< Lodewic van Twillert> Yeah so my first guess was correct I think The `PartitionedDataSet` will return a dictionary of datasets well I actually need to load the images but I can not access the data

Rosana EL-JURDI · Answer

I have removed the second part of the function which returns len num images but I still get the same error when I try to run the load func

Rosana EL-JURDI · Answer

my objective here is to loader the dataset and access the images as numpy arrays

Lodewic van Twillert · Answer

Okay so in that case following the example here <https docs kedro org en stable kedro io PartitionedDataset html> How about this ```def load pokemon images dataset `dataset` is a dictionary one entry for each image I expect num images = len dataset Load all the partitions for partition id partition load func in loaded items The actual function that loads the data image data = partition load func return num images``` Not sure what exactly what you want to do to each image but you could add it in the for loop

Rosana EL-JURDI · Answer

Thank you for your reply when I run the above code I get the following error image data = partition load func Traceback most recent call last File home rosana eljurdi local lib python3 10 site packages kedro io core py line 210 in load return self load File home rosana eljurdi Kedro Project pokemon src pokemon extras datasets image dataset py line 34 in load load path = get filepath str self get load path self protocol File home rosana eljurdi local lib python3 10 site packages kedro io core py line 596 in get load path if not self version AttributeError ImageDataSet object has no attribute version The above exception was the direct cause of the following exception Traceback most recent call last File home rosana eljurdi local lib python3 10 site packages IPython core interactiveshell py line 3508 in run code exec code obj self user global ns self user ns File lt ipython input 3 5eebafde8793 gt line 1 in lt module gt image data = partition load func File home rosana eljurdi local lib python3 10 site packages kedro io core py line 631 in load return super load File home rosana eljurdi local lib python3 10 site packages kedro io core py line 219 in load raise DatasetError message from exc <http kedro io|kedro io> core DatasetError Failed while loading data from data set ImageDataSet filepath= home rosana eljurdi Kedro Project pokemon data 01 raw pokemon images and types images images abomasnow png protocol=file ImageDataSet object has no attribute version

Rosana EL-JURDI · Answer

I can put it in a for loop my aim is to just have access the the data in numpy arrays so that I can create a CNN clarifier It is for learning purpose My application is a bit more complicated as I need to create an sitk dataloafer that deals with sitk images

Lodewic van Twillert · Answer

Hmm not sure exactly without seeing more of your custom `ImageDataSet` What is it s parent class Is it `AbstractVersionedDataset` Are you sure you need the Versioned dataset Does your custom class overwrite the ` init ` without assigning `self version` perhaps thinking face Because this line comes from the `AbstractVersionedDataset` and by default would set the `self version` attribute so I m surprised it says it doesn t have that attribute on this line in your error ``` File home rosana eljurdi local lib python3 10 site packages kedro io core py line 596 in get load path if not self version ```

Rosana EL-JURDI · Answer

for the ImageDataSet ```from pathlib import PurePosixPath from kedro io core import AbstractVersionedDataSet get filepath str get protocol and path import fsspec import numpy as np PIL is the package from Pillow from PIL import Image from typing import Dict Any class ImageDataSet AbstractVersionedDataSet def init self filepath str Creates a new instance of ImageDataSet to load save image data for given filepath Args filepath The location of the image file to load save data parse the path and protocol e g file http s3 etc protocol path = get protocol and path filepath self protocol = protocol self filepath = PurePosixPath path self fs = fsspec filesystem self protocol def load self gt np ndarray Loads data from the image file Returns Data from the image file as a numpy array using get filepath str ensures that the protocol and path are appended correctly for different filesystems load path = get filepath str self get load path self protocol with self fs open load path as f image = Image open f convert RGBA return np asarray image def save self data np ndarray gt None Saves image data to the specified filepath using get filepath str ensures that the protocol and path are appended correctly for different filesystems save path = get filepath str self get save path self protocol with self fs open save path wb as f image = Image fromarray data image save f def describe self gt Dict str Any Returns a dict that describes the attributes of the dataset return dict filepath=self filepath protocol=self protocol ```

Lodewic van Twillert · Answer

Change the parent class to `AbstractDataSet` instead and let me know how that changes things

Rosana EL-JURDI · Answer

< Lodewic van Twillert> now I get another error ImageDataSet object has no attribute get load path Traceback most recent call last File home rosana eljurdi local lib python3 10 site packages kedro io core py line 210 in load return self load File home rosana eljurdi Kedro Project pokemon src pokemon extras datasets image dataset py line 34 in load load path = get filepath str self get load path self protocol AttributeError ImageDataSet object has no attribute get load path The above exception was the direct cause of the following exception Traceback most recent call last File home rosana eljurdi local lib python3 10 site packages IPython core interactiveshell py line 3508 in run code exec code obj self user global ns self user ns File lt ipython input 1 19522f69f156 gt line 1 in lt module gt partition load func File home rosana eljurdi local lib python3 10 site packages kedro io core py line 219 in load raise DatasetError message from exc core DatasetError Failed while loading data from data set ImageDataSet filepath= home rosana eljurdi Kedro Project pokemon data 01 raw pokemon images and types images images abomasnow png protocol=file ImageDataSet object has no attribute get load path

Lodewic van Twillert · Answer

Ok two questions again then are you sure you need to use a Custom DataSet class If so you can change you ` load ` function to ``` def load self gt np ndarray with self fs open self filepath as f image = Image open f convert RGBA return np asarray image ``` Second question have you considered using <https docs kedro org en stable kedro datasets pillow ImageDataSet html>

Rosana EL-JURDI · Answer

It has worked smile smile Thank you so much

Lodewic van Twillert · Answer

For future reference and other people I see now where you got the dataset from and even better is to use the example in the tutorial from the docs Non versioned Inherits from `AbstractDataSet` More complete ` load ` method then the last example I sent +1 And the Versioned dataset variant Inherits `AbstractVersionedDataSet` Be sure to include `version` as an argument to ` init ` Call `super init version=version ` to make sure it s instantiated I think at some point you mixed those two classes and that s where it got broken

Rosana EL-JURDI · Answer

Mm Yes that is right Thank you alot for the help

Juan Luis · Answer

thanks < Lodewic van Twillert> for chiming in raised hands skin tone 3