At the end of my data science pipeline I need to save multip

Question

At the end of my data science pipeline I need to save multiple plots The number of plots depends on hyperparameters of the model and there could be around 5 30 plots How would I do this with Kedro I took a look at <https kedro readthedocs io en stable kedro extras datasets matplotlib MatplotlibWriter html> However there is only one example using YAML api which I think I need to use to be able to see pictures when looking at my experiments through kedro viz and in that example only one plot is saved There are also examples where a list of plots is saved but there the python api is used and with the python api approach I can t figure out how to get the list of images be displayed in my experiments section in Kedro viz

Ian Whalen · Answer

For the first part you could implement your own dataset that saves a list of matplotlib figures to a directory using `MatplotlibWriter` in a loop As far as displaying in kedro viz I m not sure

Deepyaman Datta · Answer

Both saving a list of plots or a single plots can be done via the YAML API the difference is whether you return a list or not from the node that saves to the MatplotlibWriter dataset

Deepyaman Datta · Answer

If you return a list from the node how does it look in Kedro Viz

Jaakko · Answer

< Deepyaman Datta> This the example from to save one plot ```output plot type matplotlib MatplotlibWriter filepath data 08 reporting output plot png save args format png``` I can t figure out how to modify the yaml in the case where there is a list of plots that we want to save What should be the filepath argument for example

Deepyaman Datta · Answer

If you look at the implementation under <https kedro readthedocs io en stable modules kedro extras datasets matplotlib matplotlib writer html MatplotlibWriter> you ll see that it will write to `data 08 reporting output plot png 0 png` `data 08 reporting output plot png 1 png` etc If you want to control the names you can return a dictionary instead of a list Ideally seeing the filepaths above you would want to specify a directory like name rather than a filename as the `filepath` argument Relevant snippet from that link ``` if isinstance data list for index plot in enumerate data full key path = get filepath str save path f index png self protocol self save to fs full key path=full key path plot=plot ``` Of course it would also be better if this were more clearly documented and you didn t have to understand the implementation but just trying to help for now

Jaakko · Answer

Awesome thanks

Jaakko · Answer

< Deepyaman Datta> I am still getting back to this I have now the following dataset entry in my catalog ```output plot type matplotlib MatplotlibWriter filepath data 08 reporting output plots save args format png versioned true``` If I have my pipeline return a single plot everything works fine and I can see the plot in the experiments section in kedro viz However if I return a list of plots the plots are still created but I can t see them through kedro viz Also I see the following warning displayed in the terminal where kedro viz was started ``` output plot with version 2022 12 22T19 00 19 079Z could not be loaded Full exception DataSetError Failed while loading data from data set experiment tracking py 101 MatplotlibWriter filepath= my project path data 08 reporting output plots protocol=file save args= format png version=Version load= 2022 12 22T19 00 19 079Z save=None Errno 21 Is a directory my project path data 08 reporting output plots 2022 12 22T19 00 19 079Z output plots ``` Is this as designed Would be nice to see list of plots displayed by kedro viz as well

Olivia Lihn · Answer

< Steeve Ndjila> same situation we are having

Deepyaman Datta · Answer

To be honest I don t know much on the Viz side but I m guessing without looking at the code it just wasn t written with the use case of handling a directory of paths in mind < Tynan> or < Rashida Kanchwala> or somebody else may know better although I think most of the core team is off Let me see if I can take a look at the code for my own knowledge later

Tynan · Answer

< Jaakko> which version of Kedro are you using

Jaakko · Answer

< Tynan> kedro version 0 18 3

Jaakko · Answer

kedro viz 5 1 1

Tynan · Answer

thanks < Deepyaman Datta> is right Viz isn t written to handle this use case what we handle is one plot per metric per run not multiple plots