I’d like to log figures in kedro-mlflow through the mlflow log_figure method, i saw it was mentioned in a discussion from quite some time ago, however never implemented. I gather i can do something like:

dependancy_figure:
    type: kedro_mlflow.io.artifacts.MlflowArtifactDataset
    dataset:
        type: kedro_datasets.plotly.JSONDataset
        filepath: ....

However when testing whether this would give the same result as

mlflow.log_figure()

i got issues with kedro assuming the dataset is versioned while the s3 bucket is not. Anyway, I’d propose to make a dataset to do this. Would the authors be open to a PR? And if so, do you have opinions on the implementation/naming, should i subclass the MlflowArtifactDataset, or another class?

However when testing whether this would give the same result as

mlflow.log_figure()

i got issues with kedro assuming the dataset is versioned while the s3 bucket is not.

Can you show the error?

catalog.yml:

shap_dependency_figure:
    type: kedro_mlflow.io.artifacts.MlflowArtifactDataset
    dataset:
        type: kedro_datasets.plotly.JSONDataset
        filepath: <s3://aw-science/performance_optimisation/${oc.env:ENV}/${oc.env:CLIENT_NAME}/data/08_reporting/shap_dependency.json>

code:

import plotly.express as px

fig = px.bar(x=["a", "b", "c"], y=[1, 3, 2])

catalog.save("shap_dependency_figure",fig)

error:

DatasetError: Cannot save versioned dataset 'shap_dependency.json' to 
'aw-science/performance_optimisation/dev/Reed/data/08_reporting' because a file with the same name already exists 
in the directory. This is likely because versioning was enabled on a dataset already saved previously. Either 
remove 'shap_dependency.json' from the directory or manually convert it into a versioned dataset by placing it in a
versioned directory (e.g. with default versioning format 
'aw-science/performance_optimisation/dev/Reed/data/08_reporting/shap_dependency.json/YYYY-MM-DDThh.mm.ss.sssZ/shap_
dependency.json').

oh this error is typical when you've used a versioned dataset a minute ago, then you modify the config, then it complains

hmm, but none of those files are versioned, there are versioned files in the bucket though

but not that folder

hmm okay, I see it now: https://kedro-mlflow.readthedocs.io/en/stable/source/04_experimentation_tracking/03_version_datasets.html#how-to-version-data-in-a-kedro-project

ahh okay, i just wrote the plot to a local directory:

shap_dependency_figure:
    type: kedro_mlflow.io.artifacts.MlflowArtifactDataset
    dataset:
        type: kedro_datasets.plotly.JSONDataset
        filepath: data/08_reporting/shap_dependency.html

The thing is, regardless of the file handle, (html, json, etc) in mlflow you will see json and not a figure. Which makes sense because the underlying dataset is saving json

so using

filepath: s3

with

MlflowArtifactDataset

isn't supported

But what i want is the same outcome as mlflow.log_figure, so either that means adding a plotly dataset that saves figures to html, and then using the

<http://kedro_mlflow.io|kedro_mlflow.io>.artifacts.MlflowArtifactDataset

or adding a MlflowFigure dataset

Sorry I forgot to answer to this thread. I indeed saw little value to add a dataset around

log_figure

because wrapping with

MlflowArtifactDataset

is supposed to cover all use cases, but I am totally open to PR if you want to avoid dealing with local path manually.

well its not so much the local path as storing my plots as html