Dear all, we are trying to set up collaborative experiment tracking in Azure Blob Storage according to Experiment tracking in Kedro-Viz. We have abfs working for our datasets in our data catalogue. However, when we use the same abfs path for SESSION_STORE_ARGS, we get the error attached. We are wondering if anybody has a working setup for collaborative experiment tracking in Azure and could kindly give some insights? Thank you so much! PS: We have set the credentials both as credentials.yml and as environment variables as outlined here without success: Publish and share on Azure — kedro-viz 10.0.0 documentation

@Ravi Kumar Pilla cc

PPS: Here is a screenshot of our credentials.yml

Hi Johann, Thank you for using experiment tracking on viz. The doc link you shared is for hosting static Kedro Viz instance on Azure. Could you see if you have the setup mentioned here - https://docs.kedro.org/projects/kedro-viz/en/stable/experiment_tracking.html#collaborative-experiment-tracking ? Thank you

Hi Ravi, the URL you posted is the same I also posted originally. This is exactly our setup, just that we try to use Azure instead of AWS, everything else identical.

okay, can you please let me know the kedro viz version you are using ?

kedro-viz 9.2.0

Thanks for the information Johann. We haven't tried Experiment tracking on Azure blob storage before. @Rashida Kanchwala do you remember trying on Azure before ?

we would also be willing to implement a bugfix and make a PR for this if anybody can give us a sketch of how to implement this.

I'd suggest try to initialise connection with SQLiteSessionStore directly.

Collaborative experiment tracking uses fsspec, so ideally it should work with Azure. during dev time; we only tested it with AWS.

yeah, but for azure kedro doesn't understand from where or how to take the credentials.

class SQLiteStore(BaseSessionStore):
    """Stores the session data on the sqlite db."""

    def __init__(self, *args, remote_path: Optional[str] = None, **kwargs):

It should support

abfs

as Rashida suggested. It would be easier to test with the class directly without involving all the Kedro stuff

One limitation with collaborative experiment tracking, credentials must be stored as environment variables, set through an

export

command.

I think they set the below credentials -

export AZURE_STORAGE_TENANT_ID="your-app-tenant-id"
export AZURE_STORAGE_CLIENT_ID="your-app-client-id"
export AZURE_STORAGE_CLIENT_SECRET="your-app-client-secret-value"

We might need to test this and include in the docs on what the credentials should be.

What Ravi says is correct. We do set the environment variables correctly.

I want to understand if the problem is at our end or fsspec as it might be the latter. Do u also save the datasets in your DataCatalog to Azure blob storage - does that work fine ?

@Rashida Kanchwala yes, saving all our datasets to azure blob storage works. here is one example of a dataset defined in our data catalog, which we find also stored on azure blob as csv file:

example:
  type: pandas.CSVDataset
  filepath: <abfs://lab-bdschad/data/stuff/example.csv>
  versioned: true
  credentials: azure_blob_storage

this is what it said online 1. Obtain the connection string for the Azure Blob Storage account. This can be found in the Azure Portal under Access keys in the storage account settings. 2. Set the AZURE_STORAGE_CONNECTION_STRING environment variable in the local environment to the connection string value.

sounds great! we'll try it out right now

Thank you, that would be very helpful. We don't need to modify the "Publish and Share" section, as it works well and is focused purely on the flowchart, not on experiment tracking. However, it would be great to add a section for the Azure credentials required in the first document 🙂

@Rashida Kanchwala i'll do that! however, i would like to note that the publish and share on azure section does not work for us as it is, but it works with AZURE_STORAGE_CONNECTION_STRING. i would recommend also updating publish and share on azure: https://docs.kedro.org/projects/kedro-viz/en/stable/publish_and_share_kedro_viz_on_azure.html#set-credentials

oh interesting in that case pls add that as well

Happy to hear this resolved the issue. As a note, in the publish and share docs, we focussed on setting credentials which will be picked by Azure ServicePrincipal. I am curious to know if your setup is any different from the docs. Thank you

i'll post you updates here as soon as we created a PR.