Hi team we are trying to use the experiment tracking feature

Question

Hi team we are trying to use the experiment tracking feature of kedro within databricks but are running into the following error ```INFO Loading data from modeling model best params JSONDataSet 8 id=949765 file databricks python lib python3 8 site packages kedro io data catalog py data catalog py 8 8 id=434875 file databricks python lib python3 8 site packages kedro io data catalog py 343 343 8 DataSetError Loading not supported for JSONDataSet ``` where we have the following catalog entry ```modeling model best params type tracking JSONDataSet filepath $ folders tracking model best params json layer reporting``` The same code runs completely fine locally but is failing within data braicks Could you please help us understand why

Zihao Xu · Answer

Using `kedro version 0 18 3`

Zihao Xu · Answer

Also in both our local test and databricks test we use the same `requirements txt` to create virtual envs

datajoely · Answer

~Is your $folders tracking local dbfs s3 ~

datajoely · Answer

Is your $folders tracking local dbfs s3

Zihao Xu · Answer

`$folders tracking` is a `dbfs` store on databricks locally it s within `data 09 tracking`

datajoely · Answer

and are you using databricks repos

Zihao Xu · Answer

Yes I am using databricks repos

Zihao Xu · Answer

but the data IO is handled by dbfs Everything was fine i e saving loading of data until we brought in the `tracking JSONDataset`

datajoely · Answer

Oh sorry I get what you re trying to do

datajoely · Answer

```DataSetError Loading not supported for JSONDataSet ```

datajoely · Answer

you don t load it

datajoely · Answer

it s for visualising in the Viz experimentation tracking view

datajoely · Answer

not using as data

Zihao Xu · Answer

yes this dataset is used in viz for tracking purposes But it is also an input to a node within our modeling pipeline hence it needs to be loaded by that node

datajoely · Answer

it s not designed for that

Zihao Xu · Answer

The interesting thing is the same setup works totally fine locally

datajoely · Answer

you would have to duplicate the return and use the duplicate one

datajoely · Answer

```DataSetError Loading not supported for JSONDataSet ``` The dataset isn t going to behave differently on different deploys

Zihao Xu · Answer

by duplicate the return do you mean have a separate node just for tracking purposes

datajoely · Answer

no just return the attribue twice in the same node

datajoely · Answer

and label one of the outputs for experiment tracking

datajoely · Answer

and the other as data references as an input to the next node

Zihao Xu · Answer

I see So one used as inputs for the next node likely will be a memory dataset and another just for tracking

Zihao Xu · Answer

got it will give this a try Thanks < datajoely>

Nok Lam Chan · Answer

The dataset doesn t support `load` so it shouldn t work for local