Zihao Xu
11/21/2022, 11:16 PMINFO Loading data from 'modeling.model_best_params_' (JSONDataSet)... ]8;id=949765;file:///databricks/python/lib/python3.8/site-packages/kedro/io/data_catalog.py\data_catalog.py]8;;\:]8;id=434875;file:///databricks/python/lib/python3.8/site-packages/kedro/io/data_catalog.py#343\343]8;;\
DataSetError: Loading not supported for 'JSONDataSet'
where we have the following catalog entry:
modeling.model_best_params_:
type: tracking.JSONDataSet
filepath: "${folders.tracking}/model_best_params.json"
layer: reporting
The same code runs completely fine locally, but is failing within data braicks.
Could you please help us understand why?kedro, version 0.18.3
requirements.txt
to create virtual envs.datajoely
11/22/2022, 10:01 AMZihao Xu
11/22/2022, 2:50 PM$folders.tracking
is a dbfs
store on databricks, locally it’s within data/09_tracking
datajoely
11/22/2022, 3:03 PMZihao Xu
11/22/2022, 4:53 PMtracking.JSONDataset
…datajoely
11/22/2022, 5:28 PMDataSetError: Loading not supported for 'JSONDataSet'
Zihao Xu
11/22/2022, 5:29 PMdatajoely
11/22/2022, 5:29 PMZihao Xu
11/22/2022, 5:29 PMdatajoely
11/22/2022, 5:29 PMDataSetError: Loading not supported for 'JSONDataSet'
The dataset isn’t going to behave differently on different deploysZihao Xu
11/22/2022, 5:30 PMdatajoely
11/22/2022, 5:30 PMZihao Xu
11/22/2022, 5:31 PMNok Lam Chan
11/22/2022, 9:40 PMload
, so it shouldn’t work for local