hey ! has anyone managed to use files on the datab...
# questionsg
Gauthier Pierard
09/01/2025, 3:47 PMhey ! has anyone managed to use files on the databricks filesystem as a data source? getting DatasetError: No partitions found in '/dbfs/FileStore/myproject/queries/' but the files are there
n
Nok Lam Chan
09/01/2025, 4:41 PMCan you give a bit more context? What dataset are you using, how are you using it, what version of kedro/kedro-datasets?
Nok Lam Chan
09/01/2025, 4:43 PM1. dbfs is not recommended by Databricks anymore, I think they switch to unity catalog in general.
2. If you are using versioned Dataset, it may not work very well with this. As Spark used it own authentication on Databricks with some magic filepath handling.
fsspec👍 1
g
Gauthier Pierard
09/01/2025, 4:51 PMhey Nok, sure, it's a partitioned text dataset. not versioned.
myns.mydataset:type: partitions.PartitionedDatasetpath: ${_mypath}dataset: text.TextDatasetGauthier Pierard
09/01/2025, 4:52 PMshould it normally work on /dbfs/ ?
n
Nok Lam Chan
09/01/2025, 5:08 PMinteresting, I expect it should. Could you first test with a single text.TextDataset?
Nok Lam Chan
09/01/2025, 5:09 PMIf this work then maybe it's an issue with
PartitionedDatasetg
Gauthier Pierard
09/01/2025, 9:27 PMworked after using native
dbutilsdbfs://dbfs/n
Nok Lam Chan
09/01/2025, 10:52 PMIt should work with both uri style or posix style.
https://docs.databricks.com/aws/en/files/
2 Views