Hello Team, We're facing a huge blocker to save pandas.ParquetDataset to /dbs/mnt/ points whereas same works on /dbfs/ on databricks ., we've a pandas dataset and team is blocked while saving the same to the /dbfs/mnt/<storage-container>. However it works with just on "/dbfs/" looks like a issue with pandas dataframe and /dbfs/mnt/<storage-container> Error from kedro: Failed while saving data to data set ParquetDataset(filepath=/dbfs/mnt/container, load_args={}, protocol=file, save_args={}). [Errno 5] Input/output error: '/dbfs/mnt/container' Also open for any alternative solutions. Please treat this with high priority as our entire pipeline run depends on it

hello @Raghunath Nair, sorry you're having a rough experience. can you write other files to

/dbfs/mnt/container

? or does it only fail with Kedro?

@Juan Luis yes

spark.SparkDataset

works fine issue and spark we are able to save with

pandas.ParquetDataset

we cannot save gets Error from kedro: Failed while saving data to data set ParquetDataset(filepath=/dbfs/mnt/container, load_args={}, protocol=file, save_args={}). [Errno 5] Input/output error: '/dbfs/mnt/container'

I remember @Shubham Agrawal was facing a similar issue, so tagging him in case he has a solution already. I’ll investigate this in the meanwhile.

thank @Ankita Katiyar appreciated please let me know!

I still have the issue. What helped me was just saving to a different location on dbfs.. check if that works for you? Is the <container> a azure data storage?

@Shubham Agrawal yes it is an

azure storage account

As i mentioned in my issue. Saving to dbfs works but does not work on the dbfs/mnt/storage

interesting.. it was the same for me too

@Shubham Agrawal yes it is an

azure storage account

As i mentioned in my issue. Saving to dbfs works but does not work on the dbfs/mnt/storage. dbfs is not meant for storing data so, we want to push into a mounted storage account and we can't use it for our security reasons without mounting

however, there were a few containers mounted on Databricks by my company IT team.. and I was able to save the pandas dataset there.. My hypothesis is it to something to do with mounting maybe? worth a try

@Shubham Agrawal

spark.SparkDataset

works fine

pandas.ParquetDataset

none of them works

Yep, I had the same issue. I tried different version of pandas, python and Databricks cluster.. but this is what eventually worked for me

@Shubham Agrawal you mean switching from

dbfs/mnt

to

dbfs

right?

@Raghunath Nair another thing you can try is launching a script that does

import pandas as pd

df = pd.DataFrame(...)  # fill it with some data

df.to_parquet("/dbfs/mnt/container/test.pq")

if it fails, it will hopefully tell you more information (and also rule out that it's a Kedro problem). if it works, we'd need to keep investigating.

Worth to note that Spark has native integration on Databricks, so the authentication mechanism and how it interact with the filesystem/dbfs could be different from others

import pandas as pd df = pd.DataFrame(...) # fill it with some data df.to_parquet("/dbfs/mnt/container/test.pq") fails with

[Errno 5] Input/output error: '/dbfs/mnt/teamdata/test.pq

@Nok Lam Chan @Juan Luis

thanks @Raghunath Nair. that's an indication that this is a pandas problem, not a Kedro problem. there's nothing we can do about it - my recommendation would be to find a workaround. given that you do want to use that location, are you able to use Spark to write Parquet? or even Polars?

Before we can conclude this is a pandas problem - can you try with something that is not pandas? maybe just try to save a file with

open

Right now you are trying to say it's not working with pandas, I tend to think it's the opposite, which is it's only working with spark but nothing else. https://forums.linuxmint.com/viewtopic.php?t=396045 The error itself suggested it's a common problem in mount drive.

@Nok Lam Chan open also gave the same error

if

import pandas as pd
df = pd.DataFrame(...)  # fill it with some data
df.to_parquet("/dbfs/mnt/container/test.pq")

fails and

spark.SparkDataset

works, then again it's not a problem of

kedro_datasets.pandas.ParquetDataset

, but

pandas

itself. am I missing something?

@Nok Lam Chan mentioned its related to mounts then my question is why then it worked with spark. So, we can confirm its a pandas issue in that case?

Raghunath Nair [3:01 PM]

@Nok Lam Chan open also gave the same error

I think this is sufficient to say that it's the mount drive issue rather than a specific library?

@Nok Lam Chan are you referring to "Work with files in cloud object storage" section of pandas? in here?

I can't point to a specific section, but I think https://docs.databricks.com/en/files/index.html#do-i-need-to-provide-a-uri-scheme-to-access-data explains a lot. There are used to be a nice page explaining how Spark treating path differently but I cannot find it anymore unfortunately they may have moved it or deleted it.

so, since databricks has native spark integration mounting works fine but with pandas it doesn't possible? i would like to rasie this issue with databricks then

It'd be great if you can get helps from Databricks, I am also curious what's the right way to handle these situatations. Just reading from the diagram though, fundamentally

dbfs/

and

dbfs/mnt

is two different things, which may explains why you can save in

dbfs/

but not the

dbfs/mnt

yes i think its one of need for the unity catalog to function