https://kedro.org/ logo
#questions
Title
# questions
m

Mo Bro

11/06/2023, 8:45 AM
#catalog #azureblobstorage Hello Kedro Community, I try to connect to a csv file in azure blobstorage with help of this example: https://docs.kedro.org/en/stable/data/data_catalog_yaml_examples.html#load-a-model-saved-as-a-pickle-from-azure-blob-storage I adjusted the catalog and created a credentials.,yml with account_name and account_key. When trying to load the csv to a dataframe in a jupyter notebook I get the following error:
Copy code
dict() got multiple values for keyword argument 'account_name'
Any body can help out on what I am missing? Thx in advance!
d

datajoely

11/06/2023, 9:33 AM
is it possible you’re providing
account_name
via env variables too?
n

Nok Lam Chan

11/06/2023, 11:01 AM
Copy code
dev_abs:
  account_name: accountname
  account_key: key
Try to update the example to
Copy code
dev_abs:
  key: accountname
  secret: key
I didn’t check the documentaiton but fsspec at some points changing the keywords (I experienced with s3 before), not sure is it the same case.
m

Mo Bro

11/07/2023, 1:06 PM
hm, I just followed the setup in the spaceflight tutorial and want to test a connection to azure blobstorage. I was not able to find the solution yet. using key and secret instead of account_name or account_key does not work in this case.
m

Mo Bro

11/07/2023, 1:20 PM
no, i get the following error:
image.png
image.png
d

datajoely

11/07/2023, 1:53 PM
could you run
pip list -V
and tell us the version of
Kedro
,
Pandas
,
fsspec
and
adlfs
m

Mo Bro

11/07/2023, 1:55 PM
adlfs 2023.10.0 kedro 0.18.14 kedro-datasets 1.7.1 pandas 2.1.1 fsspec 2023.10.0
d

datajoely

11/07/2023, 2:10 PM
Okay they look right - can I ask you to do something. Open a notebook in the same Kedro environment. I’m
from kedro.datasets.pandas import CSVDataSet
and try to configure
CSVDataSet
live in python, all we’re doing behind the scenes is passing arguments to this class so if you get the same error plugging in the credentials manually we can isolate the issue
m

Mo Bro

11/07/2023, 2:20 PM
the import already gives an error:
do I miss somehting?
d

datajoely

11/07/2023, 2:23 PM
sorry try this?
from kedro.extras.datasets.pandas import CSVDataSet
m

Mo Bro

11/07/2023, 2:34 PM
seems to work
d

datajoely

11/07/2023, 2:35 PM
can you do a
df.load()
m

Mo Bro

11/07/2023, 2:35 PM
okay, same error as before.
d

datajoely

11/07/2023, 2:37 PM
Okay can you try passing the credentials to the
fs_args
argument?
m

Mo Bro

11/07/2023, 2:41 PM
same dict error
d

datajoely

11/07/2023, 2:50 PM
Okay I’m going to escalate this
👍 1
really not sure what’s going on
n

Nok Lam Chan

11/07/2023, 2:53 PM
Can you shared what config did you pass?
Copy code
from kedro_datasets.pandas import CSVDataset
import pandas as pd

data = pd.DataFrame({"col1": [1, 2], "col2": [4, 5], "col3": [5, 6]})
cred = {"account_name":"123","account_key":"123"}
dataset = CSVDataset(filepath="<abfs://model/test.csv>", credentials=cred)

dataset.save(data)
I try this and I get authentication error not dict error
d

datajoely

11/07/2023, 2:57 PM
does it work if you do
.load()
n

Nok Lam Chan

11/07/2023, 3:01 PM
slightly awkward error but still autentication related
m

Mo Bro

11/08/2023, 9:03 AM
thanks for the escalation @datajoely
i

Iñigo Hidalgo

11/08/2023, 5:55 PM
oh i think i know what the problem is you only need to list container onwards
not the storage account url
so filepath="abfs://raw/blabla.csv"
👍 1
m

Mo Bro

11/09/2023, 8:57 PM
the filepath listing container onwards seem to have solved the dict error but now I get a weird authentication error too
i

Iñigo Hidalgo

11/10/2023, 2:00 PM
Can you share some more info? Traceback and (anonymized if necessary, but preserving the structure) how you're authenticating?