Hello First thanks a lot for creating such a community I m t Kedro #questions

Hello, First, thanks a lot for creating such a co...

Alexandra Lorenzo

01/31/2023, 4:48 PM

Hello, First, thanks a lot for creating such a community. I'm trying to connect my PartitionedDataSet to my S3 Bucket, I have the following error:

"create_client() got multiple values for keyword argument 'aws_access_key_id'."

credentials.yml

Copy code

dev_s3:
 client_kwargs:
    aws_access_key_id: AWS_ACCESS_KEY_ID
    aws_secret_access_key: AWS_SECRET_ACCESS_KEY

catalog.yml

Copy code

raw_images:
  type: PartitionedDataSet
  dataset:
    type: flair_one.extras.datasets.satellite_image.SatelliteImageDataSet
  credentials: dev_s3
  path: <s3://ignchallenge/train> 
  filename_suffix: .tif
  layer: raw

kedro = 0.17.7 s3fs = 0.4.2 Anyone as an idea ? Thanks in advance

datajoely

01/31/2023, 4:49 PM

https://kedro.readthedocs.io/en/stable/kedro.io.PartitionedDataSet.html

datajoely

01/31/2023, 4:49 PM

if you look at the example in the docs

datajoely

01/31/2023, 4:49 PM

the credentials argument needs to be under the

dataset

indent

👍 1

datajoely

01/31/2023, 4:50 PM

as it’s an argument to the wrapped dataset not the partition wrapper

Alexandra Lorenzo

01/31/2023, 4:52 PM

Thanks @datajoely , still the same error

datajoely

01/31/2023, 4:53 PM

so this one is hard to debug without seeing the implementation from your custom dataset

datajoely

01/31/2023, 4:53 PM

as it may be how that constructor is configured

👍 1

datajoely

01/31/2023, 4:54 PM

all we’re doing behind the scenes is importing that class and passing the arguments to it

datajoely

01/31/2023, 4:55 PM

I find sometimes importing it in a notebook and using the Python API is a good way to debug

👍 1

Alexandra Lorenzo

01/31/2023, 4:55 PM

Without context indeed it's not simple. The code below is working well (without S3), once I tried to read from S3 I have some errors

Copy code

raw_images:
  type: PartitionedDataSet
  dataset:
    type: flair_one.extras.datasets.satellite_image.SatelliteImageDataSet
  path: /home/ubuntu/train
  filename_suffix: .tif
  layer: raw

datajoely

01/31/2023, 5:03 PM

could you put a breakpoint in your custom dataset and work out what’s not looking correct?

Alexandra Lorenzo

01/31/2023, 5:57 PM

Found a solution, if I don't specify the credentials parameter everything looks fine, I can access to the S3 buckets. This error seems corrected in newer version of Kedro. Thanks @datajoely for your time

datajoely

01/31/2023, 5:58 PM

awesome!

4 Views

Open in Slack

Previous Next