Massinissa Saïdi
01/27/2023, 11:42 AMDataSetError:
botocore.session.session.create_client() got multiple values for keyword
argument 'aws_access_key_id'.
DataSet 'dataset' must only contain valid arguments for the
constructor of 'kedro.extras.datasets.pandas.csv_dataset.CSVDataSet'.
I run my code from a docker-compose
with only one container (for now), I write files in s3. I specified the credentials this way:
aws_credentials:
aws_access_key_id: XXXXXXX
aws_secret_access_key: XXXXXXX
and my dataframe in catalog.yml
this way:
dataset:
type: pandas.CSVDataSet
filepath: ${s3.path}/data/dataset.csv
credentials: aws_credentials
docker-compose.yml
version: '3.7'
services:
kedro:
build:
context: .
args:
PIP_USERNAME: ${PIP_USERNAME}
PIP_PASSWORD: ${PIP_PASSWORD}
PIP_REPO: ${PIP_REPO}
dockerfile: dockerfile.kedro
cache_from:
- ia-churn
image: ia-churn
command: kedro run --env prod --pipeline data-processing
volumes:
- .:/usr/src/app/
- ./data/01_raw/:/usr/src/app/data/01_raw
In conda
environement evrything works. Someone has an idea please ?
More informations: I used kedro v0.18.4 and python 3.10