https://kedro.org/ logo
#questions
Title
# questions
s

Sergei Benkovich

01/09/2023, 1:41 PM
hey i have an issue using credentials in my project. i get “ValueError: Failed to format pattern ‘${dev_s3}‘: no config value found, no default provided” credentials.yml: (in local folder), as i understand it should take the AWS_ACCESS_KEY_ID from aws cli. i tried also specifying it, and it didn’t help
Copy code
dev_s3:
     aws_access_key_id: AWS_ACCESS_KEY_ID
     aws_secret_access_key: AWS_SECRET_ACCESS_KEY
catalog.yml:
Copy code
observations:
  type: pandas.CSVDataSet
  filepath: "${s3.raw_observations_path}/commercial/observations/observations.csv"
  credentials: "${dev_s3}"
main.py
Copy code
runner = SequentialRunner()

    project_path = Path(__file__).parent.parent.parent
    conf_path = f'{project_path}/{settings.CONF_SOURCE}'
    conf_loader = CONFIG_LOADER_CLASS(conf_source=conf_path, env="local", globals_pattern='globals*')

    parameters = conf_loader.get("parameters*", "parameters*/**")
    credentials = conf_loader.get("credentials*", "credentials*/**")
    catalog = conf_loader.get("catalog*", "catalog*/**")

    data_catalog = DATA_CATALOG_CLASS(data_sets={
        'observations': CSVDataSet.from_config('observations',
                                               catalog['observations']
                                               ),
    },
        feed_dict={'params': parameters})

    result = runner.run(data_extraction.create_pipeline(), data_catalog)
    return result
settings.py
Copy code
CONF_SOURCE = "conf"

# Class that manages how configuration is loaded.
from kedro.config import TemplatedConfigLoader
CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*globals.yml",
}

# Class that manages the Data Catalog.
from <http://kedro.io|kedro.io> import DataCatalog
DATA_CATALOG_CLASS = DataCatalog
can’t get over this error…. would appreciate any help :)
j

Jordan

01/09/2023, 11:18 PM
Think for S3 you need another level in the creds:
Copy code
dev_s3:
    client_kwargs:
        aws_access_key_id: AWS_ACCESS_KEY_ID
        aws_secret_access_key: AWS_SECRET_ACCESS_KEY
Also I don’t think you need to “$” in catalog.yml.
Copy code
credentials: dev_s3
should be sufficient
s

Sergei Benkovich

01/10/2023, 8:37 AM
thanks for the reply! i tried it but it leads to ‘’str’ object is not a mapping. DataSet ‘observations’ must only contain arguments valid for the constructor of ‘kedro.extras.datasets.pandas.csv_dataset.CSVDataSet’.' which i also faced before and could resolve 🫠
12 Views