hey i have an issue using credentials in my project. i get “ValueError: Failed to format pattern ‘${dev_s3}‘: no config value found, no default provided” credentials.yml: (in local folder), as i understand it should take the AWS_ACCESS_KEY_ID from aws cli. i tried also specifying it, and it didn’t help

dev_s3:
     aws_access_key_id: AWS_ACCESS_KEY_ID
     aws_secret_access_key: AWS_SECRET_ACCESS_KEY

catalog.yml:

observations:
  type: pandas.CSVDataSet
  filepath: "${s3.raw_observations_path}/commercial/observations/observations.csv"
  credentials: "${dev_s3}"

main.py

runner = SequentialRunner()

    project_path = Path(__file__).parent.parent.parent
    conf_path = f'{project_path}/{settings.CONF_SOURCE}'
    conf_loader = CONFIG_LOADER_CLASS(conf_source=conf_path, env="local", globals_pattern='globals*')

    parameters = conf_loader.get("parameters*", "parameters*/**")
    credentials = conf_loader.get("credentials*", "credentials*/**")
    catalog = conf_loader.get("catalog*", "catalog*/**")

    data_catalog = DATA_CATALOG_CLASS(data_sets={
        'observations': CSVDataSet.from_config('observations',
                                               catalog['observations']
                                               ),
    },
        feed_dict={'params': parameters})

    result = runner.run(data_extraction.create_pipeline(), data_catalog)
    return result

settings.py

CONF_SOURCE = "conf"

# Class that manages how configuration is loaded.
from kedro.config import TemplatedConfigLoader
CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*globals.yml",
}

# Class that manages the Data Catalog.
from <http://kedro.io|kedro.io> import DataCatalog
DATA_CATALOG_CLASS = DataCatalog

can’t get over this error…. would appreciate any help :)

Think for S3 you need another level in the creds:

dev_s3:
    client_kwargs:
        aws_access_key_id: AWS_ACCESS_KEY_ID
        aws_secret_access_key: AWS_SECRET_ACCESS_KEY

Also I don’t think you need to “$” in catalog.yml.

credentials: dev_s3

should be sufficient

thanks for the reply! i tried it but it leads to ‘’str’ object is not a mapping. DataSet ‘observations’ must only contain arguments valid for the constructor of ‘kedro.extras.datasets.pandas.csv_dataset.CSVDataSet’.' which i also faced before and could resolve 🫠