Hello Team Can someone help me to use kubernetes Secrets within kedro pipelines to connect with S3 or any other external service.

If you're on K8s, best practice is to use IRSA https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

@marrrcin can you please explain this in one liner how IRSA will help me to fetch and use secrets ? Currently I have stored the credentials in env variables and using resolver to pass the credentials

training_data_insert_records:
  type: pandas.CSVDataset
  filepath: <s3://ml-datawarehouse/warehouse/extraction/doc_table_insert.csv>
  save_args:
    index: False   
  versioned: True
  credentials:
    client_kwargs:
      aws_access_key_id : ${oc.env:AWS_ACCESS_KEY_ID}
      aws_secret_access_key : ${oc.env:AWS_SECRET_ACCESS_KEY}
      aws_session_token : ${oc.env:AWS_SESSION_TOKEN}

I want to know will there be any change the way i am using datasets