I am newer to AWS and I have followed the Kedro AWS Batch Deployment Guide but I am getting a dataset error like this stackoverflow question. I do not want to put datasets into the docker container. I want to be able to read from and write to S3. The AWS tutorial puts the S3 URL as an environment variable. Do I need to do this for each dataset? I'm really looking for how to connect my docker container to S3 to run in a kedro pipeline.
Thanks in advance for your help and let me know if I need to provide more detail.