Hey everyone, In my pipeline I have one input that is optional, and depending on the run it may not exist. I was not able to add it to the catalog because it would throw an error whenever it didn't exist. So I tried to create one entry for it in the code and catch the exception whenever it doesn't exist. (Is there a better way to do this?) This worked fine locally, but I need to store it on S3, and I couldn't figure out how to pass the credentials to it in the documentation. Has anyone used a dataset that requires credentials and that was loaded programatically? I am using CSVDataset from Pandas, by the way.

For credentials, use resolvers and search Azure key vault in our docs for an example. Sorry I have a really bad network so I cannot find the link.

^ https://docs.kedro.org/en/stable/hooks/common_use_cases.html#use-hooks-to-load-external-credentials

By the way, this worked! Thanks a lot!