Hey everyone, In my pipeline I have one input that...
# questions
Hey everyone, In my pipeline I have one input that is optional, and depending on the run it may not exist. I was not able to add it to the catalog because it would throw an error whenever it didn't exist. So I tried to create one entry for it in the code and catch the exception whenever it doesn't exist. (Is there a better way to do this?) This worked fine locally, but I need to store it on S3, and I couldn't figure out how to pass the credentials to it in the documentation. Has anyone used a dataset that requires credentials and that was loaded programatically? I am using CSVDataset from Pandas, by the way.
For credentials, use resolvers and search Azure key vault in our docs for an example. Sorry I have a really bad network so I cannot find the link.
👍 1
By the way, this worked! Thanks a lot!
👍🏼 1