What is the best way to make credentials from

credentials.yml

available for use in a hook? I know I can just use the yaml loader, but this doesn’t feel correct.

Why do you need to access credentials.yml directly?

I’m trying to create a hook that adds a custom partitioned dataset to the catalog. This dataset lives in s3, so I need to give it the creds as well, at least I think

Can you define it inside catalogs.yml? Why do you need to add datasets via hooks?

I don’t strictly need to. I just have a lot of datasets to analyse. I was hoping to avoid having to write lots of catalog entires for them by creating a hook that would do this dynamically

I would stick with

catalog.yml

when possible. But if you have to do it via hooks, I would go with

data_catalog

instead of reading off from

credentials.yml

, this is something we want to avoid.

I’m not sure what you mean. Are you saying there is a way to access credentials through the data catalog?

Actually, you can access it via

ConfigLoader

With the latest

main

we are making it easier to access via

config_loader["credentials"]

in earlier version you can access it via

config_loader.get(["credentials*/*"])

Thanks for the info, I’ll check it out!