https://kedro.org/ logo
#questions
Title
# questions
j

Jordan

10/31/2022, 5:03 PM
What is the best way to make credentials from
credentials.yml
available for use in a hook? I know I can just use the yaml loader, but this doesn’t feel correct.
n

Nok Lam Chan

10/31/2022, 5:04 PM
Why do you need to access credentials.yml directly?
Could you explain what are you trying to do?
j

Jordan

10/31/2022, 5:07 PM
I’m trying to create a hook that adds a custom partitioned dataset to the catalog. This dataset lives in s3, so I need to give it the creds as well, at least I think
n

Nok Lam Chan

10/31/2022, 5:09 PM
Can you define it inside catalogs.yml? Why do you need to add datasets via hooks?
j

Jordan

10/31/2022, 5:11 PM
I don’t strictly need to. I just have a lot of datasets to analyse. I was hoping to avoid having to write lots of catalog entires for them by creating a hook that would do this dynamically
n

Nok Lam Chan

10/31/2022, 5:18 PM
I would stick with
catalog.yml
when possible. But if you have to do it via hooks, I would go with
data_catalog
instead of reading off from
credentials.yml
, this is something we want to avoid.
j

Jordan

10/31/2022, 5:20 PM
I’m not sure what you mean. Are you saying there is a way to access credentials through the data catalog?
n

Nok Lam Chan

10/31/2022, 5:28 PM
Actually, you can access it via
ConfigLoader
With the latest
main
we are making it easier to access via
Copy code
config_loader["credentials"]
in earlier version you can access it via
config_loader.get(["credentials*/*"])
j

Jordan

10/31/2022, 6:44 PM
Thanks for the info, I’ll check it out!
2 Views