https://kedro.org/ logo
#plugins-integrations
Title
# plugins-integrations
m

Muhammad Ghazalli

06/06/2023, 3:38 AM
Hi, I'm using kedro azureml plugin and want to deploy my kedro projects into azure ml pipelines, all work like a charm but I want to ask about storing the secret/credentials, what is the best practice regarding this? Is it possible to integrate it with HashiCorp Vault? Thanks all
j

Juan Luis

06/06/2023, 11:24 AM
hi @Muhammad Ghazalli, I don't think there's an existing Kedro integration for Vault. what I'm thinking is that you could define your own custom OmegaConf resolver to have something like this in your `credentials.yml`:
Copy code
dev_azure:
  azure_key_id: ${hvac:AZURE_KEY_ID}
and then use https://pypi.org/project/hvac/ to load it like this: https://hvac.readthedocs.io/en/stable/overview.html you'd need to retrieve the Vault server URL somewhere, but other than that I think it should be feasible. probably in the form of a plugin. what do you think?
m

Muhammad Ghazalli

06/09/2023, 8:57 AM
hi @Juan Luis sorry for the late reply, I'm trying to digest your proposed solution. So you're suggesting using OmegaConf and loading the Vault using HVAC? Is it possible to load it in settings.py? Or is there any way to pass the secret when uploading the pipeline to Azure In kedro azureml run?
j

Juan Luis

06/09/2023, 9:22 AM
I'm not 100 % sure what would be the blessed way to pass a secret in
settings.py
that would then be needed to load other secrets. hopefully my colleagues can chime in
y

Yetunde

06/09/2023, 9:24 AM
a

Artur Dobrogowski

06/09/2023, 9:53 AM
Summoning @marrrcin
He's off today, so unlikely to show up, but expect a follow up on monday šŸ™‚
j

Juan Luis

06/09/2023, 11:20 AM
thanks for the pointer @Yetunde, I think that should be exactly it!
m

marrrcin

06/12/2023, 10:17 AM
You can use any secret store that you want, as long as you can connect to it šŸ™‚ One way you can proceed with setting this up is to use System Managed Identity in Azure ML Compute instances - once set up, you will be able to assign roles to this identity (e.g. to read from the Azure Key Vault or other things that you have in Azure). Useful links: ā€¢ https://learn.microsoft.com/en-us/azure/machine-learning/how-to-identity-based-service-authentication?view=azureml-api-2&tabs=cli ā€¢ https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-resources-from-[ā€¦]aged-identities?view=azureml-api-2&tabs=system-identity-cli (this is for endpoints, but it should work the same for compute)
m

Muhammad Ghazalli

06/13/2023, 4:00 AM
hi @Juan Luis thank you for helping, your idea works on my case, with a few changes. big thanks to others for helping, really appreciate it, great community.
ā¤ļø 1
10 Views