Hi, I'm using kedro azureml plugin and want to deploy my kedro projects into azure ml pipelines, all work like a charm but I want to ask about storing the secret/credentials, what is the best practice regarding this? Is it possible to integrate it with HashiCorp Vault? Thanks all

hi @Muhammad Ghazalli, I don't think there's an existing Kedro integration for Vault. what I'm thinking is that you could define your own custom OmegaConf resolver to have something like this in your `credentials.yml`:

dev_azure:
  azure_key_id: ${hvac:AZURE_KEY_ID}

and then use https://pypi.org/project/hvac/ to load it like this: https://hvac.readthedocs.io/en/stable/overview.html you'd need to retrieve the Vault server URL somewhere, but other than that I think it should be feasible. probably in the form of a plugin. what do you think?

hi @Juan Luis sorry for the late reply, I'm trying to digest your proposed solution. So you're suggesting using OmegaConf and loading the Vault using HVAC? Is it possible to load it in settings.py? Or is there any way to pass the secret when uploading the pipeline to Azure In kedro azureml run?

I'm not 100 % sure what would be the blessed way to pass a secret in

settings.py

that would then be needed to load other secrets. hopefully my colleagues can chime in

I don't know if this would be of any help? https://docs.kedro.org/en/stable/hooks/common_use_cases.html#use-hooks-to-load-external-credentials

Summoning @marrrcin

thanks for the pointer @Yetunde, I think that should be exactly it!

You can use any secret store that you want, as long as you can connect to it 🙂 One way you can proceed with setting this up is to use System Managed Identity in Azure ML Compute instances - once set up, you will be able to assign roles to this identity (e.g. to read from the Azure Key Vault or other things that you have in Azure). Useful links: • https://learn.microsoft.com/en-us/azure/machine-learning/how-to-identity-based-service-authentication?view=azureml-api-2&tabs=cli • https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-resources-from-[…]aged-identities?view=azureml-api-2&tabs=system-identity-cli (this is for endpoints, but it should work the same for compute)

hi @Juan Luis thank you for helping, your idea works on my case, with a few changes. big thanks to others for helping, really appreciate it, great community.