Hi everyone, It's been really been a while !!! 🙂 I'm currently looking into solutions to make our pipelines truly "plug & play" (or "clone & run" should I say?). One block on that road is the problem of credentials. (we always end up passing around a

conf.zip

... yes.. We're ashamed 😅 ) I'm therefore exploring the different ways of committing to git gpg-encrypted credentials. So far, I've looked into

git-crypt

,

git-secret

,

blackbox

& Mozilla

sops

. After much tinkering & pondering, I think that I've settled on using

sops

but this is still an open question. 1. Has anyone here experience with those ? Would you recommend something else ? 2. I'd like

sops

to decrypt credentials on the fly at runtime ? Would you recommend creating a custom omegaconf resolver or a hook ? I can't really foresee the pros & cons of either approaches ? Thanks in advance M.

This plays well with yaml and might be a fit to extended omegaConf in your projects https://github.com/voxpupuli/hiera-eyaml

Thx @marrrcin 🙂 🙏🏼

This isn’t the solution you were asking for, but we use a key vault for this purpose https://docs.kedro.org/en/stable/hooks/common_use_cases.html#use-hooks-to-load-external-credentials Not sure if that works for you but that example should be extendable to fit other key vaults than just azures (which we happen to use)

Thx Iñigo 🙂 🙏🏼