Hi all, I want to have a config that will read/wri...
# questions
Hi all, I want to have a config that will read/write to my local file system instead of S3, when the pipeline is executed locally. A conf/dev/ folder already exists, containing configurations for our cloud dev setup. I understand that I can manually create a globals.yml in my local folder, however all developers will have to manually create these files for all the different use-cases we maintain. With conf/local/ being in the .gitignore files, what is the intended way to share local configurations?
So out of the box we provide
within the
folder. Base is committed but Local isn’t. Has this been changed in your structure?
I don't think so. But my base is now pointing towards a s3 bucket and I had to manually create a local config file (because it isn't committed) in local to overwrite it such that my datasets are reading/writing locally.
Maybe my question is: what is the idea behind not committing the /local/ configurations folder?
so in the default structure
all of the folders will be committed BUT local (that’s why they would be greyed out)
the logic is that people would only put stuff in there that they don’t want committed
you can use base (or any other config env) for shared development
you don’t have to use it, it’s for situations where there are local sensitivities
👍 1
Thanks for clarifying. Would you use some default /local/catalog.yml file, such that a new developer on the project can just "git clone" and start running things locally? Or would you expect that person to manually create that /local/catalog.yml? Or would the local configuration be in base?
well there is also an order of precedence
so if the same key exists in both base AND local, local wins
so this is useful in situations where you can have the remote execution configured, but local will win if present on someone’s local machine
for experienced developers this is less of a critical feature, but it’s a good first step for novices to minimise the risk of credential leakage etc.
👍 1