Hi all I want to have a config that will read write to my lo

Question

Hi all I want to have a config that will read write to my local file system instead of S3 when the pipeline is executed locally A conf dev folder already exists containing configurations for our cloud dev setup I understand that I can manually create a globals yml in my local folder however all developers will have to manually create these files for all the different use cases we maintain With conf local being in the gitignore files what is the intended way to share local configurations

datajoely · Answer

So out of the box we provide `base` and `local` within the `conf` folder Base is committed but Local isn t Has this been changed in your structure

Seth · Answer

I don t think so But my base is now pointing towards a s3 bucket and I had to manually create a local config file because it isn t committed in local to overwrite it such that my datasets are reading writing locally

Seth · Answer

Maybe my question is what is the idea behind not committing the local configurations folder

datajoely · Answer

so in the default structure

datajoely · Answer

all of the folders will be committed BUT local that s why they would be greyed out

datajoely · Answer

the logic is that people would only put stuff in there that they don t want committed

datajoely · Answer

you can use base or any other config env for shared development

datajoely · Answer

you don t have to use it it s for situations where there are local sensitivities

Seth · Answer

Thanks for clarifying Would you use some default local catalog yml file such that a new developer on the project can just git clone and start running things locally Or would you expect that person to manually create that local catalog yml Or would the local configuration be in base

datajoely · Answer

well there is also an order of precedence

datajoely · Answer

so if the same key exists in both base AND local local wins

datajoely · Answer

so this is useful in situations where you can have the remote execution configured but local will win if present on someone s local machine

datajoely · Answer

for experienced developers this is less of a critical feature but it s a good first step for novices to minimise the risk of credential leakage etc