Hi all,
I want to have a config that will read/wri...
# questionss
Seth
11/03/2022, 3:56 PMHi all,
I want to have a config that will read/write to my local file system instead of S3, when the pipeline is executed locally.
A conf/dev/ folder already exists, containing configurations for our cloud dev setup.
I understand that I can manually create a globals.yml in my local folder, however all developers will have to manually create these files for all the different use-cases we maintain.
With conf/local/ being in the .gitignore files, what is the intended way to share local configurations?
d
datajoely
11/03/2022, 3:57 PMSo out of the box we provide
baselocalconfs
Seth
11/03/2022, 4:01 PMI don't think so. But my base is now pointing towards a s3 bucket and I had to manually create a local config file (because it isn't committed) in local to overwrite it such that my datasets are reading/writing locally.
Seth
11/03/2022, 4:02 PMMaybe my question is: what is the idea behind not committing the /local/ configurations folder?
d
datajoely
11/03/2022, 4:03 PMso in the default structure
datajoely
11/03/2022, 4:04 PMall of the folders will be committed BUT local (that’s why they would be greyed out)
datajoely
11/03/2022, 4:04 PMthe logic is that people would only put stuff in there that they don’t want committed
datajoely
11/03/2022, 4:04 PMyou can use base (or any other config env) for shared development
datajoely
11/03/2022, 4:05 PMyou don’t have to use it, it’s for situations where there are local sensitivities
👍 1
s
Seth
11/03/2022, 4:07 PMThanks for clarifying.
Would you use some default /local/catalog.yml file, such that a new developer on the project can just "git clone" and start running things locally? Or would you expect that person to manually create that /local/catalog.yml?
Or would the local configuration be in base?
d
datajoely
11/03/2022, 4:10 PMwell there is also an order of precedence
datajoely
11/03/2022, 4:10 PMso if the same key exists in both base AND local, local wins
datajoely
11/03/2022, 4:10 PMso this is useful in situations where you can have the remote execution configured, but local will win if present on someone’s local machine
datajoely
11/03/2022, 4:11 PMfor experienced developers this is less of a critical feature, but it’s a good first step for novices to minimise the risk of credential leakage etc.
👍 1
6 Views