Fun kedro environments question. Right now I use

base

and

prod

environments to distinguish between dev and production runs where the output gets written to different databases. We also run our same pipeline across multiple customers that might have slightly different configurations. Is there a clever way to add a set of customer-specific configurations? It would be too bulky to create a

base-customer1

,

base-customer2

,

prod-customer1

,

prod-customer2

environment and have to scale that up to every customer we on board. Curious if anyone has any thoughts on this!

Someone will reply to you shortly. In the meantime, this might help:

Firstly, 👋 hey @Sanjay Hariharan hope you're smashing start up life I think passing configuration as a compressed archive is the best pattern here. You can use

kedro run --conf-source path/to/file.zip

so you can actually maintain these things independently. Longer term it might make sense to think about a building a utility that helps you manage/version this complexity. You could even think about a hook that could compare config and code versions to help manage breaking changes.

Hi @datajoely, great to see you again 🙂 . I also hope I am smashing startup life. So far it seems to be smashing me. A quick question to clarify the above: Would it be possible to combine this approach and the environment approach? If so, how do overrides work between --conf-source and --env? Ideally the code would be the same, just the configuration (catalog, params) would be slightly different across customers and dev/ prod

firstly lol

Got it, I understand. So I would still need to maintain a

conf/

for each customer, but within it I can specify my environments for dev/ prod.

yes and the fact that you can distribute / deploy with the zip file you could in theory maintain your conf/ in a completely different repo with appropriate tooling to make sure it's valid

ok understood, I like this approach. For simplicity I don't mind just having a separate folder for each customer and with it dev/prod environments, and over time can move to a separate repo and ensure validity.

yeah