Hi everyone, A super-ultra-duper-dummy question: Assuming that in

conf/base/parameters.yml

I have

model:
  init:
    k: 3
    loss: warp
    no_embeddings: 50
    learning_schedule: adagrad
    rho: 0.95
    epsilon: 1.e-6
    random_state: ${random_state}

How can I “update” a single specific field “locally” I’ve first tried in

conf/local/parameters.yml

model:
  init:
    no_embeddings: 100

But this actually completely over-writes the model section and, of course, breaks everything. Granted: I could

cp conf/base/parameters.yml conf/local/parameters

and then update

no_embeddings

But this ends up being very “noisy”, not really “highlighting” the specificities of the local config… Is there a way to do such local / “surgical over-write” ? Thx 🙂

Hi! I have the same question 🤔. The first approach that comes to my mind is to open

conf/base/parameters.yml

with a hook and update the value, maybe there is a cleaner way. Refs: • https://stackoverflow.com/questions/40762382/changing-a-value-in-a-yaml-file-using-python • Unrealated video but at minute 9:07 updates the parameters

Hi Javier Thx a lot for your answer and for those interesting pointers. I’m sure I’ll use those some day. But right now, I’m not trying to programmatically update my yaml configs 🙂 What I’m looking for is a way to “fine tune” the behavior of the pipeline to some personal / temporary needs, and a way to do so that is really “targeted, “specific” and allows to see the “uniqueness” of the local config, as opposed to copying some entire base params file and modifying some but not all keys. This really brings a lot of “noisy / unnecessary duplication”… Granted… I could copy everything then comments those the local params that differ from the base… But being acutely aware of all my “human weaknesses”, I’d rather rely on a system that automatically does it for me rather than relying on my own discipline… 😅 Thx anyhow 🙏 Marc P.S: Granted-_bis_: for super-temporary changes there is the `kedro run --params=`… But I can already see that it will be tedious… Some of those “temporary” and “personal” changes needs to be locally persistent.

I'm pretty sure that the default config loader does the merge on the top level keys only (that's why you see the full overwrite which breaks your pipeline). You can try out OmegaConfigLoader and if it doesn't help, then you can overwrite the config loading logic on your own

maybe the default logic should be as @Marc Gris is describing, but I'm not sure if I'm missing potential drawbacks 🤔 wdyt @Merel?

We’re planning to add a soft-merge strategy for non-catalog files in

0.19.0

because it is a breaking change: https://github.com/kedro-org/kedro/issues/2122

Thx everyone. @Merel I look very forward to that new feature 🙂 Till then… I’ll adapt to the current behavior 👍 Thx again M. P.S: Just to be curious: Will this “soft-merge’ ability only concern OmegaConfLoader or will it also benefit TemplatedConfigLoader ?

Only the

OmegaConfigLoader

as we’ll be removing the old legacy config loaders in

0.19.0

🙂

Ok ! Good to hear. In anticipation I’m going to switch to

OmegaConfigLoader

🙂 👍

Let us know how it goes and if you have any feedback in the meantime 🙂