https://kedro.org/ logo
#questions
Title
# questions
m

marrrcin

02/23/2024, 8:38 AM
I have a question related to `OmegaConf`and custom resolvers. Some context first - client is using https://getindata.com/blog/kedro-dynamic-pipelines/ with custom resolver
merge_dict
which allows to combine multiple keys in the configuration and merge them (deep merge) as in the example below:
Copy code
model_options:
  test_size: 0.2
  random_state: 3
  target: costs
  features:
    - engines
    - passenger_capacity
    - crew
    - d_check_complete
    - moon_clearance_complete
    - iata_approved
    - company_rating
    - review_scores_rating
  model: sklearn.linear_model.LinearRegression
  model_params: {}

price_predictor:
  _overrides:
    target: price
  model_options: ${merge:${model_options},${._overrides}}

  base:
    model_options: ${..model_options}
  
  candidate1:
    _overrides:
      features:
      - engines
      - passenger_capacity
      - crew
      - d_check_complete
      - company_rating
    model_options: ${merge:${..model_options},${._overrides}}
The problem is when the merge needs to happen between
local
and environment config - it seems like the resolvers are “resolved” ( 😄 ) for each environment (=each file) separately. Question: Is there a way to influence that behaviour and be able to do “config merge” first and then apply resolvers? 🤔
👀 1
K 1
a

Ankita Katiyar

02/23/2024, 10:31 AM
m

marrrcin

02/23/2024, 10:36 AM
I don’t think so - see, the
merge
resolver is mostly do do “inheritance” across keys - as in the example above - you have some “base parameters” that you want to use and then multiple “namespaced” instances of those params - in this case it’s a model (
price_predictor
) with multiple variants (
base
,
candidate1
, etc.). If
local
env wants to overwrite top level
model_options
, they will not be used in the
model_options: ${merge:${model_options},${._overrides}}
Btw, this doc is missing explanation with example how those two merge options work 🤔
👍 2
👍🏼 1
a

Ankita Katiyar

02/23/2024, 10:43 AM
I see, I know
globals
are loaded and merged first but even there the config is resolved per environment and then merged. But when you refer to it in
parameters.yml
, the overridden key from the local env is used for the resolution - maybe a combination of
merge
and
globals
resolver?
m

marrrcin

02/23/2024, 10:44 AM
Could you give a short example?
a

Ankita Katiyar

02/23/2024, 10:44 AM
Yep, let me actually try it out ^
base/globals.yml
:
Copy code
model_options:
  test_size: 0.3
  random_state: 3
  target: costs
  features:
    - engines
    - passenger_capacity
    - crew
    - d_check_complete
    - moon_clearance_complete
    - iata_approved
    - company_rating
    - review_scores_rating
`local/globals.yml`:
Copy code
model_options:
  test_size: 0.5
  random_state: 56
  target: costs
  features:
    - engines
    - passenger_capacity
    - crew
    - d_check_complete
    - moon_clearance_complete
    - company_rating
---- `parameters.yml`:
Copy code
candidate1:
  _overrides:
    features:
      - engines
      - passenger_capacity
      - crew
      - d_check_complete
      - company_rating
  model_options: ${merge:${globals:model_options}, ${._overrides}}
This will use the
model_options
defined in the
local/globals.yml
for the
merge
resolver - is that what you wanted to do?
m

marrrcin

02/23/2024, 11:02 AM
Yup, that might work. Thanks! 🎉
🎉 1
n

Nok Lam Chan

02/23/2024, 1:15 PM
Late to the party, but almost all configuration are contained within the environment, except
globals
, thus the name. Cross-env interpolation was discussed earlier but we decide not to do it because it creates a lot of confusion
5 Views