https://kedro.org/ logo
Powered by
#questions
Title
# questions
s

Stephane Durfort

03/22/2023, 4:22 PM
Hello, while playing with the 
OmegaConfigLoader
to eventually replace the 
TemplateConfigLoader
in my pipeline, I noticed that • variable interpolation does not seem to be applied on nested parameters (as in the 
model_options
example mentioned in the documentation) • using 
kedro run --params
only update parameters but does not propagate to references of these parameters in the configuration ? Am I doing something wrong ?
I built a simple pipeline with a single node displaying the parameters
Copy code
# conf/base/parameters_globals.yml

global: test_1

data:
  size: 0.2

# conf/base/parameters.yml

param: ${global}

nested:
  param: ${global}

model_options:
  test_size: ${data.size}
  random_state: 3

# conf/base/parameters/demo.yml

demo: ${global}

demo_nested:
  param: ${global}

# src/test_omegaconf/pipelines/demo/pipeline.py

def create_pipeline(**kwargs) -> Pipeline:
    return pipeline(
        [
            node(
                func=demo_node,
                inputs=["parameters", "params:demo_nested"],
                outputs=None,
                name="demo"
            )
        ]
    )

# src/test_omegaconf/pipelines/demo/nodes.py

def demo_node(parameters, nested_parameters):
    print(type(parameters))
    pprint(parameters)
    print(type(nested_parameters))
    pprint(nested_parameters)
running the pipeline, only top level variable reference are expanded
Copy code
❯ kedro run

                    INFO     Running node: demo: demo_node([parameters,params:demo_nested]) -> None
<class 'dict'>
{'data': {'size': 0.2},
 'demo': 'test_1',
 'demo_nested': {'param': '${global}'},
 'global': 'test_1',
 'model_options': {'test_size': '${data.size}', 'random_state': 3},
 'nested': {'param': '${global}'},
 'param': 'test_1'}
<class 'omegaconf.dictconfig.DictConfig'>
{'param': '${global}'}
passing a parameter on the command line does not propagate to reference of this parameter in the configuration
Copy code
❯ kedro run --params global:test_2

                    INFO     Running node: demo: demo_node([parameters,params:demo_nested]) -> None
<class 'dict'>
{'data': {'size': 0.2},
 'demo': 'test_1',
 'demo_nested': {'param': '${global}'},
 'global': 'test_2',
 'model_options': {'test_size': '${data.size}', 'random_state': 3},
 'nested': {'param': '${global}'},
 'param': 'test_1'}
<class 'omegaconf.dictconfig.DictConfig'>
{'param': '${global}'}
BTW: I'm testing with 
0.18.6
👍🏼 1
n

Nok Lam Chan

03/22/2023, 4:40 PM
let me run a quick test
👍 1
s

Stephane Durfort

03/22/2023, 4:53 PM
Note, if I access to the nested parameter (2nd argument) in my node, it get correctly expanded
Copy code
print(nested_parameters["param"])
-> test_1
I guess this is because 
parameters
returns a dict while 
params:xxx
returns an omegaconf object ?
n

Nok Lam Chan

03/22/2023, 5:17 PM
Thank you for your detailed example, it’s very easy to follow and reproduce your scenario
So there are two questions here: 1. Only Top-level variable are expanded 2. CLI parameters overriding the global is not propogate to the 
${global}
reference. For 1. if you access 
parameters["demo_nested"]["param"]
it will still be the correct value. 2 seems to be a bug.
s

Stephane Durfort

03/22/2023, 5:21 PM
Yes, however 1/ might be an edge case when playing with the parameters collection only
I mean if a node requires to access to all parameters, it seems it will be provided with a dictionary that no longer allow to expand variables
my bad, you're right about 1/, I did not read your answer properly
many thanks for your time and for the kedro framework, really enjoy using it!
K 1
n

Nok Lam Chan

03/22/2023, 5:38 PM
https://github.com/kedro-org/kedro/issues/2458 I created a slightly simplified version of this thread, could you check if this is what you asking?
👍 1
Appreciate it 🙂 At the moment we are trying to add more features to 
OmegaConfigLoader
, any feedback is very welcomed.
s

Stephane Durfort

03/22/2023, 5:49 PM
I'm keen to move to 
OmegaConfigLoader
when it will allow to configure catalogs as well (same concern as https://kedro-org.slack.com/archives/C03RKP2LW64/p1679009485518709) and I realize now that 1/ is not an issue but my misinterpretation of the display of parameters versus their use.
n

Nok Lam Chan

03/22/2023, 5:58 PM
If you have more comments about catalog you can add them here too. https://github.com/kedro-org/kedro/issues/2175 I agree the display could be confusing and I will see if there is anything we can improve.
2 Views
Powered by