https://kedro.org/ logo
Powered by
#questions
Title
# questions
r

Rafał Nowak

02/24/2023, 8:20 AM
Hi, I am trying to run the kedro pipeline with 
--params "section1.section2.name:value"
where 
section1.section2
is defined in 
parameters.yml
so it seems that there is some tree 
section1.section2
with some parameters. I would like to change only one of them. I think I know that kedro is not able to override only one paramter in the tree. I have to overrside the full root which is not user friendly in the CLI. I see that since kedro 0.18.5, one can use OmegaConf now. Does it change this limitatiom? If so, is it possible to use 
global_patterns
like it was in TemplatedConfigLoader ?
I think that some solution is to use interpolation strings. However I am wondering how to use interpolation string in 
catalo/*.yml
files. One cannot define the string in 
parametes/*.yml
since they are not visible. One cannot also define such string in 
catalog/*.yml
since kedro fails when loading such catalog item; for example some 
layer popup
eror appears. I thought that 
_name
would be solution, but not.
v

Vassilis Kalofolias

02/24/2023, 3:00 PM
Hi Rafal, I had a similar problem, see the discussion here: https://kedro-org.slack.com/archives/C03RKP2LW64/p1676545563999529 TLDR: • you can create a config.yml and call 
kedro run --config config.yml
, where you have defined
Copy code
# content of config.yml
run:
  params:
    section1:
      section2:
        name: value
• you can do the above in a one-liner (not my favorite) • you can do this programmatically by adding the overriding parameters in a new session with 
kedro.framework.session.Session.KedroSession(..., extra_params={"section1":{"section2":{name:value}}})
. This will replace all levels of the tree with the overriding parameters, while keeping the rest intact. • you can not use 
add_all
or 
add
or 
add_feed_dict
easily as it doesn't replace all levels of the tree.
👍 1
BTW I think if you run 
kedro run --params "section1.section2.name:value"
it should already have the same effect, as internally it uses the 
extra_params
(keeping the rest of the tree intact). Unfortunately this is not true,. While the argument is also called 
extra_params
it doesn't behave as the one of 
kedro.framework.session.Session.KedroSession
and instead it replaces the high level tree. So to make it work you have to run the full script:
Copy code
# run_experiment.py
from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from pathlib import Path

extra_params = {
  section1: {
    section2: {
      name: value
    }
  }
}

project_path = Path(".").resolve()
metadata = bootstrap_project(project_path)
session = KedroSession.create(metadata.package_name, project_path, env=None, extra_params=extra_params)
context = session.load_context()
catalog = context.catalog

assert catalog.load("params:section1.section2.name") == value
# check that other params have not disappeared
assert catalog.load("params:section1.section2.another_name") == another_value
r

Rafał Nowak

02/25/2023, 12:54 PM
Super thanks for your reply. Can anyone from kedro team take the responsibility for the complexity of this solution? In my opinion it is too complex solution for simply request
Copy code
kedro run --params "section1.section2.name:value"
I think that OmegaConf would be better solution. However I still do not know how to add interpolation string into 
catalog/*.yml
files. One cannot define globals in this folder since Kedro is trying to create DataSet objects.
v

Vassilis Kalofolias

02/27/2023, 6:17 PM
Apparently there is an issue here: https://github.com/kedro-org/kedro/issues/2357
2 Views
Powered by