https://kedro.org/ logo
Powered by
#questions
Title
# questions
m

Markus Sagen

11/16/2023, 7:47 AM
Im trying to figure out how to update the parameters used in the Kedro context and passed into the pipelines and nodes. This is my current hook implementation, but the new parameters are not passed to the pipeline. Any ideas why?
Copy code
class UpdateParamsHook:
    env: str | None

    @hook_impl
    def after_context_created(self, context: KedroContext):
        # Load base parameters
        # parameters loaded from Kedro using OmegaConf are already resolved as `context.params`
        # instead, we need to load the base parameters first.
        base_params_path = "conf/base/parameters.yml"
        base_params = OmegaConf.load(base_params_path)

        # Load environment-specific parameters
        env = context.env
        env_params_path = f"conf/{env}/parameters.yml"
        env_params = OmegaConf.load(env_params_path)

        # Merge parameters with priority to environment-specific params
        merged_params = OmegaConf.merge(base_params, env_params)

        # Ensure environment variable interpolation
        OmegaConf.resolve(merged_params)

        # Update context params
        context._params = merged_params
n

Nok Lam Chan

11/16/2023, 8:22 AM
can you explains what are you trying to do? do you just need a soft merge?
context._params
doesn’t even exist afaik
m

Markus Sagen

11/16/2023, 8:34 AM
I’m trying to pass context of what environment is currently running and would like to pass that to my pipelines and nodes. The approach I’ve tried was to update 
conf/base/parameters.yml
and merge/update them with the parameters in 
conf/prod/parameters.yml
. 
base/parameters.yml
Copy code
env: local
training:
    env: ${env}
prod/parameters.yml
Copy code
env: prod
And I would like to know what environment is targeted. I.E. get the argument from
Copy code
kedro run -e prod
n

Nok Lam Chan

11/16/2023, 8:41 AM
So you expect the resolved config as follow?
Copy code
env: prod
training:
  env: prod
m

Markus Sagen

11/16/2023, 8:42 AM
Yes, and for those parameters to be passed into the pipeline and nodes
n

Nok Lam Chan

11/16/2023, 8:43 AM
I think in this case you can use 
globals
.
The resolving order is a bit complicated. Basically, 1. Variable interpolation affect local scope (Within its env) 2. After both env’s config resolved, (base + local), a pure dictionary merge will happen (local will override base) Therefore, variable interpolation is not cross-environment, the way to get around this is using the 
globals
resolver, which happens before step1.
Powered by