Im trying to figure out how to update the parameters used in Kedro #questions

Im trying to figure out how to update the paramete...

Markus Sagen

11/16/2023, 7:47 AM

Im trying to figure out how to update the parameters used in the Kedro context and passed into the pipelines and nodes. This is my current hook implementation, but the new parameters are not passed to the pipeline. Any ideas why?

Copy code

class UpdateParamsHook:
    env: str | None

    @hook_impl
    def after_context_created(self, context: KedroContext):
        # Load base parameters
        # parameters loaded from Kedro using OmegaConf are already resolved as `context.params`
        # instead, we need to load the base parameters first.
        base_params_path = "conf/base/parameters.yml"
        base_params = OmegaConf.load(base_params_path)

        # Load environment-specific parameters
        env = context.env
        env_params_path = f"conf/{env}/parameters.yml"
        env_params = OmegaConf.load(env_params_path)

        # Merge parameters with priority to environment-specific params
        merged_params = OmegaConf.merge(base_params, env_params)

        # Ensure environment variable interpolation
        OmegaConf.resolve(merged_params)

        # Update context params
        context._params = merged_params

Nok Lam Chan

11/16/2023, 8:22 AM

can you explains what are you trying to do? do you just need a soft merge?

Nok Lam Chan

11/16/2023, 8:25 AM

context._params

doesn’t even exist afaik

Markus Sagen

11/16/2023, 8:34 AM

I’m trying to pass context of what environment is currently running and would like to pass that to my pipelines and nodes. The approach I’ve tried was to update

conf/base/parameters.yml

and merge/update them with the parameters in

conf/prod/parameters.yml

base/parameters.yml

Copy code

env: local
training:
    env: ${env}

prod/parameters.yml

Copy code

env: prod

Markus Sagen

11/16/2023, 8:36 AM

And I would like to know what environment is targeted. I.E. get the argument from

Copy code

kedro run -e prod

Nok Lam Chan

11/16/2023, 8:41 AM

So you expect the resolved config as follow?

Copy code

env: prod
training:
  env: prod

Markus Sagen

11/16/2023, 8:42 AM

Yes, and for those parameters to be passed into the pipeline and nodes

Nok Lam Chan

11/16/2023, 8:43 AM

I think in this case you can use

globals

Nok Lam Chan

11/16/2023, 8:48 AM

The resolving order is a bit complicated. Basically, 1. Variable interpolation affect local scope (Within its env) 2. After both env’s config resolved, (base + local), a pure dictionary merge will happen (local will override base) Therefore, variable interpolation is not cross-environment, the way to get around this is using the

globals

resolver, which happens before step1.

5 Views

Open in Slack

Previous Next