Hi everyone is it possible to change globals parameters conf Kedro #questions

Hi everyone, is it possible to change globals para...

Zoran

02/06/2023, 5:26 PM

Hi everyone, is it possible to change globals parameters (conf/<env>/globals.yml) dynamically at runtime (as params do with --params)?

👍 2

datajoely

02/06/2023, 6:05 PM

you can do all sorts of things with hooks if you really want https://kedro.readthedocs.io/en/stable/hooks/introduction.html

Zoran

02/06/2023, 7:06 PM

for example

before_dataset_loaded

or you have something other in mind? i tried to use

after_context_created

and tried to add in`CONFIG_LOADER_ARGS`

globals_dict

from parameters but code wouldn't accept that

datajoely

02/07/2023, 9:48 AM

so with the

after_context_created

you have access to the initialised

context.catalog

object which you can mutate with python

datajoely

02/07/2023, 9:49 AM

settings.py is the very first thing that gets executed so you don’t want to try and change that, just intercept things later on in the lifecycle

datajoely

02/07/2023, 9:49 AM

this pattern however is intentionally a little cumbersome since we do want to encourage users to prioritise reproducibility and dynamic pipelines hinder that

Zoran

02/07/2023, 10:43 AM

i forgot to add at beginning what i want to achieve, which is: catalog.yml: filepath: data/01_raw/${DATASET_NAME}.yml depending on which parameters from CLI is run on it DATASET_NAME is changed

datajoely

02/07/2023, 10:43 AM

yeah that’s possible

Zoran

02/07/2023, 10:44 AM

I need little help on that how to achieve this

datajoely

02/07/2023, 10:46 AM

https://github.com/kedro-org/kedro/issues/1647

Zoran

02/07/2023, 10:48 AM

thank you. I will try it now

Zoran

02/09/2023, 12:50 PM

@datajoely @Antony Milne i tried this: add class to settings.py

Copy code

class MyTemplatedConfigLoader(TemplatedConfigLoader):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._config_mapping.update(self.runtime_params)

CONFIG_LOADER_CLASS = MyTemplatedConfigLoader

and it works. But when i run kedro viz i getting error

Copy code

self._config_mapping.update(self.runtime_params)
TypeError: 'NoneType' object is not iterable
kedro.framework.cli.utils.KedroCliError: 'NoneType' object is not iterable
Run with --verbose to see the full exception
Error: 'NoneType' object is not iterable

Zoran

02/10/2023, 8:17 AM

i forgot to add that i am using kedro 0.18.4 and kedro viz 5.2.1

5 Views

Open in Slack

Previous Next