https://kedro.org/ logo
Powered by
#questions
Title
# questions
j

Juan Diego

04/13/2023, 3:07 PM
Hi all! Kedro 0.18.7: We’re trying to modify pipelines “extra params” on the fly, and we guessed that a 
before_pipeline_run
hook is the way to go. Can you advise us on the bests to achieve this? What we tried so far is condensed is this code.
Copy code
class ParamsHook:
    @hook_impl
    def before_pipeline_run(
        self, run_params: Dict[str, Any], pipeline: Any, catalog: DataCatalog
    ) -> None:
        catalog.add_feed_dict({"params:country": MemoryDataSet("ESP")}, replace=True)
In the hook: 1. In 
run_params
we can see: 
'extra_params': {'country': 'USA'}
2. In 
catalog.list()
this entry: 
'params:country'
,before and after invoking 
add_feed_dict
But when params are printed in the node he value persist with the original value parsed by: 
kedro run --params country=USA
Many thanks in advance! NOTE: The objective here is to be able to parse a list from the CLI, let’s say: 
--params countries="ESP<>USA"
and do the split in the hook.
f

FlorianGD

04/13/2023, 3:12 PM
If you want to pass a list, you can run the pipeline in python:
Copy code
with KedroSession.create(extra_params={"country": ["ESP", "USA"]}) as s:
    s.run(pipeline_name="your_pipeline_name")
j

Juan Diego

04/13/2023, 3:18 PM
that’s a solution, but if possible we want to address this with the hook mechanism, Thx BTW
f

FlorianGD

04/13/2023, 3:46 PM
If you want to get the list from the CLI, you can do something like this, but it seems to be what you are doing:
Copy code
countries = catalog.load("params:country").split("<>")
catalog.add_feed_dict({"params:country": MemoryDataSet(countries)}, replace=True)
or am I missing something?
I checked it with a dummy pipeline that only print its argument, and with the hook, you correctly get a list
j

Juan Diego

04/13/2023, 3:50 PM
no, that’s it, but or some reason country param retains it’s original value when reaching a node
f

FlorianGD

04/13/2023, 3:50 PM
Note that the syntax for the params is 
--params country:'ESP<>USA'
with 
:
not 
=
as a separator for key and value
👍 1
I have a pipeline that is only:
Copy code
def create_pipeline(**kwargs) -> Pipeline:
    return pipeline(
        [node(lambda x: print(repr(x)), inputs="params:country", outputs=None)]
    )
and I can correctly see the list
j

Juan Diego

04/13/2023, 3:57 PM
Running this: 
kedro run --params countries="ESP<>USA"
Copy code
class ParamsHook:
    @hook_impl
    def before_pipeline_run(
        self, run_params: Dict[str, Any], pipeline: Any, catalog: DataCatalog
    ) -> None:
        countries = catalog.load("params:countries").split("<>")
        print("countries in hook:", countries)
        catalog.add_feed_dict(
            {"params:countries": MemoryDataSet(countries)}, replace=True
        )
I get: 
countries in hook: ['ESP', 'USA']
And having this in node:
Copy code
def clean_raw(data: pd.DataFrame, params: Dict):
    countries = params["countries"]

    print("countries in node: ", countries)
countries in node:  ESP<>USA
f

FlorianGD

04/13/2023, 4:01 PM
You should not use 
params
in your node, but rather pass the correct parameter as I did above
You did not change the whole params dict, but rather the one input that is named 
params:country
j

Juan Diego

04/13/2023, 4:05 PM
so, there’s no solution where only hooks are involved?
f

FlorianGD

04/13/2023, 4:07 PM
Yes, the hook will work if you change you node to
Copy code
def clean_raw(data: pd.DataFrame, countries: List[str]):
    print("countries in node: ", countries)
and your pipeline to:
Copy code
pipeline([..., node(clean_raw, inputs={"data": ..., "countries": "params:countries"}, outputs=...)], ...)
j

Juan Diego

04/13/2023, 4:12 PM
clear, but I meant without touching anything else, just the hooks, what we’re trying to do here is an standardisation, you can use lists in the params catalog so we want to extend this behaviour to the CLI.
f

FlorianGD

04/13/2023, 4:13 PM
Copy code
class ParamsHook:
    @hook_impl
    def before_pipeline_run(self, catalog: DataCatalog) -> None:
        countries = catalog.load("params:country").split("<>")
        catalog.add_feed_dict(
            {"params:country": MemoryDataSet(countries)}, replace=True
        )
        parameters = catalog.load("parameters")
        parameters["country"] = countries
        catalog.add_feed_dict({"parameters": parameters}, replace=True)
K 1
Here, you explicitly change both the individual parameter and the whole 
parameters
But again, it is not a good practice to pass the whole 
parameters
as an input for a node
j

Juan Diego

04/13/2023, 4:17 PM
yes, we know, we have that refactoring task pending. Many thanks for your great help Florian!
f

FlorianGD

04/13/2023, 4:17 PM
You're welcome!
✔️ 1
👍 1
8 Views
Powered by