https://kedro.org/ logo
#questions
Title
# questions
s

Sergei Benkovich

03/02/2023, 12:48 PM
is there a way to use same node with different sets of input/outputs? my_node(a) -> b my_node(c) -> d i think i saw something related in the documentation but i think it looked too complicated , any easy way to do it?
r

Ricardo Araújo

03/02/2023, 12:49 PM
I think you need to do this on the level of the pipeline. That would be a modular pipeline, and then you can remap inputs and outputs. See the last example in https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html
🙏 1
kedroid 1
d

datajoely

03/02/2023, 12:50 PM
Yes exactly @Ricardo Araújo is on the money 💵 !
r

Ricardo Araújo

03/02/2023, 12:50 PM
Other than that, you could wrap the same function in different nodes with different inputs/outputs. Not as clean, though.
d

datajoely

03/02/2023, 12:51 PM
specifically look for the
Using a modular pipeline multiple times
heading
s

Sergei Benkovich

03/02/2023, 12:52 PM
yeah i think i saw it and i avoided trying it out because i have issues with the “params:override_me” no idea why… i do pass “params”, but using ‘:’ is not working and i didn’t figure out why
d

datajoely

03/02/2023, 12:53 PM
Copy code
template_pipeline = pipeline(
    [
        node(
            func=node_func1,
            inputs=["input1", "input2", "params:override_me"],
            outputs="intermediary_output",
        ),
        node(
            func=node_func2,
            inputs="intermediary_output",
            outputs="output",
        ),
    ]
)

alpha_pipeline = pipeline(
    pipe=template_pipeline,
    inputs={"input1", "input2"},
    parameters={"params:override_me": "params:alpha"},
    namespace="alpha",
)

beta_pipeline = pipeline(
    pipe=template_pipeline,
    inputs={"input1", "input2"},
    parameters={"params:override_me": "params:beta"},
    namespace="beta",
)

final_pipeline = alpha_pipeline + beta_pipeline
from the example in there
oh I see you’re saying that’s not working
what is the key you’re looking for in your override, is it nested with
.
or does it have spaces in it?
s

Sergei Benkovich

03/02/2023, 12:55 PM
i don’t have an override example written yet, but when i tried passing specific parameters using : i got an error.
d

datajoely

03/02/2023, 12:55 PM
what was the error?
s

Sergei Benkovich

03/02/2023, 12:56 PM
ValueError: Pipeline input(s) {‘params:data_extraction_params’} not found in the DataCatalog which is inside a yml inside the parameters directory
d

datajoely

03/02/2023, 12:58 PM
have you added a namespace?
s

Sergei Benkovich

03/02/2023, 12:58 PM
my usage in main and the settings file
can you explain about the namespace?
d

datajoely

03/02/2023, 12:59 PM
so this is quite an unusual way of using Kedro, is there a reason you’re not using the standard template and workflow?
it’s possible to get this working, but it’s much harder to support
s

Sergei Benkovich

03/02/2023, 1:00 PM
hm i’m not sure i understand the differences between this and the standart template and workflow 🙂
d

datajoely

03/02/2023, 1:01 PM
if you have time- I’d really encourage you to follow the spaceflights tutorial it even covers this part under this section
👍 2
it should take you through the expected usage pattern
s

Sergei Benkovich

03/02/2023, 1:04 PM
i will do that…i feel like i can optimize my kedro usage…. and i need to invest some time just going over the basiscs 🙂 thanks a lot!
d

datajoely

03/02/2023, 1:04 PM
I also have a video that takes you through some of this, a little outdated but hopefully useful

https://www.youtube.com/watch?v=NU7LmDZGb6E

🙏 1
5 Views