is there a way to use same node with different set...
# questionss
Sergei Benkovich
03/02/2023, 12:48 PMis there a way to use same node with different sets of input/outputs?
my_node(a) -> b
my_node(c) -> d
i think i saw something related in the documentation but i think it looked too complicated , any easy way to do it?
r
Ricardo Araújo
03/02/2023, 12:49 PMI think you need to do this on the level of the pipeline. That would be a modular pipeline, and then you can remap inputs and outputs. See the last example in https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html
🙏 1
kedroid 1
d
datajoely
03/02/2023, 12:50 PMYes exactly @Ricardo Araújo is on the money 💵 !
r
Ricardo Araújo
03/02/2023, 12:50 PMOther than that, you could wrap the same function in different nodes with different inputs/outputs. Not as clean, though.
d
datajoely
03/02/2023, 12:51 PMspecifically look for the
Using a modular pipeline multiple timess
Sergei Benkovich
03/02/2023, 12:52 PMyeah i think i saw it and i avoided trying it out because i have issues with the
“params:override_me”
no idea why…
i do pass “params”, but using ‘:’ is not working and i didn’t figure out why
d
datajoely
03/02/2023, 12:53 PM Copy code
template_pipeline = pipeline(
[
node(
func=node_func1,
inputs=["input1", "input2", "params:override_me"],
outputs="intermediary_output",
),
node(
func=node_func2,
inputs="intermediary_output",
outputs="output",
),
]
)
alpha_pipeline = pipeline(
pipe=template_pipeline,
inputs={"input1", "input2"},
parameters={"params:override_me": "params:alpha"},
namespace="alpha",
)
beta_pipeline = pipeline(
pipe=template_pipeline,
inputs={"input1", "input2"},
parameters={"params:override_me": "params:beta"},
namespace="beta",
)
final_pipeline = alpha_pipeline + beta_pipelinedatajoely
03/02/2023, 12:53 PMfrom the example in there
datajoely
03/02/2023, 12:54 PMoh I see you’re saying that’s not working
datajoely
03/02/2023, 12:54 PMwhat is the key you’re looking for in your override, is it nested with
.s
Sergei Benkovich
03/02/2023, 12:55 PMi don’t have an override example written yet,
but when i tried passing specific parameters using : i got an error.
d
datajoely
03/02/2023, 12:55 PMwhat was the error?
s
Sergei Benkovich
03/02/2023, 12:56 PMValueError: Pipeline input(s) {‘params:data_extraction_params’} not found in the DataCatalog
which is inside a yml inside the parameters directory
Sergei Benkovich
03/02/2023, 12:57 PMd
datajoely
03/02/2023, 12:58 PMhave you added a namespace?
s
Sergei Benkovich
03/02/2023, 12:58 PMmy usage in main and the settings file
Sergei Benkovich
03/02/2023, 12:59 PMcan you explain about the namespace?
d
datajoely
03/02/2023, 12:59 PMso this is quite an unusual way of using Kedro, is there a reason you’re not using the standard template and workflow?
datajoely
03/02/2023, 1:00 PMit’s possible to get this working, but it’s much harder to support
s
Sergei Benkovich
03/02/2023, 1:00 PMhm i’m not sure i understand the differences between this and the standart template and workflow 🙂
d
datajoely
03/02/2023, 1:01 PMif you have time- I’d really encourage you to follow the spaceflights tutorial it even covers this part under this section
👍 2
datajoely
03/02/2023, 1:01 PMit should take you through the expected usage pattern
s
Sergei Benkovich
03/02/2023, 1:04 PMi will do that…i feel like i can optimize my kedro usage…. and i need to invest some time just going over the basiscs 🙂
thanks a lot!
46 Views