Hi, Currently I built a modular pipeline that is b...
# questionss
Santiago Rua
09/27/2023, 8:05 PMHi, Currently I built a modular pipeline that is being reuse for different inputs. I was wondering whether in the same pipeline I need to add some fixed nodes at the end if it is possible? So I need to add one node to the end of data_pipeline1 and another different to data_pipeline2, should I create another different pipeline to couple with the outputs of these? Or can I add some fixed nodes in this same file?
Copy code
from kedro.pipeline.modular_pipeline import pipeline
def create_pipeline(**kwargs) -> Pipeline:
template_pipeline = pipeline(
[
node(
func=preprocess_json,
inputs="input1",
outputs="intermediary_output",
),
node(
func=expand_column_df,
inputs=["intermediary_output", "params:override_me"],
outputs="output",
),
]
)
data_pipeline1 = pipeline(
pipe=template_pipeline,
inputs={"input1":"df1"},
parameters={"params:override_me": "params:df1_param"},
outputs={"output":"df1_pos"},
namespace="df1_pos",
)
data_pipeline2 = pipeline(
pipe=template_pipeline,
inputs={"input1":"df2"},
parameters={"params:override_me": "params:df2_param"},
outputs={"output":"df2_pos"},
namespace="df2_pos",
)
final_pipeline = data_pipeline1 + data_pipeline2
return final_pipelinen
Nok Lam Chan
09/27/2023, 9:21 PMWhat do you mean by fix node?
Nok Lam Chan
09/27/2023, 9:22 PMSo i am guessing you have a few pipelines which are parallel branch and you have a final node to connect to these branches?
s
Santiago Rua
09/27/2023, 9:22 PMAssume that I want to add a new node at the end of each pipeline. This node can not belong to the modular pipeline since is going to have different functionality.
Santiago Rua
09/27/2023, 9:25 PMI just want to connect a new node called node_1 at the end of data_pipeline1 and a second node called node_2 at the end of data_pipeline2. These two nodes are different in functionality, that’s why I didn’t include in the template_pipeline. To do these operation what is the best approach?
n
Nok Lam Chan
09/27/2023, 9:26 PMThat's fine, your pipeline doesn't have to be a pure modular pipeline
Nok Lam Chan
09/27/2023, 9:27 PMYour template pipeline is fine here, then for your actual pipeline it would be equal to your modular pipeline + an extra node.
Nok Lam Chan
09/27/2023, 9:30 PMTake your example, you will end up having 3 pipelines. Take
pms
Santiago Rua
09/27/2023, 9:32 PMit makes sense. To create m1_fix_node and m2_fix_node, can i just create the node and added to the pipeline? add operation work between nodes and pipeline?
n
Nok Lam Chan
09/27/2023, 9:33 PMA modular pipeline is essentially a “high-level” node. Conceptually it’s something like this.
Nok Lam Chan
09/27/2023, 9:33 PMYes, nodes and pipeline can be
+-Nok Lam Chan
09/27/2023, 9:33 PMunder the hood it all happens at a “node” level
s
Santiago Rua
09/27/2023, 9:33 PMWow 😃
Santiago Rua
09/27/2023, 9:34 PMThanks @Nok Lam Chan
n
Nok Lam Chan
09/27/2023, 9:34 PMI mean, you can’t directly add a node to a pipeline, but you could wrap it as a single node pipeline. i.e.
pipeline([node])s
Santiago Rua
09/27/2023, 9:36 PMOk, just to recap, I create this two new nodes, wrap it into a a pipeline, and just added like you mentioned before
n
Nok Lam Chan
09/27/2023, 9:38 PMCorrect. If you look at the starter. For example you can do
kedro new -s spaceflightspipeline_registry.pysum(pipelines)🙌 1