Santiago Rua
09/27/2023, 8:05 PMfrom kedro.pipeline.modular_pipeline import pipeline
def create_pipeline(**kwargs) -> Pipeline:
template_pipeline = pipeline(
[
node(
func=preprocess_json,
inputs="input1",
outputs="intermediary_output",
),
node(
func=expand_column_df,
inputs=["intermediary_output", "params:override_me"],
outputs="output",
),
]
)
data_pipeline1 = pipeline(
pipe=template_pipeline,
inputs={"input1":"df1"},
parameters={"params:override_me": "params:df1_param"},
outputs={"output":"df1_pos"},
namespace="df1_pos",
)
data_pipeline2 = pipeline(
pipe=template_pipeline,
inputs={"input1":"df2"},
parameters={"params:override_me": "params:df2_param"},
outputs={"output":"df2_pos"},
namespace="df2_pos",
)
final_pipeline = data_pipeline1 + data_pipeline2
return final_pipeline
Nok Lam Chan
09/27/2023, 9:21 PMSantiago Rua
09/27/2023, 9:22 PMNok Lam Chan
09/27/2023, 9:26 PMp
as pipeline and m
as modular pipeline.
p1 = m1 + m1_fix_node
p2 = m2 + m2_fix_node
p_all = p1 + p2Santiago Rua
09/27/2023, 9:32 PMNok Lam Chan
09/27/2023, 9:33 PM+
or even -
(more surprisingly)Santiago Rua
09/27/2023, 9:33 PMNok Lam Chan
09/27/2023, 9:34 PMpipeline([node])
Santiago Rua
09/27/2023, 9:36 PMNok Lam Chan
09/27/2023, 9:38 PMkedro new -s spaceflights
, you will find that in pipeline_registry.py
there is a sum(pipelines)
operation.
https://github.com/kedro-org/kedro-starters/blob/8ac843863c64b98df17bdefed87ec0569[…]%7B%20cookiecutter.python_package%20%7D%7D/pipeline_registry.py