Hi everyone. I am trying to use the modular pipeli...
# questions
t
Hi everyone. I am trying to use the modular pipeline module but I am getting the error
ModularPipelineError: Failed to map datasets and/or parameters:
and a list of datasets that do exist on the catalog. This is the code:
Copy code
from kedro.pipeline import Pipeline, node
from kedro.pipeline.modular_pipeline import pipeline
from .nodes import select_columns


def create_pipeline(**kwargs) -> Pipeline:
    nominal_template = pipeline(
        [
            node(
                func=select_columns,
                inputs=["nominal_raw_data_normalized", "params:columns"],
                outputs="nominal_raw_data_features",
                name="extracting_columns_nominal_data",
                namespace="data_preprocessing"
            )
        ]
    )

    faulty_template = pipeline(
        [
            node(
                func=select_columns,
                inputs=[f"fault_{i}_raw_data_normalized", "params:columns"],
                outputs=f"fault_{i}_raw_data_features",
                name=f"extracting_columns_fault_{i}",
                namespace="data_preprocessing"
            ) for i in range(1, 29)
        ]
    )


    reactor_nominal = pipeline(
        pipe=nominal_template,
        inputs={f"fault_{i}_raw_data_normalized" for i in range(1, 29)},
        parameters={"params:columns": "params:reactor_columns"},
        namespace="reactor"
    )

    reactor_faulty = pipeline(
        pipe=faulty_template,
        inputs={f"fault_{i}_raw_data_normalized" for i in range(1, 29)},
        parameters={"params:columns": "params:reactor_columns"},
        namespace="reactors"
    )

    reactor = reactor_nominal + reactor_faulty



    return reactor
Any ideas on what is the error? Maybe I am not using the module correctly thanks in advance 🙂
j
hello @Tomás Rojas! what version of Kedro are you using?
I tried to reproduce this locally and found an interesting behavior:
kedro run
might include names in that error message of datasets that do exist in the catalog if there's a misnamed dataset in any of the subpipelines. for example, taking the tutorial: https://docs.kedro.org/en/stable/tutorial/add_another_pipeline.html#optional-extend-the-project-with-namespacing-and-a-modular-pipeline and changing in the first node the inputs to
inputs=["model_input_table_NOT_FOUND", "params:model_options"],
, will raise this error:
Copy code
ModularPipelineError: Failed to map datasets and/or parameters: model_input_table
so, have a look at your nodes inputs @Tomás Rojas, the error might be there.
t
wow in that case that might be a bug(?)
thanks for responding, at the end I changed my approach but it would be useful to do it in the proper way
kedro, version 0.18.8
j
it’s a misleading error message but if it’s raised, the pipeline still has a problem nonetheless