Hi, is there an elegant way to skip nodes in a pipeline if there is already an intermediate dataset? Currently I use a node for checking, but it doesn't seem to be the best way:

node(
                func=nd.check_intermediate,
                inputs="adatasets#{params:experiment}",
                outputs="intermediate_exists",
                name="check_intermediate",
            ),

What do you mean by "if there is already an intermediate dataset"? What is your use case / what you are trying to do?

I would like to skip nodes of a pipeline, or a subpipeline partially, if a certain intermediate result is already saved in the same version. I have found different approaches, but it feels like I am missing a core feature of Kedro.

How do you check for the existence right now?

load parameters and catalog -> pathlib check if filepath exists -> run part of pipeline or run it complete

from kedro.config import OmegaConfigLoader
from kedro.pipeline import Pipeline, pipeline

def data_exists(project_name: str = "fresh", search_var: str = "experiment") -> bool:
    conf_path = str(Path("../../conf/base").absolute())
    conf_loader = OmegaConfigLoader(conf_source=conf_path)
    experiment = conf_loader["parameters"][search_var]
    for ds_name, ds in conf_loader["catalog"].items():
        if search_var in ds_name and project_name in ds_name:
            filepath = Path(ds["filepath"].replace("{experiment}", experiment))
            return filepath.exists()


def create_pipeline(**kwargs: dict) -> Pipeline:
    data_pipe = pipeline([...])
    feature_pipe = pipeline([...])
    if data_exists():
        return feature_pipe
    return data_pipe + feature_pipe

🙈

I have already seen some of them, i will have look again.