Hi team, I'm trying to set up my modular pipelines to process new data, but I'm having problems where if one of them has no new data it crashes, and then the other pipeline (which does have new data) doesn't work either So is there a way where I can set it so that if the pipeline / node realises it has no data, it can skip the rest of the nodes without erroring?

So I have seen a pattern where a custom dataset is used to create a zero row dataframe if the data is not present. That way the pipeline ‘runs’ but without any issue. Since Kedro is built with reproducibility in mind you are pushing Kedro a little outside of its comfort zone with this overall problem, but it is solveable

Interesting okay thank you. I think I'll need a similar workaround because it doesn't seem like there's a control / setting on kedro for this