I'm noticing that warnings like SyntaxErrors and type errors are considered "warnings" by Kedro, which continues to try to run the pipeline. Is there a setting to escalate these to Errors so they can abort the pipeline run?

SyntaxErrors should be terminating, could you share your logs?

Okay I know what’s going on • There is a syntax error in one of the pipelines that get’s imported automatically • Kedro ignores this pipeline entirely, but will continue with the others that it’s able to import • Essentially the syntax error happens at import time, not execution time so won’t kill the program

@datajoely explanation is on point. To add a bit of context as to why it's implemented this way: When we introduced pipeline autodiscovery (via

find_pipelines()

), we wanted people to be able to use it by default. However, lot of users may have work-in-progress pipelines, and we didn't want to prevent them from using autodiscovery/running other pipelines just because they're still trying to get some pipeline working.

Is there a way to mark a pipeline as a work in progress and then tell

find_pipelines

to error on warnings? Or is the answer to just not use pipeline autodiscovery

good question - I’m not sure

Is there a way to mark a pipeline as a work in progress and then tell

find_pipelines

to error on warnings? Or is the answer to just not use pipeline autodiscovery

No, there's no such built-in functionality. If you think it makes sense to have

find_pipelines

raise an error if a discovered pipeline has issues, I think that could be considered--feel free to raise an issue. I don't know how many people would use it, but IMO there's little cost to adding a flag to

find_pipelines

, like

find_pipelines(raise_errors=False)

. If you think it makes sense to have a mechanism to mark a WIP pipeline and for

find_pipelines

to raise an error on a non-WIP pipeline... I mean, feel free to raise an issue, too. However, I think this is a broader change (how do you mark a pipeline as WIP? should a WIP pipeline be excluded from autodiscovery, or only excluded if it fails?), and I'm not sure how useful it would be unless see a lot of demand for it. For now, you can easily use a modified

find_pipelines

if your organization/use case needs it;

find_pipelines

is a function defined here: https://github.com/kedro-org/kedro/blob/0.18.11/kedro/framework/project/__init__.py#L302-L369 so you can create a modified function for now if there's significant benefit to doing so

What’s the reason that we can’t make

raise_error=True

a default?

When implementing this functionality, we felt that it would be frustrating for users who wish to be able to work on a new pipeline, and then can no longer run the existing pipelines because the new one is incomplete.

Ah! that rings a bell, thanks