Hello, I've worked on a lot of Kedro pipelines the past ~year and am a big fan, but there's one detail that I've seen cause some very confusing problems that id like help with. Whenever there's an error loading a pipeline, whether it be a syntax error, missing import, etc... instead of the Kedro process erroring out, it will just not use that pipeline and continue on without that pipeline. This is not only confusing, but can lead to some pretty big problems in a model without any errors occurring. I was wondering how I disable this, forcing Kedro to raise errors when loading pipelines? I tried googling but couldn't find anything. Thanks!

Someone will reply to you shortly. In the meantime, this might help:

https://docs.kedro.org/en/stable/nodes_and_pipelines/pipeline_registry.html#pipeline-autodiscovery specify

raise_errors=True

thank you! Will this also work with loading custom datasets?

Think this would be just for the pipeline code; I don't recall exactly when errors in dataset code show up.

Sounds good / no rush... the dataset one is luckily easier to deal with. I also realized I phrased the question poorly. If I have an error in a custom dataset (like missing import for example) it will error out, but instead it will say "dataset.my_dataset.FooBar" not found, install kedro-datasets This is definitely a better default behavior than the pipeline one I mentioned, and I know how to deal with it. I tell the rest of my team to try importing the dataset via the python interpreter to find the real error... All that being said, IMO it would be better if kedro would raise the actual error inside a dataset

If I have an error in a custom dataset (like missing import for example) it will error out, but instead it will say "dataset.my_dataset.FooBar" not found, install kedro-datasets

Hmm... I would think the error in custom dataset should be better/disambiguated, just looking at https://github.com/kedro-org/kedro/pull/3272#issuecomment-1798215139.

I guess the pipeline part is solved. For dataset part we tried to improve this in the past. Afaik there is still an open ticket to further improve this. But you are right, directly importing it will reveal the issue better. Maybe it helps to explain why this happens.

interesting... that explanation clears up some other weird issues i've had with custom dataset errors. I don't have any strong opinions or suggestions on how to solve this dataset import error issue, but I'll ponder it.