Kedro provides hooks for

on_node_error

and

on_pipeline_error

but neither of these appear to be triggered if there is a data load error. Is there currently any way to catch these errors gracefully, or should we raise a feature request for

on_catalog_error

or similar?

depending on what handling you want to do, if it's something specific to one type of dataset you could do something in the

_load

method, right? catch the error before the outer

load

raises the

dataseterror

if it's general dataset load error handling, i will keep an eye on the thread in case somebody has any ideas 🤣

Yeah, looking for a way to do this for all possible datasets

Just checking the source, what happens under the hood is roughly:

_run_node_sequential()
  load_data()
  _call_node_run()
    try:
       node.run(inputs) # data already loaded earlier
    except:
       on_node_error

I suspect it should try to catch this in outer function, probably even earlier in

run_node

Digging this one up, @Nok Lam Chan is there an equivalent for save error? Context, I have a pipeline that uses lazy saving with some complex logic happening in the callable, and am having trouble getting the debugging to work.

There isn't any specific hook for save/load. What error did you get? Seeing this old thread I think there are no particular issue tracking this, feel free to raise one