https://kedro.org/ logo
#questions
Title
# questions
b

Ben Horsburgh

02/22/2024, 9:25 AM
Kedro provides hooks for
on_node_error
and
on_pipeline_error
but neither of these appear to be triggered if there is a data load error. Is there currently any way to catch these errors gracefully, or should we raise a feature request for
on_catalog_error
or similar?
👀 2
i

Iñigo Hidalgo

02/22/2024, 9:47 AM
depending on what handling you want to do, if it's something specific to one type of dataset you could do something in the
_load
method, right? catch the error before the outer
load
raises the
dataseterror
if it's general dataset load error handling, i will keep an eye on the thread in case somebody has any ideas 🤣
b

Ben Horsburgh

02/22/2024, 9:56 AM
Yeah, looking for a way to do this for all possible datasets
👀 1
n

Nok Lam Chan

02/22/2024, 10:30 PM
Just checking the source, what happens under the hood is roughly:
Copy code
_run_node_sequential()
  load_data()
  _call_node_run()
    try:
       node.run(inputs) # data already loaded earlier
    except:
       on_node_error
I suspect it should try to catch this in outer function, probably even earlier in
run_node
feel free to raise an issue, I don't think there is an existing hook catching this, slightly surprised that
on_pipeline_error
doesn't work too, though admittedly I have never seen this error before, likely room for improvement
p.s. Just tried,
on_pipeline_error
get triggered, though I still think
on_node_error
doesn't quite capture what one expect during a node run
5 Views