Situation: • Kedro (0.18.8), Python (3.8.16), Databricks (9.1 LTS) • We are running

%load_ext kedro.ipython

and

%reload_kedro

to run our Kedro pipeline in Databricks environment successfully Complication: • When faced a

raise Exception

in the code, the Kedro pipeline fails, but doesn’t fail the Databricks job — causing the error to be un-raised (and users not knowing the job actually fail) • It seems to be the

%load_ext kedro.ipython

magic function which is causing the Exception to be suppressed. Before this function we are able to Raise directly in notebook and it fails the job/but afterwards it gets suppressed. Help needed: • Where should we look in the Kedro codebase in order to debug this? Anything that screams out where Exceptions might be getting suppressed?

thanks for reporting @Sohum Sachathamakul and sorry you're having trouble with this. I think we have enough information to reproduce the issue. we'll investigate a bit and get back to you.

Is this related? https://github.com/jupyter-widgets/ipywidgets/issues/3208

thanks @Kyle Chung! I'm not sure

kedro.ipython

is injecting

ipywidgets

, but will keep this in mind. @Sohum Sachathamakul and I are debugging, will report back when we know what caused this issue

Update:

pip install git+<https://github.com/kedro-org/kedro.git@noklam/tempfix-databricks>

This branch seems to fix the issue. This fork branch suppress the use of

rich.pretty.install

I suspect the issue is the older Databricks runtime 9.0 has some strange behavior because this cannot be reproduced on the current databricks version (and it should affect more people if this is a generic issue)

yep

Supplementing the source, https://docs.databricks.com/en/notebooks/ipython-kernel.html In databricks doc it indicates databricks runtime >=10 is using ipython kernel which is how rich interfering. I don’t fully understand what happened here but most likely rich is interfering it in different way. Cc @Antony Milne

It means you no longer need to do this! https://github.com/Textualize/rich/issues/2422

Yeah, I have a separate PR to make the syshook install reversible, but it seem like their team are busy working on something else so that PR has been hanging for a while.

@Nok Lam Chan thank you so much for creating a fork which did indeed fix the problem. May I check you for any short-term solution (while this goes through the proper routes to hotfix 🙂). I can think of the following: • Override the logging (?) somehow in my Kedro project — this is preferred b/c no environment change and minimal code change • Keep on increasing Databricks version until the issues is resolved

If possible, bumping databricks version should work. Alternative, if it’s acceptable you can mock the rich function so that it doesn’t do anything.

from unittest.mock import patch
patch("rich.pretty.install")

You can include this at the top of notebook.

hmm or rather

patch("kedro.logging.rich.pretty.install")

? (always confusing how mock works)

both should work, tho

kedro.logging

is introduce in some 0.18.x series, so I am not sure is this the correct namespace. The difference is

patch("kedro.logging.rich.pretty.install")

only affect the scope of Kedro, while

patch("rich.pretty.install")

affect any use of rich.pretty.install

Hey @Nok Lam Chan after bumping up the Databricks version to 11.3 it didn’t resolve the issue & also I tried to put both

patch("kedro.logging.rich.pretty.install")

and

patch("rich.pretty.install")

at the top of the notebook, didn’t work too 😢

That's strange... Did the old solution (the fork branch) still work?

Hi guys! i think i have a related issue. Running a kedro project version 0.18.0, and getting the following error:

ValueError: Cannot resolve 'kedro.logging.RichHandler': No module named 'kedro.logging'

below i can see another error:

ValueError: Unable to configure handler 'rich'

I think it's a version issue

ugh apparently it was running in 0.18.10 and now because of MLFlow plug in changed to 0.18.0 (@Elena Mironova fyi)

Glad you found a quick fix! :)