https://kedro.org/ logo
#questions
Title
# questions
j

Javier del Villar

03/21/2023, 7:35 PM
Hi everybody, i'm not new to kedro, but i'm new to using kedro, pyspark and databricks at the same time. The logs appear after all the jobs have been completed, is it there a way to see the logs as they occur? I think is more a databricks question Thanks in advance!
n

Nok Lam Chan

03/21/2023, 9:37 PM
First of all, you can use the Ipython extension in notebook which is the preferred way of running Kedro project on managed Jupyter instance https://kedro.readthedocs.io/en/stable/notebooks_and_ipython/kedro_and_notebooks.html
There should be log output for every nodes? Or this is not the case
j

Javier del Villar

03/22/2023, 6:03 PM
hello @Nok Lam Chan I followed the steps on https://docs.kedro.org/en/stable/deployment/databricks.html the logs appear after the whole excecution is completed. i'm running the same project right now. no logs so far
n

Nok Lam Chan

03/23/2023, 1:29 PM
I just test it and I couldn’t reproduce it
j

Javier del Villar

03/23/2023, 4:27 PM
hi @Nok Lam Chan, in the meantime I took databricks out of the equation, ran the project directly on spark, in different configurations, and I got the logs as the project was executing, so... it's not kedro nor spark. Now I think it's something on how my company configured databricks. Thanks for taking your time on my obscure problem 🙂
👍🏼 1
n

Nok Lam Chan

03/23/2023, 5:33 PM
You are welcome! free feel to report any issues u have.