Hi everybody, i'm not new to kedro, but i'm new to using kedro, pyspark and databricks at the same time. The logs appear after all the jobs have been completed, is it there a way to see the logs as they occur? I think is more a databricks question Thanks in advance!

First of all, you can use the Ipython extension in notebook which is the preferred way of running Kedro project on managed Jupyter instance https://kedro.readthedocs.io/en/stable/notebooks_and_ipython/kedro_and_notebooks.html

hello @Nok Lam Chan I followed the steps on https://docs.kedro.org/en/stable/deployment/databricks.html the logs appear after the whole excecution is completed. i'm running the same project right now. no logs so far

I just test it and I couldn’t reproduce it

hi @Nok Lam Chan, in the meantime I took databricks out of the equation, ran the project directly on spark, in different configurations, and I got the logs as the project was executing, so... it's not kedro nor spark. Now I think it's something on how my company configured databricks. Thanks for taking your time on my obscure problem 🙂

You are welcome! free feel to report any issues u have.