Hi Team, I’m currently working with the project using VSCode and Databricks Repos to develop Kedro project. Is there any way to show log in Databricks for running pipelines/nodes execution such as time spending to complete specific node, memory usage, or cost in graphical manner like the image below? I’d still appreciate to know the way to show the log with text if there weren’t way to show them graphically.

Hi @Sen,is the question about profiling? There is nothing specific about logging here, if you can see the log locally, you should be able to do the same on Databricks, make sure you read the custom logging file with

KEDRO_LOGGING_CONFIG

@Nok Lam Chan Thank you for your reply! And also thanks for the document, I'll take a look at it. The graph is from SSMS, but I'm not intend to extract this graph specifically , rather I just put the graph image to give a picture of what I imagine. I was just wondering there are any ways to visualize pipeline execution results/logs in graphical manner like for example, this node takes this much time to complete, etc.

Make sense, this has been asked before. We don't have have this yet, pinging our PM @Juan Luis

hi @Sen! doing so requires you picking an integration with some of the supported orchestrators, for example Airflow or Prefect. we are also considering exporting OpenLineage data that can be consumed by different frontends https://github.com/kedro-org/kedro/discussions/4054