Hi guys, Are we able to capture the lineage information of the job in kedro databricks. I mean is it possible to view the input and output of each nodes in databricks job? #databricks @Jens Peder Meldgaard

Hey @jagadeesan! Kedro has lineage tracking built into its design every node explicitly declares its inputs and outputs, so the full data flow is always traceable. For viewing that lineage, there are a couple of options depending on what you need: Kedro-Viz: gives you a visual DAG of your entire pipeline showing the inputs and outputs of each node. If you need something more specific like inspecting actual data values flowing between nodes at runtime you could also add custom hooks to log intermediate outputs.

Hey @jagadeesan, May ask how you would expect to see the lineage information in Databricks? I strive to make it so that a Databricks job shows the same graph as that in kedro-viz, and that graph is generated based on input/output.

Hi @Jens Peder Meldgaard, sorry for very late response, slack notification seems to be no enabled. Regarding the question I am looking for lineage info of upstream and downstream/ output tables of each tasks similar to azure ml pipelines, but in my kedro job i could only see the upstream table which is first table read by my workflow and no downstream table info in job level is been captured

I'm not entirely sure I understand... Do you mean in the Databricks job?