https://kedro.org/ logo
#questions
Title
# questions
y

Yifan

12/06/2022, 10:40 AM
Hey everyone! I would like to know if there is a tool or Kedro module capable of profiling each node in a pipeline? Basically I want to analyse the execution time of each node (from loading the first input dataset to the end of saving the last chunk of output to the storage) in my pipeline, and I am aware of the possibility of using log files. However, for a pipeline with hundreds of nodes, manually analysing the log files is almost impossible. Do you have any suggestions? Thank you!
d

datajoely

12/06/2022, 10:54 AM
we have a tutorial on how to do this here! https://kedro.readthedocs.io/en/stable/hooks/examples.html
👍 1
n

Nok Lam Chan

12/12/2022, 9:13 AM
https://github.com/orgs/kedro-org/projects/3 This is also on our roadmap to have native support, still early tho!
3 Views