Hi All,
In developing modular pipelines the kedro-viz tool is quite indispensable, without the viz its really quite hard to see whether inputs, outputs and parameters are connected properly.
However, the most obvious workflow for an established project to develop a new pipeline/refactor a pipeline based on the documentation, requires several steps which could be streamlined.
# The issues:
1. To have more control over over the interactions between the pipelines you want to work on, one can adjust the pipeline_registry to only return the pipelines you want. Which renders the other pipelines unusable (and could lead to bugs down the road). (this can be partly solved by filtering the pipelines in the kedro viz CLI command)
2. For every change you want to visualise, youโll need to:
a. save the file
b. stop the current kedro viz
c. run kedro viz
d. go to the browser window to view the pipeline (which is quite annoying if one has only one monitor at their disposal)
3. Lastly, this does not make for easy debugging/viewing the python objects in the pipelines/nodes.
# Half-way solution:
using run_viz in a jupyter notebook is great, and solves some issues. I personally combine it with nbdev which allows me to convert the notebook to a python file, and then do run_viz on the entire project.
This still suffers from issue 1 even more, but issue 2 and 3 are drastically reduced. Mostly because i just save/commit and then rerun the notebook and get the viz in the notebook.
# Better solution
Running the
run_viz
command almost begs for the ability to do
run_viz(pipeline)
. Where pipeline is an actual pipeline object. (although it would also be nice as to be able to pass the name of a pipeline to filter like the CLI command w.r.t issue 1).
This way, one doesnโt need nbdev (which is a slightly controversial tool), one can develop pipelines easier, without any adjustment to the original project.
Since kedro viz can already filter, i can imagine such changes being possible.
Also, debugging kedro pipelines from the vscode notebook cell debugger is actually quite nice (i would arque a lot nicer than using the debug configs).
Has anyone faced similar issues, or thought of a different solution?