Yury Fedotov
08/16/2023, 6:57 PMParallelRunner
instead of the default SequentialRunner
.
Are those conclusions correct?
1. Worst case scenario, ParallelRunner
would just yield same speed as SequentialRunner
. It can't produce different results and manages the execution order in a way that if some node expects outputs from a few nodes, it would wait until them all get generated.
2. ParallelRunner
shines when a pipeline does many similar operations on some already-available input, and it's just a matter of compute time to do each of those operations. In other words, those operations do not sequentially depend on each other. Likely, a pipeline consisting of a few namespace pipelines is a good candidate for that runner.
And a question:
3. When would you avoid using ParallelRunner
?datajoely
08/17/2023, 9:00 AM