Hello, what is the recommended way to run multiple...
# questions
e
Hello, what is the recommended way to run multiple pipelines sequentially? An obvious approach is just with multiple
kedro run --pipeline <pipeline_i>
commands defined inside a shell script, but i'm wondering if there is a better way perhaps using the run config.yml capability?
r
Are the pipelines completely independent? You could tie them together by made-up outputs at the end of each one that serves as input to the next one. Then running
kedro run
would run them in the right sequence. But I fail to understand why you need a specific ordering if they are independent.
If they are not independent, then kedro should take care of running them in the right order.
e
They are independent. I think your idea of assembling a a single pipeline with made-up in/outputs is good. Thank you.
d
hmm, I was looking for a solution for this as well. In my case, I have three pipelines and both p2 p3 are depending on p1 but p2 is independent to p3.
I tried kedro run --pipeline=p1, p2 and obviously it didn't work but that's what I expected it could
r
Won’t running just p2 work?
d
p2 depends on p1, need to run p1 then p2 that's the point
r
In that case, you can add tags to the pipeline, tagging p1 and p2 with the same tag, and then
kedro run --tag=pipe_tag
Or maybe you could use
--to_nodes
, specifying the end node in p2. That should trigger p1 as well. I think.
👍 1
I agree it should work with
--pipeline=p1,p2
. The notation exists for nodes.
👍 1