https://kedro.org/ logo
#questions
Title
# questions
e

ed johnson

03/10/2023, 9:47 PM
Hello, what is the recommended way to run multiple pipelines sequentially? An obvious approach is just with multiple
kedro run --pipeline <pipeline_i>
commands defined inside a shell script, but i'm wondering if there is a better way perhaps using the run config.yml capability?
r

Ricardo Araújo

03/10/2023, 10:31 PM
Are the pipelines completely independent? You could tie them together by made-up outputs at the end of each one that serves as input to the next one. Then running
kedro run
would run them in the right sequence. But I fail to understand why you need a specific ordering if they are independent.
If they are not independent, then kedro should take care of running them in the right order.
e

ed johnson

03/11/2023, 12:53 AM
They are independent. I think your idea of assembling a a single pipeline with made-up in/outputs is good. Thank you.
d

Dustin

03/14/2023, 3:15 AM
hmm, I was looking for a solution for this as well. In my case, I have three pipelines and both p2 p3 are depending on p1 but p2 is independent to p3.
I tried kedro run --pipeline=p1, p2 and obviously it didn't work but that's what I expected it could
r

Ricardo Araújo

03/14/2023, 1:24 PM
Won’t running just p2 work?
d

Dustin

03/14/2023, 10:24 PM
p2 depends on p1, need to run p1 then p2 that's the point
r

Ricardo Araújo

03/15/2023, 6:43 PM
In that case, you can add tags to the pipeline, tagging p1 and p2 with the same tag, and then
kedro run --tag=pipe_tag
Or maybe you could use
--to_nodes
, specifying the end node in p2. That should trigger p1 as well. I think.
👍 1
I agree it should work with
--pipeline=p1,p2
. The notation exists for nodes.
👍 1