Hello, what is the recommended way to run multiple...
# questions
Hello, what is the recommended way to run multiple pipelines sequentially? An obvious approach is just with multiple
kedro run --pipeline <pipeline_i>
commands defined inside a shell script, but i'm wondering if there is a better way perhaps using the run config.yml capability?
Are the pipelines completely independent? You could tie them together by made-up outputs at the end of each one that serves as input to the next one. Then running
kedro run
would run them in the right sequence. But I fail to understand why you need a specific ordering if they are independent.
If they are not independent, then kedro should take care of running them in the right order.
They are independent. I think your idea of assembling a a single pipeline with made-up in/outputs is good. Thank you.
hmm, I was looking for a solution for this as well. In my case, I have three pipelines and both p2 p3 are depending on p1 but p2 is independent to p3.
I tried kedro run --pipeline=p1, p2 and obviously it didn't work but that's what I expected it could
Won’t running just p2 work?
p2 depends on p1, need to run p1 then p2 that's the point
In that case, you can add tags to the pipeline, tagging p1 and p2 with the same tag, and then
kedro run --tag=pipe_tag
Or maybe you could use
, specifying the end node in p2. That should trigger p1 as well. I think.
👍 1
I agree it should work with
. The notation exists for nodes.
👍 1