Hi all, I would like to know how can I control the...
# questions
s
Hi all, I would like to know how can I control the pipelines run order? I have 3 pipelines and would like to run sequence like r_data_processing -> r_regression_training -> r_local_predict. Since r_local_predict is using same dataset as input like r_data_processing pipeline, and I simply use image and model path. So... although there is a dependency, but from kedro it looks independent. If I run like this: (in conf/hpc_run_conf.yml)
Copy code
run: 
   tags: r_data_processing, r_regression_training, r_local_predict
kedro run --config=conf/hpc_run_conf.yml
The pipeline r_local_predict would run before re_regression_training. I have checked
before_pipeline_run
, if I get it right, it's more like adding additional action before the node. But I just want to change the pipeline run order. Do you have more easier way? Or how can I add
before_pipeline_run
into my current code?
d
In your pipeline registry file, define a new named key like “full” and then add the pipelines together with the
+
operator
y
Additionally to @datajoely suggestion, you may also need to use a namespace to rename the input for r_local_predict to make sure it runs after
And I also suggest you look at https://github.com/Galileo-Galilei/kedro-mlflow-tutorial for wider thoughts about this workflow, and how you can solve it with kedro-mlflow