Hi all, I would like to disable some pipelines or tags or (some runs) from MLFLOW tracking. I tried to add below into mlflow.yml file, but if I still see it tracked on MLFLOW. On the other hand, how can I make sure the running sequence of pipelines? If I run like this, it wouldn't follow the order I wrote.

run:
  tags: r_data_processing, r_regression_training, r_local_predict

disable_tracking: pipelines: - segmentation_evaluate - segmentation_preprocessing

Hello! For those tags do you have input/output dependencies? To enforce the order

Hi @Shu-Chun Wu, The pipeline run order is determined by the dependencies (inputs) as @Emilio Vega mentioned. Kedro does a topological sort to determine the order

yes, my pipelines have the dependency. and yes, I use kedro-mlflow plugin to create mlflow.yml (if this is what you mean) except disable certain pipeline tracking, could I disable on tag level?

I think you should be able to do it. I haven't tried it but let's say the

before_pipeline_run

spec has

before_pipeline_run(run_params, pipeline, catalog)

where the pipeline object contains tags information

pipeline.tags

. Now you can have a custom logic to disable mlflow tracing if certain tag is present. I am not sure about the part on how you will get hold of the mlflow object as I haven't worked on it. But you can get hold of pipeline object, tags and add additional behavior before a pipeline is run using the hook mentioned here.

@Ravi Kumar Pilla Can you give me an example? I have 3 pipelines and would like to run sequence like r_data_processing -> r_regression_training -> r_local_predict. Since r_local_predict is using same dataset as input like r_data_processing pipeline. So it won't run in the end, if I run like this: (in conf/hpc_run_conf.yml)

run: 
   tags: r_data_processing, r_regression_training, r_local_predict

kedro run --config=conf/hpc_run_conf.yml