Hi all, does anyone have resources or tips on integrating MLflow with Kedro, with dynamic pipeline structure? Any guidance or shared experiences would be greatly appreciated! I follow this tutorial to create dynamic pipeline : https://getindata.com/blog/kedro-dynamic-pipelines/

Did you look at kedro-mlflow ? maybe mixing it with the dynamic pipelines tutorial could help

What do you want to achieve in particular? 🙂

Hi @marrrcin, I'm back and armed with a clearer question this time, after realizing my first attempt was as clear as a foggy day in London. I'm currently working on a project where I'm using Kedro to create dynamic pipelines. My goal is to integrate MLflow into these pipelines to track each run as a separate experiment, complete with its own metrics, parameters, and pipeline name. Here’s what I've done so far: • I've set up a dynamic pipeline structure using Kedro. • I'm able to run different pipeline variants based on configuration. • I've started experimenting with MLflow for basic tracking. However, I'm encountering challenges in: • Dynamically setting the MLflow experiment name to match the pipeline name. • Ensuring that each pipeline variant's run is tracked separately with its unique set of parameters and metrics. I'm looking for guidance or examples on how to: 1. Automatically set the MLflow experiment name based on the Kedro pipeline name. 2. Ensure each pipeline run logs its metrics and parameters distinctly in MLflow. Has anyone tackled a similar integration or can offer insights into best practices for this scenario? Any tips, code snippets, or resources would be greatly appreciated! Thank you in advance for your help!