https://kedro.org/ logo
#questions
Title
# questions
d

David G.

12/19/2023, 7:41 PM
Hi all, does anyone have resources or tips on integrating MLflow with Kedro, with dynamic pipeline structure? Any guidance or shared experiences would be greatly appreciated! I follow this tutorial to create dynamic pipeline : https://getindata.com/blog/kedro-dynamic-pipelines/
t

Takieddine Kadiri

12/20/2023, 9:49 AM
Did you look at kedro-mlflow ? maybe mixing it with the dynamic pipelines tutorial could help
m

marrrcin

12/20/2023, 10:32 AM
What do you want to achieve in particular? 🙂
d

David G.

12/22/2023, 4:22 PM
Hi @marrrcin, I'm back and armed with a clearer question this time, after realizing my first attempt was as clear as a foggy day in London. I'm currently working on a project where I'm using Kedro to create dynamic pipelines. My goal is to integrate MLflow into these pipelines to track each run as a separate experiment, complete with its own metrics, parameters, and pipeline name. Here’s what I've done so far: • I've set up a dynamic pipeline structure using Kedro. • I'm able to run different pipeline variants based on configuration. • I've started experimenting with MLflow for basic tracking. However, I'm encountering challenges in: • Dynamically setting the MLflow experiment name to match the pipeline name. • Ensuring that each pipeline variant's run is tracked separately with its unique set of parameters and metrics. I'm looking for guidance or examples on how to: 1. Automatically set the MLflow experiment name based on the Kedro pipeline name. 2. Ensure each pipeline run logs its metrics and parameters distinctly in MLflow. Has anyone tackled a similar integration or can offer insights into best practices for this scenario? Any tips, code snippets, or resources would be greatly appreciated! Thank you in advance for your help!