Hi team, I have a question regarding Kedro's experiment tracking feature. Situation: I have a pipeline containing a few subpipelines, let's call tlhem

[a,b,c,d]

Pipelines

c

and

d

contain some parameters that I want to do a grid search over for tuning in a notebook. My initial plan was to set up a

tuning

environment with seperate

global

paths for the inputs/outputs of

a,b

, and seperate paths for the inputs/outputs of

c,d

. I could then simply run the full pipeline using the

tuning

environment and it would all be handled. Is it possible to use Kedro's experiment tracking to do this instead? A small further complication is that

c

is very expensive. Ideally I'd like to ensure that I only run

c

the minimum amount of times. So lets say that

c

depends on a parameter

x

, and there are 2 unique values for x but 10 total combinations in my grid in which these appear, I only want to run

c

twice. I can adjust the solution above by creating a

tuning_x0

and

tuning_x1

environment, and simply hardcode the value for

x0

and

x1

in the

globals

, and use them in the path name.

have you seen this? https://getindata.com/blog/kedro-dynamic-pipelines/

I actually want to do all of this in a notebook, not sure if it still holds.

it does, but in our opinion that is often a recipe for brittle and hard to reproduce pipeline design.