https://kedro.org/ logo
#questions
Title
# questions
a

Aalok Parkash

11/09/2023, 1:08 PM
Hi team, I have a question regarding Kedro's experiment tracking feature. Situation: I have a pipeline containing a few subpipelines, let's call tlhem
[a,b,c,d]
Pipelines
c
and
d
contain some parameters that I want to do a grid search over for tuning in a notebook. My initial plan was to set up a
tuning
environment with seperate
global
paths for the inputs/outputs of
a,b
, and seperate paths for the inputs/outputs of
c,d
. I could then simply run the full pipeline using the
tuning
environment and it would all be handled. Is it possible to use Kedro's experiment tracking to do this instead? A small further complication is that
c
is very expensive. Ideally I'd like to ensure that I only run
c
the minimum amount of times. So lets say that
c
depends on a parameter
x
, and there are 2 unique values for x but 10 total combinations in my grid in which these appear, I only want to run
c
twice. I can adjust the solution above by creating a
tuning_x0
and
tuning_x1
environment, and simply hardcode the value for
x0
and
x1
in the
globals
, and use them in the path name.
d

datajoely

11/09/2023, 1:19 PM
it doesn’t cover exp tracking but it covers the parameter sweep element
a

Aalok Parkash

11/09/2023, 1:21 PM
I actually want to do all of this in a notebook, not sure if it still holds.
d

datajoely

11/09/2023, 1:30 PM
it does, but in our opinion that is often a recipe for brittle and hard to reproduce pipeline design.
2 Views