Aalok Parkash
11/09/2023, 1:08 PM[a,b,c,d]
Pipelines c
and d
contain some parameters that I want to do a grid search over for tuning in a notebook.
My initial plan was to set up a tuning
environment with seperate global
paths for the inputs/outputs of a,b
, and seperate paths for the inputs/outputs of c,d
. I could then simply run the full pipeline using the tuning
environment and it would all be handled.
Is it possible to use Kedro's experiment tracking to do this instead?
A small further complication is that c
is very expensive. Ideally I'd like to ensure that I only run c
the minimum amount of times. So lets say that c
depends on a parameter x
, and there are 2 unique values for x but 10 total combinations in my grid in which these appear, I only want to run c
twice.
I can adjust the solution above by creating a tuning_x0
and tuning_x1
environment, and simply hardcode the value for x0
and x1
in the globals
, and use them in the path name.datajoely
11/09/2023, 1:19 PMAalok Parkash
11/09/2023, 1:21 PMdatajoely
11/09/2023, 1:30 PM