Hi Kedro community! My team and are I trying to create an optimal setup for running experiments in parallel. Concerningly, it appears as though if we are to change the contents of a parameters file (i.e.

conf/local/parameters.yml

) during a run, the results of the run may be affected. For example, let's say I set

hyper_tune: False

in

parameters.yml,

and run

kedro run

in the terminal. If I change the text in

parameters.yml

to

hyper_tune: True

(for example, if I am setting up the parameters for my next experiment) before the "training" node begins executing, it appears that Kedro will then read hyper_tune: True. In this example, that would mean that Kedro would execute hyperparameter tuning (despite being instructed not to do so at the beginning of the run). Am I missing something? Is the answer as simple as passing all parameters to the pipeline one time as a whole (i.e. using a before_pipeline_runs hook) rather than to each node?

In Kedro, the parameter values are read in real-time when each node runs and not at the beginning. You can use a

before_pipeline_run

hook, you can read all the parameters at the beginning of the pipeline run and store them in memory. Then, during the pipeline run, nodes would reference these in-memory parameters, which would be unaffected by changes to the parameter file during the run.