Hugo Evers
07/12/2023, 4:35 PMkedro-mlflow
question:
I train multiple Transformer models in a single pipeline, both kedro_mlflow
and Transformers
Trainer
are logging to mlflow.
When i train a single Transformer, this is no problem, but when i train multiple transformers in the same pipeline with different settings, this becomes an issue as i get :
RestException: INVALID_PARAMETER_VALUE: Changing param values is not allowed. Params were already logged='[{'key':
'num_train_epochs', 'old_value': '3', 'new_value': '10'},
This is an example of where the issue pops up when i train a transformer on one dataset, and then tune it further on another.
I can imagine this can be solved by making kedro_mlflow
and Trainer
aware of the fact that these are actually different models (since they live inside their own namespace), by appending some prefix to the param values. But at this point it becomes quire difficult to debug (since the logging code is nested very deep) whether this should be implemented for the Trainer
, Kedro-mlflow
or both.
Any thoughts?Yolan Honoré-Rougé
07/12/2023, 5:07 PMHugo Evers
07/19/2023, 4:46 PMYolan Honoré-Rougé
07/20/2023, 7:38 PM