Hi all when running a pipeline in a `distributed` manner mul

Question

Hi all when running a pipeline in a `distributed` manner multiple GPUs using the `ThreadRunner` how can I save the metrics as versioned `kedro datasets tracking MetricsDataset` and some graphs with `matplotlib MatplotlibWriter` in just one file something like `sync dist=True` without creating a versioned file for each sub process

marrrcin · Answer

That s a nice one smile We ve stumped upon similar issue on `kedro azureml` our solution was to save only on master node <https github com getindata kedro azureml blob 8e5979f5040e03032215e9db25af51538ec6a26a kedro azureml datasets runner dataset py L82>

Gregor Höhne · Answer

Thanks for that tip The solution works but this UserWarning appears because the saving object is a versioned abstract dataset Any idea how to get rid of this warning

Gregor Höhne · Answer

Another question When running a distributed training it is recommended to use the ThreadRunner right Is there a way to run only one node with the behavior of the ThreadRunner and the rest with the behavior of the SequentialRunner Or do I have to run these parts of a pipeline separately

marrrcin · Answer

<https kedro org slack com archives C03RKP2LW64 p1699803807916089 thread ts=1699603409 411269 amp cid=C03RKP2LW64> Most likely the second option I m not aware of any options to switch between runners mid pipeline as this is against Kedro s architecture

marrrcin · Answer

<https kedro org slack com archives C03RKP2LW64 p1699802420388689 thread ts=1699603409 411269 amp cid=C03RKP2LW64> This warning is most likely from the additional processes that are trying to to ` save` on the dataset If you ve added the guardrails `is distributed master node` then the warning can be probably neglected