Daniel Kirel
08/03/2023, 8:25 PMkedro-mlflow
?
2. Is there a good way to save input datasets without needing to create separate MLFlow artifact datasets and a node to read and save datasets?
Appreciate any help/guidance on this 🙏Merel
08/04/2023, 12:52 PMkedro-mlflow
marrrcin
08/04/2023, 1:52 PMmlflow.log_param("git_sha", <value of git sha>)
. Usual place to do this is before_pipeline_run
or after_pipeline_run
.
2. Again hooks, assuming that you don’t want to read and serialize the data just for the sake of logging it as an artifact to mlflow. The problem with that is it’s not a really “Kedro way of doing things”, because you would have to access _filepath
(or similar) property of the dataset object, which is “private”. You can use before_node_run
for that https://docs.kedro.org/en/stable/kedro.framework.hooks.specs.NodeSpecs.html#kedro.framework.hooks.specs.NodeSpecs.before_node_run which has access to node, catalog and inputs in one place.Daniel Kirel
08/04/2023, 2:56 PM