How do you typically deploy Kedro with Kubernetes?
We currently use Prefect deployments to host our Kedro DS pipelines on our Kubernetes cluster but we would love to learn what are the popular ways other people are using Kedro with Kubernetes.
One of the issues we are having difficulty with is updating MLFlow deployments from the host, i.e. model registry / model deployment.
06/02/2023, 6:51 AM
We evaluated perfect as an orchestrator but found it wasn’t the easiest to integrate with kedro, they seemed to have some overlapping functionality. In the end we opted for Argo, but are still in the process of migrating from simple k8s cronjobs.
06/02/2023, 6:58 AM
Thanks @Iñigo Hidalgothanks!
How do you plan to do model rollout and rollback in production?
Let’s say that you have an inference model deployed in production.
What should be the procedure for updating the model (reference to the pickle of the trained model) and perhaps rolling back the updated model to a stable version (if for example you experience performance regression)?
06/02/2023, 10:23 AM
Unfortunately we don't have a robust procedure atm, but our model pickles are versioned through kedro, and our predict pipelines pull from latest/, so if needed we can replace the contents of latest/ to force pipelines to pull a specific version, but it isn't super sophisticated.
To avoid regressions we run updated models in a staging environment where we monitor performance for a period of time before pushing the updated model to production though.
I would actually be very curious to hear what other users' experience is like with MLOps around Kedro, since we are currently in a transition period overhauling most of our infra.