Hey fellow kedroids 🤖! Apologies if this has already been answered, but is there a standard or semi-standard approach to CI/CD for kedro projects? I.e., some example config files for CircleCI or GH actions, maybe an orb or a GH action that can be used off-the-shelf? Thanks friends!
04/14/2023, 1:54 PM
I'm not aware of any public example (very well could be some), but what do you want to test? At a high level, you can run:
• Package (and optionally publish or merge to something like JFrog or GitHub packages?)
• End-to-end tests on sample data
Except for the last piece, it's honestly not too different from a standard CI setup, and you can find a lot of examples online. If you're running into problems, I/we can potentially take a quick look.
(Since you're internal to QB, and if you're looking for a place to start, you may be able to find a DE Coffee presentation from 4-5 years ago setting up CircleCI in a few minutes with Kedro 0.15 or something.)
End-to-end testing is slightly more debated + complicated. I personally like Kedro's approach well enough and reference that.
Nok Lam Chan
04/20/2023, 9:04 PM
Agree mostly with Deepyaman, are you looking for how to CI/CD a kedro project, or in general how to CI/CD any machine learning pipeline (not necessary with Kedro)