Ryan Wendt
09/25/2023, 7:15 PMYetunde
09/25/2023, 7:47 PMkedro run --pipeline=data_preprocessing
and kedro run --pipeline=model_training
. This is best seen in our CLI guide. We mention this in our longer Spaceflights tutorial.
• And then you're pretty free to pick how you'd like these pipelines orchestrated and that's outside the scope of Kedro.Nok Lam Chan
09/25/2023, 7:48 PMRyan Wendt
09/25/2023, 8:02 PMYolan Honoré-Rougé
09/25/2023, 8:03 PMMatthias Roels
09/25/2023, 8:18 PMSo the question I'm wondering is there anywhere that shows how to neatly create these deployments of the related but separate pipelines?I think the answer depends on whether you are ok with sharing dependencies across pipelines (TensorFlow, Spark, … are big packages that you might not want to include in a container image if you don’t need them). If you are OK with sharing (which I think will be the case for the majority of kedro users), you can create one container image (Dockerfile in the root of the project). And then use that image in your deployment with overriding the
CMD
with the appropriate kedro run cmdRyan Wendt
09/25/2023, 8:55 PMMatthias Roels
09/25/2023, 9:01 PMkedro run —pipeline=inference
.
Another pattern I sometime use (usually non-prod) is to just use a container image without my project code and use a git artifact to clone the repo in a specific folder mounted to the container. From there, I can then run the image.Ryan Wendt
09/25/2023, 9:14 PM