Ryan Wendt09/25/2023, 7:15 PM
Yetunde09/25/2023, 7:47 PM
kedro run --pipeline=data_preprocessing
. This is best seen in our CLI guide. We mention this in our longer Spaceflights tutorial. • And then you're pretty free to pick how you'd like these pipelines orchestrated and that's outside the scope of Kedro.
kedro run --pipeline=model_training
Nok Lam Chan09/25/2023, 7:48 PM
Ryan Wendt09/25/2023, 8:02 PM
Yolan Honoré-Rougé09/25/2023, 8:03 PM
Matthias Roels09/25/2023, 8:18 PM
So the question I'm wondering is there anywhere that shows how to neatly create these deployments of the related but separate pipelines?I think the answer depends on whether you are ok with sharing dependencies across pipelines (TensorFlow, Spark, … are big packages that you might not want to include in a container image if you don’t need them). If you are OK with sharing (which I think will be the case for the majority of kedro users), you can create one container image (Dockerfile in the root of the project). And then use that image in your deployment with overriding the
with the appropriate kedro run cmd
Ryan Wendt09/25/2023, 8:55 PM
Matthias Roels09/25/2023, 9:01 PM
. Another pattern I sometime use (usually non-prod) is to just use a container image without my project code and use a git artifact to clone the repo in a specific folder mounted to the container. From there, I can then run the image.
kedro run —pipeline=inference
Ryan Wendt09/25/2023, 9:14 PM