Hello everyone👋, I am glad to announce the release of Kedro Boot 🚀 a new community plugin that streamlines the integration between kedro projects and any generic applications ! It will help you inject application's data into kedro projects and dynamically orchestrating multiples pipeline runs 🎉. If you are intested, you can : • 📦 Download the package with

pip install kedro-boot

• G Visit Kedro Boot github repo, give feedback and ⭐ if you like it • 🧪 Try & Learn with Kedro Boot Examples and give feedback ! ✨ This enable using Kedro pipelines in a wide range of online and low latency use cases, including model serving, data apps (streamlit, dash), statistical simulations, paralell processing of unstructured data, streaming, and more. 💥 Kedro Boot propose an answer to many issues: #2627, #2169, #1993, #2626, #devrel7, #2663, #2182 #2879 #1846 #795 #933 #2058 #143 #1041 and numerous slack questions about dynamic pipelines, injecting external data, serving pipeline as API, exposing kedro's resources to a generic application. @datajoely wanted this feature for ages 🎁 😉 If the plugin proves to be valuable and effectively addresses the problem, @Yolan Honoré-Rougé and I will ensure its maintenance and support. 🧑‍💻 Takieddine & Yolan

this is amazing, thanks a lot for sharing @Takieddine Kadiri and @Yolan Honoré-Rougé! 👏🏼👏🏼👏🏼👏🏼 I'll have a closer look at this very soon 🔍

I think this is likely worth specific documentation to explain which issues we are trying to solve, and how we do it. For now this is really intended for advanced users, but we plan to add a simplified interface for users. It is likely worth presenting in a tech design one day, but likely after 0.19 is out. Maybe as a Christmas gift 🎅 ?

for now, do you want to add it to https://github.com/kedro-org/awesome-kedro ? and @Jo Stichbury might feature it in this month's blog newsletter 🤩

@Juan Luis it would be great to have it in awesome-kedro and the blog newsletter, as this will allow us to have user feedback fairly early

feel free to send a PR!

Done !

Hey @Takieddine Kadiri Just to make sure I understand the purpose of kedro-boot, it’s a wrapper around kedro to make it easier to: 1. Run kedro. Instead of bootstrap_project and with KedroSession.create(), etc, you just call boot_session.run(name={pipeline_name}) 2. Change inputs in catalog at runtime. e.g. if I have an API wrapping kedro, this makes it much easier for me to let the user specify different input data to the kedro pipeline via API. i.e. boot_session.run(name={pipeline_name}, inputs={“companies”: “companies_df}) Is that correct?

Hey @Charlie Haley, Yes, you can see it as a wrapper that create a boot session on top of kedro session. 1. Yes, moreover, you can perform multiple runs without kedro session creation and running overhead, ensuring a low latency for your generic application. Kedro create/materialize the catalog and pipeline at session running, and enfore a 1 session -- 1 run hypothesis. 2. Yes exactly you can wrap kedro with an API, and send JSON data through an endpoint. This is typical when doing model serving or exposing a pipeline to an application through a Rest API 3. I will add a third important feature : Kedro Boot introduce a new kind of parameters called template_params that are resolved at iteration time (at each run). For example you can have a CSVDataset with filepath: data/my_data_[[ date_value ]].csv or a SqlQueryDataset with query: SELECT * from TABLE WHERE SOME_COLUMN=[[ column_value ]]. The kedro boot app can render those template params at each run time. Note that the {pipeline_name} that you use in you example, is actually the name of a "view" that was defined on top of the actual selected pipeline

from kedro.pipeline.modular_pipeline import pipeline
from kedro_boot.pipeline import app_pipeline

inference_pipeline = pipeline(......)
evaluation_pipeline = pipeline(......)

app_inference_pipeline = app_pipeline(
        inference_pipeline,
        name="inference",
        inputs="features_store",
        artifacts="training.regressor",
        outputs="inference.predictions",
    )

app_evaluation_pipeline = app_pipeline(
    evaluation_pipeline,
    name="evaluation",

all_app_pipeline = app_inference_pipeline + app_evaluation_pipeline

return {"__default__": all_app_pipeline}

In this example, you will use "inference" or "evaluation" as {pipeline_name} in your code boot_session.run(name={inference}, inputs={“companies”: “companies_df}) Note also that kedro boot will try to convert you API Json data to pandas if it see that "companies" is a pandas.*Dataset and if it is not already converted by your app code You can play with kedro boot using the examples repo https://github.com/takikadiri/kedro-boot-examples

Looks very cool! Naive question maybe, but how does it differ exactly from the pipeline serving capabilities from kedro-mflow (apart from not relying on MLflow of course)?

kedro boot doesn't offer directly a model serving or pipeline serving capabilities. It offer a frameworked way of accessing live kedro pipeline and catalog and peforming multiples runs using them (injecting external data, params, template params, ...). In a model serving scenario, it's enable you to consume dynamically your kedro pipelines from your own API that you developed using you prefered web app framework (fastapi, flask, ...). Curreently kedro-mlflow serve the model in a complete standalone and statless way, it doesn't let you (for now) to inject parameters at service startup or per requests or leverage hooks mechanisms. We could consider using kedro-boot as the backend for kedro-mlflow model serving feature. Here is some examples of using kedro-boot in a model serving scenarios https://github.com/takikadiri/kedro-boot-examples kedro boot is intended to be a framework and building block for a wide range of use cases. However, we plan to add some layers on top of kedro boot in the form of plugins. Each plugin will focus on one specific arear (pipeline serving for example) with a user friendly interface. I hope this will be clearer. Your feedback will be welcomed. simple smile

@Yolan Honoré-Rougé +1 for Tech Design demo