Hello! I would like to ask if my current way of thinking is reasonable for using kedro-boot. I want to run a KedroBootApp that constantly listens to a kafka topic and whenever a new message comes in it should run a pipeline with that data as input and send the output to another kafka topic. Is that possible with kedro-boot? And could the pipeline run be async? So that we don't need to wait for the results to start processing a new message? @Takieddine Kadiri

Hi Jan ! Yeah, kedro-boot is designed pretty much for this: Reusing your kedro pipeline inside any generic application, icluding those with long running process (as your kafka consumer/producer). It also allows you to reuse kedro's project tools for managing your application lifecycle (config loader, creds management, entry point cli, session, ...) So a KedroBootApp is any generic app that run as a kedro projet/package and have a KedroBootSession object that let it perform multiples low latency pipeline runs and access to the config_loader. Your kafka consumer would be implemented as a KedroBootApp I see from the previous thread that you've already played with KedroBootApp, i'll skip digging more into implemetation details here. For scaling your app, i suggest you to go with parallelism (multiprocessing or multithreading if your code is thread safe) instead of concurrency (async), as your code is compute-bound not I/O-bound. Moreover, in order to use async your code and libraries need to be async/await ready, which is not really the focus of kedro. keep us informed of your progress, and let us know if you've any questions

Thank you for that outline. To understand that correctly: The multiprocessing would be within the KedroBootApp or I would create multiple instances of the KedroBootApp?

It can be done both ways, but it's simpler to handle it within KedroBootApp (which is now your KafkaApp 😉) Unless you know a tool that handle starting multiple application instances and manage them, like Gunicorn but for arbitraty apps