Is anybody else using kedro in combination with streamlit (https://streamlit.io/)? We’ve had some fun combining (and abusing) the two for a few prototypes being used by clients. I’m happy to discuss this in more detail and hopefully exchange some lessons learned if anybody is interested.

Various people have tried things over the years (most recently, I've seen https://blog.streamlit.io/using-chatgpt-to-build-a-kedro-ml-pipeline/), but I don't think I've seen a plugin or authoritative way of integrating the two, so always exciting to see how people have been leveraging them.

@Elias WILLEMSE I'm interested🙂.

Following on my (2 month old 🤥) post on streamlit and kedro, I’ve finally had some time to publish a video ( ) of the prototype that I build. The frontend is streamlit, and kedro is used to orchestrate everything in the backend. It’s actually deployed on streamlit community cloud. Unfortunately, the video only shows the frontend user interaction (as it’s more of a promotional demo), Also, I can’t yet directly share the source code. Will have to first do a clone and remove sensitive data --- that is if anyone else is interested in this, in any-case 😄 tldr; It was an interesting development process. First, I developed and tested the pipelines on their own. Then hooked it up to streamlit where the user can: 1. Change and update parameters in the front end. 2. The parameter files are saved and can be loaded by the users as yml files (via kedro catalog). 3. The kedro pipelines can then be executed from streamlit by the user. 4. And all the results are stored and loaded for display in the front end via kedro catalogs (again). I’m not an export on this, but I think it’s close to a model-view-presenter framework, with pipelines being separate (basically treated like APIs) and activated through presenters. • Presenter is done via parameters updated through streamlit widgets, and the results fed into the pipelines; • View is done via streamlit. • Model is a combination of catalogs and parameter loading (via kedro) and formatting for display purposes. • The actual pipelines are treated as APIs and activated via presenters. Until I get to the time to release some more descriptive examples (with actual code), happy to try explain more and answer any questions in this thread.

This is amazing! Thanks for sharing this. I take it your Kedro runs might take a while after they are triggered, was this easy to communicate to the user? Streamlit's

spinner

is good for this. You can still see Kedro's log output in the shell that was used to start streamlit, am I right? 🙂 Also, just an FYI but the way your hyperlink is setup in your vid's description is a bit off. It is

<https://wastelabs.co/>)

, where the closing bracket is part of the hyperlink and it 404s 🙂 loved checking out your company's page! Awesome work.

Cheers! Yeh, the kedro runs take a while to initiate and start (maybe 0.5 second), their execution time depends on the specific pipeline. There's around 6 used in the app. And yup, kedro logs are displayed in streamlit shell, and also viable in streamlit cloud — very useful. As for user coms on the delays, some of the pipelines actually solve combinatorial optimisation problems, which can take a few minutes, so the user is already used to that, so they are not too concerned with the delays. Also, they are enterprise clients used to enterprise software. For them, even streamlit feels super reactive and modern in comparison to most other tools. PS. Thanks for the website link error catch!

Will have to first do a clone and remove sensitive data --- that is if anyone else is interested in this, in any-case 😄

absolutely yes! 🙌🏼

This is really cool stuff, amazing work @Elias WILLEMSE!

This is amazing @Elias WILLEMSE

This is enough motivation for me to make something sharable: code and the hosted app. Will probably be a bit simpler, just to make it digestible, and to get something out sooner (instead of only posting again in two months 😄)