Is anybody else using kedro in combination with streamlit (https://streamlit.io/)? Weโve had some fun combining (and abusing) the two for a few prototypes being used by clients. Iโm happy to discuss this in more detail and hopefully exchange some lessons learned if anybody is interested.
d
Deepyaman Datta
04/19/2023, 1:51 AM
Various people have tried things over the years (most recently, I've seen https://blog.streamlit.io/using-chatgpt-to-build-a-kedro-ml-pipeline/), but I don't think I've seen a plugin or authoritative way of integrating the two, so always exciting to see how people have been leveraging them.
๐ 2
o
Oleg Pilipenok
04/19/2023, 11:57 AM
https://www.youtube.com/watch?v=fYkVtzXUEBEโพ
๐ 1
๐๐ผ 1
n
Nero Okwa
04/20/2023, 9:27 AM
@Elias WILLEMSE I'm interested๐.
e
Elias WILLEMSE
06/05/2023, 8:47 PM
Following on my (2 month old ๐คฅ) post on streamlit and kedro, Iโve finally had some time to publish a video (
https://youtu.be/UIhiluz2vDEโพ
) of the prototype that I build. The frontend is streamlit, and kedro is used to orchestrate everything in the backend. Itโs actually deployed on streamlit community cloud.
Unfortunately, the video only shows the frontend user interaction (as itโs more of a promotional demo), Also, I canโt yet directly share the source code. Will have to first do a clone and remove sensitive data --- that is if anyone else is interested in this, in any-case ๐
tldr;
It was an interesting development process.
First, I developed and tested the pipelines on their own.
Then hooked it up to streamlit where the user can:
1. Change and update parameters in the front end.
2. The parameter files are saved and can be loaded by the users as yml files (via kedro catalog).
3. The kedro pipelines can then be executed from streamlit by the user.
4. And all the results are stored and loaded for display in the front end via kedro catalogs (again).
Iโm not an export on this, but I think itโs close to a model-view-presenter framework, with pipelines being separate (basically treated like APIs) and activated through presenters.
โข Presenter is done via parameters updated through streamlit widgets, and the results fed into the pipelines;
โข View is done via streamlit.
โข Model is a combination of catalogs and parameter loading (via kedro) and formatting for display purposes.
โข The actual pipelines are treated as APIs and activated via presenters.
Until I get to the time to release some more descriptive examples (with actual code), happy to try explain more and answer any questions in this thread.
๐ฎ 3
1000 11
j
Jannic Holzer
06/05/2023, 9:11 PM
This is amazing! Thanks for sharing this. I take it your Kedro runs might take a while after they are triggered, was this easy to communicate to the user? Streamlit's
spinner
is good for this. You can still see Kedro's log output in the shell that was used to start streamlit, am I right? ๐
Also, just an FYI but the way your hyperlink is setup in your vid's description is a bit off. It is
<https://wastelabs.co/>)
, where the closing bracket is part of the hyperlink and it 404s ๐ loved checking out your company's page! Awesome work.
๐ 1
e
Elias WILLEMSE
06/05/2023, 10:56 PM
Cheers!
Yeh, the kedro runs take a while to initiate and start (maybe 0.5 second), their execution time depends on the specific pipeline. There's around 6 used in the app. And yup, kedro logs are displayed in streamlit shell, and also viable in streamlit cloud โ very useful.
As for user coms on the delays, some of the pipelines actually solve combinatorial optimisation problems, which can take a few minutes, so the user is already used to that, so they are not too concerned with the delays. Also, they are enterprise clients used to enterprise software. For them, even streamlit feels super reactive and modern in comparison to most other tools.
PS. Thanks for the website link error catch!
๐ 1
j
Juan Luis
06/06/2023, 7:44 AM
Will have to first do a clone and remove sensitive data --- that is if anyone else is interested in this, in any-case ๐
absolutely yes! ๐๐ผ
๐ฅณ 1
๐ 1
a
Antony Milne
06/06/2023, 10:10 PM
This is really cool stuff, amazing work @Elias WILLEMSE!
๐ 1
FYI @Joseph Perkins ๐
n
Nero Okwa
06/07/2023, 8:46 AM
This is amazing @Elias WILLEMSE
FYI @Jo Stichbury
thankyou 1
๐ 1
e
Elias WILLEMSE
06/07/2023, 1:57 PM
This is enough motivation for me to make something sharable: code and the hosted app. Will probably be a bit simpler, just to make it digestible, and to get something out sooner (instead of only posting again in two months ๐)