https://kedro.org/ logo
#random
Title
# random
e

Elias WILLEMSE

04/19/2023, 1:18 AM
Is anybody else using kedro in combination with streamlit (https://streamlit.io/)? Weโ€™ve had some fun combining (and abusing) the two for a few prototypes being used by clients. Iโ€™m happy to discuss this in more detail and hopefully exchange some lessons learned if anybody is interested.
d

Deepyaman Datta

04/19/2023, 1:51 AM
Various people have tried things over the years (most recently, I've seen https://blog.streamlit.io/using-chatgpt-to-build-a-kedro-ml-pipeline/), but I don't think I've seen a plugin or authoritative way of integrating the two, so always exciting to see how people have been leveraging them.
๐Ÿ‘ 2
o

Oleg Pilipenok

04/19/2023, 11:57 AM

https://www.youtube.com/watch?v=fYkVtzXUEBEโ–พ

๐Ÿ™ 1
๐Ÿ™๐Ÿผ 1
n

Nero Okwa

04/20/2023, 9:27 AM
@Elias WILLEMSE I'm interested๐Ÿ™‚.
e

Elias WILLEMSE

06/05/2023, 8:47 PM
Following on my (2 month old ๐Ÿคฅ) post on streamlit and kedro, Iโ€™ve finally had some time to publish a video (

https://youtu.be/UIhiluz2vDEโ–พ

) of the prototype that I build. The frontend is streamlit, and kedro is used to orchestrate everything in the backend. Itโ€™s actually deployed on streamlit community cloud. Unfortunately, the video only shows the frontend user interaction (as itโ€™s more of a promotional demo), Also, I canโ€™t yet directly share the source code. Will have to first do a clone and remove sensitive data --- that is if anyone else is interested in this, in any-case ๐Ÿ˜„ tldr; It was an interesting development process. First, I developed and tested the pipelines on their own. Then hooked it up to streamlit where the user can: 1. Change and update parameters in the front end. 2. The parameter files are saved and can be loaded by the users as yml files (via kedro catalog). 3. The kedro pipelines can then be executed from streamlit by the user. 4. And all the results are stored and loaded for display in the front end via kedro catalogs (again). Iโ€™m not an export on this, but I think itโ€™s close to a model-view-presenter framework, with pipelines being separate (basically treated like APIs) and activated through presenters. โ€ข Presenter is done via parameters updated through streamlit widgets, and the results fed into the pipelines; โ€ข View is done via streamlit. โ€ข Model is a combination of catalogs and parameter loading (via kedro) and formatting for display purposes. โ€ข The actual pipelines are treated as APIs and activated via presenters. Until I get to the time to release some more descriptive examples (with actual code), happy to try explain more and answer any questions in this thread.
๐Ÿ˜ฎ 3
1000 11
j

Jannic Holzer

06/05/2023, 9:11 PM
This is amazing! Thanks for sharing this. I take it your Kedro runs might take a while after they are triggered, was this easy to communicate to the user? Streamlit's
spinner
is good for this. You can still see Kedro's log output in the shell that was used to start streamlit, am I right? ๐Ÿ™‚ Also, just an FYI but the way your hyperlink is setup in your vid's description is a bit off. It is
<https://wastelabs.co/>)
, where the closing bracket is part of the hyperlink and it 404s ๐Ÿ™‚ loved checking out your company's page! Awesome work.
๐Ÿ‘ 1
e

Elias WILLEMSE

06/05/2023, 10:56 PM
Cheers! Yeh, the kedro runs take a while to initiate and start (maybe 0.5 second), their execution time depends on the specific pipeline. There's around 6 used in the app. And yup, kedro logs are displayed in streamlit shell, and also viable in streamlit cloud โ€” very useful. As for user coms on the delays, some of the pipelines actually solve combinatorial optimisation problems, which can take a few minutes, so the user is already used to that, so they are not too concerned with the delays. Also, they are enterprise clients used to enterprise software. For them, even streamlit feels super reactive and modern in comparison to most other tools. PS. Thanks for the website link error catch!
๐Ÿ™Œ 1
j

Juan Luis

06/06/2023, 7:44 AM
Will have to first do a clone and remove sensitive data --- that is if anyone else is interested in this, in any-case ๐Ÿ˜„
absolutely yes! ๐Ÿ™Œ๐Ÿผ
๐Ÿฅณ 1
๐Ÿ‘ 1
a

Antony Milne

06/06/2023, 10:10 PM
This is really cool stuff, amazing work @Elias WILLEMSE!
๐Ÿ™ 1
FYI @Joseph Perkins ๐Ÿ™‚
n

Nero Okwa

06/07/2023, 8:46 AM
This is amazing @Elias WILLEMSE
FYI @Jo Stichbury
thankyou 1
๐Ÿ™‚ 1
e

Elias WILLEMSE

06/07/2023, 1:57 PM
This is enough motivation for me to make something sharable: code and the hosted app. Will probably be a bit simpler, just to make it digestible, and to get something out sooner (instead of only posting again in two months ๐Ÿ˜„)
๐Ÿฅณ 2
4 Views