https://kedro.org/ logo
#questions
Title
# questions
r

Ryoji Kuwae Neto

04/07/2023, 9:05 PM
Is there a good way on using Kedro with structured streaming? Customizing runner or doing other similar things? I just started studying streaming and would be nice to have some good scaffolding on this. Also making the overall project structure similar between batch and streaming would be interesting in a learning curve as people would only see one single structure.
d

Deepyaman Datta

04/07/2023, 9:10 PM
You can see an old example a friend and I put together here during a hackathon: https://github.com/deepyaman/kedro-streaming Definitely not production ready, but it worked and could serve as inspiration. I know that there are some people who work at QuantumBlack who have done something more recently, but I haven't seen it/no idea how it differs.
Also making the overall project structure similar between batch and streaming would be interesting in a learning curve as people would only see one single structure.
I have some thoughts I can hopefully share here in the next month or two (once things are public). 😉
❤️ 2