https://kedro.org/ logo
#random
Title
# random
d

Deepyaman Datta

02/12/2024, 5:52 PM
If you found last month's article on scaling Kedro pipelines with Ibis interesting (https://kedro.org/blog/building-scalable-data-pipelines-with-kedro-and-ibis ICYMI), you may be excited to learn that you can now also do real-time data processing using the same interface: https://ibis-project.org/posts/flink-announcement/ You can also watch a short demo video on LinkedIn.
🔥 4
n

Nok Lam Chan

02/12/2024, 5:57 PM
awesome! you may want to try https://asciinema.org/ to speed up the setup bit, since the actual interesting part is the last 20 seconds or so. (you can also post-edit it because it's just a text file)
👀 1
Did you set some kind of
sleep
or interval time? Why is it feels a bit laggy when it prints the message at the end?
d

Deepyaman Datta

02/12/2024, 6:10 PM
Did you set some kind of
sleep
or interval time? Why is it feels a bit laggy when it prints the message at the end?
There's a sleep in the data producer: https://github.com/ibis-project/ibis-flink-example/blob/main/generate_source_data.py (not shown in the video, but repo is linked in the article) Other than that, nothing.
👍🏼 1
n

Nok Lam Chan

02/12/2024, 6:13 PM
make sense!
3 Views