If you found last month's article on scaling Kedro pipelines with Ibis interesting (https://kedro.org/blog/building-scalable-data-pipelines-with-kedro-and-ibis ICYMI), you may be excited to learn that you can now also do real-time data processing using the same interface: https://ibis-project.org/posts/flink-announcement/ You can also watch a short demo video on LinkedIn.

awesome! you may want to try https://asciinema.org/ to speed up the setup bit, since the actual interesting part is the last 20 seconds or so. (you can also post-edit it because it's just a text file)

Did you set some kind of

sleep

or interval time? Why is it feels a bit laggy when it prints the message at the end?

There's a sleep in the data producer: https://github.com/ibis-project/ibis-flink-example/blob/main/generate_source_data.py (not shown in the video, but repo is linked in the article) Other than that, nothing.

make sense!