deepseek release a DuckDB based data processing en...
# resourcesn
Nok Lam Chan
02/28/2025, 12:08 PMdeepseek release a DuckDB based data processing engine: https://github.com/deepseek-ai/smallpond
Nok Lam Chan
02/28/2025, 12:08 PM@datajoely
Nok Lam Chan
02/28/2025, 12:08 PMthought you may be interested, the codebase actually looks small enough that I may spend some time to look at it
j
Juan Luis
02/28/2025, 12:09 PMwhat is "data processing framework" in this context? a dataframe library?
n
Nok Lam Chan
02/28/2025, 12:09 PMSo it's DuckDB + 3FS (deepseek's own filesystem)
d
datajoely
02/28/2025, 12:09 PMI saw!
datajoely
02/28/2025, 12:09 PMcool stuff
n
Nok Lam Chan
02/28/2025, 12:09 PMI guess it's an orchestrated Duckdb run in "local" ssd?
🤔 1
Nok Lam Chan
02/28/2025, 12:11 PMI haven't read, but I am guess the 3FS deal with the I/O & network part while Duckdb handle compute local mostly.
Nok Lam Chan
02/28/2025, 12:11 PMimage.png
Nok Lam Chan
02/28/2025, 12:11 PMlooks like it has some lightweight DB components there
Nok Lam Chan
02/28/2025, 12:23 PMplaying with the example, it also use
ray[core]👀 1
j
Juan Luis
02/28/2025, 12:24 PMinteresting
Juan Luis
02/28/2025, 12:29 PMn
Nok Lam Chan
02/28/2025, 4:50 PMhttps://mehdio.substack.com/p/duckdb-goes-distributed-deepseeks
There is now a more detailed blog about this
j
Juan Luis
03/02/2025, 8:16 PMthe name of the project seems to be a reference to Google's BigLake 🙃
Juan Luis
03/02/2025, 8:41 PMMehdi's writeup is great 💯
16 Views