[Not a Kedro Question] - What’s the most efficient way to find out some basic statistic of a CSV (filesize/number of rows and columns) ? Requirements: • Memory efficient (cannot load the full csv into a dataframe and do

df.describe()

• Need to work in Windows and Linux so

wc

is not an option • Need to be fast • Bonus: is it possible to generalised to Excel filetype?

You may enjoy this option: https://github.com/BurntSushi/xsv 🙂

• You can check columns cheaply because you only need to read 1 line • You can check filesize from the filesystem cheaply • You could estimate the rows from a sample filesize, but also report the filesize

VisiData is quite famous too https://www.visidata.org/

xsv looks good! Maybe too much overhead and makes the dependency management more complicated tho.

you can also try Polars 🐻‍❄️

python -c 'import polars as pl; print(pl.scan_csv("companies.csv").select(pl.count()).collect())' 
shape: (1, 1)
┌───────┐
│ count │
│ ---   │
│ u32   │
╞═══════╡
│ 77096 │
└───────┘

(number of columns and file size left as an exercise to the reader)

scan_csv is similar to using pandas read_csv with chunksize ? I am not sure. we want to get the file metadata (local file/remote file) without loading the file in memory

you can also try Polars 🐻‍❄️ @Juan Luis

I was waiting for your polar’s solution

scan_csv

does not load the file, only creates a lazy representation. it's more similar to how

dask.dataframe

works

Any idea how it works under the hood to achieve this? not familiar with `polars`codebase, I think these are all expression and it get evaluated in a smart way.

I have no idea to be honest! maybe using Arrow memory mapping https://dev.to/astrojuanlu/demystifying-apache-arrow-5b0a/#arrow-awesomeness

remind me this answer using

np.memmap

https://stackoverflow.com/questions/64744161/best-way-to-find-out-number-of-rows-in-csv-without-loading-the-full-thing

yea I saw this which is similar to what you suggested of counting lines

Whilst this is an interesting problem, I wouldn’t overindex on CSV - any real pipeline is using an adult fileformat

yep, there's

scan_parquet

. no

scan_excel

though 😄

adult fileformat

lol…

I think I mean Parquet, Delta and SQL Tables these days

I would say, go with Polars or Duckdb. Both will perform just fine and are extremely memory efficient. Also, they are lightweight when it comes to dependencies… Edit: oh and I forgot to mention, they are fast too 😅. Been using polars for a couple of week now and I will never switch back to pandas unless I really really really have to!

> adult fileformat

Do you mean Excel?

I think Excel counts as a senior citizen by now

Time to send it to the farm