Has anyone ever used one of the kedro polars datasets to load excel files from s3? It seems the calamine engine doesn't like

fsspec

objects and expects

str

or

os.pathlike

bytes

.

Someone will reply to you shortly. In the meantime, this might help:

thinking I might just need to use pandas and convert to polars

Haven't checked, but if it doesn't work the more correct approach should be to fetch locally from S3? Using a different engine and then converting feels bad

Yes, it does feel bad. What do you mean by fetch locally?

Essentially download file into temp path using fsspec, and point polars to the tempfile

Got it, will give it a try. Also I raised an issue, happy to continue discussion there.

I've been using

engine: xlsx2csv

for reading Excel with Polars due to this.

Since the creation of these datasets. Polars has evolved significantly rendering the implementation of these datasets severely outdated. Nowadays, polars natively supports reading from objects stores so that both fsspec and pyarrow (used in the lazy version) are no longer needed

Sounds like what @Chris Schopp said is the right current solution, and then that the datasets should be updated

I have an implementation ready to ship for the Lazy dataset…