Hi all, I have a niche question about

lazy_polars_dataset

. Abbreviated, this has:

if self._protocol == "file":
            return pl.scan_parquet(load_path, **self._load_args)
        return pl.scan_pyarrow_dataset(<something>)

But, for my GCS dataset,

scan_parquet

works, and is more reliable than

scan_pyarrow_dataset

. So the question is, why is

scan_pyarrow_dataset

favoured here? And, given that

scan_parquet

works fine for various cloud datasets (incl partitioned GCS parquet datasets), maybe there is a valid use-case for an option here to just use

scan_parquet

no matter the protocol?

Hey Simon, I think this is mostly historical. When

LazyPolarsDataset

was originally implemented may be 2-3 years ago, native cloud reading for Parquet in Polars was newer and less established. PyArrow was therefore used for non-local files as the more established option. Polars has moved on quite a bit since then, and as you pointed out,

scan_parquet()

now works directly with cloud-hosted and partitioned Parquet data, and Polars itself recommends using it when possible. So yes, I think there's a reasonable case for supporting a native

scan_parquet()

path for cloud Parquet datasets, while retaining the existing PyArrow path for compatibility. Would you be open to creating a GitHub issue, or even a draft PR, for this in

kedro-plugins

?

I would not retain the existing PyArrow path because Polars does not recommend doing it that way. Another reason for getting rid of that is in case you want to override schema; with e.g.

scan_parquet

You have to use Polars dtypes while with

scan_pyarrow_dataset

, you’ll have to use pyarrow dtypes, which adds confusion