Hi team I am trying to load a spark parquet as a `polars Laz Kedro #questions

Hi team! I am trying to load a spark parquet as a ...

Puneet Saini

04/28/2025, 8:20 AM

Hi team! I am trying to load a spark parquet as a

polars.LazyPolarsDataset

for which I assume the filepath needs to be a glob pattern. But since kedro-datasets>=6.0.0, we are checking the availability of the file itself without expanding the glob pattern if passed in. Is this a bug or am I doing something wrong?

Elena Khaustova

04/28/2025, 1:07 PM

Hi Puneet, could you elaborate a bit on the issue you’re having? Do you mean that globals are not resolving in your dataset path?

Puneet Saini

04/28/2025, 1:39 PM

Seems like when I pass

filepath: path/to/my/parquet_folder/*.parquet

, on this line we are checking whether the file exists or not. Since, filepath is a glob pattern and not a static filepath it fails to load the parquet.

👍 1

Elena Khaustova

04/28/2025, 1:45 PM

Does it show you unresolved path in the raised error? https://github.com/kedro-org/kedro-plugins/blob/da4af7e783ae38241175cb197ad419b9f52aa0f6/kedro-datasets/kedro_datasets/polars/lazy_polars_dataset.py#L205

Puneet Saini

04/28/2025, 1:53 PM

Copy code

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/circleci/project/project_folder/venv-env/lib/python3.10/site-packages/kedr │
│ o/io/core.py:245 in load                                                     │
│                                                                              │
│   242 │   │   │   self._logger.debug("Loading %s", str(self))                │
│   243 │   │   │                                                              │
│   244 │   │   │   try:                                                       │
│ ❱ 245 │   │   │   │   return load_func(self)                                 │
│   246 │   │   │   except DatasetError:                                       │
│   247 │   │   │   │   raise                                                  │
│   248 │   │   │   except Exception as exc:                                   │
│                                                                              │
│ /home/circleci/project/project_folder/venv-env/lib/python3.10/site-packages/kedr │
│ o_datasets/polars/lazy_polars_dataset.py:205 in load                         │
│                                                                              │
│   202 │   def load(self) -> pl.LazyFrame:                                    │
│   203 │   │   load_path = str(self._get_load_path())                         │
│   204 │   │   if not self._exists():                                         │
│ ❱ 205 │   │   │   raise FileNotFoundError(errno.ENOENT, os.strerror(errno.EN │
│   206 │   │                                                                  │
│   207 │   │   if self._protocol == "file":                                   │
│   208 │   │   │   # With local filesystems, we can use Polar's build-in I/O  │
╰──────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: 
'/home/circleci/project/project_folder/data/pipeline_1/*.parquet'

👍 1

Elena Khaustova

04/28/2025, 2:46 PM

@Deepyaman Datta, hey, are we aiming to support glob patterns here?

Puneet Saini

04/28/2025, 2:48 PM

I can raise a quick fix if that works?

Puneet Saini

04/28/2025, 2:48 PM

Let me know if checking glob pattern for exists is a good enough solution

👍 1

Elena Khaustova

04/28/2025, 2:50 PM

Yeah, thank you, @Puneet Saini Give us some time to align on what this check was originally for

Deepyaman Datta

04/28/2025, 3:01 PM

Let me know if checking glob pattern for exists is a good enough solution

I think that's fine. It looks like the CI logs before implementing this fix are gone since it's been a while, but it was just added to fix breaking tests, and I think the reason for this would be because Polars lazyframe doesn't otherwise test file existence until you actually go to collect the result of some operation. You could potentially try removing that and seeing which test fails.

thankyou 1

Elena Khaustova

04/28/2025, 3:31 PM

@Puneet Saini feel free to open an issue with a suggested fix, or we’ll take this into the sprint ourselves

2 Views

Open in Slack

Previous Next