hey, just to confirm there is no `AbstractDataset`...
# questions
g
hey, just to confirm there is no
AbstractDataset
predefined currently for polars to delta table? would something like this do the job?
Copy code
class PolarsDeltaDataset(AbstractDataset):
    def __init__(self, filepath: str, mode: str = "append"):
        self.filepath = filepath
        self.mode = mode

    def _load(self) -> pl.DataFrame:
        return pl.read_delta(self.filepath)

    def _save(self, data: pl.DataFrame) -> None:
        write_deltalake(
            self.filepath,
            data,
            mode=self.mode
        )

    def _describe(self):
        return dict(
            filepath=self.filepath,
            mode=self.mode
        )
l
We don't currently have a Polars to DeltaTable dataset. I think this implementation should work, if all you need to do is save your dataframe as a deltatable.
👍 1
isn't there already a file_format argument? I would expect Polar support Delta out of the box since there is a delta rust implemnetation
(I haven't used this personally, just reference from docs)
g