https://kedro.org/ logo
#questions
Title
# questions
r

Rafał Nowak

02/05/2023, 6:54 PM
Hello all kedro enthusiasts, I am looking for implementation of kedro dataset
json.JSONDataSet
supporting gzip compression, so the filepath would be
*.json.gz
I haven’t found such backend in
kedro.datasets
Have anyone already implemented such dataset?
d

datajoely

02/05/2023, 7:20 PM
Since we use
fsspec
behind the scenes I think it’s possible that you can provide
compression='gzip'
as an an argument to
fs_args
https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open If that doesn’t work it should be very simple to subclass the dataset and provide your own implementation
r

Rafał Nowak

02/05/2023, 10:26 PM
I will check if fs-args solve my problem. Thanks.
4 Views