Akash Agnihotri10/23/2023, 10:42 AM
to save and load torch object. When we use this dataset as part of a kedro pipeline, the node does not create local folder when saving the data ? Question is, do we need to implement this functionality ourself in our custom dataset, or is it somehting provided by
or ``AbstractDataSet` ?
Deepyaman Datta10/23/2023, 11:05 AM
Akash Agnihotri10/23/2023, 5:29 PM
def _save(self, data: pl.LightningDataModule) -> None: """Save data module to filepath.""" with open(self.filepath, "wb+") as file: torch.save(data, file)
Deepyaman Datta10/23/2023, 6:12 PM
. As mentioned, directories aren't created magically by
, but IIRC
implementations will handle the create (which is part of why we use it). Also, it will enable use on other filesystems, like if you want to save to s3.