Luis Cano11/18/2022, 4:53 AM
Is there a way of saving pickles this way? or maybe what would be a better way of doing this? any thoughts?
s3_path/train_outputs/ ├── 202201.pkl ├── 202202.pkl ├── 202203.pkl ├── 202204.pkl ... so on.
datajoely11/18/2022, 10:07 AM
will do this on a normal dataset, but we don’t currently give you control over the timestamp format so you’ll get one that it’s a lot more precise rather than month level
Yetunde11/18/2022, 10:16 AM
Luis Cano11/18/2022, 3:27 PM
Since it was already looping for each period, every model was being saved in a dictionary and then with this way of defining the catalog it allowed me to save it the way it was before using the datacatalog. It might not be the most optimal thing though !
model_name: type: PartitionedDataSet path: "s3_path/folder_name/" dataset: type: pickle.PickleDataSet filename_suffix: ".pkl"