Hi all, Any specific reason why versioning is not supported in PartitionedDataset ?

hi @Rahul Kumar,

The main reason we haven't implemented this functionality yet, is because it's very complex.

more discussion in https://github.com/kedro-org/kedro/issues/2043, current issue https://github.com/kedro-org/kedro/issues/2115

It's possible; but you have to choose between versioning of the PartitionedDataSet as a whole, or of the underlying dataset. There was some back and forth about what route to take, and it eventually didn't get the green light, but I honestly so no reason why it can't be added.

good insight @Deepyaman Datta. @Rahul Kumar could you detail a bit more what you had in mind?

Thanks @Juan Luis @Deepyaman Datta For me actually any one of the approach will work actually, would prefer both so user have more choice and based on scenario user can choose 😅

choose between versioning of the PartitionedDataSet as a whole, or of the underlying dataset.

I believe treating the whole dataset as a whole will be easier for implementation, but might not be the most efficient way to store the data?

I think it's worth reading the discussion on https://github.com/kedro-org/kedro/pull/521 (and maybe especially starting from https://github.com/kedro-org/kedro/pull/521#issuecomment-744653023), rather than rethinking this from scratch. I think there was a pretty clear path forward with: 1. having the

versioned: true

be set on the top-level dataset 2. versioning the underlying dataset with that key I would be down to close this out in a tech design session or something.