ladies&gents, any plan to make partitioned dat...
# questionsg
Gauthier Pierard
04/01/2025, 12:14 PMladies&gents, any plan to make partitioned datasets compatible with versioning?
n
Nok Lam Chan
04/01/2025, 12:18 PMIIRC, it support versioning already? https://github.com/kedro-org/kedro-plugins/pull/447
g
Gauthier Pierard
04/01/2025, 1:02 PM@Nok Lam Chan any idea why I get this on kedro, version 0.19.12 ?
Copy code
DatasetError:
PartitionedDataset.__init__() got an unexpected keyword argument 'version'.
Dataset 'usp.contextualized_deltav' must only contain arguments valid for the constructor of
'kedro_datasets.partitions.partitioned_dataset.PartitionedDataset'. Copy code
mynamespace.mydataset:
versioned: true
type: partitions.PartitionedDataset
path: ${_data_prefix}/TEMP/PARTITION_TEST/
dataset: pandas.ParquetDataset
filename_suffix: ".parquet"
overwrite: false
credentials: adls_credsn
Nok Lam Chan
04/01/2025, 1:24 PMhm
Nok Lam Chan
04/01/2025, 1:25 PMI think the versioned key may goes under your
DatasetNok Lam Chan
04/01/2025, 1:25 PM Copy code
mynamespace.mydataset:
type: partitions.PartitionedDataset
path: ${_data_prefix}/TEMP/PARTITION_TEST/
dataset: pandas.ParquetDataset
versioned: true <- something like this I think
filename_suffix: ".parquet"
overwrite: false
credentials: adls_credsg
Gauthier Pierard
04/01/2025, 2:24 PMthanks, that works. however not suited to my case where I'd like to save all files in one folder
n
Nok Lam Chan
04/01/2025, 2:32 PMhmm
Nok Lam Chan
04/01/2025, 2:32 PMWhat's the current behavior?
Nok Lam Chan
04/01/2025, 2:34 PM@Deepyaman Datta do you remember this?
g
Gauthier Pierard
04/01/2025, 2:34 PMeach key of the partition has its own folder in which the timestamped versions exist (each one in a subfolder)
Gauthier Pierard
04/01/2025, 2:35 PMby the way my current implementation is described here
https://kedro-org.slack.com/archives/C03RKP2LW64/p1743172644361429
basically creating a dynamic dataset in one of the nodes
d
Deepyaman Datta
04/01/2025, 2:44 PMIt's been a while. https://github.com/kedro-org/kedro-plugins/pull/447 introduced versioning of the underlying dataset.
Deepyaman Datta
04/01/2025, 2:46 PMSo yes, what @Nok Lam Chan said; the key should be on the underlying dataset
👍 1
Deepyaman Datta
04/01/2025, 2:52 PMIf I understand @Gauthier Pierard you want to define partitions (and potentially how they evolve) for each version? I think that's very reasonable, but confusing to support in Kedro due to how partitions and versions are both defined by folder structure. It was a very long battle to even agree to get this way of supporting versioning in, while making assumptions.
Your options are (1) create a custom dataset, take some assumptions or (2) look into using something like Iceberg for storage, which should define this behavior much more reasonably.
g
Gauthier Pierard
04/01/2025, 2:54 PMYes the users want all their files in one folder per run basically. Thanks for the suggestions
d
Deepyaman Datta
04/01/2025, 2:56 PMImproving partitioned and incrementql dataset is something want to tackle at some point, but I don't know when that would be unfortunately (and it's not at all clear what that would mean)
Cc @Juan Luis @Merel just in case you think it's worth recording from a priorities perspective :)