Hi all 👋 A quick question about Kedro versioning behaviour. Is it possible to do folder-level versioning rather than dataset-level versioning? My use case: I have a single node that outputs a dozen CSV files each week. We want to keep weekly snapshots, but downloading each versioned dataset individually is a bit painful ideally we’d like all files stored under a single timestamped folder. And also to me that's a much cleaner foldering way of storing the files and understanding the weekly snapshot. At the moment I’ve implemented this by generating a timestamp myself and returning a dictionary of partition keys, e.g.:

timestamp = datetime.utcnow().strftime("%Y-%m-%dT%H-%M-%S")

return {
    f"{timestamp}/national_ftds_ftus_ratio_df": national_ftds_ftus_ratio_df,
    f"{timestamp}/future_ftds_predictions_by_month_df": future_ftds_predictions_by_month_df,
    ...
}

And my catalog entry is:

forecast_outputs:
  type: partitions.PartitionedDataset
  dataset: pandas.CSVDataset
  path: s3://.../forecast/
  filename_suffix: ".csv"

This works, but I’m not sure if I’m using

PartitionedDataset

in the most “Kedro-native” way or if there’s a better supported pattern for grouping multiple outputs under a single version. It’s a minor problem, but I’d love to hear any thoughts, best practices, or alternative approaches. Thanks!

Hey Zubin! We have documented a few options of integration between Kedro and other versioning software. Maybe one of those might be hepful to you? DVC: https://docs.kedro.org/en/stable/integrations-and-plugins/dvc/ Iceberg: https://docs.kedro.org/en/stable/integrations-and-plugins/iceberg_versioning/ Delta Lake: https://docs.kedro.org/en/stable/integrations-and-plugins/deltalake_versioning/

Thanks. Will give those a read through!

@Zubin Roy The answer is, kind of. Out of the box, Kedro is quite rigid about how it does versioning. However, you could override (or patch) the way Kedro generates versioned paths: https://github.com/kedro-org/kedro/blob/1.1.1/kedro/io/core.py#L812-L813 I don't know that this would create any problems off the top of my head, other than with versioned partitioned datasets, but you should test it. You should also be aware that this is not part of the public API; I don't personally expect it to change frequently, but it's not something you can technically rely on.

Cool. Thanks @Deepyaman Datta will check that out as well.