https://kedro.org/ logo
#questions
Title
# questions
k

Kenny B

03/23/2023, 10:27 PM
hello, I'm trying see if the following functionality exists for versioned datasets: 1. list all available versions of the catalog item 2. limit the number of versions created of this dataset, ie - limit is 10, clean up the oldest 11th version when I save a newer version
d

Deepyaman Datta

03/23/2023, 10:43 PM
On the first point, not exactly, but it's not difficult to do by accessing the dataset object's internals: https://github.com/kedro-org/kedro/blame/0.18.7/kedro/io/core.py#L537 (hasn't changed in years) Related to the second point: https://github.com/kedro-org/kedro/issues/406
❤️ 1
d

datajoely

03/24/2023, 8:56 AM
You can deffo do this in hooks too
k

Kenny B

03/24/2023, 12:01 PM
im looking at hooks but it doesn't seem easily done? I'm trying to load the last version of my dataset to compare with the new dataframe, and then clean up old versions based on a limit. seems like a custom dataset implementation is the path forward
d

datajoely

03/24/2023, 12:02 PM
in the hook you can get the
catalog._datasets
object
❤️ 1
and introspect from there
but custom dataset should work too
k

Kenny B

03/24/2023, 2:26 PM
i ended up implementing hook for after node run. thanks for all the help
👍 1
d

Deepyaman Datta

03/24/2023, 2:58 PM
If you don't mind, feel free to throw your hook implementation on that issue above; it is something that comes up every now and then, so I'm sure it would be appreciated!
👍 1
k

Kenny B

03/24/2023, 3:11 PM
done, thank you!
❤️ 1