https://kedro.org/ logo
#plugins-integrations
Title
# plugins-integrations
i

Iñigo Hidalgo

02/12/2024, 5:14 PM
Hiya, is https://github.com/deepyaman/kedro-accelerator still supported? @Deepyaman Datta I assume not considering its age, but asking can't hurt. If it is no longer functional, do you have any recommendations from modern kedro functionality to replicate it? Is this the usecase that cached dataset solves? https://docs.kedro.org/en/stable/_modules/kedro/io/cached_dataset.html EDIT: answered myself: yes, cacheddataset is the kedro-native implementation
n

Nok Lam Chan

02/12/2024, 5:20 PM
I think they are solving similar problems, however if you are saving the data I think the "saving" part is still a blocking operation, while
load
is directly fetch from memory.
👍 1
It's also worth to mention, if you are not using "type safe" dataset, there could be differences. For example "save" and "load" a CSVdataset doesn't guarantee you the same result because of automatic casting. If you are converting to Cachedataset, I would recommend to check if your output are identical after that.
i

Iñigo Hidalgo

02/12/2024, 5:22 PM
Okay, for me the feature I was looking at was the loading directly from memory instead of loading from disk, so Cache works for me
Also good point about the second, thank you!
Btw @Nok Lam Chan I know you mentioned you've been tweaking the search result weighting on the page: if there is some documentation about the cached dataset I did not find it
👀 1
(other than the entries under the API documentation, I meant some documentation about it in the "main" docu sections)
n

Nok Lam Chan

02/12/2024, 5:24 PM
https://noklam.github.io/blog/posts/2021-07-02-kedro-datacatalog.html There are still a bug about Cachedataset that I haven't been able to fix.
I don't recall any specific documentation other than API docs. IMO it's a useful feature particular for interactive workflow, but there haven't been a lot of development since it is implemented a few years ago.
feel free to create more github issues :D
i

Iñigo Hidalgo

02/12/2024, 5:42 PM
Ok, I posted a docs issue just for reference https://github.com/kedro-org/kedro/issues/3616 For us it's covering a very narrow usecase (a badly designed pipeline tbh) so I'm not surprised it's not more widely-used
👍🏼 1
thankyou 1
n

Nok Lam Chan

02/12/2024, 5:45 PM
I think the default should be as efficient as possible, as I understand there were issues to make this generic enough. It's also a bit closer to the "orchestrator" mode where nodes doesn't communicate through shared memory, arguably not necessary for local development.
d

Deepyaman Datta

02/12/2024, 8:19 PM
@Nok Lam Chan's top answer is the big difference; if you are writing a lot of datasets, the idea is there's no need to block on that. As for maintenance, I maintained it for a while, but there were no users. 🤷 I'm still not sure why. I'm open to updating it if you're going to use it; also, you're Kedro expert enough that you could probably update the hook pretty easily. :)
😂 2
3 Views