Hiya, is <https://github.com/deepyaman/kedro-accel...
# plugins-integrationsi
IƱigo Hidalgo
02/12/2024, 5:14 PMHiya, is https://github.com/deepyaman/kedro-accelerator still supported? @Deepyaman Datta
I assume not considering its age, but asking can't hurt.
If it is no longer functional, do you have any recommendations from modern kedro functionality to replicate it? Is this the usecase that cached dataset solves? https://docs.kedro.org/en/stable/_modules/kedro/io/cached_dataset.html
EDIT: answered myself: yes, cacheddataset is the kedro-native implementation
n
Nok Lam Chan
02/12/2024, 5:20 PMI think they are solving similar problems, however if you are saving the data I think the "saving" part is still a blocking operation, while
loadš 1
Nok Lam Chan
02/12/2024, 5:22 PMIt's also worth to mention, if you are not using "type safe" dataset, there could be differences.
For example "save" and "load" a CSVdataset doesn't guarantee you the same result because of automatic casting. If you are converting to Cachedataset, I would recommend to check if your output are identical after that.
i
IƱigo Hidalgo
02/12/2024, 5:22 PMOkay, for me the feature I was looking at was the loading directly from memory instead of loading from disk, so Cache works for me
IƱigo Hidalgo
02/12/2024, 5:22 PMAlso good point about the second, thank you!
IƱigo Hidalgo
02/12/2024, 5:23 PMBtw @Nok Lam Chan I know you mentioned you've been tweaking the search result weighting on the page: if there is some documentation about the cached dataset I did not find it
š 1
IƱigo Hidalgo
02/12/2024, 5:23 PM(other than the entries under the API documentation, I meant some documentation about it in the "main" docu sections)
n
Nok Lam Chan
02/12/2024, 5:24 PMhttps://noklam.github.io/blog/posts/2021-07-02-kedro-datacatalog.html
There are still a bug about Cachedataset that I haven't been able to fix.
Nok Lam Chan
02/12/2024, 5:25 PMI don't recall any specific documentation other than API docs. IMO it's a useful feature particular for interactive workflow, but there haven't been a lot of development since it is implemented a few years ago.
Nok Lam Chan
02/12/2024, 5:25 PMfeel free to create more github issues :D
i
IƱigo Hidalgo
02/12/2024, 5:42 PMOk, I posted a docs issue just for reference https://github.com/kedro-org/kedro/issues/3616
For us it's covering a very narrow usecase (a badly designed pipeline tbh) so I'm not surprised it's not more widely-used
šš¼ 1
thankyou 1
n
Nok Lam Chan
02/12/2024, 5:45 PMI think the default should be as efficient as possible, as I understand there were issues to make this generic enough. It's also a bit closer to the "orchestrator" mode where nodes doesn't communicate through shared memory, arguably not necessary for local development.
d
Deepyaman Datta
02/12/2024, 8:19 PM@Nok Lam Chan's top answer is the big difference; if you are writing a lot of datasets, the idea is there's no need to block on that.
As for maintenance, I maintained it for a while, but there were no users. š¤· I'm still not sure why. I'm open to updating it if you're going to use it; also, you're Kedro expert enough that you could probably update the hook pretty easily. :)
š 2
3 Views