Hey, I hope you are doing well. Our team is evaluating to use kedro as a framework. I was wondering if some of you have some experience with kedro integration with huggingface library. I was wondering how you are handleling models after training.
My first reflex was to implement a custom dataset for this usecase not sure how much effort it is though
Then maybe use a string as the model path (not very clean) for next node.
03/24/2023, 4:10 PM
We’re very much overdue a native HuggingFace dataset, people have built custom datasets like you suggest
Happy to coach you through building it as it should be relatively simple and we’d love a contribution 🙂
Nok Lam Chan
03/24/2023, 9:29 PM
CustomDataSet should work, shout if you have need more help for implementation