Hey, I hope you are doing well. Our team is evaluating to use kedro as a framework. I was wondering if some of you have some experience with kedro integration with huggingface library. I was wondering how you are handleling models after training. My first reflex was to implement a custom dataset for this usecase not sure how much effort it is though Then maybe use a string as the model path (not very clean) for next node.

We’re very much overdue a native HuggingFace dataset, people have built custom datasets like you suggest

CustomDataSet should work, shout if you have need more help for implementation