A language-only question-answering reference game gives language models a communicative reward signal, but fine-tuning with it changes speaker behavior without improving grammatical ability.
nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
State-of-the-art language models like T5 have revolutionized the NLP landscape, but their computational demands hinder a large portion of the research community. To address this challenge, we present nanoT5, a specially-optimized PyTorch framework for efficient pre-training and fine-tuning of T5 models. Drawing on insights from optimizer differences and prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a single GPU in just 16 hours, without any loss in performance. With the introduction of this open-source framework, we hope to widen the accessibility to language modelling research and cater to the community's demand for more user-friendly T5 (Encoder-Decoder) implementations. We make our contributions, including configurations, codebase, pre-training insights, and pre-trained models, available to the public.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models
A language-only question-answering reference game gives language models a communicative reward signal, but fine-tuning with it changes speaker behavior without improving grammatical ability.