REVIEW 15 cited by
When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning -- full-model tuning (FMT) and parameter efficient tuning (PET, including prompt tuning and LoRA), and explore their scaling behaviors in the data-limited regime where the LLM model size substantially outweighs the finetuning data size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and experiments on bilingual machine translation and multilingual summarization benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative joint scaling law between finetuning data size and each other scaling factor; 2) LLM finetuning benefits more from LLM model scaling than pretraining data scaling, and PET parameter scaling is generally ineffective; and 3) the optimal finetuning method is highly task- and finetuning data-dependent. We hope our findings could shed light on understanding, selecting and developing LLM finetuning methods.
Forward citations
Cited by 15 Pith papers
-
TacReasoner: A Dynamic Tactile-Language Framework for Interactive Reasoning in Real-World Scenarios
A dynamic-aware tactile encoder plus TouchCoT-10k chain-of-thought data lets a 7B model outperform larger tactile-language baselines on physical-property and real-world reasoning tasks.
-
Understanding Generative Recommendation with Semantic IDs from a Model-scaling View
Semantic-ID-based generative recommenders saturate as model size grows, while directly using an LLM as the recommender keeps improving with scale and learns collaborative filtering signals better.
-
Training language models to be warm and empathetic makes them less reliable and more sycophantic
Fine-tuning five LLMs for warmth raised errors by about 5 to 15 percentage points on safety-critical questions and increased sycophancy, especially when users sounded sad.
-
EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corpora
EraRAG uses hyperplane-based locality-sensitive hashing to build a hierarchical retrieval graph whose affected regions only are re-summarized when new documents arrive, cutting update cost by up to an order of magnitude.
-
Gradients: When Markets Meet Fine-tuning -- A Distributed Approach to Model Optimisation
Gradients reports that competitive, reward-driven fine-tuning beats centralized AutoML in 82 to 100 percent of comparisons, but its evaluation does not isolate competition from a much larger compute budget.
-
Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation
MFT, a corrective self-distillation objective, reports 2 to 10 times better specialization-to-degeneralization ratios than standard finetuning across many models and three specialized domains.
-
LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
LlamaRL is a distributed asynchronous RL framework that reports up to 10.7x faster RL post-training steps for 405B-parameter LLMs than a co-located synchronous baseline, using off-policy corrections and GPU-direct wei...
-
Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward
A new RL framework that rewards fine-grained reasoning steps, called Atomic Thoughts, claims better agentic deep research on seven benchmarks.
-
Can Past Experience Accelerate LLM Reasoning?
LLMs can cut reasoning compute by up to 56% on similar questions when memory of past correct solutions is combined with adaptive early stopping, though the benchmark is small and synthetic.
-
Scaling behavior of large language models in emotional safety classification across sizes and tasks
Fine-tuning a 1B LLaMA model on a synthetic emotional-safety benchmark matches or beats 70B few-shot performance and a BERT baseline on three high-data categories, using under 2GB VRAM.
-
Collaborative Editable Model
CoEM scores user-contributed knowledge fragments using user ratings and LLM attribution, keeps the high scorers in a prompt-level knowledge pool, and reports 76% agreement with FinGPT on fragment value.
-
Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.
-
Foundation Model Empowered Synesthesia of Machines (SoM): AI-native Intelligent Multi-Modal Sensing-Communication Integration
The paper proposes a systematic classification and two roadmaps for using foundation models (LLMs and wireless foundation models) to design Synesthesia of Machines systems for 6G, with preliminary case-study evidence ...
-
Rethinking the Understanding Ability across LLMs through Mutual Information
The paper uses token-level recoverability as a computable lower bound on mutual information to compare LLMs and to fine-tune them, finding encoder-only models preserve information better than decoder-only models.
-
LLM Harms: A Taxonomy and Discussion
This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.
Discussion (0). Sign in to comment.