Stacking Small Language Models for Generalizability

Laurence Liang

RECORD

Stacking Small Language Models for Generalizability

Laurence Liang

Computation and Language · Submitted 21 Oct 2024

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim By evaluating FSLM on common natural language benchmarks, this paper highlights promising early results toward generalizable performance using FSLM as a cost-effective alternative to LLMs.

how it is stated · hedged possibility 0.56 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 4,959 input tokens.

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.48 · human participants 0.04 · controlled comparison 0.15 · randomized assignment 0.01 · states quantitative result 0.04 · compares with prior results 0.48 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

Recent advances show that large language models (LLMs) generalize strong performance across different natural language benchmarks. However, the large size of LLMs makes training and inference expensive and impractical to run in resource-limited settings. This paper introduces a new approach called fine-tuning stacks of language models (FSLM), which involves stacking small language models (SLM) as an alternative to LLMs. By fine-tuning each SLM to perform a specific task, this approach breaks down high level reasoning into multiple lower-level steps that specific SLMs are responsible for. As a result, FSLM allows for lower training and inference costs, and also improves model interpretability as each SLM communicates with the subsequent one through natural language. By evaluating FSLM on common natural language benchmarks, this paper highlights promising early results toward generalizable performance using FSLM as a cost-effective alternative to LLMs.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Recent advances show that large language models (LLMs) generalize strong performance across different natural language benchmarks.	background 1.00	0.05	0.03	asserted 1.00
s1	However, the large size of LLMs makes training and inference expensive and impractical to run in resource-limited settings.	background 0.81	0.09	0.02	asserted 1.00
s2	This paper introduces a new approach called fine-tuning stacks of language models (FSLM), which involves stacking small language models (SLM) as an alternative to LLMs.	contribution 0.90	0.79	0.09	asserted 1.00
s3	By fine-tuning each SLM to perform a specific task, this approach breaks down high level reasoning into multiple lower-level steps that specific SLMs are responsible for.	method 0.99	0.51	0.06	asserted 1.00
s4	As a result, FSLM allows for lower training and inference costs, and also improves model interpretability as each SLM communicates with the subsequent one through natural language.	interpretation 0.43	0.65	0.19	asserted 0.99
s5	By evaluating FSLM on common natural language benchmarks, this paper highlights promising early results toward generalizable performance using FSLM as a cost-effective alternative to LLMs.	result 0.80	0.94	0.87	hedged possibility 0.56

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.95.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Recent advances show that large language models (LLMs) generalize strong performance across different natural language benchmarks. However, the large size of LLMs makes training and inference expensive and impractical to run in resource-limited settings. This paper introduces a new approach called fine-tuning stacks of language models (FSLM), which involves stacking small language models (SLM) as an alternative to LLMs. By fine-tuning each SLM to perform a specific task, this approach breaks down high level reasoning into multiple lower-level steps that specific SLMs are responsible for. As a result, FSLM allows for lower training and inference costs, and also improves model interpretability as each SLM communicates with the subsequent one through natural language. By evaluating FSLM on common natural language benchmarks, this paper highlights promising early results toward generalizable performance using FSLM as a cost-effective alternative to LLMs.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 2 sentences without a confident label

Uses the method or tool of 1 paper

[ref] Pythia also integrates well with LM-eval, which we use to evaluate FSLM on natural language benchmarks.

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling · Choice of Models · 0.84

Cites as support 1 paper

[ref] then show that LoRA can maintain or even improve model performance.

LoRA: Low-Rank Adaptation of Large Language Models · Model Fine-Tuning · 0.96

Cites as background 1 paper

[ref] introduced Low-Rank Adaptation (LoRA) as a fine-tuning approach.

LoRA: Low-Rank Adaptation of Large Language Models · Model Fine-Tuning · 0.80

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.