Pith. sign in

REVIEW 3 major objections 4 minor 50 references

Tiny language models under 3B parameters, fine-tuned with a discriminative classification head, beat label-generation training by 2–3 points and match much larger frontier models on commonsense multiple-choice tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 23:51 UTC pith:ZLR3EUYQ

load-bearing objection Clean controlled result that CLS-SFT beats label-gen by ~2–3 pts on sub-3B Qwen3 for MC tasks; the SOTA/frontier framing is softer because HellaSwag/PIQA numbers are validation, not private test. the 3 major comments →

arxiv 2607.03801 v1 pith:ZLR3EUYQ submitted 2026-07-04 cs.LG cs.AIcs.CL

Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks

Bhavesh Sood , Jaromir Savelka This is my paper
classification cs.LG cs.AIcs.CL
keywords tiny language modelsclassification-head fine-tuningLoRAmultiple-choice evaluationcommonsense reasoningon-device inferenceQwen3
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most published language-model research targets systems too large to run on phones and ordinary laptops. This paper defines Tiny Language Models (TLMs) as models below roughly three billion parameters whose weights fit in the eight-gigabyte memory budgets of mainstream consumer devices, then asks how such models should be adapted for verifiable multiple-choice work—tasks with a finite set of candidates and one correct answer. Across four Qwen3 sizes and five standard benchmarks, three LoRA fine-tuning recipes are compared under identical training budgets: ordinary label-generation (train the model to emit A/B/C/D), gold-only continuation, and a discriminative classification head that scores every candidate sequence and trains with cross-entropy over those scores. At the 0.6B and 1.7B scales the classification-head approach wins by two to three absolute points on the main commonsense suites; the resulting models reach or exceed published zero- and few-shot numbers from GPT-3 (175B), PaLM (540B) and GPT-4, and set the highest reported scores for these Qwen3 checkpoints on HellaSwag, WinoGrande and PIQA. The gap shrinks once models grow past three billion parameters, and memorization-heavy science tasks benefit less. The practical message is that previously reported weak TLM numbers were often an artifact of sub-optimal training and evaluation interfaces rather than an inherent capacity ceiling.

Core claim

For language models below roughly three billion parameters, replacing next-token label generation with a linear classification head that scores each full candidate sequence and optimizes a softmax over those scores yields a reliable two-to-three-point accuracy gain on verifiable multiple-choice benchmarks, producing systems competitive with far larger zero- and few-shot frontier models.

What carries the argument

Classification-head supervised fine-tuning (CLS-SFT): the language-modeling head is replaced by a single linear projection that maps the final non-padding hidden state of each candidate sequence to a scalar score; training applies cross-entropy over the K candidate scores of each example so the model is forced to rank gold above distractors.

Load-bearing premise

The gains and the SOTA claims rest on a single model family (Qwen3), single-task LoRA adapters, fixed chat templates, and public validation splits for two of the three main benchmarks; if those choices do not transfer, the reported superiority and frontier comparisons may not hold.

What would settle it

Repeat the identical LoRA and hyper-parameter protocol on at least one other open decoder-only family (for example Llama-3.2 1B/3B) at the same sizes; if the classification-head advantage disappears or falls below statistical significance on HellaSwag, WinoGrande and PIQA, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper defines Tiny Language Models (TLMs) as sub-3B-parameter models that fit on mainstream consumer devices (motivated by ~8 GB BF16/FP16 memory budgets). It compares three LoRA fine-tuning paradigms—instruction-formatted label generation (Label-Gen SFT), gold-only continuation, and a discriminative classification-head objective (CLS-SFT)—on Qwen3 models (0.6B–8B) across five multiple-choice benchmarks (HellaSwag, WinoGrande, PIQA, SciQ, ARC-Challenge). CLS-SFT is reported to outperform Label-Gen by +2–3 points at the 0.6B/1.7B scales (with McNemar significance on most commonsense tasks), to yield SOTA numbers for those Qwen3 TLMs on HellaSwag/WinoGrande/PIQA, and to make the fine-tuned TLMs competitive with zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4. The advantage shrinks or vanishes beyond ~3B, and is weaker on memorization-heavy science tasks.

Significance. If the CLS-SFT advantage and the absolute performance levels hold under true held-out evaluation and broader model families, the work supplies a concrete, low-cost recipe for adapting on-device TLMs to verifiable multiple-choice tasks and challenges the common assumption that sub-3B models are inadequate for commonsense reasoning. The unified LoRA budget, native-interface evaluation, McNemar tests, and extensive log-probability diagnostics (LP-Full/Sum/Avg/Char) are methodological strengths that make the internal comparison reproducible and useful to practitioners. The operational TLM definition tied to consumer memory tiers is also a clear contribution relative to looser SLM cutoffs.

major comments (3)
  1. [§5.3, Table 6, Figure 1] §5.3 and Table 6 / Figure 1: For HellaSwag and PIQA the paper evaluates on the public validation split because test labels are private, yet the abstract, introduction, and Figure 1 claim SOTA for Qwen3-0.6B/1.7B and competitiveness with GPT-3/PaLM/GPT-4 (whose published numbers are typically test-set or differently protocolled). Hyperparameters are fixed once and applied uniformly with no multi-seed or private-test confirmation. This makes the absolute SOTA and “punching above their weight” claims load-bearing but currently unsupported by true held-out numbers; either obtain/report test-set results (or a public proxy) or substantially qualify the claims and remove the SOTA framing from the abstract/title/Figure 1.
  2. [§5.2, Limitations] Limitations and §5.2: All scaling and paradigm comparisons use only the Qwen3 family. The central claim that “classification-head fine-tuning reliably outperforms label generation … at the 0.6B and 1.7B scales” and the practical TLM recipe are therefore family-specific. Without at least one additional decoder-only family (e.g., Llama-3.2 or Gemma) under the same LoRA budget, the generalization of both the TLM definition and the recommended fine-tuning interface remains untested; this is load-bearing for the practitioner takeaway in the Conclusion.
  3. [§6, Figure 1] §6.1–6.4 and Table 6: External baselines (UNICORN T5-11B SFT, GPT-3 few-shot, PaLM 0-shot, GPT-4 few-shot) are taken from published numbers that differ in training regime, prompt format, and evaluation protocol from the authors’ single-task LoRA setup. While the internal CLS-vs-Label-Gen deltas are cleanly controlled, the cross-model “competitive with 175B–540B” statements in the abstract and Figure 1 therefore mix apples and oranges; either re-evaluate the large models under a matched protocol or clearly separate the controlled ablation from the external reference comparison.
minor comments (4)
  1. [Table 3, §2] Table 3 and §2: Zero-shot numbers under LABEL-GEN vs LOGPROB already show large protocol sensitivity; a short note on why the authors chose not to report few-shot or chain-of-thought variants (even as a diagnostic) would help readers gauge how much of the TLM gap is interface versus capacity.
  2. [Appendix C.1] Appendix C.1: LoRA dropout is 0.0 for CLS and 0.05 for the causal-LM baselines; a one-sentence justification would remove any appearance of an uncontrolled hyperparameter difference.
  3. [Figure 1] Figure 1 caption and §1: “exceeds the reported GPT-3 and PaLM 540B results” should explicitly flag that the Qwen3 numbers are fine-tuned while the large-model numbers are zero-/few-shot, to avoid visual over-claim.
  4. Throughout: minor typographical inconsistencies (e.g., “TLMs” vs “Tiny LM”, occasional missing spaces around em-dashes) and the repeated arXiv-style line breaks in the provided text should be cleaned for the camera-ready version.

Circularity Check

0 steps flagged

No circularity: purely empirical LoRA comparison on held-out splits against external published baselines; no fitted constants re-used as predictions and no load-bearing self-citation chain.

full rationale

The paper is an experimental comparison of three fine-tuning objectives (CLS-SFT classification head, Label-Gen SFT, Gold-Only SFT) under a unified LoRA setup on Qwen3 models. All accuracy numbers, McNemar tests, and deltas are obtained by training on the public train split and evaluating on the designated eval split (validation for HellaSwag/PIQA because private test labels are unavailable; test otherwise). External reference numbers (UNICORN T5-11B, GPT-3, PaLM, GPT-4) are taken from the original publications and plotted for context; they are not fitted parameters of the present work. The classification head is a standard linear projection trained with cross-entropy over candidate scores and evaluated with the same head; matching train/eval interface is ordinary supervised learning, not a self-definitional reduction. No uniqueness theorem, ansatz, or prior result by the same authors is invoked to force the method. The SOTA claim is simply that the reported numbers are the highest published for these specific Qwen3 checkpoints; it does not rest on a circular derivation. Validation-versus-test protocol differences affect absolute comparability to external models but constitute an evaluation-protocol concern, not circularity of the claimed derivation. Hence score 0 with empty steps.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

Empirical ML paper; the load-bearing content is experimental rather than axiomatic. Free parameters are the usual training hyper-parameters chosen by the authors. Axioms are standard ML assumptions. The only invented entity is the operational category “TLM.”

free parameters (3)
  • LoRA rank r / alpha = r=16, α=32
    Fixed at r=16, α=32 for all runs; not swept.
  • learning rate / epochs / batch = 2e-4 / 2 / 64
    2e-4, 2 epochs, effective batch 64; chosen once and applied uniformly.
  • TLM parameter cutoff = ~3B / 8 GB
    Operational definition “below roughly 3 B / <8 GB BF16” chosen to match consumer-device memory; not derived.
axioms (3)
  • domain assumption LoRA adapters on a frozen backbone plus a linear classification head are sufficient to adapt decoder-only models for candidate ranking.
    Standard PEFT assumption used throughout §4–5; not re-proved.
  • standard math Cross-entropy over scalar candidate scores is a valid discriminative objective for verifiable multiple-choice tasks.
    Ordinary multi-class CE; §4.3.
  • domain assumption Public validation splits of HellaSwag and PIQA are adequate proxies for the private test sets.
    Stated in §5.3; common practice but unverified here.
invented entities (1)
  • Tiny Language Models (TLMs) independent evidence
    purpose: Operational category of models <~3 B that fit consumer devices with headroom for OS/KV cache.
    New named cut-off motivated by 2025–2026 device memory tables; independent evidence is the hardware survey in Appendix A, but the label itself is author-defined.

pith-pipeline@v1.1.0-grok45 · 26818 in / 2827 out tokens · 27220 ms · 2026-07-11T23:51:29.571495+00:00 · methodology

0 comments
read the original abstract

We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices. We study how to adapt them for and use them on verifiable multiple-choice tasks. We compare three LoRA-based fine-tuning paradigms (label generation, gold only, and our discriminative classification head) on a unified setup across several Qwen3 models from 0.6B to 8B and five benchmarks: HellaSwag, WinoGrande, PIQA, SciQ and ARC-C. Classification-head fine-tuning reliably outperforms label generation (+2-3%) at the 0.6B and 1.7B scales. Further, TLMs fine-tuned using the discriminative method are competitive to zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4. The performance we report for Qwen3-0.6B and Qwen3-1.7B are SOTA on HellaSwag, WinoGrande, and PIQA.

Figures

Figures reproduced from arXiv: 2607.03801 by Bhavesh Sood, Jaromir Savelka.

Figure 1
Figure 1. Figure 1: Accuracy on three commonsense benchmarks for Qwen3 + LoRA models fine-tuned with Label-Gen [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 10 linked inside Pith

  1. [2]

    arXiv preprint arXiv:2410.20011 , year=

    A Survey of Small Language Models , author=. arXiv preprint arXiv:2410.20011 , year=

    Pith/arXiv arXiv

  2. [3]

    Instruction-Based Approaches , author =

    Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches , author =. 2025 , eprint =

    2025

  3. [4]

    2025 , publisher =

    Srivastava, Gaurav and Cao, Shuxiang and Wang, Xuan , booktitle =. 2025 , publisher =

    2025

  4. [5]

    Advances in Neural Information Processing Systems , volume =

    Language Models are Few-Shot Learners , author =. Advances in Neural Information Processing Systems , volume =

  5. [6]

    2024 , eprint =

    Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models , author =. 2024 , eprint =

    2024

  6. [7]

    Right Answer, Wrong Score: Uncovering the Inconsistencies of

    Molfese, Francesco Maria and Moroni, Luca and Gioffr. Right Answer, Wrong Score: Uncovering the Inconsistencies of. Findings of the Association for Computational Linguistics: ACL 2025 , pages =. 2025 , publisher =

    2025

  7. [8]

    Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with

    Sanz-Guerrero, Mario and Bui, Minh Duc and von der Wense, Katharina , booktitle =. Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with. 2025 , publisher =. doi:10.18653/v1/2025.emnlp-main.988 , url =

    doi:10.18653/v1/2025.emnlp-main.988 2025

  8. [9]

    Efficient Multi-Prompt Evaluation of

    Polo, Felipe Maia and Xu, Ronald and Weber, Lucas and Silva, M. Efficient Multi-Prompt Evaluation of. 2024 , eprint =

    2024

  9. [10]

    arXiv preprint arXiv:2409.15790 , year =

    Small Language Models: Survey, Measurements, and Insights , author =. arXiv preprint arXiv:2409.15790 , year =

    Pith/arXiv arXiv

  10. [11]

    arXiv preprint arXiv:2501.05465 , year =

    Small Language Models (SLMs) Can Still Pack a Punch: A Survey , author =. arXiv preprint arXiv:2501.05465 , year =

    Pith/arXiv arXiv

  11. [12]

    arXiv preprint arXiv:2411.03350 , year =

    A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness , author =. arXiv preprint arXiv:2411.03350 , year =

    Pith/arXiv arXiv

  12. [13]

    arXiv preprint arXiv:2308.14363 , year =

    Mobile Foundation Model as Firmware , author =. arXiv preprint arXiv:2308.14363 , year =

    Pith/arXiv arXiv

  13. [14]

    Scientific Reports , volume =

    On-device Query Intent Prediction with Lightweight LLMs to Support Ubiquitous Conversations , author =. Scientific Reports , volume =

  14. [15]

    Haque, Mohd Ariful and Rahman, Fahad and Gupta, Kishor Datta and Shujaee, Khalil and George, Roy , journal =

  15. [16]

    Sensors , volume =

    Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions , author =. Sensors , volume =. 2025 , doi =

    2025

  16. [17]

    Kandala, Savitha Viswanadh and Medaranga, Pramuka and Varshney, Ambuj , journal =

  17. [18]

    Proceedings of the 41st International Conference on Machine Learning , year =

    Rethinking Optimization and Architecture for Tiny Language Models , author =. Proceedings of the 41st International Conference on Machine Learning , year =

  18. [19]

    A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with

    Wang, Fali and Zhang, Zhiwei and Zhang, Xianren and Wu, Zongyu and Mo, Tzuhao and Lu, Qiuhao and Wang, Wanjing and Li, Rui and Xu, Junjie and Tang, Xianfeng and others , journal=. A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with

  19. [20]

    Journal of Machine Learning Research , volume =

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , author =. Journal of Machine Learning Research , volume =. 2020 , url =

    2020

  20. [21]

    2021 , url =

    Lourie, Nicholas and Le Bras, Ronan and Bhagavatula, Chandra and Choi, Yejin , booktitle =. 2021 , url =

    2021

  21. [22]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Language Models are Few-Shot Learners , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  22. [23]

    Chowdhery, Aakanksha and Narang, Sharan and Devlin, Jacob and Bosma, Maarten and Mishra, Gaurav and Roberts, Adam and Barham, Paul and Chung, Hyung Won and Sutton, Charles and Gehrmann, Sebastian and Schuh, Parker and Shi, Kensen and Tsvyashchenko, Sasha and Maynez, Joshua and Rao, Abhishek and Barnes, Parker and Tay, Yi and Shazeer, Noam and Prabhakaran,...

    2023

  23. [24]

    International Conference on Learning Representations , year =

    LoRA: Low-Rank Adaptation of Large Language Models , author =. International Conference on Learning Representations , year =

  24. [25]

    arXiv preprint arXiv:2505.09388 , year =

    Qwen3 Technical Report , author =. arXiv preprint arXiv:2505.09388 , year =

    Pith/arXiv arXiv

  25. [26]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    Transformers: State-of-the-Art Natural Language Processing , author =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =. 2020 , url =

    2020

  26. [27]

    Psychometrika , volume =

    Note on the sampling error of the difference between correlated proportions or percentages , author =. Psychometrika , volume =. 1947 , publisher =

    1947

  27. [28]

    Neural Computation , volume =

    Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , author =. Neural Computation , volume =. 1998 , publisher =

    1998

  28. [29]

    Gunter, Tom and Wang, Zirui and Wang, Chong and Pang, Ruoming and Narayanan, Andy and Zhang, Aonan and Zhang, Bowen and Chen, Chen and Chiu, Chung-Cheng and Qiu, David and others , journal=

  29. [30]

    arXiv preprint arXiv:2507.13575 , year=

    Pith/arXiv arXiv

  30. [31]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and others , journal=. The

  31. [32]

    2024 , howpublished=

    2024

  32. [33]

    Liu, Zechun and Zhao, Changsheng and Iandola, Forrest and Lai, Chen and Tian, Yuandong and Fedorov, Igor and Xiong, Yunyang and Chang, Ernie and Shi, Yangyang and Krishnamoorthi, Raghuraman and Chandra, Vikas , booktitle=

  33. [34]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  34. [35]

    Publications Manual , year = "1983", publisher =

    1983

  35. [36]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    doi:10.1145/322234.322243 1981

  36. [37]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  37. [38]

    Dan Gusfield , title =. 1997

    1997

  38. [39]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  39. [40]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  40. [41]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =

    Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin , title =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =. 2019 , publisher =. doi:10.18653/v1/P19-1472 , url =

    doi:10.18653/v1/p19-1472 2019

  41. [42]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Sakaguchi, Keisuke and Le Bras, Ronan and Bhagavatula, Chandra and Choi, Yejin , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2020 , doi =

    2020

  42. [43]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Bisk, Yonatan and Zellers, Rowan and Le Bras, Ronan and Gao, Jianfeng and Choi, Yejin , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2020 , doi =

    2020

  43. [44]

    2024 , eprint =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

    2024

  44. [45]

    2503.19786 , archivePrefix =

    Gemma 3 Technical Report , year =. 2503.19786 , archivePrefix =

    Pith/arXiv arXiv

  45. [46]

    2024 , eprint =

    The. 2024 , eprint =

    2024

  46. [47]

    Proceedings of the 3rd Workshop on Noisy User-generated Text , pages=

    Crowdsourcing multiple choice science questions , author=. Proceedings of the 3rd Workshop on Noisy User-generated Text , pages=

  47. [48]

    arXiv preprint arXiv:1803.05457 , year=

    Think you have solved question answering? try arc, the ai2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    Pith/arXiv arXiv

  48. [49]

    Advances in Neural Information Processing Systems , volume=

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning , author=. Advances in Neural Information Processing Systems , volume=

  49. [50]

    arXiv preprint arXiv:2305.15717 , year=

    The false promise of imitating proprietary llms , author=. arXiv preprint arXiv:2305.15717 , year=

    Pith/arXiv arXiv

  50. [51]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Small models struggle to learn from strong reasoners , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025