Pith. sign in

REVIEW 3 major objections 5 minor 10 references

This paper argues that once a LoRA adapter has enough capacity, the quality of the training QA data — especially brevity of gold answers — is the dominant lever on closed-book question answering, with a single curation pass raising accuracy

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:26 UTC pith:Z2UOLCYY

load-bearing objection Honest, well-measured paper with a real effect, but the headline claim about data quality being the dominant lever is not fully identified; one more ablation would settle it. the 3 major comments →

arxiv 2607.21861 v1 pith:Z2UOLCYY submitted 2026-07-23 cs.CL cs.AI

Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

Joan Figuerola Hurtado This is my paper
classification cs.CL cs.AI
keywords closed-book QALoRAknowledge internalizationdata qualityanswer brevitycapacity scalingretrieval-augmented generationmetric bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that internalizing documents into a small language model's weights via LoRA (no retrieval, no context window) is governed more by the quality of the training question-answer data than by model capacity or architecture. Across about 100 runs, the authors find that short, canonical 1–6 word gold answers and dropping trivia produce the largest accuracy gains — +28 points on 15 documents — exceeding rank changes, learning-rate tuning, and two alternative architectures. The paper also shows that capacity is a hard gate: below a minimum rank, no data intervention helps, and rank must grow with corpus size. A real retrieval baseline shows the internalized adapter outperforms BM25-RAG and a gold-chunk oracle on the same questions and scorer.

Core claim

The central claim is that data learnability, measured by how short and how low-entropy the gold answers are, is the dominant lever for closed-book QA once adapter capacity is sufficient. A single curation pass — shortening answers to 1–6 word spans, removing trivia, de-duplicating — moved overall accuracy from 57.7% to 85.7% on 15 documents, a gain larger than any architectural change tested. The effect is not a scoring artifact: re-scoring under exact match widens the answer-length gradient. The paper also reports a capacity trend (rank must double roughly when documents double) and a rank/learning-rate coupling that initially masqueraded as a capacity ceiling.

What carries the argument

The central object is a low-rank adapter (LoRA) trained in two stages — completion-style pretraining on chunks plus paraphrases, then closed-book SFT with answer-token loss — on a 4-bit Gemma model. The load-bearing identity is the relation between gold-answer length/entropy and the achievable validation loss floor: short canonical answers yield loss floors near 0.05 and high recall, while verbose high-entropy trivia yield floors near 0.29 and poor recall. The paper uses this loss-floor signature to diagnose capacity saturation versus interference and to attribute gains to data quality.

Load-bearing premise

The load-bearing premise is that the accuracy gain from curation is caused by learnability of short answers and removal of trivia, not by the intrinsic difficulty of reproducing longer gold answers or by the deterministic scorer's stylistic match to the fine-tuned adapter; the paper explicitly leaves the answer-length-difficulty confound to a controlled ablation and flags the metric bias.

What would settle it

A controlled ablation that canonicalizes gold answers to short spans while keeping the same question set and document content, scored by a paraphrase-invariant LLM judge across multiple seeds; if accuracy does not rise (or the gain shrinks below the RAG margin), the data-quality-dominates claim is weakened. Also, re-running the 15-doc comparison with a judge that ignores phrasing; if the internalized adapter no longer beats the gold-chunk oracle, the retrieval-baseline superiority is partly a metric artifact.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If data quality dominates, practitioners should invest in answer canonicalization and trivia filtering before spending on larger adapters or fancier architectures.
  • The rank-must-grow-with-corpus trend and rank-LR coupling give a practical scaling recipe: double rank roughly with corpus size, and halve learning rate when rank doubles.
  • Internalization can beat retrieval at small corpus scale on both accuracy and latency, at least under a style-aligned deterministic scorer.
  • Distinguishing capacity saturation (uniform collapse, rising loss floor) from interference (rising mis-binding) tells you whether to add capacity or change data.
  • The loss-floor diagnostic can be used as a cheap early-warning signal during training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's +28 point curation gain likely transfers to other closed-book tasks but may shrink under a paraphrase-invariant judge; the reported margin over RAG is a lower bound for RAG's knowledge.
  • A controlled ablation that canonicalizes answers while holding the question set and document content fixed would directly test whether intrinsic difficulty of long answers contributes; if it does, the 'data quality' claim should be restated as 'data learnability at fixed difficulty.'
  • The capacity gate suggests a practical rule-of-thumb: before blaming data, check whether rank is adequate for corpus size; the paper's own misdiagnoses show LR retuning is the first thing to try.
  • Multi-seed and a reader-independent metric (LLM judge) would firm up the RAG comparison; until then, the 19–25 point margins are upper bounds for the advantage under a fairer scorer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies closed-book question answering by fine-tuning a 4-bit Gemma model with LoRA adapters that internalize a document corpus. Across roughly 100 single-seed runs, it reports that once adapter capacity is adequate, training-data quality—specifically the brevity and canonicality of gold answers—is the dominant lever, outweighing LoRA rank, learning-rate changes, and two alternative architectures. A single curation pass (short canonical 1–6 word answers, dropped trivia, deduplication) is claimed to raise 15-document closed-book accuracy from 57.7% to 85.7%. The paper also reports capacity must scale with corpus size, that larger rank requires a gentler learning rate, and that the internalized adapter beats a BM25-RAG baseline and a gold-chunk oracle on the same deterministic scorer. The paper is unusually transparent about its threats: single-seed runs, a metric that favors the adapter, a 4-bit quantization confound, and an unresolved answer-length difficulty confound.

Significance. If the central claim holds, the paper provides a practically important and counterintuitive empirical result: data curation can outperform capacity and architectural innovation for knowledge internalization. The work is also valuable as an honest case study of empirical debugging, with explicit documentation of misdiagnoses (the 'oracle RAG' copy-task, the sequence-length confound, the rank/LR entanglement). Strengths include Appendix B's exact-match rescoring showing the brevity gradient is not a scoring artifact, the CL-100 cohort re-authoring replicating the data-quality finding at roughly 7× scale, and the retained evaluation harness allowing re-scoring from saved predictions. However, the headline causal claim—that answer brevity specifically drives the gain—is not yet identified, because the curation pass changed multiple factors and the paper explicitly leaves the intrinsic-difficulty confound to future work. The single-seed nature of all runs also limits the strength of the comparative claims.

major comments (3)
  1. [§3, Table 1; Appendix B] The paper's central claim—that data quality, specifically answer brevity, is the dominant lever—rests on the +28-point curation gain (57.7% → 85.7%). The curation pass changed at least three correlated factors: gold-answer length, trivia content, and deduplication. Appendix B's final sentence explicitly acknowledges that 'longer answers are intrinsically harder to reproduce' remains a residual confound. This is load-bearing, because if longer gold answers are intrinsically harder for any model to reproduce, then canonicalization makes the task easier independently of 'learnability' or 'quality.' I request the proposed canonicalization-only ablation: fix the question set, fact content, and deduplication policy, and compare training against (a) original verbose gold answers and (b) canonical short gold answers for the same facts, holding all other recipe elements fixed. Additionally, re-sc
  2. [§4, Threats; Table 1] All runs are single-seed (seed 42). The abstract's claim that data quality 'outweighs' capacity, learning rate, and architecture changes, and the specific null result for EntiGraph (84.0% → 82.1%) and the rank/LR interaction (55.2% → 61.6%), are presented without variance estimates. The paper is transparent about this, but the strength of the comparative claims requires at least a small multi-seed replication (3 seeds) of the key rows in Table 1 and Table 2. Without it, sub-2-point differences cannot be distinguished from noise, and the 'dominant lever' framing is stronger than the evidence supports.
  3. [Table 3; §4] The internalization-beats-RAG comparison uses the same deterministic token-F1/exact-match scorer that the paper concedes 'favors the style-matched adapter.' The BM25 hit-rate is a useful reader-independent control, but the headline accuracy margins (84.2% vs. 58.9% for BM25-RAG; 84.2% vs. 65.6% for the gold-chunk oracle) may substantially understate the RAG reader's knowledge if it produces valid but paraphrastic answers. To support the abstract's claim that internalization beats retrieval, the same predictions should be re-scored with a paraphrase-invariant judge or human annotations. Until then, the RAG accuracy numbers should be described as lower bounds, as the Discussion already partially does.
minor comments (5)
  1. [§3, Table 1] The caption reports 'N=1,772–3,990 held-out questions' but does not specify which row corresponds to which N. Please make the per-row question counts explicit.
  2. [§3, EntiGraph sentence] The text says 'EntiGraph-style synthetic-CPT augmentation ... moved overall accuracy 84.0%→ 82.1%,' but Table 1 does not include an 84.0% baseline. Clarify which configuration was augmented to produce the 84.0% starting point.
  3. [§3, 'paid-teacher data'] The term 'paid-teacher data' is introduced without definition. Provide a brief description of how this data was produced (model, prompt, verification) or point to a supplement.
  4. [Appendix B] Appendix B reports re-scoring 3,792 saved predictions, while Table 3 reports N=2,009 for the 15-document evaluation. Clarify the relationship between these question sets (e.g., known-answer recall subset vs. full held-out set).
  5. [Abstract and §3] The phrase 'closed-book accuracy' is used to mean accuracy on questions about documents seen during training but with no context in the prompt. Consider using 'closed-book QA accuracy' consistently to avoid ambiguity with unseen-domain closed-book benchmarks.

Circularity Check

0 steps flagged

No circular derivation: the curation result is an empirical comparison with an acknowledged but non-circular confound.

full rationale

The paper's central claim is that training-data quality (answer brevity, dropped trivia) is a larger lever than capacity or architecture. This is an empirical finding from controlled runs, not a result derived from its own conclusion. The curation gain is not obtained by fitting a parameter to the held-out metric and then reporting that fit as a prediction; it is a direct before/after comparison under a fixed scorer. The metric-bias concern is handled by an exact-match rescoring in Appendix B, which shows the brevity gradient widens rather than disappears. The paper explicitly flags the remaining limitation: 'The residual confound is that longer answers are intrinsically harder to reproduce, which a canonicalization-only ablation on a fixed question set would isolate' (Appendix B) and 'Metric favors the adapter' (§4). These are threats to causal identification, not circularity: the paper does not define data quality in terms of the metric, does not rely on self-citations for any load-bearing step, and does not rename a known result. The acknowledged confound means the +28-point gain may be partly explained by intrinsic difficulty of longer gold answers, but that is a validity risk, not a case of the derivation being equivalent to its inputs by construction. Honest non-finding: no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper is an empirical recipe study; it introduces no mathematical derivation, so its 'free parameters' are hand-set curation/eval thresholds and important hyperparameters. Its axioms are mostly domain assumptions about the faithfulness of self-generated QA data, the diagnostic meaning of loss floors, and single-seed representativeness - each flagged by the authors as a threat in §4. No new entities are postulated.

free parameters (3)
  • Gold answer brevity cap = 1-6 words
    The curation pass enforces canonical 1-6 word gold answers; the +28-point effect is defined relative to this rule, and the paper does not vary the cap to test sensitivity (Abstract, §3, Appendix B).
  • Token-F1 acceptance threshold = 0.74
    A prediction is correct on exact substring match or token-F1 >= 0.74; changing this threshold changes pass rates and could alter the ranking of levers, though exact-match rescoring partially addresses it (§2, Appendix B).
  • Peak learning rate = 4e-5 (2e-5 for rank 256)
    Hand-tuned per rank; the rank-256 result depended on halving LR, showing the claim 'rank 256 regresses' was an LR artifact (Table 2, §3).
axioms (4)
  • domain assumption The self-generated QA data is factually faithful enough that its effect on model accuracy can be attributed to learnability (answer length/trivia) rather than factual error rate
    The paper distinguishes learnability from factual quality (§3, 'Ablations ruled out ... factual quality') but does not show an independent factuality audit, so a hidden factuality difference between teacher and self-generated data could confound the headline.
  • domain assumption Rising training loss floor is a valid signature of adapter capacity saturation, separable from optimization/quantization effects
    The capacity-gate claim (rank 64 collapses at CL-50; rank 128 restores) is read off loss-floor signatures (Allen-Zhu and Li 2024b; Back et al. 2026); the paper itself notes 4-bit quantization could set part of the floor (§4).
  • ad hoc to paper Single-seed runs (seed 42) are representative of the stated comparisons
    All numbers are single-seed; the paper states sub-2-point deltas are within noise but treats 3-28 point deltas as load-bearing without multi-seed replication (§1, §4).
  • domain assumption The 4-bit Gemma-4-e4b base model retains enough pretrained competence that LoRA-on-MLP-only can internalize document facts
    The recipe assumes quantization and MLP-only LoRA do not preclude knowledge storage; no higher-precision control is run (§4 explicitly lists this as an open threat).

pith-pipeline@v1.3.0-alltime-deepseek · 5165 in / 16820 out tokens · 162702 ms · 2026-08-01T06:26:15.569547+00:00 · methodology

0 comments
read the original abstract

We study baking documents directly into the weights of a 4-bit Gemma-4-e4b model via LoRA, so a system can answer questions about a corpus closed-book: no retrieval and no context-window budget. Across roughly 100 training runs from single documents to a 99-document corpus, we find that once adapter capacity is adequate, training-data quality is the dominant lever on closed-book accuracy, outweighing LoRA rank, learning rate, and two alternative architectures combined; capacity itself is a hard gate below which no data intervention helps. A single curation pass (shortening gold answers to canonical 1-6 word spans and dropping trivia) moved closed-book accuracy from 57.7% to 85.7% on a 15-document corpus, a larger jump than any architectural change. We confirm a capacity trend (rank must grow with corpus size) entangled with a coupling between rank and learning rate that we initially misdiagnosed. On a 15-document slice we add a real retrieval baseline: the internalized adapter (84.2% recall) beats a BM25-RAG pipeline with a base reader (58.9%) and even a realistic gold-chunk oracle (65.6%) at lower latency. We report the full arc, including three misdiagnoses, as a case study in debugging LLM training empirically.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

10 extracted references · 10 linked inside Pith

  1. [1]

    Zeyuan Allen-Zhu and Yuanzhi Li. 2024 a . https://arxiv.org/abs/2309.14316 Physics of language models: Part 3.1, knowledge storage and extraction . arXiv preprint arXiv:2309.14316. ICML 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Zeyuan Allen-Zhu and Yuanzhi Li. 2024 b . https://arxiv.org/abs/2404.05405 Physics of language models: Part 3.3, knowledge capacity scaling laws . arXiv preprint arXiv:2404.05405

    Pith/arXiv arXiv 2024

  3. [3]

    Seungwoo Back et al. 2026. https://arxiv.org/abs/2603.01097 Understanding LoRA as knowledge memory: An empirical analysis . arXiv preprint arXiv:2603.01097. ICML 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Sabri Eyuboglu et al. 2025. https://arxiv.org/abs/2506.06266 Cartridges: Lightweight and general-purpose long context representations via self-study . arXiv preprint arXiv:2506.06266

    Pith/arXiv arXiv 2025

  5. [5]

    Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. https://arxiv.org/abs/2405.05904 Does fine-tuning LLMs on new knowledge encourage hallucinations? In Proceedings of EMNLP

    Pith/arXiv arXiv 2024

  6. [6]

    Momchil Hardalov, Gonzalo Iglesias, and Adri\`a de Gispert. 2026. https://arxiv.org/abs/2606.04557 Cartridges at scale: Training modular KV caches over large document collections . arXiv preprint arXiv:2606.04557

    Pith/arXiv arXiv 2026

  7. [7]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 LoRA : Low-rank adaptation of large language models . arXiv preprint arXiv:2106.09685

    Pith/arXiv arXiv 2021

  8. [8]

    Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. https://arxiv.org/abs/2401.16380 Rephrasing the web: A recipe for compute and data-efficient language modeling . arXiv preprint arXiv:2401.16380

    Pith/arXiv arXiv 2024

  9. [9]

    Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2024. https://arxiv.org/abs/2312.05934 Fine-tuning or retrieval? comparing knowledge injection in LLMs . arXiv preprint arXiv:2312.05934

    Pith/arXiv arXiv 2024

  10. [10]

    Zitong Yang, Neil Band, Shuangping Li, Emmanuel Cand\`es, and Tatsunori Hashimoto. 2024. https://arxiv.org/abs/2409.07431 Synthetic continued pretraining . arXiv preprint arXiv:2409.07431

    Pith/arXiv arXiv 2024