Pith. sign in

REVIEW 3 major objections 3 minor 47 references

Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A benchmark gain is not evidence of expanded capability unless the reachable set also grows; in DAPO/DART, deployed accuracy rose 14.7 points while oracle reachability fell 13.3 points.

desk verdict A real and useful evaluation distinction, with a fixed-probe ceiling that probably overstates how little RLVR expands reachability. read the letter →

arxiv 2608.03219 v1 pith:PMDL2TSU submitted 2026-08-04 cs.AI cs.CL

Yanchao Li , Wanhao Liu , Jiaqing Xie , Ben Gao , Yanbo Wang , Tianfan Fu , Yuqiang Li This is my paper · ORCID
classification cs.AIcs.CL
keywords LLMevaluationreachabilityrealizationbenchmarkgainsRLVRlayerroutingMLPlocalizationquestion-levelaudit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a rising benchmark score cannot be read as a growing capability unless we know whether the model is solving new questions or merely producing answers it could already reach. To separate these, it defines a question as realized when the default deployment produces the correct answer, and reachable when a fixed sampling probe finds the correct answer within a specified budget. Auditing layer routing, MLP-block interventions, and six matched RLVR checkpoints, the paper finds that random layer routes match or beat structured search, that silencing one identified MLP block repairs most failures where the model recognises but cannot produce the correct answer, and that RLVR training raises deployed performance while the reachable ceiling stays flat or falls—in the DAPO case, +14.7 points deployed and −13.3 points reachable. The conclusion is that realization and reachability move apart, so capability claims should report both under matched conditions.

What carries the argument

The load-bearing machinery is the two-number question-level audit $(d_i, o_i(K))$. $d_i$ flags whether the deployment output is correct; $o_i(K)$ flags whether any of $K$ probe candidates is correct. Their aggregates $D$ and $O_K$ are called deployed performance and oracle reachability (or the reachable ceiling). The same pair is recorded for every question on both sides of a matched comparison, so a training run or inference intervention can be classified as changing realization, reachability, both, or neither. The audit is only defined relative to a stated protocol—question set, answer format, grading rule, temperature, probe, and budget—and the oracle label means correctness is used after

What would settle it

Run the DAPO/DART matched comparison (base vs trained, n=300, K=200, temperature 0.8) again with K=800 and a second temperature (e.g. 1.0) and an alternative prompt format. If the trained model's $O_K$ recovers to at least the base level under any of those probes, the 13.3-point reachability drop fails to generalize and the 'reachable ceiling' is probe-dependent. Also check whether the newly realized questions are exactly the base model's high-hit-rate bin (above 35% at K=200); a different propensity profile would falsify the concentration claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that a benchmark gain is underdetermined: the same aggregate score change can come from expanding the set of questions a model can reach, or from making already-reachable answers appear more reliably. It makes the distinction operational by scoring every question twice: $d_i$ for whether the default deployment produces the correct answer, and $o_i(K)$ for whether any of $K$ candidates from a fixed probe is correct. Aggregated, these are deployed performance $D$ and oracle reachability $O_K$, and the comparison is only meaningful when base and trained checkpoints share question set, answer format, temperature, budget, and grader. The empirical payload is threefold

Load-bearing premise

The load-bearing premise is that $O_K$—sampling $K$ candidates at temperature 0.8 under a fixed format and grader—is the right operational definition of reachability; the paper itself concedes that an answer not found by a finite probe is unobserved, not impossible, so a flat or falling ceiling after RLVR could be a probe artifact if training shifts the sampling distribution (format collapse, output length changes, or reduced diversity).

Editorial extensions

If this is right

  • A rising deployed score with a flat or falling $O_K$ is a realization gain, not a reachability expansion; capability claims should report both $D$ and $O_K$ under matched conditions.
  • RLVR tends to realize questions the base model already sampled frequently, so training gains can be concentration effects rather than additions to the solvable set.
  • Structured layer routing should not be credited with expanding what a model can answer unless it beats budget-matched random routes under the same scoring; answer-blind selection recovers almost none of the oracle headroom.
  • Recognition-correct/generation-wrong failures can be causally localized to a single MLP block in some models, giving an intervention-level handle on where realized answers are lost.
  • Whether RLVR raises or lowers the reachable ceiling depends on the recipe—math-only RLVR lowered oracle reachability in one OLMo lineage while SFT-containing variants raised it—so training-recipe controls are part of any reachability claim.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the finite-probe caveat opens a direct artifact check. Re-probe DAPO/DART with higher $K$, a different temperature, and a different prompt format; if the trained checkpoint's $O_K$ stops falling, the measured reachability contraction is an artifact of the probe's sampling distribution rather than a loss of solvable questions.
  • Editorial extension: the routing result suggests the limiting factor for test-time scaling is candidate selection, not candidate generation; ordinary sampling already reproduces the gap without layer routing, so any new answer-blind selector should be compared against majority voting at equal budget.
  • Editorial extension: the predefined failure set (recognizes correctly, generates wrong) can serve as a public benchmark for mechanistic interventions—if a proposed edit repairs a large fraction of that panel without breaking healthy questions, it has a concrete measure to beat.
  • Editorial extension: benchmark designers could publish $D$ and $O_K$ as a standard two-number scorecard, making 'capability expansion' claims falsifiable and comparable across labs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces a question-level audit framework that distinguishes 'realized' answers (produced by the default deployment procedure) from 'reachable' answers (found by a fixed probe within a budget K). It then applies this framework to three settings: inference-time layer routing, MLP-block silencing, and RLVR training. The routing experiments report that random layer paths match or exceed structured oracle searches in 43 'clean' model–task cells, while answer-blind selectors recover little of the oracle gain. The MLP experiments localize a predefined recognition-correct/generation-wrong failure to a single MLP block in six cases, with component, specificity, and reciprocal controls. The RLVR experiments report that deployed accuracy rises while oracle reachability stays flat or falls, with DAPO as the sharpest case (+14.7 deployed points, -13.3 reachability points), and that newly realized questions were already base-reachable. The paper concludes that benchmark gains do not by themselves establish capability expansion, and that both realized performance and reachability should be reported.

Significance. If the empirical claims hold, the distinction between realization and reachability is a useful evaluation discipline and could change how benchmark gains are interpreted. The paper has real strengths: matched budgets and temperatures, random baselines for routing, predefined failure sets for MLP localization, reciprocal perturbation controls, and public code. The central caveat is that reachability is defined by a fixed finite probe, and several headline conclusions depend on treating that probe's success count as a faithful 'reachable ceiling' across checkpoints whose output distributions differ. This is an empirical audit rather than a derivation, and the main claims need to be robust to probe sensitivity before the reporting prescription is fully grounded.

major comments (3)
  1. [Setup; RLVR and Reachability (Fig. 6b, Tables 5-6)] The Setup explicitly says 'an answer not found by a finite probe is unobserved under that protocol, not impossible for the model,' yet the RLVR conclusion treats a drop in O_K as a drop in the reachable ceiling. For DAPO, Table 5 reports -13.3 points (40 fewer reachable questions), and Table 6 shows this persists at K=400. This is only valid if the fixed probe (temperature 0.8, base prompt, K<=400) has comparable sensitivity for base and trained checkpoints. RLVR is known to alter output format and diversity; Table 8 attributes 41-50% of the realization gain to prompt format, and Figure 2a elsewhere excludes 'format-collapse' cells. If DAPO's trained distribution collapses under the base-format probe, the -13.3 point drop may be a probe-sensitivity artifact rather than a contraction of solvable questions. Please provide candidate-output diagnostics (format, length, diversity) for the DAP
  2. [Figure 2a and Abstract] The Abstract claims random routes 'match or exceed structured search in all 43 model and task settings,' but Figure 2a reports '43 clean model–task cells; three format-collapse cells are excluded.' This is a post hoc exclusion, not 'all settings.' The paper never states the exclusion criterion or reports what happens in the excluded cells. Because the routing result is one of the paper's four contributions, the universal phrasing is misleading unless the exclusion rule is defined before the comparison and the excluded cells are shown. Please report the three excluded cells individually and restate the claim as 'all non-collapsed cells' or provide a pre-specified definition of 'clean.'
  3. [Tables 2, 5, 6, 7; Figure 6] Most headline tables and figures report point estimates without confidence intervals, and some of the supporting differences are small relative to the sample size. For example, Table 5's Qwen3.5 row has n=150 and a reachable-count change of -1; Table 6's SimpleRL row has n=150 and +1; these are within sampling noise. Yet the aggregate claim 'deployed performance rises while the reachable ceiling remains flat or falls' in 'five of six' matched evaluations depends on distinguishing zero from small nonzero changes. Please report paired bootstrap or McNemar confidence intervals for D, O_K, and their difference, and include the per-question transition matrices behind Figure 6a. This would also clarify whether the DAPO result is the only statistically secure divergence or whether the other cells support it.
minor comments (3)
  1. [Table 1] Table 1 mixes different budgets across rows, and the caption says the matched comparison is 'given in Figures 1 and 2.' For readability, add a budget column or use the same K across rows in the table; otherwise the row-to-row comparison is hard to interpret.
  2. [Setup (Recognition)] The 'recognition' auxiliary readout is defined as 'ranks the correct option under multiple-choice likelihood,' but the paper also uses tasks such as GSM8K and DART-Math that are not multiple-choice. State how recognition is operationalized for free-form generation tasks, or clarify that the MLP-localization census is restricted to multiple-choice tasks.
  3. [Figure 2a caption] The phrase 'clean model–task cells' appears without definition in the main text. Specify in the caption or Setup which cells are clean and why the three format-collapse cells are excluded.

Circularity Check

1 steps flagged · score 3.0 of 10

MLP repair rate is selected from the very failure set it then explains; the reachability/realization distinction itself is operational rather than derivationally circular.

  1. fitted input called prediction [Experiment / MLP Localization (Figure 4b, Table 3)]
    "Magnitude pruning reproduced the same recognition-correct, generation-wrong pattern. Llama-3.1-8B showed a sharp 96.6% repair peak at layer 20, whereas Qwen2.5-7B showed a broader response across layers 12 and 16. ... Silencing the identified MLP block repaired the predefined failure more consistently than silencing attention in the same layer (Figure 5a; Table 3). MLP-block silencing repaired 67.6–92.0% of the evaluated failure set."

    The layer identity is selected from the repair profile on the predefined failure set (the '96.6% repair peak at layer 20' defines which layer is 'identified'), and the same failure set is then used to report the repair percentage. This is selection-on-the-test-set: the headline 67.6–92.0% is the maximum over the searched layers used for identification, so the reported effect size is fitted to the data it is claimed to explain. The healthy-panel and reciprocal controls provide independent evidence of localization, but the specific repair magnitude is a selected statistic rather than an unbiased causal estimate.

full rationale

The central reachability/realization claim rests on an operational definition, not a derivation: reachability is explicitly protocol-relative ('an answer not found by a finite probe is unobserved under that protocol, not impossible for the model'), so flat or falling O_K is a statement about the measured probe, not about absolute model capability. The routing and RLVR comparisons are empirical and self-contained, with matched budgets and controls; answer-blind selectors are measured, not assumed to fail. The only notable circularity-like step is in the MLP localization analysis, where the layer whose silencing repairs the failure set is identified from that same failure set and then reported as repairing 68–92% of it. That inflates the effect magnitude by construction, though the specificity and reciprocal controls keep the localization claim partially independent. No equation reduces to a fitted parameter, and no load-bearing self-citation chain is present. Overall circularity is therefore partial and localized, not a collapse of the paper's main distinction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its contributions are operational metrics and experimental protocols. The main burden is that 'reachability' is an artifact of the chosen probe (K, temperature, answer format, grader), and the central claim that reachability does not expand is contingent on that probe. The MLP localization claim additionally depends on a data-selected layer index.

free parameters (5)
  • candidate budget K = 8, 16, 64, 200, 400 depending on cell
    Reachability is defined by the probe and its budget K; most headline claims compare O_K at K=200 (Tables 5-6). The result 'ceiling flat or falls' is K-relative.
  • sampling temperature and subset size n = temperature 0.8; n=150 or 300
    Matched protocol choices; changing temperature changes the oracle-majority gap (Figure 3b), and all central tables use these small subsets.
  • silenced MLP layer index = L20 for Llama-3.2-3B; L29 for Gemma-4-12B; others per case
    The repair layer is selected from per-layer silencing sweeps (Figure 4b), then reported as the repair locus in Table 3; this is a data-selected parameter for the MLP localization claim.
  • top-32 directional unit set = 32 units per case
    The directional subset analysis uses a fixed 32-unit cutoff; recovery varies widely across cases (5.6% to over 87.5%).
  • alpha dose for directional perturbations = 2, 4, or 8 depending on case
    Reciprocal rescue and induction use a dose selected per case (Table 4).
assumptions (4)
  • domain assumption The K-sample oracle maximum is treated as the 'reachable ceiling' for the fixed protocol.
    Setup, Eq. 1 defines o_i(K); the paper later compares ceilings across checkpoints. The authors explicitly note that unobserved does not equal impossible.
  • domain assumption The multiple-choice recognition readout (correct option ranked first under likelihood) indicates the model has the answer when direct generation fails.
    Used to predefine the failure set in MLP Localization (Figure 4a); if recognition is an artifact of the MC format, the failure set and repair results are readout-specific.
  • domain assumption Random routes with the same budget span the candidates structured search can find.
    Routing experiment; this is the interpretation of the main random-versus-structured claim, and it assumes random search is not unduly handicapped by the action space definition.
  • domain assumption The n=150/300 question subsets are representative for each model-task cell.
    All central tables use these small subsets (Tables 2, 5-8); no test-retest or bootstrap evidence is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains." pith.science (2026). https://pith.science/paper/PMDL2TSU

@misc{pith2026260803219,
  author       = {Pith},
  title        = {Pith review of: Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMDL2TSU}},
  note         = {Machine review of arXiv:2608.03219}
}
read the original abstract

Benchmark gains are often treated as evidence of greater LLM capability. Yet the same gain can reflect different changes in model behavior. A model may reach new answers, or produce answers that were already within reach. Aggregate scores do not distinguish these changes question by question. We establish a question-level audit under fixed budgets, temperatures, and answer formats. A question is realized when the default deployment procedure produces the correct answer. A question is reachable when a specified probe finds that answer within a fixed budget. We first test whether inference-time layer routing can expand reachability. Under a matched budget, random routes match or exceed structured search in all 43 model and task settings. Answer-blind procedures retain almost none of this gain, which instead requires access to the correct answer. We then ask why reachable answers sometimes fail to appear. Across six cases spanning 0.5B to 31B, silencing one identified MLP block repairs 68 to 92 percent of a predefined failure set. We next test whether training closes the gap by expanding reachability. In five of six matched evaluations, deployed performance rises while the reachable ceiling remains flat or falls. For DAPO, the deployed score rises by 14.7 points while the reachable ceiling falls by 13.3 points. Across the settings we audit, realization and reachability therefore do not always change together. Claims of capability expansion should report both realized performance and reachability under matched evaluation conditions. Code is available at https://github.com/LiZaiyuan0619/reachability-not-realization

Figures

Figures reproduced from arXiv: 2608.03219 by the authors.

Figure 1
Figure 1. Budget-matched routing trajectories. a–d, Random-path and structured-search oracle reachability, answer-blind voting, and the full-model baseline across four model–task cells. All procedures share the same K = 1 origin. Oracle curves use correctness after generation and are not deployable selectors. consistently expanding reachability. The three analyses share a measurement distinction rather than a common mechanism… view at source ↗
Figure 2
Figure 2. Breadth and tested robustness of the random-path oracle advantage. a, Random-minus-structured oracle differ￾ences across 43 clean model–task cells; three format-collapse cells are excluded. b, Directional counts across three action spaces and differences against hill climbing and genetic search at K = 64. Dots are cells and bars are medians. Setup Evaluation coordinates. Reachability is defined relative to a protoco… view at source ↗
Figure 3
Figure 3. Answer-blind selectors recover little of the routing oracle gap. a, Recovery of the oracle–greedy gap by six answer￾blind channels at K = 32, defined as (selector − greedy)/(oracle − greedy). Bars are clean-cell medians; hollow points are excluded format-collapse cells. b, Oracle-minus-majority gaps across measured temperatures. channel ARC-C ARC-E DART pool learned router +0.1 −0.1 −0.1 +0.5 random vote (K = 16) −2… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Bounded occurrence and pruning-created MLP-sensitive failures. a, Recognition versus direct-generation scores across 303 records from 284 checkpoint–task cells; highlighted records meet the predefined failure criterion. b, MLP-silencing repair across layers in two prun…
Figure 5
Figure 5. Figure 5: Selective causal localization to an identified MLP block. a, Fraction of the predefined failure set repaired by silencing the identified MLP block, attention in the same layer, or the whole layer in six cases. b, Failure-set repair and healthy-panel breakage under MLP …
Figure 6
Figure 6. Figure 6: Question-level fate after RLVR training. a, Gains from base-reachable and base-unreachable questions, losses, and net deployed change. b, Deployed and oracle changes in four question-matched cells (n = 300, K = 200). matched checkpoint pair training regime n K base D/O…
Figure 7
Figure 7. Figure 7: Base propensity predicts post-RLVR realization. Realization by base hit-rate bin at K = 200 with 95% Wilson intervals. pair K oracle B/T (%) gap (pp) ∆nr DAPO 200 95.3/85.3 +10.0 −15 400 96.0/85.3 +10.7 −16 SimpleRL 200 94.0/94.7 −0.7 +1 400 94.0/94.7 −0.7 +1 [PITH_FU…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 29 canonical work pages

  1. [1]

    Ahmed Heakl and Martin Gubri and Salman Khan and Sangdoo Yun and Seong Joon Oh , booktitle=. Dr. 2026 , url=

    work page 2026

  2. [2]

    2026 , eprint=

    Skip a Layer or Loop It? Learning Program-of-Layers in LLMs , author=. 2026 , eprint=

    work page 2026

  3. [3]

    2024 , eprint=

    Dynamic layer selection in decoder-only transformers , author=. 2024 , eprint=

    work page 2024

  4. [4]

    2024 , eprint=

    Mixture-of-Depths: Dynamically allocating compute in transformer-based language models , author=. 2024 , eprint=

    work page 2024

  5. [5]

    L ayer S kip: Enabling Early Exit Inference and Self-Speculative Decoding

    Elhoushi, Mostafa and Shrivastava, Akshat and Liskovich, Diana and Hosmer, Basil and Wasti, Bram and Lai, Liangzhen and Mahmoud, Anas and Acun, Bilge and Agarwal, Saurabh and Roman, Ahmed and Aly, Ahmed and Chen, Beidi and Wu, Carole-Jean. L ayer S kip: Enabling Early Exit Inference and Self-Speculative Decoding. Proceedings of the 62nd Annual Meeting of ...

    doi:10.18653/v1/2024.acl-long.681 2024

  6. [6]

    Filipe Laitenberger and Dawid Jan Kopiczko and Cees G. M. Snoek and Yuki M Asano , booktitle=. What Layers When: Learning to Skip Compute in. 2026 , url=

    work page 2026

  7. [7]

    Router-Tuning: A Simple and Effective Approach for Dynamic Depth

    He, Shwai and Ge, Tao and Sun, Guoheng and Tian, Bowei and Wang, Xiaoyang and Yu, Dong. Router-Tuning: A Simple and Effective Approach for Dynamic Depth. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.99

    work page doi:10.18653/v1/2025.emnlp-main.99 2025

  8. [8]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

Show all 47 references
  1. [9]

    The Eleventh International Conference on Learning Representations , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. The Eleventh International Conference on Learning Representations , year=

  2. [10]

    2026 , eprint=

    When LLM Judge Scores Look Good but Best-of-N Decisions Fail , author=. 2026 , eprint=

    2026

  3. [11]

    2026 , eprint=

    ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation , author=. 2026 , eprint=

    2026

  4. [12]

    2026 , eprint=

    LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning , author=. 2026 , eprint=

    2026

  5. [13]

    2026 , eprint=

    Beyond Majority Voting: Efficient Best-Of-N with Radial Consensus Score , author=. 2026 , eprint=

    2026

  6. [14]

    2026 , eprint=

    Boosting Self-Consistency with Ranking , author=. 2026 , eprint=

    2026

  7. [15]

    2026 , eprint=

    KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling , author=. 2026 , eprint=

    2026

  8. [16]

    Bridging the Knowledge-Prediction Gap in

    Yoonah Park and Haesung Pyun and Yohan Jo , booktitle=. Bridging the Knowledge-Prediction Gap in. 2026 , url=

    2026

  9. [17]

    2026 , eprint=

    Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer , author=. 2026 , eprint=

    2026

  10. [18]

    2026 , eprint=

    From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks , author=. 2026 , eprint=

    2026

  11. [19]

    2026 , eprint=

    Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models , author=. 2026 , eprint=

    2026

  12. [20]

    2026 , eprint=

    MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models , author=. 2026 , eprint=

    2026

  13. [21]

    2026 , eprint=

    Wrong Before Right: Late Rescue and Interface Failure in Aligned Language Models , author=. 2026 , eprint=

    2026

  14. [22]

    2026 , eprint=

    PRISM Edit: One Vector for All Temporal Answers , author=. 2026 , eprint=

    2026

  15. [23]

    From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled D eep S eek R1 Models

    Zhang, Jue and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei. From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled D eep S eek R1 Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 202...

    2025 doi

  16. [24]

    2026 , eprint=

    Multi-component Causal Tracing in Large Language Models , author=. 2026 , eprint=

    2026

  17. [25]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in

    Yang Yue and Zhiqi Chen and Rui Lu and Andrew Zhao and Zhaokai Wang and Yang Yue and Shiji Song and Gao Huang , booktitle=. Does Reinforcement Learning Really Incentivize Reasoning Capacity in. 2025 , url=

    2025

  18. [26]

    2026 , eprint=

    Understanding Diversity Collapse in RLVR via the Lens of Overtraining , author=. 2026 , eprint=

    2026

  19. [27]

    2025 , eprint=

    The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models , author=. 2025 , eprint=

    2025

  20. [28]

    2025 , eprint=

    The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View , author=. 2025 , eprint=

    2025

  21. [29]

    2026 , eprint=

    Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model , author=. 2026 , eprint=

    2026

  22. [30]

    2026 , eprint=

    Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation , author=. 2026 , eprint=

    2026

  23. [31]

    2025 , eprint=

    Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs , author=. 2025 , eprint=

    2025

  24. [32]

    2018 , eprint=

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , author=. 2018 , eprint=

    2018

  25. [33]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    2021

  26. [34]

    2019 , eprint=

    PIQA: Reasoning about Physical Commonsense in Natural Language , author=. 2019 , eprint=

    2019

  27. [35]

    2020 , eprint=

    LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning , author=. 2020 , eprint=

    2020

  28. [36]

    2024 , eprint=

    DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving , author=. 2024 , eprint=

    2024

  29. [37]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  30. [38]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  31. [39]

    arXiv preprint arXiv:2607.02770 , year=

    Gemma 4 technical report , author=. arXiv preprint arXiv:2607.02770 , year=

    arXiv

  32. [40]

    2025 , eprint=

    SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild , author=. 2025 , eprint=

    2025

  33. [41]

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Dai, Weinan and Fan, Tiantian and Liu, Gaohong and Liu, Lingjun and others , journal=

  34. [42]

    2024 , eprint=

    Large Language Monkeys: Scaling Inference Compute with Repeated Sampling , author=. 2024 , eprint=

    2024

  35. [43]

    2024 , eprint=

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters , author=. 2024 , eprint=

    2024

  36. [44]

    Transformer Feed-Forward Layers Are Key-Value Memories

    Geva, Mor and Schuster, Roei and Berant, Jonathan and Levy, Omer. Transformer Feed-Forward Layers Are Key-Value Memories. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021.emnlp-main.446

    2021 doi

  37. [45]

    Locating and Editing Factual Associations in GPT , url =

    Meng, Kevin and Bau, David and Andonian, Alex and Belinkov, Yonatan , booktitle =. Locating and Editing Factual Associations in GPT , url =. doi:10.52202/068431-1262 , editor =

    doi

  38. [46]

    Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , pages =

    Random Search and Reproducibility for Neural Architecture Search , author =. Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , pages =. 2020 , editor =

    2020

  39. [47]

    arXiv preprint arXiv:2501.00656 , year=

    2 OLMo 2 Furious , author=. arXiv preprint arXiv:2501.00656 , year=

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.