Pith. sign in

REVIEW 2 major objections 5 minor 29 references

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that benchmark accuracy conflates whether a model produced an evaluable answer with whether that answer is correct, and reports execution-state measurements showing that equal output caps produce very different failure…

desk verdict A solid, honest evaluation-methodology paper: the two-layer execution/scoring separation is useful, the headline counts are transparent, and the main weakness -- an unvalidated answer extractor and missing Qwen raw bodies -- is disclosed but needs a sensitivity check before the numbers are treated as final. read the letter →

arxiv 2607.24268 v2 pith:QNGUHUXX submitted 2026-07-27 cs.CL

Zongyou Yang , Yinghan Hou This is my paper · ORCID
classification cs.CL
keywords languagemodelevaluationexecutionstatesoutputbudgetstruncationmissingfinalanswerverificationpolicyLLM-as-judgeaccuracymetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's target is a measurement reform: language-model accuracy, as usually reported, compresses two distinct questions into one bit—did the response reach an evaluable state, and was its answer right. The authors propose a two-layer evaluation that fixes execution evidence (termination reason, answer exposure, parseability, completion length) before any grader touches the output, and only then applies a scorer. Across 2,550 outputs from three Qwen and two DeepSeek configurations on MATH and ARC-Challenge, matching the output cap at 2,048 tokens does not match the execution mixture: Qwen MATH truncates without a final answer in 49 of 450 outputs, DeepSeek MATH in 5 of 300, and ARC in none. A coverage-audited targeted verification panel then flips the DeepSeek-minus-Qwen MATH accuracy comparison from −7.89 to +6.89 percentage points, showing that verification policy is as load-bearing as grader choice. The consequence is that evaluations of test-time methods should report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.

What carries the argument

The carrying mechanism is a two-layer measurement architecture with a hard separation line. The execution/observability layer freezes four scorer-independent endpoints—termination reason, whether a final answer was exposed, parser status, and completion length—before any correctness adjudication, producing states LMF (length termination without a parsed final answer), LFP (length termination with one), and NLU (non-length termination without a parseable answer). A shared answer-reading rule that takes the final FINAL: line or last balanced boxed expression decides answer exposure. The scoring layer then applies a stated scorer, including the mechanical baseline and the targeted verification panel, with candidate selection and aggregation policy made explicit and coverage-audited. Matched question–model–cap pairs and question-clustered bootstrap intervals allow within-family cap effects and cross-family contrasts to be estimated without pretending the models are random draws.

What would settle it

Run a fresh batch of the same 450 Qwen MATH questions under identical settings while saving full response bodies, and have an independent parser or human label whether a final answer is exposed; if many of the 49 truncations contain a real answer the extractor failed to parse, the LMF definition and the headline contrast would need revision.

Watch

Extended reading notes

Core claim

The central discovery is that matched output budgets do not imply matched execution case mixes, and that accuracy scores therefore bundle a model's failure distribution with its verification policy. In the measured fixed configurations, LMF—length termination with no parsed final answer—occurs in 49 of 450 Qwen MATH outputs (10.89%) versus 5 of 300 DeepSeek MATH outputs (1.67%) at the same 2,048-token cap, and in none of the 750 ARC outputs. Answer availability follows the same pattern: Qwen MATH at 88.44%, DeepSeek MATH at 97.67%, and ARC at 100%. Raising the cap to 8,192 removes all five DeepSeek LMF events among the same question–model pairs, while the Qwen ARC control shows no execution-state movement. A targeted verification audit limited to suspected false negatives changes the DeepSeek-minus-Qwen MATH accuracy contrast from −7.89 points under the mechanical scorer to +6.89 under unanimous panel adjudication, showing that candidate selection and aggregation are part of the measurement instrument rather than external noise.

Load-bearing premise

The headline counts hinge on the answer-reading rule correctly spotting a final answer; if it mistakes a present but oddly formatted answer for a missing one, the 49-versus-5 gap would shift.

Editorial extensions

If this is right

  • Benchmark reports for test-time methods should include an execution-state table—LMF, LFP, NLU, and answer availability—alongside accuracy, because the starting failure distribution shapes what any recovery method can do.
  • Accuracy gaps between models or tasks should not be interpreted as capability gaps until verification coverage, candidate selection rules, and scorer provenance are matched or disclosed.
  • Paired cap transitions, such as the DeepSeek MATH 2,048-to-8,192 comparison, can reveal how much of an apparent failure is budget-bound rather than competence-bound; the five LMF events vanish with more tokens.
  • Scorer choice is not a nuisance parameter: in this study, switching from mechanical scoring to a unanimous panel reverses the sign of the DeepSeek-minus-Qwen MATH difference.
  • A minimal evaluation protocol would report unconditional response-state rates before intervention and state transitions afterward, making post-intervention accuracy interpretable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this pattern generalizes, many published comparisons of test-time compute methods may be partly artifacts of the underlying truncation mix, since methods that benefit from cheap continuation will look stronger on models with high LMF rates.
  • A testable extension is to rerun the same two-layer protocol with a parser that accepts arbitrary free-form answers; the Qwen-versus-DeepSeek LMF gap would shrink or grow depending on how much of the truncation is format failure rather than missing content.
  • Applying the framework to code generation and open-ended QA, where there is no FINAL-line marker, would show whether the ARC-like zero-LMF pattern is an artifact of answer format or a real task difference.
  • Reporting execution states would let meta-evaluations decompose accuracy into availability and correctness, turning a single number into a two-attribute signature that may be more stable across prompts and scorers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that benchmark accuracy conflates two distinct measurement questions: whether a response reached an evaluable state and whether its answer was judged correct. It proposes a two-layer evaluation framework, separating an execution/observability layer (termination reason, answer exposure, parseability, completion length) from a scoring layer (correctness under a stated scorer). Using 2,550 outputs from three Qwen and two DeepSeek configurations on MATH and ARC-Challenge, it reports that at a matched 2,048-token cap, 49 of 450 Qwen MATH outputs are length-terminated without a parsed final answer (LMF), versus 5 of 300 DeepSeek MATH outputs and none of 750 ARC outputs. Within DeepSeek MATH, raising the cap to 8,192 removes all five LMF events. A targeted verification study, in which 164 suspected MATH false negatives were adjudicated by three LLM judges, reverses the mechanical DeepSeek-minus-Qwen MATH comparison from -7.89 to +6.89 percentage points under unanimity; the authors attribute this reversal to the combined selection-and-scoring pipeline rather than to a pure grader effect. The paper concludes that evaluations of test-time methods should report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.

Significance. If the measurements hold, the paper makes a useful methodological point: matched output budgets do not imply matched execution mixtures, and comparative accuracy estimates can be sensitive to candidate selection and aggregation policies. The paper is unusually transparent: it separates primary from exploratory endpoints, prespecifies bootstrap seeds and resampling units, explicitly labels the verification reversal as descriptive of a specific audit pipeline, and releases item-level labels and offline analysis code. The main execution-state measurements are mechanical and do not depend on the LLM judges, and the limitations are stated frankly. The contribution is a demonstration plus a reporting recommendation rather than a new recovery method, which is appropriate for an evaluation-focused venue. The strengths of the artifact and the generally careful hedging make this a worthwhile paper, provided the primary LMF measurements are shown to be robust to the parser and tokenizer concerns raised below.

major comments (2)
  1. [III.B, IV.A, V] The abstract and Section IV.A state that 49 of 450 Qwen MATH outputs 'terminate without a final answer,' but the LMF definition in Section III.A is 'length termination without a parsed final answer,' and the shared extractor (Section III.B) accepts only the final FINAL: line or the last balanced boxed expression. No independent validation of this answer-presence judgment is reported, and Section V states that the Qwen ledger did not retain complete response bodies, so the largest LMF count cannot be re-derived from raw text. If many of the 49 Qwen outputs contain human-readable final answers in formats the extractor misses (lower-case 'final:', 'Final Answer:', unboxed text, or truncated boxed expressions), the headline Qwen-versus-DeepSeek gap would be an answer-format compliance difference rather than the claimed difference in execution case mix. Please report parser precision/recall on a labeled sample, at least for the configurations whose raw bodies are available, and align the abstract and conclusion wording with 'without a parsed final answer' unless validation supports the stronger claim.
  2. [III.B, IV.A] The 'matched 2,048-token caps' are matched in configured token counts, but different models and serving stacks may tokenize text differently, so 2,048 tokens can correspond to different amounts of visible text. Since no tokenizer alignment or length-normalization check is reported, the Qwen-versus-DeepSeek LMF contrast could partly be an artifact of where each tokenizer happens to truncate a response. Please report tokenizer characteristics or show robustness to alternative length normalizations (for example, character counts, word counts, or a common tokenizer applied to both model families).
minor comments (5)
  1. [III.A] Please define 'length termination' explicitly as hitting the configured token cap, and distinguish it from a response that ends at a natural end-of-text token, since the LMF taxonomy depends on that distinction.
  2. [III.B] The exact prompt templates are described but not quoted; because the extractor's behavior depends on the required FINAL: line and on answer-format instructions, include the full prompts in the artifact or an appendix.
  3. [IV.C / Figure 3a] Figure 3a should state the Qwen candidate-coverage percentages explicitly in the caption, since the text reports them but the figure alone does not convey the 31.2% versus 93.3% coverage asymmetry.
  4. [V] Consider moving the statement that the Qwen Natural ledger did not retain complete response bodies to Section III.B, where LMF is first defined, because the limitation bears directly on the primary endpoint and not only on the later verification study.
  5. [II / Reference [7]] Add one sentence summarizing the audit protocol of the cited prior framework [7], since the verification study's design relies on it and readers should not need to consult the preprint to understand the coverage-audit terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's headline counts are direct measurements under an explicit operational definition, and the verification reversal is admitted to be a coverage-dependent sensitivity analysis rather than a derived prediction.

full rationale

The paper's derivation chain is self-contained: Section III.A defines LMF/LFP/NLU and answer availability operationally, and Section IV.A reports direct counts from fixed Qwen/DeepSeek outputs under matched caps. No parameter is fitted to produce the headline 49/450 vs 5/300 contrast, and no equation in the paper reduces a conclusion to an input by construction. The 'accuracy conflates execution case mix with verification policy' claim is supported by the empirical execution-state counts and by the targeted-verification experiment, whose reversal Section IV.C explicitly labels as sensitivity of the audit pipeline ('jointly reflecting scorer and selection policy; it does not identify a general grader effect'), not as an independent ground-truth prediction. The only self-citation is reference [7], used as background for LLM-judge measurement validity; the paper's own measurements, not [7], carry the load. The extractor-validation concern raised in Section V (Qwen raw bodies unavailable) is a reproducibility/validity limitation, not circularity, because the reported LMF counts are defined as 'without a parsed final answer' under the stated shared extractor rather than claimed as a model-independent ground truth.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities, particles, or forces. It relies on a mechanical parser, deterministic equivalence rules, and LLM judge instruments whose limitations are stated. No fitted free parameters appear in the primary execution-state analyses; the panel aggregation thresholds are policy choices, not fitted values.

assumptions (5)
  • domain assumption The shared extractor (final FINAL: line or last balanced boxed expression) correctly identifies final answer presence across all outputs.
    LMF and availability counts depend on parse success; Section III.B describes the extractor, but no independent validation of parser accuracy is reported.
  • domain assumption Temperature-zero hosted endpoints are treated as deterministic for single-call measurements.
    Section III.B notes hosted temperature-zero calls need not be bitwise deterministic, so small off-diagonal transitions are interpreted cautiously.
  • standard math Question-clustered bootstrap resampling over the fixed question sets provides valid confidence intervals for these fixed configurations.
    Common practice, but models are fixed cases, not population draws; Section III.D states this explicitly.
  • domain assumption LLM judges' labels, when unanimous, are treated as a plausible audit policy rather than a gold standard.
    Section III.C and V acknowledge shared training data and correlated errors; agreement does not imply truth.
  • domain assumption MATH training-split questions may have appeared in model pretraining; the study measures response states under possible contamination.
    Section V acknowledges the contamination risk; it affects capability interpretation but not execution-state measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets." pith.science (2026). https://pith.science/paper/QNGUHUXX

@misc{pith2026260724268,
  author       = {Pith},
  title        = {Pith review of: Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNGUHUXX}},
  note         = {Machine review of arXiv:2607.24268}
}
read the original abstract

Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.

Figures

Figures reproduced from arXiv: 2607.24268 by the authors.

Figure 1
Figure 1. Two-layer evaluation architecture. Execution evidence is frozen before correctness adjudication, while the scoring layer makes candidate selection, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Matched-cap execution mixtures and paired budget transitions. Panel (a) reports missing-final termination and answer unavailability before correctness [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Targeted-verification pipeline and exploratory sensitivity. Panel (a) shows configuration-specific candidate coverage. Panel (b) reports the matched [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 23 canonical work pages

  1. [1]

    G-eval: NLG evaluation using GPT-4 with better human alignment,

    Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: NLG evaluation using GPT-4 with better human alignment,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 2511–2522. [Online]. Available: https://aclanthology.org/2023.emnlp-main.153/

    work page 2023

  2. [2]

    Judging LLM-as-a-judge with MT-Bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-Bench and chatbot arena,” inAdvances in Neural Information Processing Systems, vol. 36, 2023, pp. 46 595–46 623. [Online]. Available: https://proceedings.neurips. cc/paper files/paper/2...

    work page 2023

  3. [3]

    Large language models are not fair evaluators,

    P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y . Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui, “Large language models are not fair evaluators,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 9440–

    work page 2024

  4. [4]

    Length- controlled AlpacaEval: A simple way to debias automatic evaluators,

    Y . Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto, “Length- controlled AlpacaEval: A simple way to debias automatic evaluators,” inFirst Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=CybBmzWBX0

    work page 2024

  5. [5]

    Finding blind spots in evaluator LLMs with interpretable checklists,

    S. Doddapaneni, M. S. U. R. Khan, S. Verma, and M. M. Khapra, “Finding blind spots in evaluator LLMs with interpretable checklists,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 16 279–16 309. [Online]. Available: https://aclanthology.org/2024.emnlp-main.911/

    work page 2024

  6. [6]

    A survey on LLM-as-a-judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y . Wang, W. Gao, L. Ni, and J. Guo, “A survey on LLM-as-a-judge,”arXiv preprint arXiv:2411.15594, 2024. [Online]. Available: https://arxiv.org/abs/2411.15594

    arXiv 2024

  7. [7]

    When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

    Z. Yang, Y . Hou, and X. Yang, “When the judge changes, so does the measurement: Auditing LLM-as-judge reliability,”arXiv preprint arXiv:2607.08535, 2026. [Online]. Available: https://arxiv.org/abs/2607. 08535

    work page Pith review arXiv 2026

  8. [8]

    Self- refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self- refine: Iterative refinement with self-feedback,” inAdvances in Neural Information Processing Systems, vol. 36, 2023, pp. 46 534–46 594. [On- line]. Available...

    work page 2023

Show all 29 references
  1. [9]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” inAdvances in Neural Information Processing Systems, vol. 36, 2023, pp. 8634–8652. [Online]. Available: https://proceedings.neurips.cc/paper files/pap...

    2023

  2. [10]

    Large language models cannot self- correct reasoning yet,

    J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “Large language models cannot self- correct reasoning yet,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=IkmD3fKBPQ

    2024

  3. [11]

    Large language models can self-correct with key condition verification,

    Z. Wu, Q. Zeng, Z. Zhang, Z. Tan, C. Shen, and M. Jiang, “Large language models can self-correct with key condition verification,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 12 846–12 867. [Online]. Available: https://a...

    2024

  4. [12]

    When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs,

    R. Kamoi, Y . Zhang, N. Zhang, J. Han, and R. Zhang, “When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs,”Transactions of the Association for Computational Linguistics, vol. 12, pp. 1417–1440, 2024. [Online]. Available: https://acla...

    2024

  5. [13]

    Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies,

    L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, and W. Y . Wang, “Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies,”Transactions of the Association for Computational Linguistics, vol. 12, pp. 484–506,

  6. [14]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inAdvances in Neural Information Processing Systems, vol. 35, 2022, pp. 24 824–24 837. [Online]. Available: ...

    2022

  7. [15]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openre...

    2023

  8. [16]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning,

    C. Snell, J. Lee, K. Xu, and A. Kumar, “Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://proceedings.iclr.cc/paper fil...

    2025

  9. [17]

    Adaptive inference-time compute: LLMs can predict if they can do better, even mid- generation,

    R. Manvi, A. Singh, and S. Ermon, “Adaptive inference-time compute: LLMs can predict if they can do better, even mid- generation,”arXiv preprint arXiv:2410.02725, 2024. [Online]. Available: https://arxiv.org/abs/2410.02725

    2024 arXiv

  10. [18]

    s1: Simple test-time scaling,

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Cand `es, and T. Hashimoto, “s1: Simple test-time scaling,”arXiv preprint arXiv:2501.19393, 2025. [Online]. Available: https://arxiv.org/abs/2501.19393

    2025 arXiv

  11. [19]

    Beyond accuracy: Behavioral testing of NLP models with CheckList,

    M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, “Beyond accuracy: Behavioral testing of NLP models with CheckList,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 4902–4912. [Online]. Available: https://aclanthology.org/2...

    2020

  12. [20]

    Evaluating evaluation metrics: A framework for analyzing NLG evaluation metrics using measurement theory,

    Z. Xiao, S. Zhang, V . Lai, and Q. V . Liao, “Evaluating evaluation metrics: A framework for analyzing NLG evaluation metrics using measurement theory,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 10 967– 10 982. [Online]...

    2023

  13. [21]

    Capturing failures of large language models via human cognitive biases,

    E. Jones and J. Steinhardt, “Capturing failures of large language models via human cognitive biases,” inAdvances in Neural Information Processing Systems, vol. 35, 2022, pp. 11 785–11 799. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2022/hash/ 4d13b2d...

    2022

  14. [22]

    What will it take to fix benchmarking in natural language understanding?

    S. R. Bowman and G. Dahl, “What will it take to fix benchmarking in natural language understanding?” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 4843–4855. [Online]....

    2021

  15. [23]

    A systematic classification of knowledge, reasoning, and context within the ARC dataset,

    M. Boratko, H. Padigela, D. Mikkilineni, P. Yuvraj, R. Das, A. McCallum, M. Chang, A. Fokoue-Nkoutche, P. Kapanipathi, N. Mattei, R. Musa, K. Talamadupula, and M. Witbrock, “A systematic classification of knowledge, reasoning, and context within the ARC dataset,” inProceedings...

    2018

  16. [24]

    Towards a consensus taxonomy for annotating errors in automatically generated text,

    R. Huidrom and A. Belz, “Towards a consensus taxonomy for annotating errors in automatically generated text,” inProceedings of the 14th International Conference on Recent Advances in Natural Language Processing. Varna, Bulgaria: INCOMA Ltd., Shoumen, Bulgaria, 2023, pp. 527–54...

    2023

  17. [25]

    Measuring mathematical problem solving with the MATH dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the MATH dataset,” inProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1, 2021. [Online]. A...

    2021

  18. [26]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try ARC, the AI2 reasoning challenge,”arXiv preprint arXiv:1803.05457, 2018. [Online]. Available: https://arxiv.org/abs/1803.05457

    2018 arXiv

  19. [27]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. L...

    2025 arXiv

  20. [2024]

    Available: https://aclanthology.org/2024.tacl-1.27/

    [Online]. Available: https://aclanthology.org/2024.tacl-1.27/

    2024

  21. [9450]

    Available: https://aclanthology.org/2024.acl-long.511/

    [Online]. Available: https://aclanthology.org/2024.acl-long.511/

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.