Pith. sign in

REVIEW 4 major objections 3 minor 4 cited by

Answer Matching Outperforms Multiple Choice for Language Model Evaluation

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that grading language models by matching free-form answers to a reference answer is now as reliable as human grading, while multiple-choice grading is not.

desk verdict A genuinely useful empirical case that answer matching beats MCQ and judge-only evaluation, with a MATH-grounded core and a human-annotation layer that deserves scrutiny but not dismissal. read the letter →

arxiv 2507.02856 v1 pith:CMTADOIO submitted 2025-07-03 cs.CL cs.AIcs.LG

Nikhil Chandak , Shashwat Goel , Ameya Prabhu , Moritz Hardt , Jonas Geiping This is my paper
classification cs.CLcs.AIcs.LG
keywords answermatchinggenerativeevaluationmultiplechoiceshortcutsLLM-as-a-judgehumanalignmentScott'spiMMLU-ProGPQA-Diamond
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that the standard way of evaluating language models, multiple-choice questions, measures the wrong ability and systematically overstates generative skill. The proposed alternative is answer matching: let the model answer the question in free form, then have a second, recent language model check whether the response is semantically or functionally equivalent to a reference answer. Using human-annotated subsets of MMLU-Pro and GPQA-Diamond, answer matching reaches Scott's pi of 0.81 to 0.91 against human grading, close to the human-human range of 0.89 to 0.95, while multiple choice scores only 0.26 to 0.47 and LLM-as-judge without a reference reaches 0.23 to 0.48. The paper also shows that model rankings change substantially when evaluations switch to answer matching, and that the protocol costs no more than multiple-choice evaluation. This matters because benchmark scores currently guide model selection, yet they may be rewarding discriminative shortcuts rather than the generative capabilities users actually interact with.

What carries the argument

The load-bearing object is the answer-matching protocol: the evaluated model receives only the question, generates a free-form response, and a separate matcher language model judges whether the response is semantically or functionally equivalent to a provided reference answer in the context of the question. Alignment with ground truth is measured with Scott's pi, a chance-corrected inter-annotator agreement metric chosen because it avoids the marginal-inflation problem of Cohen's kappa. The protocol also depends on a filtering step: only questions that human annotators rate as specific enough to answer without choices and having a unique correct answer enter the comparison. This filtering turns grading into an equivalence check rather than a full question-solving task, which is why small recent models can perform it well.

What would settle it

Have independent expert annotators re-grade a random sample of the released 800 responses without seeing the authors' labels, then recompute Scott's pi for each matcher against that new ground truth; if answer matching's pi falls well below the new human-human agreement, the 'human-level grading' claim fails. Alternatively, test the matchers on a random sample of unfiltered MMLU-Pro questions covering non-STEM subjects and check whether alignment collapses.

Watch

Extended reading notes

Core claim

The central claim is that LLM-based answer matching has only recently become a valid, scalable way to grade generative responses, and that it now outperforms both multiple choice and LLM-as-a-judge without a reference answer. The paper frames generation as a membership test: a response is correct if it belongs to the set of correct answers for the question, and multiple choice dodges this test by reducing the task to picking one of several provided choices. Evidence for the shortcut problem comes from a classifier finetuned on choices only, which reaches 39 to 83 percent accuracy on MMLU, TruthfulQA-v2, and related benchmarks without ever seeing the question. Answer matching replaces discrimination with equivalence checking: a matcher model sees the question, a reference answer, and the candidate response, and outputs match or no-match. Across 800 manually graded responses from MMLU-Pro and GPQA-Diamond, recent matchers, including a 4B-parameter model, achieve near-human Scott's pi agreement with human graders, whereas multiple choice and ungrounded judges align poorly, mostly because they over-credit incorrect responses. The paper concludes that answer matching should inform future benchmark design because it measures a capability users actually rely on.

Load-bearing premise

The result stands or falls on whether the two authors' manual grading of 800 responses, together with the filtered STEM-heavy question subsets, is treated as unbiased ground truth for what answer matching should predict.

Editorial extensions

If this is right

  • Multiple-choice scores overstate generative ability: models that appear to saturate a benchmark in MCQ format can drop by over 20 percentage points when asked to produce free-form answers, so supposedly saturated benchmarks still have headroom.
  • Leaderboard rankings are protocol-dependent: chat-optimised proprietary models tend to rise and several open-weight models fall when evaluations switch to answer matching, so model selection decisions would change.
  • Existing multiple-choice datasets can be reused for generative evaluation by filtering to questions with a specific, unique answer, or by rewriting questions and reference answers to make them answerable in free form.
  • Answer matching is scalable: even small recent matchers achieve near-human alignment, and the overall cost is no higher than multiple-choice evaluation because free-form responses are typically shorter than MCQ responses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If answer matching truly tracks human grading, then leaderboards should probably report it alongside multiple choice, since a model that scores high on MCQ but low on free-form generation is being used in ways its benchmark score does not certify.
  • The filtering step that produces the human-verified subsets skews toward STEM and numeric-answer questions, so extrapolating the reported agreement values to humanities or open-ended reasoning questions is an untested extrapolation.
  • A testable extension is to stress matchers with adversarial responses, such as vague answers, multiple answers, or near-paraphrases that are actually wrong; the paper explicitly leaves matcher gaming unstudied.
  • The results suggest dataset designers should follow the 'single, indisputable, short answer' guideline instead of investing in better distractors, since answer matching rewards answer specificity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper argues that multiple-choice (MCQ) evaluation of language models is compromised by discriminative shortcuts that let a model pick the correct option without understanding the question, and proposes 'answer matching' as a scalable generative alternative: the candidate model answers the question without options, and a second LLM checks whether the free-form response is semantically or functionally equivalent to a reference answer. To support the proposal, the authors (i) demonstrate choice-only shortcut accuracy on several benchmarks, (ii) measure alignment with ground truth on MATH using the rule-based MATH-Verify system, and (iii) collect human annotations on MMLU-Pro and GPQA-Diamond subsets, reporting Scott's π between automatic graders and human graders. They report that answer matching achieves near-inter-annotator agreement (π ≈ 0.81–0.91), far above MCQ (π ≈ 0.26–0.47) and LLM-as-judge without reference answers (π ≈ 0.23–0.48), and that model rankings change substantially when switching from MCQ to answer matching. The paper also analyzes evaluation cost and discusses how existing MCQ benchmarks can be converted to answer-matching evaluations.

Significance. If the central claim holds, the paper provides a concrete, cheap, and scalable alternative to multiple-choice evaluation that is better aligned with generative use, and it supports this with a machine-checked MATH experiment, public release of human annotations and filtered subsets, and a careful use of Scott's π rather than raw agreement. The choice-only shortcut experiments (Section 2, Figures 3 and 15) are an effective demonstration of a known but underappreciated problem. The MATH result in Figure 4 is strong evidence that answer matching can outperform MCQ on a task with an objective ground truth. However, the headline natural-language alignment claims in Section 3.2 rest on human labels produced by the authors under a rubric that closely matches the matcher prompt, which substantially weakens the claim of 'human-level' grading. The filtered subsets also skew heavily toward STEM and numeric questions, limiting generalization. With these caveats, the paper is a valuable contribution to evaluation methodology, but the central validity claim needs stronger support.

major comments (4)
  1. [§3.2, Appendix B.1, Appendix F.1] The human ground truth used in Figure 5 is not independent of the answer-matching procedure being evaluated. Appendix B.1 states that two of the authors performed the annotation, and the annotation guidelines require marking a response correct only if it is a 'super-set of reference' and instruct annotators to 'use the options' when in doubt; this is the same superset rule and reference-answer grounding used in the matcher prompt in Appendix F.1. On GPQA-Diamond, disagreements were discussed and annotations updated. Consequently, the reported Scott's π between matchers and Human 2 largely measures how faithfully the matcher implements the authors' rubric, not how well it tracks an independent judgment of correctness. The authors should either obtain independent, blinded annotations with a rubric that is not identical to the matcher prompt, or explicitly reframe the results as agreement with a specific rubric and temper the validity claims.
  2. [§3.2, Figures 4 and 5] No uncertainty quantification is reported for any Scott's π value. With only 126 questions in the filtered GPQA-Diamond set and 493 in MMLU-Pro, the differences between matchers (e.g., 0.81 vs 0.87 for Qwen3-4B vs Llama-4-Scout on GPQA) and the difference between matcher and Human 1 may be within sampling noise. The paper should report bootstrap confidence intervals or a significance test for the π values, and it should test whether the matcher–human gap is within the human–human inter-annotator interval before claiming 'human-level' grading.
  3. [§3.2, Figures 10 and 11] The filtered subsets used for the natural-language alignment analysis are not representative of the original benchmarks: after filtering to 'specific' and 'unique answer' questions, MMLU-Pro loses most law, psychology, and history questions, and the retained set skews heavily toward STEM and numeric-answer items (Figures 10 and 11). The abstract and conclusion state the superiority of answer matching without this caveat. The paper should either report alignment on a broader sample, or restrict the generalizability claims to the filtered, largely quantitative question distribution for which the evidence exists.
  4. [§4, Figures 6 and 14] The ranking-change analysis is built on matcher-generated labels, so the ranking differences between MCQ and answer matching in Figure 6 conflate a genuine change in the measured construct with any systematic matcher bias (e.g., the false-positive tendency noted for LLM-as-judge). The robustness analysis in Figure 14 shows only that different matchers rank similarly, not that the matcher agrees with an independent ground truth on the ranking task. At minimum, the paper should state that the ranking conclusions inherit the limitations of the human ground-truth discussion above, and ideally should validate a subset of the ranking differences against independent human preferences.
minor comments (3)
  1. [Section 4, paragraph after Figure 6] The word 'consierably' should be 'considerably'.
  2. [Appendix B.1] The text 'For GQPA, we limit to the Diamond set' uses 'GQPA' where 'GPQA' is intended.
  3. [Appendix F.1] The matcher prompt includes the instruction 'The response can have more information than the ground-truth... it must cover everything mentioned in the ground-truth' twice, once in a comment and once in the f-string; the duplication is unnecessary and should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central alignment claims are measured against external ground truth (MATH-Verify for MATH, and human annotations for MMLU-Pro/GPQA-Diamond), with no fitted parameter renamed as a prediction.

full rationale

The paper's central claim is an empirical measurement, not a derivation from a premise. For MATH, ground truth is the external rule-based MATH-Verify library, and the matcher achieves Scott's pi = 0.97 against it. For MMLU-Pro and GPQA-Diamond, ground truth consists of manual grading by two authors (Appendix B.1) with a rubric that says responses are correct only if they are a 'super-set of reference'. The matcher prompt in Appendix F.1 uses the same 'must cover everything mentioned in the ground-truth' rule. This is a shared criterion, but it is not a circular reduction: the human labels were made by people, not by the matcher models, and no matcher output or parameter was fitted to those labels. The reported alignment is therefore an agreement measurement between independent raters applying a common gold-standard definition. The MATH result provides an independent rule-based confirmation of the same ordering, so the overall conclusion does not reduce to the design of the human rubric. The paper's self-citations (e.g., Goel et al., 2025; Hardt, 2025) support background claims and are not load-bearing for the main result. The main caveat is that the human rubric and matcher prompt overlap, which can inflate agreement if the matcher simply follows the rubric, but this is a construct-validity concern rather than a circularity by construction. No equation, fitted parameter, or self-citation chain makes the prediction equivalent to its input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests primarily on human annotations performed by the authors and on a filtered subset of questions; no numerical model parameters are fitted, but several hand-chosen thresholds shape the data and results.

free parameters (4)
  • Filtering threshold for answerable questions = rating >= 8 out of 10 by DeepSeek-V3-0324
    Used in Appendix B.1 to select MMLU-Pro questions answerable without options; affects which questions are included in the human study and hence all alignment comparisons.
  • Human annotation thresholds = >= 4 on a 1 to 5 scale for specificity, uniqueness, and match
    Used in Appendix B.1 to filter to valid questions and decide correct matches; directly determines the ground truth labels.
  • Relative error tolerance for numeric answers = 1%
    Used in the judge prompt (Appendix F) to decide whether numeric responses match; affects both matcher and human guidelines.
  • Temperature settings and max tokens = 0.6 for thinking models, 0.3 for non-thinking; max tokens 16384
    Experimental choices that may affect generation and grading; not central but part of the protocol.
assumptions (4)
  • domain assumption Human grading with access to the reference answer is a valid ground truth for free-form correctness.
    Section 3.2 and Appendix B.1 rely on author annotations as ground truth for MMLU-Pro and GPQA-Diamond; if this is biased, the alignment results are compromised.
  • domain assumption The filtered subset of questions with unique answers and specific wording is representative enough to support general conclusions about evaluation methods.
    The paper reduces MMLU-Pro from 12,000 to 493 questions and acknowledges skew toward STEM (Figures 10 to 11); generality of the comparison depends on this.
  • standard math Scott's pi is the appropriate chance-corrected agreement metric.
    Used in Section 3 and Appendix B.2 to compare graders; relies on standard assumptions of the metric.
  • domain assumption Choice-only classifier accuracy provides a lower bound on shortcut susceptibility.
    Section 2 uses a finetuned classifier on choices-only to demonstrate shortcuts; the paper notes this is a lower bound because a single classifier may not exploit all separability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Answer Matching Outperforms Multiple Choice for Language Model Evaluation." pith.science (2026). https://pith.science/paper/CMTADOIO

@misc{pith2026250702856,
  author       = {Pith},
  title        = {Pith review of: Answer Matching Outperforms Multiple Choice for Language Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMTADOIO}},
  note         = {Machine review of arXiv:2507.02856}
}
read the original abstract

Multiple choice benchmarks have long been the workhorse of language model evaluation because grading multiple choice is objective and easy to automate. However, we show multiple choice questions from popular benchmarks can often be answered without even seeing the question. These shortcuts arise from a fundamental limitation of discriminative evaluation not shared by evaluations of the model's free-form, generative answers. Until recently, there appeared to be no viable, scalable alternative to multiple choice--but, we show that this has changed. We consider generative evaluation via what we call answer matching: Give the candidate model the question without the options, have it generate a free-form response, then use a modern language model with the reference answer to determine if the response matches the reference. To compare the validity of different evaluation strategies, we annotate MMLU-Pro and GPQA-Diamond to obtain human grading data, and measure the agreement of each evaluation approach. We find answer matching using recent models--even small ones--achieves near-perfect agreement, in the range of inter-annotator agreement. In contrast, both multiple choice evaluation and using LLM-as-a-judge without reference answers aligns poorly with human grading. Improving evaluations via answer matching is not merely a conceptual concern: the rankings of several models change significantly when evaluating their free-form responses with answer matching. In light of these findings, we discuss how to move the evaluation ecosystem from multiple choice to answer matching.

Figures

Figures reproduced from arXiv: 2507.02856 by the authors.

Figure 1
Figure 1. In this work, we show how multiple choice evaluations measure a discriminative [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Shortcut accuracy achieved by finetuning a discriminative classifier that sees only [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. The correct answer set AQ may contain multiple correct answers a1, a2—Evaluations thus involve test￾ing the membership of response R in AQ. Multiple choice evaluation, in contrast, tests whether a model can dis￾criminate between a candidate answer a1 and incorrect choices {w1, w2, . . .} Circumventing this challenge is what popular question answering formats like multiple choice attempt to solve. Multiple choice eva… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Accuracy estimated by different graders and their alignment with ground truth evaluation on MATH. We evaluate the responses of Qwen2.5-7B on MATH Level 5 using different grading schemes. Left: Each bar represents a grader and its corresponding number is the accuracy es…
Figure 5
Figure 5. Figure 5: Human-agreement comparison on GPQA-Diamond (a, left) and MMLU-Pro (b, right). Each panel plots alignment (Scott’s π) between Human 2 and a range of automatic graders. Solid green bars (top) show human–human consistency, followed closely by several tiny-to-mid-sized LM …
Figure 6
Figure 6. Figure 6: Leaderboard rankings change when moving from MCQ to answer-matching on generative responses in the filtered subset of GPQA-Diamond (L) and MMLU-Pro (R): Thick lines represent statistically significant changers based on the Compact Letter Display algo￾rithm (Piepho, 200…
Figure 7
Figure 7. Figure 7: Breakdown of evaluation cost averaged across 17 models. Bars rank different grading schemes from worst (lowest Scott’s π) to best alignment with human evaluation while showing their total cost. The cost of generating a response from a model, including its input and out…
Figure 8
Figure 8. Figure 8: Discrimination, Verification and Generation Performance on MMLU-Pro (L) and MATH (R): Discrimination accuracies are significantly higher. Verification and Generation accuracies are close together, with smaller models being worse at verification. While it can seem that …
Figure 9
Figure 9. Figure 9: Screenshot of the annotation interface used by the authors to grade model responses [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Change in subject distribution of MMLU Pro before and after filtering to questions [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Change in subject distribution of GPQA Diamond before and after filtering to [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Accuracy estimated by different graders and their alignment with human evaluation [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Leaderboard rankings change on the whole unfiltered dataset when moving from MCQ to answer-matching on generative responses in GPQA-Diamond (L) and MMLU-Pro (R): Thick lines represent statistically significant changers based on the Compact Letter Display algorithm (Pi…
Figure 14
Figure 14. Figure 14: Ranking changes across matchers on GPQA-Diamond and MMLU Pro. We find that [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Shortcut accuracy achieved by finetuning DeBerta that sees only the answer choices, [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 16
Figure 16. Figure 16: Comparison of the number of output tokens in MCQ vs Free Form evaluation on GPQA-Diamond and MMLU-Pro. Each panel shows a bar plot on the mean output tokens produced by the models evaluated. We consistently find that the number of output tokens in free form (blue) is …

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Hiding future trajectory information until after a driving model forms its decision reduces rationalization and improves verifiable autonomous-driving reasoning in the proposed AD-MCQ and DEFT-RLVR framework.

  2. Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

    cs.SE 2026-03 conditional novelty 7.0 of 10

    Map-reduce scaffolding degrades measured safety mainly by stripping multiple-choice options (40–89% of the loss is format conversion); scaffold architecture explains only 0.4% of variance and composite safety scores h...

  3. Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets

    cs.AI 2026-08 conditional novelty 6.0 of 10

    LLMs under one-shot greedy decoding enumerate or materialize acceptable sets much worse than they judge membership, a gap that persists across scale, family, and generation and is dominated by omissions.

  4. Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions

    cs.CL 2025-09 conditional novelty 6.0 of 10

    KAPPA reduces the knowledge-prediction gap in LLMs by aligning a prediction-direction coordinate to a knowledge-direction coordinate in the residual stream, yielding accuracy gains on binary-choice MCQs and modest gai...

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages · cited by 4 Pith papers

  1. [1]

    • Reversible: Expansion is infinitesimally slow, maintaining equilibrium

    Understand the Process • Isothermal: Temperature remains constant at 298.15 K. • Reversible: Expansion is infinitesimally slow, maintaining equilibrium. • Nitrogen Gas: Treated as an ideal gas. • Doubling Volume: Final volume V2 = 2V1

    work page

  2. [2]

    Formula for Isothermal Reversible Expansion The entropy change is given by: ∆S = nR ln V2 V1 Where: • n = number of moles of gas • R = ideal gas constant = 8.314 J/mol·K • V2, V1 = final and initial volumes, respectively

    work page

  3. [3]

    Calculate the Number of Moles (n) • Molar mass of N 2 = 28.0134 g/mol • Mass = 14 g • n = 14 g 28.0134 g/mol ≈ 0.5 mol

    work page

  4. [4]

    Calculate the total change in entropy, when a sample of nitrogen gas of mass 14 g at 298 K and 1.00 bar doubles its volume in an isothermal reversible expansion

    Apply the Formula Since V2/V1 = 2: ∆S = (0.5 mol) · (8.314 J/mol·K) · ln(2) ∆S = 0.5 · 8.314 · 0.6931 ≈ 2.887 J/K Answer: The total change in entropy is approximately: 2.89 J/K 27 Preprint Mechanical Engineering MCQ Example — Question ID: 11475 Question: Assume that all gases are perfect and that data refer to 298.15 K unless otherwise stated. Calculate t...

    work page

  5. [5]

    We find that MCQ estimates the highest accuracy followed by LLM-judges

    Here, we also plot the same for GPQA-Diamond in Figure 12(a) and for MMLU-Pro in Figure 12(b). We find that MCQ estimates the highest accuracy followed by LLM-judges. These judges overestimate the performance as they often quickly conclude responses to be correct at surface level without engaging deeper. Meanwhile, language models as matcher give accuracy...

    work page 2004

  6. [6]

    " " # The response can have more information than the ,→ ground− t r u t h . # I t can be more s p e c i f i c ( f o r example ,

    Choose the Closest Answer The closest multiple-choice option to 2.88 J/K is: E. 3.2 J/K Therefore, the answer is E. 28 Preprint Llama 4 Scout Qwen3 4B 56.3 Grok 3 Mini57.9 57.9 54.0 44.4 Llama 4 Maverick45.2 45.2 42.945.2 DeepSeek V343.7 43.7 42.9 44.4 Qwen 3 32B42.1 42.1 39.7 43.7 R1 Distill Llama 3.3 70B39.7 39.7 41.3 39.7 Mistral Medium 338.1 38.1 42.9...

    work page

  7. [7]

    • Reversible: Expansion occurs slowly enough to maintain equilibrium

    Understand the Process • Isothermal: Temperature remains constant at 298.15 K. • Reversible: Expansion occurs slowly enough to maintain equilibrium. • Expansion: Gas volume increases during the process

    work page

  8. [8]

    Formula for Isothermal Reversible Expansion ∆S = nR ln V2 V1 Where: • n = number of moles of gas • R = ideal gas constant = 8.314 J/(mol · K) • V2 = final volume • V1 = initial volume

    work page

Show all 12 references
  1. [9]

    Calculate the Number of Moles (n) • Molar mass of N 2 = 28 g/mol • Mass of gas = 14 g • n = 14 g 28 g/mol = 0.5 mol

  2. [10]

    Determine the Volume Ratio The volume doubles, so: V2 V1 = 2

  3. [11]

    Calculate the Entropy Change ∆S = nR ln V2 V1 = (0.5)(8.314) ln(2) ∆S = 0.5 · 8.314 · 0.693 ≈ 2.88 J/K

  4. [2025]

    cloze procedure

    URL https://openreview.net/forum?id=QSQEUJfhen. Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities of large language models. In The Thirteenth International Conference on Learning Rep...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.