{"id":"242b816a-8969-42fd-8d72-f645200146ad","arxiv_id":"2507.02856","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Answer matching with recent language models aligns with human grading at near-inter-annotator levels, substantially outperforming multiple-choice evaluation and LLM-as-judge without reference answers.","lead":"This paper shows that many multiple-choice LLM benchmarks can be gamed by looking only at the answer options, and proposes 'answer matching' as a more valid evaluation method: let the model answer freely, then use a language model to check the answer against a reference. The authors find that even small recent language models used as matchers agree with human grading almost as well as humans agree with each other, while multiple-choice and judge-only evaluations agree poorly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline alignment numbers are measured against a non-independent human gold standard: the two author-annotators graded with a rubric that is essentially identical to the matcher prompt, so high matcher–human agreement may reflect shared criterion rather than independent correctness.","rationale":"The reader's weakest assumption identifies the validity of the two authors' manual grading as the key risk, and my stress-test concurs. I add a specific mechanism: the annotation rubric is nearly identical to the matcher prompt, so the high matcher–human agreement is at least partly a self-consistency check rather than an independent validation. This is the most load-bearing concern because every headline number (alignment in Figures 4 and 5, ranking changes in Figure 6) is derived from these human labels, and the paper's central claim—that answer matching reaches human-level agreement—cannot be separated from the non-independence of the gold standard. The filtered-subset generalizability issue is real but secondary: the paper explicitly restricts its claims to filtered questions, and even the unfiltered ranking analysis in Appendix C shows similar conclusions. The annotation-bias concern, by contrast, threatens the validity of the ground truth itself. A concrete external re-annotation study would settle the question. The verdict remains CONDITIONAL: the paper is a valuable, well-executed study, but acceptance should be conditional on independent human validation before the 'human-level grading' claim is accepted as established.","tokens_in":30703,"tokens_out":4046,"duration_ms":55076,"concrete_test":"Recruit two or three external annotators with relevant STEM expertise, blinded to the paper's hypotheses and to the authors' labels, to re-grade the same 800 MMLU-Pro/GPQA-Diamond responses. Give them an independent rubric (e.g., 'mark correct if the response is a correct answer to the question, regardless of whether it is a superset of the provided reference') and allow web search and calculators. Compute Scott's pi between each matcher (Qwen3-4B, DeepSeek-v3, Llama-4-Scout) and the external majority label, and compare with the paper's reported values. If the external-human pi drops below roughly 0.7 for the best matcher, the central alignment claim is not robust to gold-standard bias; if it stays above 0.8, the concern is resolved. Report bootstrap confidence intervals for all pi values.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim—that answer matching attains near-inter-annotator agreement with human grading (Figures 4 and 5)—rests entirely on human labels produced by the authors themselves. Appendix B.1 shows both annotators are coauthors, and the annotation guidelines encode the same equivalence rule used in the matcher prompt: responses are marked correct only if they are a 'super-set of reference', exactly the instruction given to the matcher in Appendix F.1 ('The response can have more information than the ground-truth... it must cover everything mentioned in the ground-truth'). The LLM-as-judge prompt, by contrast, asks for open-ended correctness without this superset rule. Consequently, the reported Scott's pi between matchers and Human 2 partially measures how faithfully the matcher follows the authors' rubric, not how well it tracks an independent judgment of correctness. The issue is compounded because the two annotators are not blind to the study's hypothesis, and on GPQA-Diamond they explicitly discussed disagreements and updated annotations (Appendix B.1), which can inflate the reported human–human agreement of 0.95. The rubric also tells annotators to 'use the options' when in doubt, tying human labels to the multiple-choice reference answer and further favoring reference-based matchers over independent judges. Because the same human labels are used both to filter questions and to evaluate responses, any systematic annotator bias affects every headline number in the paper. If an external gold standard disagrees with the authors' labels, the central comparison—matchers at 0.81–0.91 vs. MCQ at 0.26–0.47—could shift materially. The paper acknowledges individual annotations 'may be noisy' but does not address the more serious threat that the gold standard is not independent of the method being validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that multiple-choice (MCQ) evaluation of language models is compromised by discriminative shortcuts that let a model pick the correct option without understanding the question, and proposes 'answer matching' as a scalable generative alternative: the candidate model answers the question without options, and a second LLM checks whether the free-form response is semantically or functionally equivalent to a reference answer. To support the proposal, the authors (i) demonstrate choice-only shortcut accuracy on several benchmarks, (ii) measure alignment with ground truth on MATH using the rule-based MATH-Verify system, and (iii) collect human annotations on MMLU-Pro and GPQA-Diamond subsets, reporting Scott's π between automatic graders and human graders. They report that answer matching achieves near-inter-annotator agreement (π ≈ 0.81–0.91), far above MCQ (π ≈ 0.26–0.47) and LLM-as-judge without reference answers (π ≈ 0.23–0.48), and that model rankings change substantially when switching from MCQ to answer matching. The paper also analyzes evaluation cost and discusses how existing MCQ benchmarks can be converted to answer-matching evaluations.","tokens_in":31116,"tokens_out":4408,"duration_ms":51858,"significance":"If the central claim holds, the paper provides a concrete, cheap, and scalable alternative to multiple-choice evaluation that is better aligned with generative use, and it supports this with a machine-checked MATH experiment, public release of human annotations and filtered subsets, and a careful use of Scott's π rather than raw agreement. The choice-only shortcut experiments (Section 2, Figures 3 and 15) are an effective demonstration of a known but underappreciated problem. The MATH result in Figure 4 is strong evidence that answer matching can outperform MCQ on a task with an objective ground truth. However, the headline natural-language alignment claims in Section 3.2 rest on human labels produced by the authors under a rubric that closely matches the matcher prompt, which substantially weakens the claim of 'human-level' grading. The filtered subsets also skew heavily toward STEM and numeric questions, limiting generalization. With these caveats, the paper is a valuable contribution to evaluation methodology, but the central validity claim needs stronger support.","major_comments":[{"comment":"The human ground truth used in Figure 5 is not independent of the answer-matching procedure being evaluated. Appendix B.1 states that two of the authors performed the annotation, and the annotation guidelines require marking a response correct only if it is a 'super-set of reference' and instruct annotators to 'use the options' when in doubt; this is the same superset rule and reference-answer grounding used in the matcher prompt in Appendix F.1. On GPQA-Diamond, disagreements were discussed and annotations updated. Consequently, the reported Scott's π between matchers and Human 2 largely measures how faithfully the matcher implements the authors' rubric, not how well it tracks an independent judgment of correctness. The authors should either obtain independent, blinded annotations with a rubric that is not identical to the matcher prompt, or explicitly reframe the results as agreement with a specific rubric and temper the validity claims.","section":"§3.2, Appendix B.1, Appendix F.1"},{"comment":"No uncertainty quantification is reported for any Scott's π value. With only 126 questions in the filtered GPQA-Diamond set and 493 in MMLU-Pro, the differences between matchers (e.g., 0.81 vs 0.87 for Qwen3-4B vs Llama-4-Scout on GPQA) and the difference between matcher and Human 1 may be within sampling noise. The paper should report bootstrap confidence intervals or a significance test for the π values, and it should test whether the matcher–human gap is within the human–human inter-annotator interval before claiming 'human-level' grading.","section":"§3.2, Figures 4 and 5"},{"comment":"The filtered subsets used for the natural-language alignment analysis are not representative of the original benchmarks: after filtering to 'specific' and 'unique answer' questions, MMLU-Pro loses most law, psychology, and history questions, and the retained set skews heavily toward STEM and numeric-answer items (Figures 10 and 11). The abstract and conclusion state the superiority of answer matching without this caveat. The paper should either report alignment on a broader sample, or restrict the generalizability claims to the filtered, largely quantitative question distribution for which the evidence exists.","section":"§3.2, Figures 10 and 11"},{"comment":"The ranking-change analysis is built on matcher-generated labels, so the ranking differences between MCQ and answer matching in Figure 6 conflate a genuine change in the measured construct with any systematic matcher bias (e.g., the false-positive tendency noted for LLM-as-judge). The robustness analysis in Figure 14 shows only that different matchers rank similarly, not that the matcher agrees with an independent ground truth on the ranking task. At minimum, the paper should state that the ranking conclusions inherit the limitations of the human ground-truth discussion above, and ideally should validate a subset of the ranking differences against independent human preferences.","section":"§4, Figures 6 and 14"}],"minor_comments":[{"comment":"The word 'consierably' should be 'considerably'.","section":"Section 4, paragraph after Figure 6"},{"comment":"The text 'For GQPA, we limit to the Diamond set' uses 'GQPA' where 'GPQA' is intended.","section":"Appendix B.1"},{"comment":"The matcher prompt includes the instruction 'The response can have more information than the ground-truth... it must cover everything mentioned in the ground-truth' twice, once in a comment and once in the f-string; the duplication is unnecessary and should be cleaned up.","section":"Appendix F.1"}],"recommendation":"major_revision","confidential_remarks":"The central validity claim is currently supported mainly by a non-independent human annotation protocol. I would encourage the editor to seek a reviewer with expertise in human annotation design and inter-annotator agreement statistics. The MATH-based results are solid and likely sufficient for a strong paper if the natural-language claims are appropriately reframed or supplemented with independent annotations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper makes a real empirical contribution. It shows that answer matching—grading free-form responses against a reference with a modern LLM—aligns far better with human judgment than multiple choice or LLM-as-judge without a reference. The MATH experiment is the cleanest part: there the ground truth is rule-based, and even a 1.7B matcher hits 0.97 Scott's pi while MCQ gets 0.26. That is a solid, reproducible anchor for the whole argument. The choice-only shortcut results are also useful, though they extend prior work rather than break new ground.\n\nWhat the paper does well beyond the headline: it releases human annotations and filtered free-form subsets for MMLU-Pro and GPQA-Diamond, reports ranking changes with significance tests, breaks down costs, and shows that matcher rankings are stable across different matcher models. The choice of Scott's pi over Cohen's kappa is justified. The error analysis for judges (mostly false positives) is a nice touch.\n\nThe soft spot is the natural-language human gold standard, and the stress-test note is partly right. The two annotators are the authors, and the annotation rubric encodes roughly the same superset rule that the matcher prompt uses. So the matcher-human agreement on MMLU-Pro and GPQA-Diamond is in part measuring how faithfully the matcher follows the authors' rubric, not purely how well it tracks an independent judgment. On GPQA they also discussed disagreements, which can inflate the reported human-human agreement. No confidence intervals are given for the alignment values, and the filtered subset skews heavily to STEM and numeric answers. All of that limits how far the natural-language results generalize.\n\nBut I would not call this a fatal flaw. The MATH result is external and independent, and it already demonstrates the core superiority of answer matching over MCQ and judge-only evaluation. The natural-language results are consistent with that anchor, even if their absolute pi values are less trustworthy than they look. The authors openly acknowledge the filtering, the noise in individual annotations, and the lack of adversarial robustness testing. The central qualitative claim—answer matching is now a viable, cheap, and more valid evaluation strategy than MCQ—holds up.\n\nWho is this for? Anyone building or maintaining leaderboards, benchmark designers, and practitioners doing model selection. It deserves a serious referee. The revision should add external or at least blinded human annotation, confidence intervals around the pi values, and a robustness check on the filtered subset.","headline":"A genuinely useful empirical case that answer matching beats MCQ and judge-only evaluation, with a MATH-grounded core and a human-annotation layer that deserves scrutiny but not dismissal.","tokens_in":31596,"tokens_out":1111,"would_cite":true,"duration_ms":16609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that grading language models by matching free-form answers to a reference answer is now as reliable as human grading, while multiple-choice grading is not.","keywords":["answer matching","generative evaluation","multiple choice shortcuts","LLM-as-a-judge","human alignment","Scott's pi","MMLU-Pro","GPQA-Diamond"],"falsifier":"Have independent expert annotators re-grade a random sample of the released 800 responses without seeing the authors' labels, then recompute Scott's pi for each matcher against that new ground truth; if answer matching's pi falls well below the new human-human agreement, the 'human-level grading' claim fails. Alternatively, test the matchers on a random sample of unfiltered MMLU-Pro questions covering non-STEM subjects and check whether alignment collapses.","tokens_in":30512,"feed_emoji":"✅","tokens_out":5958,"duration_ms":59979,"temperature":0.7,"pith_summary":"This paper is trying to establish that the standard way of evaluating language models, multiple-choice questions, measures the wrong ability and systematically overstates generative skill. The proposed alternative is answer matching: let the model answer the question in free form, then have a second, recent language model check whether the response is semantically or functionally equivalent to a reference answer. Using human-annotated subsets of MMLU-Pro and GPQA-Diamond, answer matching reaches Scott's pi of 0.81 to 0.91 against human grading, close to the human-human range of 0.89 to 0.95, while multiple choice scores only 0.26 to 0.47 and LLM-as-judge without a reference reaches 0.23 to 0.48. The paper also shows that model rankings change substantially when evaluations switch to answer matching, and that the protocol costs no more than multiple-choice evaluation. This matters because benchmark scores currently guide model selection, yet they may be rewarding discriminative shortcuts rather than the generative capabilities users actually interact with.","feed_headline":"Free-form answer matching beats multiple choice at grading LLMs","feed_subtitle":"Reference-checked grading matches human judgment almost as closely as humans match each other.","key_machinery":"The load-bearing object is the answer-matching protocol: the evaluated model receives only the question, generates a free-form response, and a separate matcher language model judges whether the response is semantically or functionally equivalent to a provided reference answer in the context of the question. Alignment with ground truth is measured with Scott's pi, a chance-corrected inter-annotator agreement metric chosen because it avoids the marginal-inflation problem of Cohen's kappa. The protocol also depends on a filtering step: only questions that human annotators rate as specific enough to answer without choices and having a unique correct answer enter the comparison. This filtering turns grading into an equivalence check rather than a full question-solving task, which is why small recent models can perform it well.","core_discovery":"The central claim is that LLM-based answer matching has only recently become a valid, scalable way to grade generative responses, and that it now outperforms both multiple choice and LLM-as-a-judge without a reference answer. The paper frames generation as a membership test: a response is correct if it belongs to the set of correct answers for the question, and multiple choice dodges this test by reducing the task to picking one of several provided choices. Evidence for the shortcut problem comes from a classifier finetuned on choices only, which reaches 39 to 83 percent accuracy on MMLU, TruthfulQA-v2, and related benchmarks without ever seeing the question. Answer matching replaces discrimination with equivalence checking: a matcher model sees the question, a reference answer, and the candidate response, and outputs match or no-match. Across 800 manually graded responses from MMLU-Pro and GPQA-Diamond, recent matchers, including a 4B-parameter model, achieve near-human Scott's pi agreement with human graders, whereas multiple choice and ungrounded judges align poorly, mostly because they over-credit incorrect responses. The paper concludes that answer matching should inform future benchmark design because it measures a capability users actually rely on.","pith_inferences":["If answer matching truly tracks human grading, then leaderboards should probably report it alongside multiple choice, since a model that scores high on MCQ but low on free-form generation is being used in ways its benchmark score does not certify.","The filtering step that produces the human-verified subsets skews toward STEM and numeric-answer questions, so extrapolating the reported agreement values to humanities or open-ended reasoning questions is an untested extrapolation.","A testable extension is to stress matchers with adversarial responses, such as vague answers, multiple answers, or near-paraphrases that are actually wrong; the paper explicitly leaves matcher gaming unstudied.","The results suggest dataset designers should follow the 'single, indisputable, short answer' guideline instead of investing in better distractors, since answer matching rewards answer specificity."],"forward_implications":["Multiple-choice scores overstate generative ability: models that appear to saturate a benchmark in MCQ format can drop by over 20 percentage points when asked to produce free-form answers, so supposedly saturated benchmarks still have headroom.","Leaderboard rankings are protocol-dependent: chat-optimised proprietary models tend to rise and several open-weight models fall when evaluations switch to answer matching, so model selection decisions would change.","Existing multiple-choice datasets can be reused for generative evaluation by filtering to questions with a specific, unique answer, or by rewriting questions and reference answers to make them answerable in free form.","Answer matching is scalable: even small recent matchers achieve near-human alignment, and the overall cost is no higher than multiple-choice evaluation because free-form responses are typically shorter than MCQ responses."],"supporting_citations":[{"why":"Supplies GPQA-Diamond, the graduate-level benchmark whose responses the paper annotates and uses for alignment comparisons.","marker":"Rein et al., 2024"},{"why":"Supplies MMLU-Pro, the 10-choice benchmark whose free-form subset is annotated and evaluated.","marker":"Wang et al., 2024b"},{"why":"Provides MATH-Verify, the rule-based ground-truth grader used for the MATH alignment experiment.","marker":"Kydlicek et al., 2025"},{"why":"Defines LLM-as-a-judge, the reference-free baseline that the paper compares against answer matching.","marker":"Zheng et al., 2023"},{"why":"Supplies the Scott's pi alignment methodology and the recommendation to use it for judge evaluation.","marker":"Thakur et al., 2024"},{"why":"Provides the rubric-based filtering prompt used to select MMLU-Pro questions answerable without choices.","marker":"Myrzakhan et al., 2024"},{"why":"Reports the parallel result that small models with a reference answer beat larger models without one for grading, which the paper cites as consistent evidence.","marker":"Krumdick et al., 2025"},{"why":"Documents choice-only shortcuts in multiple-choice evaluation, which the paper extends to finetuned classifiers.","marker":"Balepur et al., 2024"}],"fun_headline_variants":["Answer matching beats multiple choice for grading LLMs","Free-form answer matching: new benchmark for LLM eval","Multiple choice shortcuts exposed; answer matching fixes grading","Reference-grounded grading matches human judgment in LLMs","Why LLM benchmarks should switch to answer matching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands or falls on whether the two authors' manual grading of 800 responses, together with the filtered STEM-heavy question subsets, is treated as unbiased ground truth for what answer matching should predict.","fun_headline_variants_meta":{"raw":{"variants":["Answer matching beats multiple choice for grading LLMs","Free-form answer matching: new benchmark for LLM eval","Multiple choice shortcuts exposed; answer matching fixes grading","Reference-grounded grading matches human judgment in LLMs","Why LLM benchmarks should switch to answer matching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000564,"raw_usage":{"total_tokens":2719,"prompt_tokens":1032,"completion_tokens":1687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1613}},"tokens_in":648,"tokens_out":1687,"duration_ms":15476,"temperature":1.0,"reasoning_tokens":1613,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:19:04.396604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent expert annotators re-grade a random sample of the released 800 responses without seeing the authors' labels, then recompute Scott's pi for each matcher against that new ground truth; if answer matching's pi falls well below the new human-human agreement, the 'human-level grading' claim fails. Alternatively, test the matchers on a random sample of unfiltered MMLU-Pro questions covering non-STEM subjects and check whether alignment collapses.","supporting_citations":[],"review_version":1}