{"id":"d376b33b-f004-4c10-88d9-dc4c9d737820","arxiv_id":"2607.24268","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Matched output limits still produce very different execution-failure mixtures across model families, so benchmark accuracy alone hides how models fail.","lead":"Language-model accuracy scores mix up whether a model produced a usable answer and whether that answer was right. This paper measures those two things separately across thousands of Qwen and DeepSeek outputs and shows the hidden mix varies sharply by model and task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Primary LMF counts depend on an unvalidated extractor, and Qwen raw bodies were discarded; if many of the 49 Qwen MATH LMF outputs contain human-present answers in alternate formats, the headline gap is an answer-format artifact, not answer absence.","rationale":"The reader's weakest assumption already identifies the extractor, and I agree that it is the most load-bearing point. My reading strengthens it: the concern is not merely 'no validation reported' but 'validation is impossible for the primary Qwen count after the fact because raw response bodies were not retained.' This makes the 49/450 figure a single-extractor, single-parse measurement. If the extractor is too strict for Qwen's answer formatting, the headline 'terminate without a final answer' overstates answer absence; the gap could shrink or change character. I do not see this as evidence of any misrepresentation: Section V discloses the ledger limitation and DeepSeek/ARC raw bodies are retained, and the paper hedges its causal language throughout. But the disclosure does not remove the need to verify the key statistic. The framework and the DeepSeek cap contrast provide independent support for the general message, so the paper should not be rejected. Acceptance should be conditional on either validating the extractor on re-run Qwen outputs or re-scoping the abstract/headline claim to 'no extractor-recognized final answer' and reporting how many of the 49 contain unparseable-but-present answers. If the re-run/annotator check confirms LMF stability, the condition is discharged and the reader's ACCEPT stands.","tokens_in":8963,"tokens_out":11610,"duration_ms":107447,"concrete_test":"Use the frozen question identifiers and prompt/model bindings to re-run the 450 Qwen MATH question-model pairs at cap 2,048 with identical settings, capture raw completions, and confirm that their hashes match the stored ledger entries (if byte-identical re-runs are impossible, report the mismatch explicitly). Have two independent annotators classify each of the 49 stored LMF output tails (plus a sample of non-LMF controls) for whether a final answer is clearly present, present as a fragment, or absent, without using the extractor. Also re-run the shared extractor alongside variant parsers (case-insensitive FINAL, 'Answer:', last boxed expression regardless of balance, and last-20-token window) on the same tails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section III.B defines LMF as length termination without a parsed final answer, with the shared extractor taking only the final FINAL: line or the last balanced boxed expression. The primary empirical contrast (49/450 Qwen MATH LMF vs 5/300 DeepSeek and 0/750 ARC at 2,048 tokens) therefore inherits every error the extractor makes in judging whether an answer was exposed. No independent validation of that answer-presence judgment is reported, and the largest LMF count cannot be re-checked from raw text: Section V states that the Qwen Natural ledger retained only parsed answers and response hashes, not complete response bodies. If a substantial fraction of the 49 Qwen LMF outputs actually ended with a human-readable final answer in a format the extractor misses (lower-case 'final:', 'Final Answer:', unboxed text, or a boxed expression truncated just after 'boxed{'), they are not 'terminating without a final answer' in the abstract's sense; they are answer-format failures. Because DeepSeek may follow the required FINAL: convention more consistently, the Qwen-vs-DeepSeek gap would then be an instruction-following/formatting difference rather than the claimed difference in pre-intervention execution mixtures. The two-layer framework remains defensible even under a parser-dependent definition, but the specific headline number needs to be shown robust to reasonable parser variants.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that benchmark accuracy conflates two distinct measurement questions: whether a response reached an evaluable state and whether its answer was judged correct. It proposes a two-layer evaluation framework, separating an execution/observability layer (termination reason, answer exposure, parseability, completion length) from a scoring layer (correctness under a stated scorer). Using 2,550 outputs from three Qwen and two DeepSeek configurations on MATH and ARC-Challenge, it reports that at a matched 2,048-token cap, 49 of 450 Qwen MATH outputs are length-terminated without a parsed final answer (LMF), versus 5 of 300 DeepSeek MATH outputs and none of 750 ARC outputs. Within DeepSeek MATH, raising the cap to 8,192 removes all five LMF events. A targeted verification study, in which 164 suspected MATH false negatives were adjudicated by three LLM judges, reverses the mechanical DeepSeek-minus-Qwen MATH comparison from -7.89 to +6.89 percentage points under unanimity; the authors attribute this reversal to the combined selection-and-scoring pipeline rather than to a pure grader effect. The paper concludes that evaluations of test-time methods should report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.","tokens_in":9212,"tokens_out":8598,"duration_ms":81533,"significance":"If the measurements hold, the paper makes a useful methodological point: matched output budgets do not imply matched execution mixtures, and comparative accuracy estimates can be sensitive to candidate selection and aggregation policies. The paper is unusually transparent: it separates primary from exploratory endpoints, prespecifies bootstrap seeds and resampling units, explicitly labels the verification reversal as descriptive of a specific audit pipeline, and releases item-level labels and offline analysis code. The main execution-state measurements are mechanical and do not depend on the LLM judges, and the limitations are stated frankly. The contribution is a demonstration plus a reporting recommendation rather than a new recovery method, which is appropriate for an evaluation-focused venue. The strengths of the artifact and the generally careful hedging make this a worthwhile paper, provided the primary LMF measurements are shown to be robust to the parser and tokenizer concerns raised below.","major_comments":[{"comment":"The abstract and Section IV.A state that 49 of 450 Qwen MATH outputs 'terminate without a final answer,' but the LMF definition in Section III.A is 'length termination without a parsed final answer,' and the shared extractor (Section III.B) accepts only the final FINAL: line or the last balanced boxed expression. No independent validation of this answer-presence judgment is reported, and Section V states that the Qwen ledger did not retain complete response bodies, so the largest LMF count cannot be re-derived from raw text. If many of the 49 Qwen outputs contain human-readable final answers in formats the extractor misses (lower-case 'final:', 'Final Answer:', unboxed text, or truncated boxed expressions), the headline Qwen-versus-DeepSeek gap would be an answer-format compliance difference rather than the claimed difference in execution case mix. Please report parser precision/recall on a labeled sample, at least for the configurations whose raw bodies are available, and align the abstract and conclusion wording with 'without a parsed final answer' unless validation supports the stronger claim.","section":"III.B, IV.A, V"},{"comment":"The 'matched 2,048-token caps' are matched in configured token counts, but different models and serving stacks may tokenize text differently, so 2,048 tokens can correspond to different amounts of visible text. Since no tokenizer alignment or length-normalization check is reported, the Qwen-versus-DeepSeek LMF contrast could partly be an artifact of where each tokenizer happens to truncate a response. Please report tokenizer characteristics or show robustness to alternative length normalizations (for example, character counts, word counts, or a common tokenizer applied to both model families).","section":"III.B, IV.A"}],"minor_comments":[{"comment":"Please define 'length termination' explicitly as hitting the configured token cap, and distinguish it from a response that ends at a natural end-of-text token, since the LMF taxonomy depends on that distinction.","section":"III.A"},{"comment":"The exact prompt templates are described but not quoted; because the extractor's behavior depends on the required FINAL: line and on answer-format instructions, include the full prompts in the artifact or an appendix.","section":"III.B"},{"comment":"Figure 3a should state the Qwen candidate-coverage percentages explicitly in the caption, since the text reports them but the figure alone does not convey the 31.2% versus 93.3% coverage asymmetry.","section":"IV.C / Figure 3a"},{"comment":"Consider moving the statement that the Qwen Natural ledger did not retain complete response bodies to Section III.B, where LMF is first defined, because the limitation bears directly on the primary endpoint and not only on the later verification study.","section":"V"},{"comment":"Add one sentence summarizing the audit protocol of the cited prior framework [7], since the verification study's design relies on it and readers should not need to consult the preprint to understand the coverage-audit terminology.","section":"II / Reference [7]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is careful, honest, and methodologically interesting, and the artifact and reproducibility practices are above the norm. My main reservation is that the headline LMF counts rest on an unvalidated extractor and on tokenizer-specific budgets; both are addressable in revision. I do not see grounds for rejection, but the current evidence does not support the abstract's stronger 'without a final answer' wording, and the primary empirical contrast needs a robustness check before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth reading. It separates execution evidence (termination, answer exposure, parseability) from correctness adjudication, and shows empirically that equal token caps produce very different failure mixtures across models and tasks. The headline numbers -- 49/450 Qwen MATH LMF at 2,048 tokens vs 5/300 DeepSeek and 0/450 ARC -- are simple counts with question-clustered bootstrap CIs, and the paired cap contrast for DeepSeek (5 LMF at 2,048, zero at 8,192) is clean. The verification reversal is honestly labeled as a joint selection-plus-scoring effect, not a pure grader effect. That is the right way to present it.\n\nThe main soft spot is the one the stress-test flags. LMF is defined by a shared extractor that takes only the final FINAL: line or last balanced boxed expression. No independent validation of that answer-presence judgment is reported, and for Qwen the raw response bodies were not retained, so the 49 count cannot be re-checked against the actual text. Since DeepSeek may follow the FINAL: convention more consistently, part of the Qwen-vs-DeepSeek gap could be formatting or instruction-following differences rather than pure answer absence. The paper actually discloses both facts (parser definition in III.B, missing bodies in V), so this is not hidden, but the abstract's phrase \"terminate without a final answer\" overstates what is measured. The authors should either validate the extractor on a hand-labeled sample or soften the language to \"without a parsed final answer\" throughout.\n\nThe other weaknesses are minor. LLM judges are not a gold standard, and the paper says so. The sample sizes are modest, and cross-configuration comparisons are associational. None of this undercuts the central claim that accuracy conflates execution case mix with verification policy. The two-layer framework is a genuinely useful reporting protocol for test-time compute studies.\n\nWho is this for: people building or evaluating test-time scaling, self-correction, and budget-forcing methods. It deserves a serious referee. I would ask for a parser-validation sensitivity analysis and a small wording fix, but the paper is in good shape for review.","headline":"A solid, honest evaluation-methodology paper: the two-layer execution/scoring separation is useful, the headline counts are transparent, and the main weakness -- an unvalidated answer extractor and missing Qwen raw bodies -- is disclosed but needs a sensitivity check before the numbers are treated as final.","tokens_in":9763,"tokens_out":1832,"would_cite":true,"duration_ms":16526,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that benchmark accuracy conflates whether a model produced an evaluable answer with whether that answer is correct, and reports execution-state measurements showing that equal output caps produce very different failure…","keywords":["language model evaluation","execution states","output budgets","truncation","missing final answer","verification policy","LLM-as-judge","accuracy metrics"],"falsifier":"Run a fresh batch of the same 450 Qwen MATH questions under identical settings while saving full response bodies, and have an independent parser or human label whether a final answer is exposed; if many of the 49 truncations contain a real answer the extractor failed to parse, the LMF definition and the headline contrast would need revision.","tokens_in":8740,"feed_emoji":"📊","tokens_out":10230,"duration_ms":77848,"temperature":0.7,"pith_summary":"The paper's target is a measurement reform: language-model accuracy, as usually reported, compresses two distinct questions into one bit—did the response reach an evaluable state, and was its answer right. The authors propose a two-layer evaluation that fixes execution evidence (termination reason, answer exposure, parseability, completion length) before any grader touches the output, and only then applies a scorer. Across 2,550 outputs from three Qwen and two DeepSeek configurations on MATH and ARC-Challenge, matching the output cap at 2,048 tokens does not match the execution mixture: Qwen MATH truncates without a final answer in 49 of 450 outputs, DeepSeek MATH in 5 of 300, and ARC in none. A coverage-audited targeted verification panel then flips the DeepSeek-minus-Qwen MATH accuracy comparison from −7.89 to +6.89 percentage points, showing that verification policy is as load-bearing as grader choice. The consequence is that evaluations of test-time methods should report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.","feed_headline":"At the same token cap, Qwen misses 49 final answers; DeepSeek misses 5","feed_subtitle":"Accuracy collapses two questions into one number; this paper separates reaching an answer from being right.","key_machinery":"The carrying mechanism is a two-layer measurement architecture with a hard separation line. The execution/observability layer freezes four scorer-independent endpoints—termination reason, whether a final answer was exposed, parser status, and completion length—before any correctness adjudication, producing states LMF (length termination without a parsed final answer), LFP (length termination with one), and NLU (non-length termination without a parseable answer). A shared answer-reading rule that takes the final FINAL: line or last balanced boxed expression decides answer exposure. The scoring layer then applies a stated scorer, including the mechanical baseline and the targeted verification panel, with candidate selection and aggregation policy made explicit and coverage-audited. Matched question–model–cap pairs and question-clustered bootstrap intervals allow within-family cap effects and cross-family contrasts to be estimated without pretending the models are random draws.","core_discovery":"The central discovery is that matched output budgets do not imply matched execution case mixes, and that accuracy scores therefore bundle a model's failure distribution with its verification policy. In the measured fixed configurations, LMF—length termination with no parsed final answer—occurs in 49 of 450 Qwen MATH outputs (10.89%) versus 5 of 300 DeepSeek MATH outputs (1.67%) at the same 2,048-token cap, and in none of the 750 ARC outputs. Answer availability follows the same pattern: Qwen MATH at 88.44%, DeepSeek MATH at 97.67%, and ARC at 100%. Raising the cap to 8,192 removes all five DeepSeek LMF events among the same question–model pairs, while the Qwen ARC control shows no execution-state movement. A targeted verification audit limited to suspected false negatives changes the DeepSeek-minus-Qwen MATH accuracy contrast from −7.89 points under the mechanical scorer to +6.89 under unanimous panel adjudication, showing that candidate selection and aggregation are part of the measurement instrument rather than external noise.","pith_inferences":["If this pattern generalizes, many published comparisons of test-time compute methods may be partly artifacts of the underlying truncation mix, since methods that benefit from cheap continuation will look stronger on models with high LMF rates.","A testable extension is to rerun the same two-layer protocol with a parser that accepts arbitrary free-form answers; the Qwen-versus-DeepSeek LMF gap would shrink or grow depending on how much of the truncation is format failure rather than missing content.","Applying the framework to code generation and open-ended QA, where there is no FINAL-line marker, would show whether the ARC-like zero-LMF pattern is an artifact of answer format or a real task difference.","Reporting execution states would let meta-evaluations decompose accuracy into availability and correctness, turning a single number into a two-attribute signature that may be more stable across prompts and scorers."],"forward_implications":["Benchmark reports for test-time methods should include an execution-state table—LMF, LFP, NLU, and answer availability—alongside accuracy, because the starting failure distribution shapes what any recovery method can do.","Accuracy gaps between models or tasks should not be interpreted as capability gaps until verification coverage, candidate selection rules, and scorer provenance are matched or disclosed.","Paired cap transitions, such as the DeepSeek MATH 2,048-to-8,192 comparison, can reveal how much of an apparent failure is budget-bound rather than competence-bound; the five LMF events vanish with more tokens.","Scorer choice is not a nuisance parameter: in this study, switching from mechanical scoring to a unanimous panel reverses the sign of the DeepSeek-minus-Qwen MATH difference.","A minimal evaluation protocol would report unconditional response-state rates before intervention and state transitions afterward, making post-intervention accuracy interpretable."],"supporting_citations":[{"why":"Supplies the 150 stratified MATH training items whose format and deterministic equivalence rules define what counts as a parseable final answer in the mechanical scorer.","marker":"[25]"},{"why":"Supplies the 150 ARC-Challenge validation items whose option-letter format and option-count validation underlie the ARC extractor behavior with zero LMF states.","marker":"[26]"},{"why":"Defines budget forcing, the output-cap manipulation the paper holds fixed while varying task, model, and cap.","marker":"[18]"},{"why":"Frames evaluator replacement as a measurement-validity problem requiring workload-specific evaluation, bias probes, and audit trails, motivating the coverage audit and scorer provenance reporting.","marker":"[7]"},{"why":"Documents LLM-based evaluation with improved human alignment, the class of instrument used in the targeted verification panel.","marker":"[1]"},{"why":"Documents biases and limitations of LLM-as-judge, supporting the paper's decision to report mechanical baselines and panel agreement statistics.","marker":"[2]"},{"why":"Motivates the test-time-compute setting in which pre-intervention execution states matter, since adaptive compute decides where to allocate extra inference.","marker":"[16]"}],"fun_headline_variants":["Same token cap: Qwen drops 49 finals, DeepSeek drops 5","Accuracy masks failure states: verifier flips the winner","Matching output budgets doesn't match failure rates","Executable vs. correct: accuracy conflates two questions","Verifier choice flips which model wins on MATH"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline counts hinge on the answer-reading rule correctly spotting a final answer; if it mistakes a present but oddly formatted answer for a missing one, the 49-versus-5 gap would shift.","fun_headline_variants_meta":{"raw":{"variants":["Same token cap: Qwen drops 49 finals, DeepSeek drops 5","Accuracy masks failure states: verifier flips the winner","Matching output budgets doesn't match failure rates","Executable vs. correct: accuracy conflates two questions","Verifier choice flips which model wins on MATH"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00054,"raw_usage":{"total_tokens":2602,"prompt_tokens":974,"completion_tokens":1628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1544}},"tokens_in":590,"tokens_out":1628,"duration_ms":10586,"temperature":1.0,"reasoning_tokens":1544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:25:51.108369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a fresh batch of the same 450 Qwen MATH questions under identical settings while saving full response bodies, and have an independent parser or human label whether a final answer is exposed; if many of the 49 truncations contain a real answer the extractor failed to parse, the LMF definition and the headline contrast would need revision.","supporting_citations":[{"cited_title":"Measuring mathematical problem solving with the MATH dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the 150 stratified MATH training items whose format and deterministic equivalence rules define what counts as a parseable final answer in the mechanical scorer."},{"cited_title":"When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability","cited_arxiv_id":"2607.08535","evidence_quote":"Frames evaluator replacement as a measurement-validity problem requiring workload-specific evaluation, bias probes, and audit trails, motivating the coverage audit and scorer provenance reporting."},{"cited_title":"G-eval: NLG evaluation using GPT-4 with better human alignment,","cited_arxiv_id":null,"evidence_quote":"Documents LLM-based evaluation with improved human alignment, the class of instrument used in the targeted verification panel."},{"cited_title":"Judging LLM-as-a-judge with MT-Bench and chatbot arena,","cited_arxiv_id":null,"evidence_quote":"Documents biases and limitations of LLM-as-judge, supporting the paper's decision to report mechanical baselines and panel agreement statistics."},{"cited_title":"Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning,","cited_arxiv_id":null,"evidence_quote":"Motivates the test-time-compute setting in which pre-intervention execution states matter, since adaptive compute decides where to allocate extra inference."}],"review_version":2}