{"id":"ad5b7e82-33a6-440e-ada5-f7b130645e43","arxiv_id":"2507.10541","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Asking a reasoning model several problems at once reveals large accuracy drops and exposes differences that single-question benchmarks miss.","lead":"This paper proposes a new way to evaluate AI reasoning models: instead of giving one question at a time, it packs several questions into one prompt and measures how well the model handles them together. The authors find that even top models like DeepSeek-R1 lose a large chunk of accuracy under this stress, and the test separates models that look equally strong on standard benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rule-based answer extraction may systematically under-score verbose REST responses, potentially inflating the headline degradation and the long2short advantage; the paper's artifact-free claim is not supported.","rationale":"The reader identified rule-based answer extraction as the weakest assumption, and I agree: it is the single most load-bearing step in the measurement chain. Every headline number in the paper is a function of this extractor, and the paper itself provides evidence that it loses 3-5 points relative to LLM extraction on REST outputs. That magnitude is not negligible when the reported effects include a 22-point gap between R1-7B and R1-32B on MATH500 and a 16-point long2short advantage at high stress levels. Because the extraction bias is plausibly correlated with verbosity and format cleanliness, it could selectively affect exactly the models and training methods the paper contrasts. The concern is testable: the authors already have an LLM-based extractor and the stored responses, so re-scoring a subset would settle it directly. I am not treating this as a rejection because the available evidence suggests the degradation is large even with LLM extraction on the two models checked, but the paper's own Appendix C makes clear that the default numbers are not final. The condition for acceptance should be that the reported REST accuracies are re-computed or at least validated with the more accurate extractor. This leaves the reader's CONDITIONAL verdict unchanged; it does not move it to ACCEPT or REJECT because the core finding is credible but the measurement pipeline needs verification.","tokens_in":31290,"tokens_out":5756,"duration_ms":72400,"concrete_test":"Re-score the stored REST responses for at least the headline models (DeepSeek-R1, R1-7B, R1-32B, AReaL-boba-RL-7B, Light-R1-7B-DS, Efficient-R1-7B) using the LLM-based extractor described in Appendix C (gemma-3-27b-it with the prompt in Fig. 12), then recompute (a) single-to-stress accuracy drops, (b) REST rankings, and (c) the long2short vs standard-trained comparisons. If LLM extraction raises any headline stress accuracy by more than 5 points, or if any model ordering among post-trained vs base models flips, the rule-based pipeline is load-bearing and the headline claim is not settled. If all shifts are within 1-2 points and the conclusions are unchanged, the extraction concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that REST reveals genuine, large accuracy drops and sharper model discrimination. But the measurement pipeline is the default rule-based extractor, and Appendix C shows it under-scores REST responses relative to LLM-based extraction by 3-5 points on the two models tested (e.g., R1-7B on MATH500: 66.75 vs 70.59; on AMC23: 36.06 vs 41.05). If this parsing penalty grows with response verbosity or with answers embedded in long chain-of-thought summaries, then the models labeled as 'overthinkers' are doubly penalized: they reason less effectively and their correct answers are harder to extract. Conversely, long2short models produce concise, well-formatted outputs that are easier to parse, so the paper's long2short advantage could be partly an extraction artifact. Section 5.1 states that no Format Violation errors were found and concludes the degradation is intrinsic, but that only rules out one artifact class. The paper's decision to default to rule-based extraction while reporting only a two-model check in Appendix C leaves the headline numbers unvalidated for the remaining 32 models. This is the most load-bearing assumption because it directly conditions every reported accuracy, drop, and ranking in Tables 1 and 2.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"REST (Reasoning Evaluation through Simultaneous Testing) transforms existing reasoning benchmarks by concatenating s consecutive questions into a single prompt and measures accuracy at each stress level s, reporting the average over s > 1. The paper evaluates 34 large reasoning models on seven benchmarks and reports three main findings: (i) even DeepSeek-R1 drops substantially under REST (29.1 points on AIME24); (ii) REST distinguishes models that are near-tied in single-question accuracy; and (iii) verbose \"overthinking\" behavior and the absence of long2short compression training are associated with larger degradation. It also analyzes error types, output truncation, question position and order, and compares rule-based vs LLM-based answer extraction.","tokens_in":1367,"tokens_out":1667,"duration_ms":160439,"significance":"The main observation that multi-question prompting sharply lowers accuracy on challenging reasoning benchmarks is large and practically important, and the paper's breadth (34 models, 7 benchmarks), public code, 128K token-limit control in Appendix D, and the detailed error taxonomy in Appendix G are genuine strengths. The enhanced-discriminative-power and long2short conclusions are, however, only as credible as the answer-extraction pipeline, and the validation of that pipeline covers only two models. If the extraction gap is resolved and uncertainty is reported, REST would be a valuable evaluation contribution; as it stands, the artifact-free claim in Section 5.1 is too strong.","major_comments":[{"comment":"The default rule-based extractor under-scores REST responses relative to the LLM-based extractor by 3-5 points for R1-7B on MATH500 and AMC23 (Table 4: 66.75 vs 70.59; 36.06 vs 41.05) and by smaller amounts for R1-32B. This is large enough to affect some rankings in Tables 1 and 2. Section 5.1's statement that the degradation is intrinsic because no Format Violation errors were found only rules out one artifact class; it does not address parsing or answer-to-question alignment. The long2short advantage (Section 4.2, Figures 3-4) could be partly an extraction artifact if concise outputs are easier to parse. The headline drops are much larger than the extraction gap, so the central observation is not overturned, but the paper should either adopt LLM-based extraction for the main tables or report parser-error bounds and re-check the key conclusions for all 34 models.","section":"§5.1; Appendix C, Table 4"},{"comment":"Tables 1-2 report single point estimates for most stress accuracies; only AIME24, AIME25, and AMC23 use 8 sampling runs, and no confidence intervals or significance tests are reported anywhere. Some of the discriminative-power claims rest on margins of a few points (e.g., Table 1: Qwen-QwQ-32B vs R1-32B on MATH500 differ by 3.52 points; Table 2: O3-mini vs Gemini-2.5-Flash-Thinking on LiveCodeBench differ by 0.02 points). Without an estimate of run-to-run variance, \"stronger discriminative power\" is not established for these small margins. Please add bootstrap confidence intervals or per-seed results, at least for the models used in the headline comparisons.","section":"§4.1; Tables 1-2"},{"comment":"Section 5.2 uses R1-7B's fail rate as a difficulty proxy but does not specify whether the proxy is computed from single-question or REST responses or how the ordering is generated for each stress level. Because the hard-first condition always places the hard question at position 1, the observed effect is confounded with the positional bias documented in Figure 7; a design that counterbalances difficulty and position (or conditions on first-question difficulty) is needed to conclude that presenting questions from easy to hard consistently yields better overall accuracy.","section":"§5.2, Figure 8"}],"minor_comments":[{"comment":"The header says \"GQPA Diamond\"; it should be \"GPQA Diamond\".","section":"Table 2"},{"comment":"\"supervised fune-tuning\" should be \"supervised fine-tuning\".","section":"§4.2"},{"comment":"The statement that LLM-based extraction \"may introduce additional instability\" is not supported by any measurement; either report an agreement or stability statistic or omit the claim.","section":"Appendix C"},{"comment":"The no-Format-Violation statement is made for the four models in Figure 5 only; the following sentence generalizes to all 34 models and should be qualified.","section":"§5.1, Figure 5"},{"comment":"The text refers to \"Nemotron-7B\" while Table 1 uses \"Nemotron-Nano-8B\"; use one consistent model name throughout.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is a competent empirical evaluation and is within scope for a general ML/NLP venue. The core degradation effect is far too large to be an extraction artifact, but the paper overclaims that all reported effects are intrinsic, and the long2short and discriminative-power conclusions need the parser-sensitivity analysis described in the major comments. No concerns about novelty disclosure; the related work appears fairly cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central finding holds up; the measurement and order concerns are real but not fatal. This is a useful empirical paper worth a serious referee, not a desk reject.\n\nWhat is actually new: the paper takes the old batch-prompting trick and applies it to hard reasoning benchmarks at scale. That combination produces new, concrete findings: state-of-the-art reasoning models lose 20-40 points on AIME-class problems when several are asked in one prompt; models that look near-identical on single-question evals separate sharply under stress; question omission and summary errors are common failure modes; position and question-order effects are large; and long2short-trained models retain more accuracy. Previous multi-question work stopped at simple classification and QA, so the empirical body here is a genuine contribution.\n\nThe paper is also honest in the right places: it reports the rule-based vs LLM-based extraction comparison in the appendix, tests the max-token effect, runs multiple samples on small benchmarks, and gives a clear error taxonomy. That makes the headline results more credible than the average benchmark paper.\n\nWhere the soft spots are, in proportion:\n\nRule-based extraction: the stress-test note has a point, but it is overstated. LLM-based extraction adds 3-5 points on the two models checked. The headline drops are 20-40 points, so the central claim is robust. What is not robust is the fine-grained ranking of closely spaced models and the specific magnitude of the long2short advantage. The claim that 'no Format Violation errors' proves the degradation is intrinsic is too strong: that only rules out one artifact class. I would not trust the exact ordering of mid-ranked models until it is confirmed with LLM-based extraction.\n\nQuestion order: the main evaluation uses the benchmark's original order, and the paper's own Section 5.2 shows easy-first versus hard-first changes accuracy by several points. That is a genuine confound for the reported stress-level curves. They should either randomize order or show the aggregate result is stable across orderings.\n\nError bars: for most benchmarks we get one run per model. A few bootstrap intervals would help the reader assess which differences are real.\n\nContamination framing: the abstract's 'future-proof' and contamination-resistance language is marketing. Concatenation does not clean contaminated questions. That should be toned down, but it is not load-bearing.\n\nWho it is for: anyone building or evaluating reasoning models—the overthinking and long2short findings give practical training signals, and the position/order analysis is a useful warning for prompt design.\n\nRecommendation: send it to peer review. The protocol is simple, the empirical base is wide, and even after tightening the extraction and order robustness checks, the core result will stand. A good referee will ask for those checks; the paper should survive them.","headline":"Central claim survives the extraction concern; a solid empirical contribution that deserves referee time, with extraction and order-robustness checks to add.","tokens_in":32071,"tokens_out":3145,"would_cite":true,"duration_ms":36502,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Asking large reasoning models to solve several problems in one prompt causes large accuracy drops, exposing a hidden weakness that single-question benchmarks miss.","keywords":["stress testing","multi-question prompting","large reasoning models","benchmark saturation","overthinking","long2short training","discriminative power","reasoning evaluation"],"falsifier":"Re-score all REST responses with an LLM-based extractor and compare the stress-level accuracy curves for verbose overthinking models against concise long2short models; if the extraction gap is systematically larger for verbose outputs and closes the performance gap, the overthinking-based explanation is a measurement artifact.","tokens_in":31043,"feed_emoji":"🧠","tokens_out":6656,"duration_ms":71205,"temperature":0.7,"pith_summary":"REST (Reasoning Evaluation through Simultaneous Testing) claims that current large reasoning models are not reliable multi-problem solvers: when several questions are concatenated into one prompt, even state-of-the-art models lose substantial accuracy, with DeepSeek-R1 dropping 29.1% on AIME24. The paper argues this effect is not just output truncation but reflects deeper misbehaviors, including question omission, summary errors, and a tendency to overthink early questions at the expense of later ones. If true, REST gives a low-cost way to make saturated benchmarks discriminative again and a new axis for evaluating real-world multi-context reasoning. It also identifies a promising remedy: models trained with long2short (length-rewarded) techniques preserve more of their single-question accuracy under stress.","feed_headline":"Asking multiple math problems at once drops top model accuracy by 29%","feed_subtitle":"Concatenated prompts expose hidden reasoning failures that single-question benchmarks miss.","key_machinery":"The load-bearing object is the stress-level-$s$ prompt set $P^s$, built by composing $s$ consecutive questions into one instruction with cyclic indexing, so each benchmark question appears exactly once in every position. The protocol measures the model's accuracy on each $P^s$ by extracting every answer from the single generated response, then compares accuracy across stress levels. This construction is what converts a saturated benchmark into a graded stress test: it imposes controlled multi-problem cognitive load, makes positional and cross-question interference visible, and produces an error taxonomy (question omission, summary error, output truncation, endless repetition, format violation, reasoning error) that single-question testing cannot reveal.","core_discovery":"On the paper's own terms, the central discovery is that REST, which concatenates $s$ consecutive benchmark questions into one prompt with cyclic indexing, reveals a reasoning-capability gap invisible to single-question evaluation. Across 34 models and 7 benchmarks, accuracy falls as $s$ grows, and models that look nearly identical on single questions separate sharply under stress: R1-7B and R1-32B score 93.0% and 94.6% on MATH500 singly but 66.75% and 88.97% under REST. Error analysis attributes the drop to a small set of failure modes, mainly question omission and reasoning errors in large models and endless repetition and truncation in small ones. The paper also finds a clear position effect, with earlier questions answered far more accurately, and an order effect, with easy-to-hard prompts beating hard-to-easy ones, and it identifies overthinking of early questions as a central mechanism: models that burn too many tokens on the first question leave too little for the rest.","pith_inferences":["The authors do not claim this, but REST's format should make memorized single-question answers harder to apply directly, which could give existing benchmarks a partial, cheap defense against contamination.","Because REST accuracy depends strongly on question position and order, REST scores are prompt-sensitive; comparing models fairly will require position-balanced and order-controlled prompt construction.","A direct test of the measurement pipeline would be to re-score all responses with an LLM-based extractor; the paper's own appendix reports 3 to 5 percentage point gaps, so if verbose models are systematically under-extracted, part of the overthinking penalty is a measurement artifact.","REST could be extended to other reasoning domains and to more stress levels to map how degradation scales with model size and training objective."],"forward_implications":["Existing benchmarks like MATH500 and AMC23 can be made challenging again by concatenation, without requiring new human-written questions.","REST rank-orders models that appear tied at near-ceiling single-question accuracy, giving evaluation signal where standard benchmarks have saturated.","The overthinking trap is a concrete failure mechanism: models that allocate reasoning effort poorly across multiple questions suffer the largest drops.","Long2short or length-rewarded training appears to be a practical mitigation, preserving more single-question accuracy under stress.","Post-training gains measured on single questions do not automatically transfer to multi-question scenarios, so REST offers a complementary check on model improvement claims."],"supporting_citations":[{"why":"Supplies the state-of-the-art model DeepSeek-R1, whose 29.1% AIME24 drop under REST anchors the central degradation claim.","marker":"[13]"},{"why":"Earlier batch-prompting work on simple tasks that the paper extends and challenges by applying multi-question prompting to complex reasoning.","marker":"[6]"},{"why":"Prior multi-problem prompting study whose claim of negligible position effects REST contradicts for complex reasoning.","marker":"[54]"},{"why":"Earlier multi-task inference benchmark on simple tasks that motivates REST's focus on simultaneous instruction following.","marker":"[44]"},{"why":"MATH500, the saturated benchmark whose single-question near-ceiling scores REST makes discriminative.","marker":"[17]"},{"why":"Documents the overthinking phenomenon that REST identifies as a key cause of degradation.","marker":"[45]"},{"why":"Introduces the L1 long2short training that REST finds preserves accuracy under stress.","marker":"[1]"},{"why":"Introduces length-rewarded Efficient-R1 training, another long2short method that beats standard distillation under REST.","marker":"[3]"},{"why":"Provides post-trained Light-R1 models whose single-question gains fail to transfer under REST.","marker":"[55]"}],"fun_headline_variants":["Multitasking problems drop top AI accuracy by 29%, study finds","Single-question benchmarks miss real AI reasoning gaps; REST finds them","Connecting math problems reveals AI's 'overthinking trap'","REST stress test separates top AI models that looked equal before","Asking many problems at once reveals hidden AI reasoning gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported REST accuracy gaps are real model behavior rather than artifacts of the rule-based answer extractor, which the paper's own appendix shows scores 3 to 5 percentage points below LLM-based extraction.","fun_headline_variants_meta":{"raw":{"variants":["Multitasking problems drop top AI accuracy by 29%, study finds","Single-question benchmarks miss real AI reasoning gaps; REST finds them","Connecting math problems reveals AI's 'overthinking trap'","REST stress test separates top AI models that looked equal before","Asking many problems at once reveals hidden AI reasoning gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":3164,"prompt_tokens":1051,"completion_tokens":2113,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":2025}},"tokens_in":667,"tokens_out":2113,"duration_ms":19482,"temperature":1.0,"reasoning_tokens":2025,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:28:26.353128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score all REST responses with an LLM-based extractor and compare the stress-level accuracy curves for verbose overthinking models against concise long2short models; if the extraction gap is systematically larger for verbose outputs and closes the performance gap, the overthinking-based explanation is a measurement artifact.","supporting_citations":[{"cited_title":"Batch prompting: Efficient inference with large language model apis","cited_arxiv_id":null,"evidence_quote":"Earlier batch-prompting work on simple tasks that the paper extends and challenges by applying multi-question prompting to complex reasoning."},{"cited_title":"Evaluating llms with multiple problems at once: A new paradigm for probing llm capabilities.arXiv e-prints, pages arXiv–2406, 2024","cited_arxiv_id":null,"evidence_quote":"Prior multi-problem prompting study whose claim of negligible position effects REST contradicts for complex reasoning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier multi-task inference benchmark on simple tasks that motivates REST's focus on simultaneous instruction following."}],"review_version":1}