{"id":"b94cc2d4-6d0d-4dec-aee9-b33c11b0a876","arxiv_id":"2506.08446","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":1.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.","lead":"This paper surveys how large language models (LLMs) learn and perform mathematical reasoning, organizing the field into two phases: comprehension and answer generation. It reviews prompting, fine-tuning, reinforcement learning, test-time scaling, and open challenges for researchers and practitioners.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The two-phase comprehension/generation framework does not actually organize the survey's own Section 4 methods, undermining the claimed coherent synthesis.","rationale":"The reader's weakest assumption pointed to the two-phase dichotomy being untested and compared against alternatives. My concern is closely related but more specific: the dichotomy is not merely untested; it is inconsistent with the survey's own structure. The paper introduces the framework in the abstract and Section 3, but Section 4 (which covers half of the surveyed methods) is presented as a separate category that is never mapped onto the comprehension/generation axis. This is an internal inconsistency rather than a dispute with external consensus, so it is a legitimate load-bearing issue. The central claim of the survey is that it 'consolidate[s] the fragmented developments into a coherent framework'; if the framework cannot accommodate SFT, RL, test-time scaling, or external knowledge, the consolidation is incomplete and the abstract overstates the contribution. I do not think this warrants rejection: the survey is still a useful reference with a broad and mostly accurate literature coverage, and the framework can be salvaged by explicitly reframing it as a heuristic organization rather than a literal cognitive decomposition. The reader's conditional verdict already accommodates this kind of revision, so I see no need to change the verdict. I grant credit for the paper's extensive reference list and the informative synthesis of CoT, RL, and search methods, which are independent of the framework's validity. The concern is therefore about the packaging and the central claim, not about the factual content of the individual sections.","tokens_in":36077,"tokens_out":3379,"duration_ms":42191,"concrete_test":"Construct a classification matrix assigning every method discussed in §4 (SFT, RL, DPO, MCTS/search, self-improvement, external tools, RAG) to either 'comprehension' or 'answer generation' as defined in §3.1 and §3.2. For each method, determine whether the assignment follows from the paper's definitions without expanding them (e.g., 'RL improves comprehension' or 'search is a kind of generation'). If a substantial fraction (say, more than a third) of §4 methods cannot be cleanly assigned, then the two-phase framework does not organize the survey's content and the abstract's consolidation claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 3 assert that LLM mathematical reasoning develops through two cognitive phases: comprehension (pretraining, data) and answer generation (CoT, prompting). The claim is that this framework 'consolidate[s] the fragmented developments into a coherent framework' (Section 1). However, the survey's own structure and content contradict this dichotomy. Figure 2 places 'Comprehension and Generation (§3)' as one box and 'Methods for Boosting Reasoning (§4)' as a separate parallel box. Section 3.2 defines answer generation as direct prediction evolving into CoT, while Section 3.1 defines comprehension as training on diverse mathematical corpora. Section 4 then reviews SFT, RL, DPO, test-time structural search, self-improvement, and external knowledge/tools. None of these methods is explicitly classified into either phase: SFT and RL are parameter updates, not pretraining; and they do not fit 'direct prediction to CoT' because they shape the generator itself. Test-time scaling and tree search are inference-time generation processes but are treated as a separate methodological family. The paper never explains how RL for long CoT, RAG, or tool integration relate to comprehension or answer generation. Thus the central organizing claim — that the field naturally decomposes into these two phases — is unsupported by the paper's own content, and the claimed coherence is asserted rather than demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey reviews research on mathematical reasoning in large language models (LLMs). It organizes the field around two proposed cognitive phases: comprehension (acquiring mathematical understanding through pretraining on diverse corpora) and answer generation (progressing from direct prediction to Chain-of-Thought reasoning). The paper covers background on pretraining, SFT, RL, and prompting; methods for boosting reasoning (SFT, RL, DPO, test-time structural search, self-improvement, external knowledge/tools); discussion of limitations; and future research directions. It also provides tables summarizing pretraining datasets, SFT datasets, and evaluation benchmarks.","tokens_in":36348,"tokens_out":5016,"duration_ms":56849,"significance":"The paper's compilation of recent work on long CoT, rule-based RL, test-time scaling, and self-improvement is timely, and its dataset/benchmark tables (Tables 1–3) are a useful reference. The discussion in Section 5 of RL as unlocking latent capabilities rather than creating new ones is a fair and well-supported observation. However, the central claim—that the paper consolidates fragmented developments into a coherent framework based on comprehension and generation—is not actually realized: the methods in Section 4 are never mapped onto the proposed phases. As a synthesis, the contribution is therefore currently incomplete, though the underlying survey content is largely accurate and potentially valuable after restructuring.","major_comments":[{"comment":"The comprehension/generation dichotomy is asserted in the abstract and Section 3, and Figure 2 presents it as the organizing structure, but Section 4 never uses these categories. Section 4.1 (SFT), Section 4.2 (RL), Section 4.3 (test-time search), Section 4.4 (self-improvement), and Section 4.5 (external knowledge) are not classified as either comprehension or answer generation, and the text provides no explanation of how, say, RL with rule-based rewards or RAG relates to the two phases. Because the claimed 'coherent framework' (Section 1) is the paper's stated contribution, this omission is load-bearing: the framework is decorative rather than organizing.","section":"Section 3 / Figure 2"},{"comment":"The roadmap at the start of Section 6 promises three directions but cites 'Sections 6 and 6', '(6)', and 'Section 6'—all self-references. The actual subsections are not numbered, so the reader cannot map the three promises (extending performance boundaries, enhancing efficiency, cross-domain generalization) onto the content that follows. This structural error undermines the paper's forward-looking contribution and needs to be fixed, either with proper subsection numbers and cross-references or by removing the broken pointers.","section":"Section 6"}],"minor_comments":[{"comment":"The first sentence is grammatically incomplete: 'align a pretrained language model with high-quality, human-crafted by supervised learning' appears to be missing a noun such as 'data'.","section":"Section 4.1"},{"comment":"The 'Synth.' column uses '%' to indicate non-synthetic datasets, which is confusing; use 'No' or a dash instead, since '%' reads as a percentage or a formatting artifact.","section":"Tables 1–3"},{"comment":"Figure 2 renders 'Comprehension and Generation (§3)' and 'Methods for Boosting Reasoning (§4)' as sibling boxes, visually contradicting the text's claim that Section 4 methods fit within the comprehension/generation framework.","section":"Figure 2"},{"comment":"Besta et al. 2024a and 2024b are the same paper (Graph of Thoughts, AAAI 2024) listed twice; similarly, Sprague et al. 2024a and 2024b are duplicates. These should be merged into single entries with consistent in-text citations.","section":"References"},{"comment":"Model naming is inconsistent: 'DeepSeek R1' appears both as 'DeepSeek' and 'Deepseek', and 'Kimi K1.5' appears also as 'Kimi k1.5'. Please standardize.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like an early preprint that was not fully proofread: the self-referential Section 6 and duplicate references are clear signs. More substantively, the mismatch between the claimed two-phase framework and the actual structure of Section 4 means the central contribution—a coherent synthesis—is not yet delivered. I believe this is salvageable: either the authors should reorganize Section 4 so that methods are explicitly classified as supporting comprehension or generation, or they should soften the 'coherent framework' claim and present the two phases as a discussion lens rather than the organizing principle. The survey content itself is broadly accurate and would be useful after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is a genuinely current survey of LLM mathematical reasoning, with good coverage of the post-o1/DeepSeek-R1 wave — long CoT, RL with rule-based rewards, DPO variants, tree search, self-improvement, tool use, and RAG. The benchmark appendix and the dataset tables are handy. If you need a quick map of the recent landscape, this is a reasonable place to start.\n\nThe organizing idea is a two-phase story: comprehension (pretraining, data formats) and answer generation (direct prediction → CoT). That framing is plausible as a narrative device, but the stress-test is right that it doesn't do much organizing work. Section 4 reviews SFT, RL, DPO, test-time search, self-improvement, and external knowledge without ever mapping these back onto comprehension vs. generation. RL, for example, shapes the generator itself, but it's neither pretraining nor CoT prompting. So the claim that the framework 'consolidates fragmented developments' is asserted rather than demonstrated. That said, it's not a fatal flaw for a survey — many surveys have a loose taxonomy. The paper explicitly promises the framework will organize things, so the disconnect is a genuine soft spot, but a fixable one.\n\nThe bigger immediate issues are structural. Section 6 has broken cross-references that point to itself ('Sections 6 and 6', '(6)', 'Section 6'). Two references are duplicated (Besta et al. 2024a/b; Sprague et al. 2024a/b). Those are the kind of things a referee would catch, and the authors should fix them before publication.\n\nThe citation pattern looks honest. The authors do cite their own work (BWArea, CoLA, SIRLC) as promising directions, but that's not inappropriate in context.\n\nVerdict: worth a serious referee, conditional on fixing the production errors and ideally tightening the framework or explicitly treating it as a loose organizing story rather than a demonstrated synthesis. I'd cite it as a reference survey, and I'd probably bring it to a reading group focused on math reasoning, though not as a paper to dissect.","headline":"A useful, up-to-date survey of LLM math reasoning that is let down by a framework that doesn't actually organize the content and some fixable production errors.","tokens_in":36818,"tokens_out":3272,"would_cite":true,"duration_ms":32796,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey argues that mathematical reasoning in LLMs advances through two phases—comprehension, built by pretraining, and answer generation, led by chain-of-thought reasoning—and that this split organizes the field's methods and open…","keywords":["mathematical reasoning","large language models","chain-of-thought","reinforcement learning","test-time scaling","survey"],"falsifier":"A controlled comparison of models matched on pretraining but differing only in CoT training, and vice versa, would show whether one phase alone drives the accuracy gains on benchmarks like MATH or AIME.","tokens_in":1795,"feed_emoji":"🧮","tokens_out":1884,"duration_ms":78517,"temperature":0.7,"pith_summary":"This survey argues that progress in LLM mathematical reasoning is best organized into two high-level phases: comprehension, built by pretraining on diverse mathematical corpora, and answer generation, which has moved from direct prediction to step-by-step chain-of-thought (CoT) reasoning. The survey uses this split to organize the field's methods—prompting, supervised fine-tuning, reinforcement learning, test-time scaling, self-improvement, and external tools—and to identify the open problems. The synthesis matters because it gives researchers a map of where current techniques act and where future progress is bottlenecked.","feed_headline":"Two cognitive phases drive LLM math gains","feed_subtitle":"Survey sorts pretraining, chain-of-thought, RL, and test-time scaling under one math-reasoning map.","key_machinery":"The organizing device is the two-phase division of comprehension and answer generation. Comprehension names the mathematical understanding internalized during pretraining; answer generation names the production of solutions, from direct prediction to structured step-by-step reasoning. Within the generation phase, CoT prompting is the key mechanism: intermediate tokens give transformers greater computational reach, so CoT is the load-bearing object connecting the two phases.","core_discovery":"On the paper's own terms, the central claim is that LLMs acquire mathematical competence through comprehension first and answer generation second. Comprehension comes from pretraining on large, curated mathematical corpora; answer generation is the reasoning behavior elicited at inference or fine-tuning time, whose keystone is CoT prompting. The survey reads recent advances as a sequence of ways to strengthen the generation phase, and it gathers evidence that these methods largely unlock latent capabilities rather than create new ones.","pith_inferences":["If reinforcement learning mainly unlocks latent capability, then measuring a base model's Pass@k before training could predict how much RL can help, which the survey does not explicitly propose.","The comprehension-versus-generation framing could transfer to non-mathematical reasoning domains, predicting that CoT and search help most when the domain's knowledge is already in pretraining."],"forward_implications":["CoT is more than a prompting trick; it lets models emulate structured procedures, so extending CoT with long reasoning or structural search should keep improving accuracy on multi-step mathematics problems.","Reinforcement learning with rule-based rewards improves Pass@1 and, with long CoT, Pass@k, but mostly by activating capabilities already present in the base model.","Test-time scaling offers a practical alternative to scaling model parameters for mathematics benchmarks."],"supporting_citations":[{"why":"Establishes CoT prompting as the method that elicits step-by-step reasoning.","marker":"Wei et al. [2022b]"},{"why":"DeepSeek-R1, the pure-RL long CoT system whose rule-based rewards anchor the RL discussion.","marker":"Guo et al. [2025]"},{"why":"Shows that intermediate CoT steps set a transformer's computational capacity.","marker":"Merrill and Sabharwal [2024]"}],"fun_headline_variants":["Survey finds LLM math reasoning flows through two phases","Comprehension and answer generation define LLM math ability","LLM math skills emerge from pretraining and chain-of-thought","Two cognitive phases underpin LLM mathematical reasoning"],"cache_read_input_tokens":39040,"weakest_assumption_plain":"The survey's map rests on the assumption that LLM mathematical ability cleanly splits into a comprehension phase and an answer-generation phase.","fun_headline_variants_meta":{"raw":{"variants":["Survey finds LLM math reasoning flows through two phases","Comprehension and answer generation define LLM math ability","LLM math skills emerge from pretraining and chain-of-thought","Two cognitive phases underpin LLM mathematical reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00085,"raw_usage":{"total_tokens":3631,"prompt_tokens":812,"completion_tokens":2819,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":2753}},"tokens_in":428,"tokens_out":2819,"duration_ms":24981,"temperature":1.0,"reasoning_tokens":2753,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:10:07.903282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled comparison of models matched on pretraining but differing only in CoT training, and vice versa, would show whether one phase alone drives the accuracy gains on benchmarks like MATH or AIME.","supporting_citations":[],"review_version":1}