{"id":"52b69951-5090-4da7-ab18-00d2cc3dad9e","arxiv_id":"2608.00605","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Evolutionary Decoding, a selection-and-mutation test-time strategy, improves LLaDA 2.0 mathematical reasoning on AIME and AMC benchmarks.","lead":"Diffusion language models often generate confident but wrong math answers. This paper introduces a training-free decoding tweak that boosts math tokens and explores alternative continuations, improving accuracy on several math benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ED's reported gains may stem from its 1.5–2× increase in decoding steps rather than from the evolutionary selection/mutation mechanism; no compute-matched baseline is provided.","rationale":"The reader's weakest_assumption concerns the regex token coverage (variables, inequalities, powers, etc.). While that is a valid generalizability limitation, it is not the most load-bearing issue: even on the tested benchmarks, the reported gains could be an artifact of unequal compute. The paper's own numbers show ED uses substantially more decoding steps, yet no baseline with matched compute is reported. This confound is more fundamental because it threatens the internal validity of the central claim, not just its external validity. I therefore identify the missing compute-matched control as the primary concern. The proposed concrete test is straightforward and would settle whether the gains are mechanistic or merely computational. The reader's verdict of CONDITIONAL remains appropriate: the paper is promising but requires this essential control experiment before the claim can be accepted. I disagree with the reader's choice of weakest assumption because it targets a secondary limitation rather than the primary confound.","tokens_in":27400,"tokens_out":4382,"duration_ms":57706,"concrete_test":"Run LLaDA2.0-Flash with confidence-based decoding (the default baseline) but with a lowered confidence threshold τ (e.g., 0.90) or an increased diffusion-step budget so that its average steps per problem match ED's reported values in Table 1 (e.g., AIME24 ≈960, AIME25 ≈1085, AIME26 ≈1113, AMC23 ≈878, MATH500 ≈310). Keep all other settings, prompts, and answer extraction identical. If this compute-matched baseline achieves pass@1 accuracy within noise of ED on these benchmarks, the reported ED gains are attributable to added computing rather than to the evolutionary mechanism. If the baseline remains clearly below ED, the mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Evolutionary Decoding (ED) improves LLaDA 2.0 over confidence-based decoding. Table 1 shows that ED consistently uses far more decoding steps than the baseline: e.g., AIME24 960 vs 670, AIME25 1085 vs 669, AIME26 1113 vs 686, AMC23 878 vs 320, MATH500 310 vs 154. The paper interprets the resulting accuracy gains as evidence for the value of its selection and mutation operations, but it never compares against a confidence-based baseline with the same compute budget. A test-time scaling method should be evaluated at fixed compute; otherwise, the gains could simply reflect the extra diffusion steps, additional branch explorations from mutation, or a best-of-N effect over neutral/numerical/symbolic/mixed branches. This is not a minor omission: it directly challenges whether the proposed mechanism, rather than the added compute, causes the improvement. The paper's own Figure 4 shows ED moving upward and rightward in the steps-accuracy plane, but no iso-compute line is drawn. Without a compute-matched control, the headline claim is underdetermined.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies mathematical reasoning failures in the diffusion LLM LLaDA 2.0 and attributes them to a \"diffusion confidence trap\" in which local token confidence is misaligned with global reasoning correctness. It identifies two failure regimes, sampling-sensitive and sampling-consistent, and proposes a training-free test-time decoding framework, Evolutionary Decoding (ED). ED combines step-wise selection, which boosts the release score of numerical/symbolic tokens and penalizes repetitive blocks, with block-wise mutation, which runs four logit-bias branches and keeps the highest-confidence candidate. Experiments on AIME 2024/2025/2026, AMC 2023, MATH500, and GSM8K report accuracy gains for LLaDA2.0-Flash and LLaDA2.0-Mini over confidence-based decoding, with per-question trajectory analyses and component ablations.","tokens_in":27741,"tokens_out":5121,"duration_ms":66485,"significance":"If the effect is real, ED is a simple, training-free intervention for improving mathematical reasoning in diffusion LLMs and is compatible with existing test-time scaling strategies. The paper makes a useful conceptual contribution by moving from output-level repeated sampling to trajectory-level selection and mutation, and it provides unusually detailed per-question trajectory diagnostics. The calibration of alpha and beta in Appendix D is a genuine attempt to ground hyperparameters in the release-threshold boundary rather than pure grid search. The main limitation is that the empirical evidence for the central causal claim is currently confounded with additional compute: ED consistently uses substantially more decoding steps and up to four mutation branches, and no compute-matched baseline is reported. The manuscript is therefore promising but needs targeted experiments to distinguish the proposed mechanism from a best-of-N / more-compute effect.","major_comments":[{"comment":"AIME25 is not held out. The text says all hyperparameters are selected on Flash-AIME 2025 (Tables 4 and 5) and then kept fixed, but Table 1 reports AIME25 as a main result for the same configuration. The AIME25 gains (+13.3 pts for both models) are therefore development-set results, not independent evidence. This does not invalidate the other held-out benchmarks, but the claim of \"consistent gains on most competition-style benchmarks\" should be re-stated with AIME25 labeled as the tuning set or removed from the headline table.","section":"§4.1, §4.2, Tables 1, 4, 5"},{"comment":"The central claim is underdetermined by a compute confound. ED uses far more decoding steps than the confidence baseline: e.g., Flash AIME24 960 vs 670, AIME25 1085 vs 669, AIME26 1113 vs 686, AMC23 878 vs 320, MATH500 310 vs 154 (Table 1). Appendix G states the overall decoding time is 2–3× higher. Figure 4 plots the steps–accuracy trade-off but draws no iso-compute line. Because the mutation stage (Eq. (6)) runs up to four parallel branches and survival selection (Eq. (8)) keeps the highest-confidence one, ED is partly a best-of-4 test-time scaling method. Without a compute-matched baseline—e.g., confidence-based decoding with the same step budget, or repeated independent samples/temperature draws with the same total forward passes—the gains cannot be attributed specifically to step-wise selection and structured mutation. This is the most important issue.","section":"§4.2, Table 1, Figure 4, Appendix B/G"},{"comment":"The ablations do not isolate the value of structured mutation from the effect of branch exploration. Table 5 varies δ, τ_mut, and m_max but always retains the four-branch structure with survival selection. There is no control with four identical neutral branches or with random logit perturbations at the same compute budget. Given Eq. (8) selects the branch with the highest average confidence, part of the observed improvement on AIME/AMC could be a generic best-of-N effect rather than the specific numerical/symbolic mutation directions. Adding such a control is necessary to support the claimed mechanism.","section":"§4.3, Tables 4/5, Eqs. (6)–(8)"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any accuracy number. The AIME sets have only 30 problems: several headline differences are 3.3 pts, i.e., one problem, and one result (Mini AIME26) is exactly 0.0. With n=30, these differences are within binomial sampling noise. I recommend reporting variance over multiple decoding seeds or bootstrap intervals, and applying a simple paired test for the small AIME/AMC sets. This is particularly important because the claim of consistency across benchmarks is central to the paper.","section":"Table 1, Section 4.2"}],"minor_comments":[{"comment":"The appendix heading reads \"AMIE-2025\"; this should be \"AIME-2025\".","section":"Appendix F title"},{"comment":"The phrase \"prior to mutation\" in the definition of \\bar{c}_m^{(q)} is ambiguous: Eq. (7) defines c_m^{(t,q)} on the mutated logits, so \"prior to mutation\" seems to contradict the equation. Please clarify whether the selection score is computed on neutral or post-mutation confidence.","section":"Eq. (8), §3.3"},{"comment":"The calibration of α and β depends on hand-set targets c_tar and r_tar. The text says the hyperparameters are \"determined from the score formulations,\" but these targets are user choices. Please state explicitly which values of c_tar and r_tar are used and how sensitive the final α/β are to them.","section":"Appendix D"},{"comment":"The numerical/symbolic token detector is limited to digits and a small symbol set; the paper acknowledges this in the appendix. This limitation should also be stated in Section 3 or 5, since the method's motivation is general mathematical reasoning but variables, inequalities, and powers are not covered.","section":"Appendix A.3"},{"comment":"Several trajectory figures show answer-extraction mismatches, e.g., Q5 pred=360 gt=336^\\circ. It would help to note whether these are formatting artifacts of the extraction rule or genuine model errors.","section":"Figures 8/9"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the trajectory analysis is valuable, but the main experimental claim needs to be re-supported with compute-matched baselines and statistical error bars. The AIME25 contamination and lack of significance testing are fixable within the manuscript's scope, so I do not recommend rejection. If the authors can show that ED outperforms a confidence-based baseline at matched compute and that structured mutation beats equal-cost neutral/random branch exploration, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful engineering paper with a genuinely new decoding mechanism, but its central claim is underdetermined. ED costs 1.5-2x more decoding steps than the baseline, and the paper never compares against a confidence-based baseline at the same compute budget. Without that, the accuracy gains could just be test-time scaling in disguise.\n\nWhat's new and good: the two-regime failure analysis (sampling-sensitive vs sampling-consistent) is a clear framing, and the specific operations - entropy-boosted numeric/symbolic token selection, block repetition penalty with a fallback, and logit-level mutation with survival selection - are not in the cited prior work. The authors are transparent about picking hyperparameters on Flash-AIME25 and then fixing them elsewhere, and Appendix D derives alpha and beta from release-boundary calibration rather than pure grid search. The per-problem trajectory figures are overkill but honest. The GSM8K appendix shows they know mutation can hurt short trajectories.\n\nSoft spots, in order: (1) No compute-matched control. Selection alone captures much of the AIME gain at lower cost; ED's extra steps on AMC23 (878 vs 320) buy 5 points, and that could be from the added branching/best-of-N effect rather than the evolutionary mechanism. (2) No error bars or significance tests on any accuracy. AIME has 30 problems, so a 3-13 point delta is inside the noise if there is per-problem variance. (3) The AIME25 number is on the tuning split, so treat it as validation, not test. (4) The regex token detection misses variables, inequalities, powers - authors admit it, and it limits generalizability. (5) Abstract overclaims: GSM8K is flat/negative.\n\nNone of these are fatal. The idea is plausible, the paper is well written, and the evaluation is broader than most dLLM decoding papers. It deserves a serious referee, but the referee should ask for a compute-matched baseline (e.g., run baseline with more diffusion steps or repeated sampling to match ED's average steps) and error bars before accepting the headline. I'd bring it to reading group to discuss the right way to evaluate test-time scaling in dLLMs.","headline":"Useful decoding intervention for dLLM math reasoning, but the headline gain is confounded by extra compute; needs a compute-matched baseline and error bars before I'd believe the mechanism does the work.","tokens_in":28238,"tokens_out":2982,"would_cite":true,"duration_ms":35665,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion language models decide which masked tokens to reveal using a local confidence score; this paper argues that for mathematics this score misleads, and it shows a training-free selection-and-mutation decoding scheme that improves LLa","keywords":["diffusion language models","mathematical reasoning","test-time scaling","confidence-based decoding","evolutionary decoding","block-wise unmasking","LLaDA","decoding trajectories"],"falsifier":"Take a benchmark whose solutions rely on alphabetic variables and inequalities (a symbolic algebra set), keep the paper's exact digit/symbol regex, and compare ED against the confidence baseline: if the accuracy gain shrinks to near zero or reverses there while staying positive on a matched numeric-only set, the regex scoping is what carries the result. A second check: on AIME problems, record which of the four mutation branches wins survival selection and whether that branch actually yields a higher final-answer correctness rate than the neutral branch; if the survival rule's choice correlate","tokens_in":27336,"feed_emoji":"🧮","tokens_out":11186,"duration_ms":115756,"temperature":0.7,"pith_summary":"Diffusion large language models generate text block by block, unmasking whichever tokens reach a confidence threshold. This paper asks whether that confidence-driven procedure can support mathematical reasoning, where a few digits and operators decide correctness, and answers no — it identifies a 'diffusion confidence trap' in which local confidence ratings diverge from global correctness: useful but uncertain digits get suppressed, while repetitive, confident, wrong continuations get locked in. Eight sampling runs per problem reveal two failure regimes, and pass@8 stalls around 70% on AIME, showing that simply sampling more does not escape the trap. The proposed fix, Evolutionary Decoding, treats decoding as evolution over candidate reasoning states and applies two training-free operations: step-wise selection, which boosts numerical-symbolic tokens and penalizes repetition, and block-wise mutation, which injects structured digit/symbol alternatives before a block collapses. Across six benchmarks it improves LLaDA 2.0's pass@1 over confidence-based decoding on AIME 2024/2025/2026, AMC 2023, and MATH500, and raises the pass@8 ceiling under stochastic sampling — evidence for the paper's core thesis that reliable math reasoning in diffusion LLMs requires steering the trajectory, not just increasing samples.","feed_headline":"Two training-free tweaks lift diffusion LLM contest-math scores","feed_subtitle":"Selection rescues number tokens; mutation busts wrong loops, lifting LLaDA 2.0 pass rates on AIME, AMC, and MATH500.","key_machinery":"Two modifications to the block-diffusion loop of LLaDA 2.0. Step-wise selection replaces the confidence-only release rule with a score s = c + α·η·H − β·r: confidence c, plus an entropy-weighted boost (α·η·H) applied only when the top token is numerical-symbolic (η=1), minus a block repetition penalty (β·r); a fallback reveals top-K high-confidence positions if the penalty would stall the block. Block-wise mutation biases logits toward numerical and symbolic vocabulary subsets, spawning neutral, numerical, symbolic, and mixed branches when the neutral branch's confidence falls below a mutation threshold, then keeps the branch with the highest average pre-mutation confidence. The 'numerical-s","core_discovery":"Diffusion LLMs reveal masked tokens by local confidence; the paper claims this score misaligns with global math correctness — a 'diffusion confidence trap' in two regimes: sampling-sensitive failures (correct paths lost at prose-to-symbol transitions) and sampling-consistent failures (all runs reach one high-confidence wrong continuation). Pass@8 caps near 70% on AIME, so repeated sampling cannot escape it. The training-free Evolutionary Decoding intervenes in the trajectory: step-wise selection boosts uncertain digit/symbol tokens and penalizes block repetition; block-wise mutation biases logits toward digits and symbols in unsupported blocks, keeping the most confident branch. Reported res","pith_inferences":["Extension: the gains shown likely concentrate on numerically driven solutions, since the regex recognises only digits and a fixed symbol set; on problems dominated by alphabetic variables, inequalities, powers, or percents, selection and mutation would intervene at the wrong positions, and a token-aware classifier would be needed for those domains.","Extension: the two-regime diagnosis suggests a cheap online diagnostic — tracking block repetition ratio and digit/symbol density during decoding to predict when a trajectory needs mutation; the gated trigger is a first version of this and could be tuned per task.","Extension: if the confidence trap is a general property of block-wise masked decoding, analogous selection and mutation rules should transfer to other diffusion LLM families and to other precision domains such as code or formal reasoning, where the informative-token class would differ.","Extension: ED's step cost grows with trajectory length, so a compute-aware variant that stops mutating once a branch exits the low-support zone could keep most of the accuracy benefit at a fraction of the step budget."],"forward_implications":["On LLaDA 2.0-Flash, ED lifts pass@1 from 56.7 to 70.0 on AIME 2025 and from 90.0 to 95.0 on AMC 2023; on LLaDA 2.0-Mini it lifts AIME 2025 from 30.0 to 43.3, with consistent gains on AIME 2024/2026 and MATH500 as well.","Under eight stochastic attempts, ED improves both pass@1 and pass@8 relative to the baseline, meaning the benefit is not confined to one deterministic trajectory but improves the candidate pool itself.","The gains come at roughly two-to-three times the decoding steps on hard sets; the additional compute pays off on long competition-style problems, while on GSM8K's short four-to-five-block trajectories the standard mutation budget hurts and must be cut to m_max≤4 to restore a small gain.","A gated variant that triggers mutation only when the selection trajectory looks likely to fail (late blocks, low confidence, high repetition) raises solved AIME 2025 problems from 20/30 to 22/30 while cutting the average block index to 50.6.","Because the method is training-free, it can sit on top of downstream selection strategies such as majority voting or best-of-N confidence selection, whose oracle ceiling (pass@8) ED also raises."],"supporting_citations":[{"why":"Supplies LLaDA 2.0, the base model whose block-wise decoding is analyzed and improved.","marker":"[2]"},{"why":"Defines the large language diffusion model family and the block-wise unmasking paradigm.","marker":"[1]"},{"why":"Provides the SGLang inference engine used for all decoding runs and repeated-sampling experiments.","marker":"[37]"},{"why":"Grounds the default confidence-threshold release rule that Evolutionary Decoding modifies.","marker":"[13]"},{"why":"Complementary analysis of locally confident, globally stuck decoding that supports the confidence-trap diagnosis.","marker":"[14]"},{"why":"The repeated-sampling/self-consistency baseline whose saturation motivates trajectory-level intervention.","marker":"[11]"},{"why":"Prior chain-of-thought reasoning in diffusion LLMs, used as an anchor for the reasoning-quality perspective.","marker":"[4]"},{"why":"Prior finding that reasoning in diffusion LLMs concentrates in locally plausible but globally wrong zones, supporting the trap diagnosis.","marker":"[6]"},{"why":"External test-time scaling method whose relative gains ED is compared against in the reference comparison.","marker":"[32]"}],"fun_headline_variants":["Evolutionary decoding escapes diffusion LLM confidence trap","Training-free evolution improves diffusion LLM math reasoning","Diffusion LLM math trap bypassed by selection and mutation","Evolutionary decode lifts diffusion LLM math benchmarks","Confidence trap in diffusion LLMs tamed by evolution"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The mechanism's selection and mutation only recognise tokens that are digits or members of a short fixed symbol list; if correctness hinges on alphabetic variables, inequalities, powers, percentages, or other operators, the method cannot tell which positions matter, so its gains should shrink or vanish on such problems.","fun_headline_variants_meta":{"raw":{"variants":["Evolutionary decoding escapes diffusion LLM confidence trap","Training-free evolution improves diffusion LLM math reasoning","Diffusion LLM math trap bypassed by selection and mutation","Evolutionary decode lifts diffusion LLM math benchmarks","Confidence trap in diffusion LLMs tamed by evolution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1472,"prompt_tokens":744,"completion_tokens":728,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":652}},"tokens_in":488,"tokens_out":728,"duration_ms":9333,"temperature":1.0,"reasoning_tokens":652,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:30:31.044727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark whose solutions rely on alphabetic variables and inequalities (a symbolic algebra set), keep the paper's exact digit/symbol regex, and compare ED against the confidence baseline: if the accuracy gain shrinks to near zero or reverses there while staying positive on a matched numeric-only set, the regex scoping is what carries the result. A second check: on AIME problems, record which of the four mutation branches wins survival selection and whether that branch actually yields a higher final-answer correctness rate than the neutral branch; if the survival rule's choice correlate","supporting_citations":[{"cited_title":"Sglang: Efficient execution of structured language model programs,","cited_arxiv_id":null,"evidence_quote":"Provides the SGLang inference engine used for all decoding runs and repeated-sampling experiments."},{"cited_title":"Diffusion of thought: Chain-of-thought reasoning in diffusion language models,","cited_arxiv_id":null,"evidence_quote":"Prior chain-of-thought reasoning in diffusion LLMs, used as an anchor for the reasoning-quality perspective."},{"cited_title":"Advancing block diffusion language models for test-time scaling,","cited_arxiv_id":null,"evidence_quote":"External test-time scaling method whose relative gains ED is compared against in the reference comparison."}],"review_version":1}