{"id":"38666353-09f0-4c38-8ecf-6d059b42993e","arxiv_id":"2608.02867","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"RLVR-trained LLMs show a stronger collapse in semantic branching entropy than in syntactic style entropy, suggesting the policy prunes genuinely distinct reasoning continuations.","lead":"This paper measures whether reinforcement learning with verifiable rewards (RLVR) makes language models less willing to explore different reasoning paths, beyond just changes in wording. Using maze tasks and a new tree representation of math reasoning traces, it finds that RLVR shrinks the space of semantically distinct solutions, even as it improves rule-following and recovery from dead ends.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The syntax-vs-semantics comparison in §4.4 is confounded by surface form; semantic pairs are likely more token-distant than syntactic pairs, so the reported difference may not isolate semantic branching.","rationale":"The reader's CONDITIONAL verdict is appropriate, and the reader's weakest_assumption—that the LLM judge M(x,y) is validated only against other LLMs—is a genuine concern about the BODHI-TREE construction. My analysis points to a distinct and, in my view, more targeted gap: even if the judge were perfect, §4.4's comparison of syntactic and semantic pairs does not control for surface-form distance. The two continuations in a 'semantic' pair come from different BODHI children and are therefore likely to be far more lexically divergent than the two syntactic variants of the same child; the CPE metric in Eq. 3 operates on raw token sequences. A model whose preferences are purely syntactic—e.g., a preference for shorter traces or a recurring formula template—would naturally show a larger entropy drop for the more token-distant semantic pair. This means the headline claim that RLVR 'prunes semantically distinct verifier-equivalent continuations, not just stylistic variation' rests on a comparison that has not isolated semantic equivalence from surface dissimilarity. The concern is load-bearing because §4.4 is the only experiment that directly addresses RQ2 ('is the policy concentration merely syntactic/stylistic?'). Table 1 and the maze results support a reduction in rollout diversity, but they do not establish that the reduction is specifically semantic. I do not think this requires moving to REJECT: the finding that RLVR constricts continuation diversity is supported by multiple strands, and the semantic interpretation is plausible; the appropriate response is to require the matched-pair control before accepting the stronger semantic-collapse interpretation. Hence the reader's CONDITIONAL verdict stands, and my agreement with the reader is partial: we identify related weak points, but the surface-distance confound is the more decisive one for the central claim.","tokens_in":16896,"tokens_out":5975,"duration_ms":59865,"concrete_test":"Run a matched-pair reanalysis on 200 sampled BODHI branch nodes: for each (a,b) syntactic pair and (a,g) semantic pair, compute BLEU/Jaccard/token-overlap and length ratio. If semantic pairs are significantly more surface-distant, construct new semantic pairs with surface distance matched to syntactic pairs—e.g., semantically different but near-identical phrasing such as 'x=2' versus 'x=3'—and re-estimate ΔCPE_RLVR−ΔCPE_Distil for §4.4. If the semantic-vs-syntactic difference disappears under matching, the 'not merely syntactic' claim is not established; if it persists, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To establish that RLVR entropy collapse is 'not merely syntactic,' §4.4 compares ΔCPE for (a,b)—two continuations in the same BODHI child, judged semantically equivalent but syntactically different—with ΔCPE for (a,g), where g is from a different child and semantically distinct. The CPE in Eq. 3 is computed from token-level average log-probabilities of the two candidate continuations. No control is reported for lexical distance, length, or token overlap between a and g versus a and b. Semantic pairs are selected precisely because they are different solution strategies, so they will typically have much lower surface similarity than the syntactic variants. An RLVR policy that simply develops stronger preferences for a surface form—shorter traces, a particular notation, or a formula layout—would show a larger CPE drop for the more surface-distant pair without any specifically 'semantic' calcification. Thus the difference plotted in Figure 4 (µ_sem − µ_synt) can be an artifact of token-level distance, not a demonstration that semantically distinct verifier-equivalent continuations are pruned. The BODHI-TREE node labels cannot rule this out, because the model never sees the labels; it scores raw token strings. The paper's judge validation (Appendix A, κ≈0.8 against LLMs, with no human evaluation) is a related but distinct weakness: errors in M(x,y) can misplace segments, but even with a perfect judge the surface-form confound remains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether RLVR training collapses test-time exploration in LLMs in a way that is merely syntactic/stylistic or genuinely semantic. It contributes a maze-solving probe with three model families and the BODHI-TREE dataset for AIME mathematics, where reasoning traces are segmented and merged into a tree by an LLM-based semantic-equivalence judge. Defining candidate preference entropy (CPE) from token-level log-probabilities at branch nodes, the authors report that RLVR models have lower branching entropy than distillation-only models, that the drop is stronger for semantically distinct continuations than for syntactic variants, and that RLVR improves legality and backtracking at the cost of rollout diversity. The paper closes with an intervention analysis suggesting RLVR's sample-efficiency gains come from constricting the accessible continuation space.","tokens_in":17141,"tokens_out":3893,"duration_ms":36588,"significance":"If the central claim holds, the paper makes a valuable contribution to the debate on whether RLVR expands reasoning capabilities or merely sharpens sampling: it provides evidence that entropy collapse is not reducible to surface style. The strengths are the controlled matched training setup, the open-sourced artifacts and checkpoints, and the careful use of paired bootstrap confidence intervals and KS tests across four model families. The significance is conditional, however, because the semantic-versus-syntactic comparison rests on an unvalidated LLM judge and on an uncontrolled surface-form distance between the compared continuation pairs, as detailed below.","major_comments":[{"comment":"The comparison that supports the paper's central claim is not controlled for surface-form distance. The semantic pair (a,g) is drawn from different BODHI children, so it will typically have less token overlap and different lengths than the syntactic pair (a,b), while CPE is computed from per-token average log-probabilities of the raw continuations. An RLVR policy that simply sharpens preferences for a surface form—shorter traces, particular notation, or formula layout—would produce a larger ΔCPE for the more surface-distant pair without any specifically semantic calcification. Please add a matched analysis (e.g., pairs balanced for length and token overlap, or regression with edit distance as a covariate) or otherwise show that the μ_sem − μ_synt difference survives when surface distance is equated.","section":"§4.4, Eqs. (3)–(4)"},{"comment":"The semantic equivalence function M(x,y) is validated only against two other LLMs (Cohen's κ = 0.815 and 0.837), and the paper explicitly states that no rigorous human evaluation was performed. If M merges or splits reasoning steps incorrectly, the BODHI branch structure itself is mismeasured, and the semantic CPE values in §4.2 and §4.4 inherit those errors. A human-annotated sample (even a few hundred pairs) with disagreement analysis would materially strengthen the claim that the entropy collapse is semantic rather than an artifact of tree construction.","section":"Appendix A and Limitations"},{"comment":"The CPE computation truncates candidate continuations to 20 tokens. For mathematical reasoning steps, the point at which two strategies diverge semantically can occur after the first 20 tokens, so the reported CPE may measure only a shared surface prefix. Please report sensitivity to the truncation length (e.g., 20/50/100 tokens) or verify on a subset that branch identity is already evident within 20 tokens.","section":"Appendix B.2"}],"minor_comments":[{"comment":"The caption and text contain placeholder glyphs (□) and the typo 'dashedlines'; the rendered means and confidence intervals should be checked.","section":"Figure 4"},{"comment":"Table 3 is difficult to read: the 'Method' and 'Original(%)/Distracted(%)' columns are not cleanly separated in the text, and the 'Acc. [Pass@3]' header appears twice; reformatting is needed.","section":"Table 3"},{"comment":"The distractor example includes an unrelated quadratic step ('Solving gives c=...') that may confuse readers; a cleaner example or a clearer annotation would help.","section":"Figure 8"},{"comment":"The 'Avoid Short Path' intervention states that another verifier-equivalent continuation is guaranteed to exist, but the guarantee is not justified; please specify the maze property that ensures this.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for cs.CL and the open-sourced artifacts are a useful contribution. The main risk is the §4.4 surface-form confound; the revision can address it with matched controls and additional judge validation, so I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is the BODHI-TREE construction and the way the authors use it to ask whether RLVR prunes semantically distinct verifier-equivalent continuations or just stylistic variation. They open-source data, code, and intermediate checkpoints, and they run the analysis across four model families with paired bootstrap CIs and KS tests. The maze experiments are a nice controlled complement. That is real work and the paper is readable.\n\nThe core finding—that RL-trained models show a larger drop in candidate preference entropy for semantically distinct continuations than for syntactic variants—is new relative to the token-level entropy papers, and it is the result that matters. I think the direction of the claim is probably right, but the paper does not yet establish it cleanly.\n\nThe load-bearing issue is Section 4.4. The comparison between ΔCPE for (a,b) within a BODHI child and ΔCPE for (a,g) across children does not control for surface form. Semantic pairs are, by construction, more different in wording and often in length than syntactic variants. Since CPE is computed from token-level average log-probabilities, a policy that sharpens preferences on any surface feature—notation, phrasing, length—will show a larger drop on the more surface-distant pair without any specifically semantic calcification. The paper reports no control for token overlap, length, or lexical distance between the two kinds of pairs. I don't think this breaks the paper, but it means the headline \"not merely syntactic\" is not as clean as stated.\n\nThe secondary weakness is the semantic equivalence judge. The paper validates GPT-oss-120b only against other LLMs (κ≈0.8) and explicitly says no human evaluation was done. The authors are honest about this in the Limitations section, and the inter-annotator agreement is decent, but the judge is the definition of semantic branches, so it deserves at least a small human-validated sample.\n\nMinor: the CPE truncation at 20 tokens is a free parameter, and the maze entropy reduction is partly explained by improved legality, though the paper partially addresses that with the hall-node analysis. I would not call either disqualifying.\n\nWho is this for? Anyone working on RLVR training dynamics or diversity-encouraging objectives. It deserves a serious referee: the flaws are cost-addressable (human eval, matched surface-form pairs) and the measurement framework is reusable. I would send it to review, with a request to redo the Section 4.4 analysis with surface-form controls and to add a small human validation of the judge.","headline":"A solid, honest measurement study of RLVR entropy collapse—new enough to referee, but the semantic-vs-syntactic conclusion is not yet nailed down because the comparison lacks surface-form controls.","tokens_in":17711,"tokens_out":3459,"would_cite":true,"duration_ms":29021,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning with verifiable rewards prunes semantically distinct reasoning branches in LLMs, not just stylistic variation, and this trade-off drives sample efficiency at the cost of diversity.","keywords":["reinforcement learning with verifiable rewards","RLVR","policy entropy collapse","semantic branching entropy","BODHI-TREE","candidate preference entropy","test-time exploration","mathematical reasoning diversity"],"falsifier":"If human mathematicians re-label a sample of BODHI-TREE branch points and the CPE drop between distilled and RLVR models disappears or reverses, the semantic-collapse claim falls. A more mechanical version: rerun the experiment on a domain where semantic equivalence is decidable (e.g., rewritten symbolic algebra expressions) and test whether RLVR still reduces branching at provably distinct nodes.","tokens_in":16667,"feed_emoji":"🌳","tokens_out":7617,"duration_ms":56830,"temperature":0.7,"pith_summary":"This paper asks whether reinforcement learning with verifiable rewards (RLVR) expands an LLM's reasoning ability or merely makes it sample more efficiently. The authors build a tree of semantically equivalent reasoning steps (BODHI-TREE) from thousands of correct AIME solutions, separating stylistic variation from genuine inferential branching. Using this tree plus maze-solving probes, they show that RLVR-trained models have significantly lower branching entropy: they prefer fewer continuations, and the collapse is stronger when the alternatives are semantically distinct rather than merely syntactic variants. The paper concludes that RLVR works by constricting the space of continuations, improving constraint adherence and backtracking at the cost of genuine trajectory diversity.","feed_headline":"RLVR prunes LLM reasoning branches, not just style","feed_subtitle":"Maze and AIME math-trace experiments show RLVR trades trajectory diversity for constraint adherence and backtracking.","key_machinery":"The BODHI-TREE, a prefix tree whose nodes are sets of semantically equivalent mathematical statements (merged by an LLM-as-a-judge matching function), and the candidate preference entropy (CPE) computed at branch nodes. The tree lets the authors separate syntactic variation (two continuations that say the same thing differently) from semantic variation (continuations that use different mathematical approaches), and CPE quantifies how strongly a model prefers one continuation over another at a given branch.","core_discovery":"On the paper's own terms, the central discovery is that the policy entropy collapse observed in RLVR-trained LLMs is not merely syntactic. By segmenting correct math traces into reasoning nodes and merging semantically equivalent statements via an LLM judge, the authors construct BODHI-TREEs in which each branching node offers distinct continuations. They define candidate preference entropy (CPE) at these branch points and find that RLVR-trained models show a statistically significant drop in CPE relative to distilled models across four model families, with the drop larger for semantically distinct continuations than for syntactic variants. Maze experiments confirm reduced test-time exploration, while also showing that RLVR improves the model's ability to avoid wall collisions and to backtrack out of dead-ends. The authors interpret this as evidence that RLVR's sample-efficiency gains come from pruning verifier-invalid continuations and ossifying trajectory preferences, at the cost of semantic rollout diversity.","pith_inferences":["The stronger collapse for semantically distinct continuations suggests that the reward signal itself, which checks only final answers, implicitly penalizes intermediate steps that are verifier-equivalent but less probable; an explicit diversity reward computed at the semantic level (e.g., over BODHI-TREE nodes) could counteract this without hurting validity.","A direct testable extension: build BODHI-TREEs using a human-annotated equivalence function on a small subset of AIME questions and recompute the CPE drop; if the drop persists, the semantic collapse is robust to judge error.","The maze 'Legal Only' intervention shows that masking invalid moves recovers most of RLVR's gain on a distilled model, which suggests that a lightweight decoding-time constraint could substitute for some RLVR training while preserving diversity.","The paper's distinction between syntactic and semantic branching entropy could be applied to other post-training schemes (e.g., DPO variants) to see whether all preference optimization methods calcify semantic choices or only reward-based ones."],"forward_implications":["If RLVR inherently prunes semantically distinct reasoning paths, then techniques that sample many rollouts at test time, such as self-consistency or tree-of-thought, will see diminishing returns on RLVR-trained models unless the training objective includes a semantic-diversity term.","The paper's maze results imply that RLVR's pass@1 gains come largely from learning environmental constraints (avoiding invalid moves) and better backtracking, not from discovering novel solution strategies.","Monitoring CPE on BODHI-TREEs during RL training could serve as an early-warning signal for reasoning-diversity collapse, before it shows up in pass@k metrics.","Long-CoT distillation preserves more semantic branching than RLVR on the same base model, suggesting that distillation and RLVR occupy different points on a diversity-versus-validity trade-off.","If the effect generalizes beyond AIME-style math, RLVR-trained coding or agentic models may also suffer from reduced exploration, limiting their ability to handle novel or adversarial states."],"supporting_citations":[{"why":"Claimed RL-trained LLMs have degraded pass@k scaling and do not discover novel reasoning pathways; the paper's main target.","marker":"Yue et al. (2025)"},{"why":"Showed the RL policy is driven by a minority of high-entropy forking tokens; motivates measuring semantic rather than token-level diversity.","marker":"Wang et al. (2025)"},{"why":"Posited RL-trained models are over-confident with lower policy entropy; provides context for the CPE drop.","marker":"Saha et al. (2026)"},{"why":"Supplied the Maze Dataset used for the controlled exploration probes.","marker":"Ivanitskiy et al. (2023)"},{"why":"Provided the DAPO-Math-17k dataset and RL training recipe (including DAPO-Qwen2.5-32B) for the math experiments.","marker":"Yu et al. (2026)"},{"why":"The model card for the LLM judge used to instantiate the semantic-equivalence matching function in Algorithm 1.","marker":"OpenAI et al. (2025)"}],"fun_headline_variants":["RLVR trades reasoning diversity for constraint adherence","RLVR shrinks semantic branching, not just style","BODHI: RLVR cuts genuine inference branching","RLVR's efficiency comes at cost of rollout diversity","RLVR prunes inferential branches, gains backtracking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The BODHI-TREE construction assumes that the LLM judge used for semantic equivalence correctly decides when two reasoning steps are mathematically equivalent; the authors validate it only against other LLMs (Cohen's kappa ~0.82-0.84) and note they could not perform rigorous human evaluation, so the 'semantic' branches may partly reflect judge artifacts.","fun_headline_variants_meta":{"raw":{"variants":["RLVR trades reasoning diversity for constraint adherence","RLVR shrinks semantic branching, not just style","BODHI: RLVR cuts genuine inference branching","RLVR's efficiency comes at cost of rollout diversity","RLVR prunes inferential branches, gains backtracking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1218,"prompt_tokens":892,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":251}},"tokens_in":508,"tokens_out":326,"duration_ms":12614,"temperature":1.0,"reasoning_tokens":251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:56:57.097195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If human mathematicians re-label a sample of BODHI-TREE branch points and the CPE drop between distilled and RLVR models disappears or reverses, the semantic-collapse claim falls. A more mechanical version: rerun the experiment on a domain where semantic equivalence is decidable (e.g., rewritten symbolic algebra expressions) and test whether RLVR still reduces branching at provably distinct nodes.","supporting_citations":[{"cited_title":"Transactions of the Association for Computational Linguistics , volume =","cited_arxiv_id":null,"evidence_quote":"Posited RL-trained models are over-confident with lower policy entropy; provides context for the CPE drop."}],"review_version":2}