{"id":"b12c522b-ff7e-44f9-84a7-6ca6026a05b1","arxiv_id":"2608.07581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MBPO uses branch-level, sibling-relative advantages in tree-structured rollouts to improve credit assignment when training multimodal large language models with reinforcement learning.","lead":"A new training method, MBPO, grows trees of parallel reasoning paths for multimodal models, so each segment of an answer receives its own credit instead of one score for the whole response. It reports gains over standard RL baselines on geometry and chart benchmarks, but key comparisons and reproducibility details are missing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing evidence for the causal premise: sibling branches are never shown to represent distinct visual hypotheses, so Eq. (10)'s branch-relative advantage may be a noisy token-level contrast rather than a visual-commitment signal.","rationale":"The reader's weakest-assumption diagnosis is the same one I would make: the method's causal story depends on sibling branches embodying alternative visual interpretations, yet no metric in the paper establishes that. I checked whether any other concern was more load-bearing. The compute-matched analysis in Table 6, the retrained baselines in Table 1, and the reported GPU-hour parity partially address the 'more compute' objection, so that is not the central risk. The internal numbering inconsistencies (Eq. (7) vs Eq. (10) in Algorithm 1, and the 'same sequence length' claim despite <look>-based truncation) are real but presentation-level, not enough to overturn the claim. The decisive gap remains the unmeasured visual diversity of sibling branches: without it, the branch-relative advantage in Eq. (10) could be doing nothing more than renormalizing outcome differences among random samples, which is unlikely to provide a principled improvement over GRPO. The proposed test would settle this by measuring whether siblings actually split on visual groundings. Because the reader already conditioned acceptance on evidence of this kind, the verdict should remain CONDITIONAL rather than being moved to accept or reject.","tokens_in":15626,"tokens_out":6279,"duration_ms":65197,"concrete_test":"On a stratified sample of about 200 held-out Geo3K/MMK12 questions, record for every sibling branch the set of image regions or patches referenced after the <look> marker (using the model's grounding outputs or attention maps), then compute the average pairwise IoU or region-overlap diversity within sibling groups and between independent GRPO rollouts. If within-sibling diversity is not significantly higher than between-rollout diversity, sibling branches are not exploring distinct visual hypotheses, and Eq. (10) is not isolating visual-commitment advantages.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that branch-level sibling-relative advantages improve credit assignment rests on interpreting sibling branches under a shared prefix as distinct visual hypotheses. This premise enters at Eq. (3), where children are sampled i.i.d. from the same policy conditioned on the same context, and is exploited in Eq. (10), where normalized differences in recursively averaged rewards are assigned to whole segments. The paper never measures whether siblings actually differ in visual grounding. The only supporting evidence offered—replay-buffer question diversity in Fig. 7 and self-correction rates in Tables 4–5—is about surface text and cue-word heuristics ('however', 'but', 'actually'), not about whether branches commit to different image interpretations. Since sibling branches are drawn i.i.d. from the same policy under the same prefix, most of their variance could be lexical, arithmetic, or formatting variation; in that case the normalized advantage in Eq. (10) is a noisy token-level contrast with no special connection to perceptual uncertainty, and the method has no principled advantage over GRPO even if the empirical gains survive. The self-reported limitation about difficulty in broader multimodal settings does not resolve this, because the missing evidence is about the mechanism itself, not about scope.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Multi-Branch Policy Optimization (MBPO), a tree-structured reinforcement learning framework for multimodal large language models that constructs reasoning trees via breadth-first parallel expansion, computes segment-level sibling-relative advantages as in Eq. (10), and reuses past segments through a temporal replay buffer with question-balanced sampling. The authors report consistent gains over GRPO, DAPO, GSPO, Shuffle-R1, and MM-Eureka on Geometry3K, K12, and several out-of-domain benchmarks, and argue that branching at vision-language decision boundaries improves credit assignment and mitigates relative advantage degeneration.","tokens_in":15891,"tokens_out":5023,"duration_ms":46922,"significance":"If the central claims hold, MBPO would be a useful contribution to RL training of multimodal LLMs, offering a concrete alternative to trajectory-level advantage assignment and providing a released open-source implementation. The reported gains on MathVerse, MathVision, ChartQA, HallusionBench, and WeMath are encouraging. However, the significance is substantially weakened by (i) the absence of comparisons to tree-based RL baselines (TreePO, TreeRPO, TreeRL) that are most relevant to the method, (ii) a lack of direct evidence that sibling branches correspond to distinct visual hypotheses, and (iii) a compute-controlled analysis whose control conditions are not clearly specified. The paper's own ablation (Table 7) suggests that fixed-length branching, not vision-language-boundary branching, drives most of the improvement, which undercuts the proposed mechanism.","major_comments":[{"comment":"The central mechanism claim that MBPO branches at vision-language decision boundaries to explore distinct visual hypotheses is not supported by the ablation. Table 7 shows that <look>-only branching achieves 45.17, worse than fixed-M branching at 48.35, and the hybrid's gain over fixed-M is only 1.56 points (49.91 vs 48.35). If visual boundary branching were the source of the improvement, one would expect it to outperform or at least match fixed-length segmentation. The paper does not measure whether sibling branches commit to different visual interpretations; the diversity statistics in Figure 7 are about distinct questions, not visual hypotheses, and the self-correction analysis in Tables 4-5 relies on textual cue words. This leaves the causal premise of Eq. (3) and Eq. (10) unverified.","section":"Section 4.4, Table 7"},{"comment":"The 'relative advantage degeneration' motivation is partly circular. The valid advantage ratio in Eq. (1) counts advantages with |A_i| > 0. For GRPO, advantages are group-relative and can be exactly zero when a response equals the group mean or when all rewards in the group are identical. For MBPO, Eq. (10) normalizes sibling-relative differences by std + epsilon, so advantages are non-zero by construction whenever sibling values differ, and the epsilon term ensures non-zero values even when they do not. Thus Figure 2's near-1.0 VAR for MBPO may be an artifact of the advantage definition rather than evidence of a better learning signal. The paper should report VAR for MBPO computed using the same trajectory-level advantage formula as GRPO on the same rollouts to make the comparison meaningful.","section":"Section 3.1, Eq. (1), Figure 2"},{"comment":"No tree-based RL baselines are evaluated. The related work explicitly positions MBPO against TreePO, TreeRPO, and TreeRL, which are the closest methods in using tree-structured rollouts for policy optimization. Without including these or equivalent tree-based methods in Tables 1-3, the observed gains cannot be attributed to branch-level sibling-relative advantages over trajectory-level advantages; they could simply stem from the tree-structured sampling strategy or from a larger number of effective samples per question. Adding at least one tree-based baseline on the same benchmarks is essential for the paper's central comparative claim.","section":"Section 2.2 and Section 4.2"},{"comment":"The compute-controlled analysis is not convincingly controlled. The paper states that per-epoch GPU cost is nearly identical (19.48 vs 19.32 GPU-hours on 4 GPUs) despite MBPO generating K^D children per question with branching factors such as [6,6,6]. For a fixed token budget per segment, this implies MBPO generates far more tokens per question than GRPO, so the claim that the gains are not due to increased rollout expenditure requires an explicit specification of the total token budget or rollout count per question per method. Table 6 aligns methods by cumulative TFLOPs but does not state how many generated tokens each method consumed at each aligned checkpoint. Without this information, the 'compute-matched' comparison does not rule out the possibility that MBPO simply uses more rollouts.","section":"Section 4.3, Table 6"},{"comment":"All reported results are single-run point estimates without standard deviations, confidence intervals, or significance tests. The claimed in-domain gains over the best baselines are often small (e.g., 1.0 and 2.5 points on MathVerse and MathVision in Table 1; 1.63 points on Geo3K Math Avg in Table 2), and the K12 comparison in Table 3 shows MBPO-Qwen-7B is within 0.03 points of DAPO on K12 accuracy (67.68 vs 67.65). Given the known variance of RL fine-tuning runs, these differences may not be statistically robust. The paper should report multiple seeds or at least a significance analysis for the central comparisons.","section":"Tables 1-3"}],"minor_comments":[{"comment":"Line 26 says 'compute normalized sibling-relative advantages by Eq. (7)', but Eq. (7) is the parent-child difference A(n) = R(n) - R(parent(n)); the normalized sibling-relative advantage is defined in Eq. (10). The algorithm should refer to Eq. (10).","section":"Algorithm 1, line 26"},{"comment":"The text states that sibling nodes share the same sequence length, but the <look>-based branching can produce segments of varying lengths, as evidenced by the average segment-length values in Table 7 (e.g., 287.4 tokens for <look>-only). Please clarify how Eq. (10) ensures a fair comparison when sibling subsequences have different lengths.","section":"Section 3.3, Eq. (8)-(10)"},{"comment":"Figure 7 measures 'Distinct Questions' in the replay buffer, which is a question-coverage statistic, not a measure of visual hypothesis diversity among sibling branches. The caption and the surrounding text should be careful not to imply that this measures perceptual diversity.","section":"Figure 7 and Section 4.3"},{"comment":"The footnote says all RL methods are grouped and retrained on MMRL18K, but the table includes cold-start models marked with † and models from different base architectures. Please specify the exact training protocol, data, and base model for each baseline, and clarify which values are taken from original papers versus re-implemented.","section":"Table 1 footnote"},{"comment":"The dataset is referred to as both 'MMK12' and 'K12' in different places (e.g., Section 4.1 vs Tables 2-3). Please unify the terminology.","section":"General notation"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and reports promising empirical results, but the central mechanism is under-supported and the comparison to tree-based baselines is missing. The ablation in Table 7 is particularly concerning for the paper's narrative: if fixed-length branching alone nearly matches the full method, the 'vision-language boundary' framing may be overstated. I would advise the editor that the revision should either provide direct evidence of visual diversity among sibling branches or reframe the method as a generic tree-based RL approach. The compute-matched analysis also needs a clear statement of token budgets, as the current GPU-hours claim is not credible without it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the method is a sensible combination of existing ideas—tree rollouts, sibling-relative advantages, replay—applied to multimodal reasoning at vision-language boundaries. The empirical gains are plausible, and the compute-matched comparison is a real plus. But the paper's own mechanism story is shakier than the results: the \"distinct visual hypotheses\" premise is never directly measured, and the VAR-based evidence is partially circular. Missing tree-based baselines also weaken the novelty claim.\n\nWhat's actually new: TreePO/TreeRPO/TreeRL already do tree-structured RL for text; MBPO's contribution is branching at <look> boundaries and segment-level sibling-relative advantages with a temporal replay buffer for MLLMs. That combination is new enough to matter, and the authors cite the concurrent tree work fairly.\n\nWhat it does well: Table 6 matches TFLOPs and shows MBPO ahead at every checkpoint, which addresses the usual \"tree search just spends more compute\" objection. Table 7's ablation of branching strategies is informative. Code is public. The limitations section names real constraints.\n\nWhere it's soft: no TreePO/TreeRPO/TreeRL adaptation is run, so we don't know if the gain is from the tree structure or the advantage rule. All numbers are single runs; the 1-2 point leads over MM-Eureka/Shuffle-R1 could move with seeds. The VAR plot in Figure 2 compares a normalized advantage that is essentially guaranteed non-zero by the epsilon in Eq. (10) with GRPO's raw advantages; that comparison doesn't support the \"degeneration\" story. And the visual-diversity premise—that siblings represent different image interpretations—is asserted, not measured. The self-correction analysis uses cue-word heuristics, which is weak but not central. Also, hyperparameters (M, branch factors, Tmax) are tuned on Geo3K/K12 and then evaluated there; some overfitting risk.\n\nNone of this kills the result. The performance comparison is independent of the mechanism story, and the compute-matched evidence is solid enough to believe tree-structured branch-level credit assignment helps. But the paper overclaims the visual interpretation mechanism.\n\nWho gets value: anyone working on RL for multimodal LLMs or credit assignment in long visual reasoning chains. It deserves a serious referee; a good referee would ask for tree baselines, seeds, and a direct measure of branch diversity in revision. I'd accept it for review rather than desk reject.","headline":"A plausible and compute-controlled method for tree-structured RL in multimodal LLMs, but the visual-hypothesis mechanism is asserted rather than measured and the nearest tree-based baselines are missing.","tokens_in":16425,"tokens_out":2590,"would_cite":true,"duration_ms":25140,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that multimodal reinforcement learning should assign credit at vision-language decision points, where a model commits to one visual interpretation, rather than across entire response trajectories.","keywords":["Multi-Branch Policy Optimization","credit assignment","multimodal large language models","reinforcement learning","sibling-relative advantage","reasoning tree","temporal replay buffer","visual reasoning"],"falsifier":"Measure the visual diversity of sibling branches: for a sample of questions, take a parent node's $K$ sampled continuations and compare their image-attention maps or the image regions they refer to. If sibling continuations mostly share the same groundings and differ only in wording or arithmetic, or if the reward spread across siblings disappears when the image is replaced by a blank placeholder, then the sibling-relative advantage is not isolating visual interpretation and the claimed mechanism would fail.","tokens_in":15404,"feed_emoji":"🌳","tokens_out":4530,"duration_ms":39316,"temperature":0.7,"pith_summary":"This paper argues that reinforcement learning for multimodal large language models fails when it credits every token in a response with the same advantage, because visual reasoning contains decision points where the model commits to one interpretation of an image over another. It proposes Multi-Branch Policy Optimization (MBPO), which grows a tree of rollouts at these vision-language boundaries, so that sibling branches share the same prefix and differ in the visual hypothesis they pursue. A sibling-relative advantage, formed by normalizing each branch's propagated value against its siblings under the same parent, then assigns segment-level credit to the branch rather than the whole trajectory. The authors report that this keeps the fraction of non-zero advantages high throughout training, where GRPO's decays toward zero, and yields consistent gains over GRPO, DAPO, GSPO, Shuffle-R1, and MM-Eureka on geometry and multimodal reasoning benchmarks.","feed_headline":"Branch-level rewards beat GRPO on vision-language math","feed_subtitle":"A tree of sibling rollouts with segment-level credit keeps advantages from collapsing, raising MathVerse to 52.6.","key_machinery":"The load-bearing object is the reasoning tree with sibling-relative branch advantage. A node's value is the mean reward of its children for internal nodes; Eq. (10) defines $A_{\\text{sib}}(n)$ as the normalized difference between a node's value and the mean of its siblings, divided by the sibling standard deviation, which isolates how much better one continuation is than alternatives generated from the same prefix. Branching is triggered by a vision-language boundary marker `<look>` plus fixed-length segmentation, and the temporal replay buffer (Eq. 12) restricts reuse to recent segments and samples balanced by question. The hybrid `<look>`-plus-fixed segmentation is what aligns branches with visual interpretation changes rather than arbitrary token positions.","core_discovery":"The central claim is that representing each candidate answer as a multimodal reasoning tree, with branching at points where the model re-examines the image, converts credit assignment from a trajectory-level average into a comparison among sibling hypotheses. The tree is built by breadth-first parallel expansion with branch factor $K$ per depth; after leaves are scored, rewards propagate bottom-up as means over children (Eq. 6), and each node's advantage is its value minus the mean of its siblings, divided by their standard deviation (Eq. 10). This sibling-relative advantage is then used in a PPO update with a temporal replay buffer that keeps only segments younger than $T_{\\max}$ and balances questions in each mini-batch. The paper's evidence is that MBPO keeps the valid advantage ratio near 1.0 while GRPO declines, produces a denser non-zero advantage distribution, and on MMRL18K MBPO-Qwen-VL-7B reaches 52.6 on MathVerse and 30.6 on MathVision, beating MM-Eureka-Qwen-7B by 1.0 and 2.5 points.","pith_inferences":["A testable extension is to measure, on a held-out set, whether sibling branches generated under the same prefix actually differ in visual interpretation (for example, in image-attention maps) rather than only in wording or arithmetic; if visual diversity is low, branch-relative advantages may reduce to noisy token-level comparisons.","The mechanism suggests an early-termination rule: if the local parent-child value difference in Eq. (7) is below a threshold, stop expanding that branch. The paper lists this as future work rather than testing it.","The same tree-relative credit idea could transfer to text-only reasoning if branching is defined at uncertainty boundaries instead of `<look>` markers, but the paper only demonstrates the multimodal setting.","The temporal replay buffer with question-balanced sampling could be ablated independently to see whether its gains persist without tree-structured rollouts, isolating the two contributions."],"forward_implications":["If sibling-relative advantages work as claimed, then RL for multimodal LLMs should be designed around decision boundaries rather than whole responses, making segment-level credit the default learning signal.","The relative advantage degeneration that GRPO suffers is avoidable: MBPO maintains a near-1.0 valid advantage ratio and a denser non-zero advantage distribution, so training stays informative longer.","Compute-matched comparisons on Geo3K show gains of +2.0 to +8.7 accuracy points at the same cumulative TFLOPs, suggesting the gains come from credit quality rather than a larger rollout budget.","The observed self-correction behavior rises with MBPO training (self-correction rate from 13.3% to 60.9%, successful corrections from 4.7% to 29.8%), implying branch-level credit encourages the model to revisit images and revise its reasoning.","Larger tree branch factors (8-8-8) increase replay-buffer question diversity and late-training accuracy, while 6-6-6 offers a better efficiency-accuracy trade-off."],"supporting_citations":[{"why":"Supplies the GRPO baseline and the trajectory-level group-relative advantage scheme that MBPO replaces.","marker":"[37]"},{"why":"Provides the DAPO baseline with token-level policy gradient loss that MBPO compares against.","marker":"[33]"},{"why":"Shuffle-R1 addresses advantage collapsing and is a key baseline; its valid advantage ratio measurement is reused by MBPO.","marker":"[39]"},{"why":"MM-Eureka provides the MMRL18K training data source and the main large-scale RL baseline on MathVerse and MathVision.","marker":"[22]"},{"why":"R1-ShareVL introduces Share-GRPO and hierarchical advantage estimation, and is the source of the valid advantage ratio tracking.","marker":"[32]"},{"why":"R1-VL adapts GRPO to MLLMs with stepwise rewards and is a baseline for the RL comparison.","marker":"[34]"},{"why":"Geometry3K is the small-scale training and in-domain evaluation dataset used for main experiments and ablations.","marker":"[19]"}],"fun_headline_variants":["Sibling branch comparisons keep reward signals from vanishing","Tree-based policy optimization prevents advantage decay in MLLMs","Segment-level rewards from sibling rollouts improve math reasoning","Branch-relative advantages stop reward collapse in vision-language RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes sibling branches generated from the same prefix are genuinely distinct visual hypotheses, so that the normalized value difference between a branch and its siblings measures the causal effect of choosing one visual interpretation; if siblings differ mostly for non-visual reasons or are near-duplicates, the branch advantages inherit that noise.","fun_headline_variants_meta":{"raw":{"variants":["Sibling branch comparisons keep reward signals from vanishing","Tree-based policy optimization prevents advantage decay in MLLMs","Segment-level rewards from sibling rollouts improve math reasoning","Branch-relative advantages stop reward collapse in vision-language RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000505,"raw_usage":{"total_tokens":2461,"prompt_tokens":942,"completion_tokens":1519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":1455}},"tokens_in":558,"tokens_out":1519,"duration_ms":11786,"temperature":1.0,"reasoning_tokens":1455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:32:23.295606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the visual diversity of sibling branches: for a sample of questions, take a parent node's $K$ sampled continuations and compare their image-attention maps or the image regions they refer to. If sibling continuations mostly share the same groundings and differ only in wording or arithmetic, or if the reward spread across siblings disappears when the image is replaced by a blank placeholder, then the sibling-relative advantage is not isolating visual interpretation and the claimed mechanism would fail.","supporting_citations":[],"review_version":1}