{"id":"6cab45aa-9fff-4537-8659-d2fb42aa7f26","arxiv_id":"2505.16673","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Share-GRPO creates paraphrased and visually augmented versions of reasoning questions and shares answers and reward signals across versions, improving multimodal reasoning without cold-start SFT.","lead":"This paper introduces Share-GRPO, a reinforcement learning method that rewrites and augments each question, then shares reasoning answers across the variants during training. The authors report consistent accuracy gains over standard GRPO and prior RL-based reasoning models on six multimodal benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 8 uses an invalid importance ratio for cross-variant sharing: responses drawn from π_old(·|Q_j) are scored with π_θ/π_old under Q_k, so the surrogate is biased even when SCT preserves semantics; the method's core sharing step lacks a sound policy-gradient derivation.","rationale":"The reader's weakest assumption, that SCT variants preserve semantics, is a legitimate and testable concern, but it is a data-quality premise that could be repaired by auditing generated variants. The more fundamental issue is the off-policy validity of Eq. 8: even with perfectly consistent variants, the cross-variant terms use an incorrect importance ratio, so the shared objective is not a faithful policy-gradient surrogate. This directly threatens the central claim that sharing responses and rewards effectively mitigates sparse reward and advantage vanishing, because the learning signal itself is biased. The paper does provide ablations and positive benchmark results, which are real empirical evidence, but the missing derivation is not a stylistic gap; it means the updates optimize an uncontrolled objective. I therefore keep the reader's conditional verdict, with an additional required condition: either correct Eq. 8 with a proper importance weight or explicitly treat the objective as a heuristic and measure its bias. My disagreement with the reader is one of emphasis: the single most load-bearing assumption should be the objective's off-policy validity, with SCT consistency as a secondary, though important, condition.","tokens_in":14699,"tokens_out":8314,"duration_ms":73278,"concrete_test":"On a few hundred rollouts from the actual training distribution, compute Δ(o) = log π_old(o|Q_k) − log π_old(o|Q_j) for the m = 2 variants, both before training and at an intermediate checkpoint. If the empirical distribution of Δ is materially nonzero (e.g., average |Δ| exceeds 0.1 nats per token, or the variance is large), the k ≠ j terms in Eq. 8 are biased and the shared objective needs an importance-sampling correction; if Δ ≈ 0 for essentially all responses, the off-policy concern would be vacuous.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central mechanism of Share-GRPO is the shared policy objective in Eq. 8, which applies every sampled response o_i^{Q_j} to every variant Q_k. For k ≠ j, the PPO-style ratio is π_θ(o_i^{Q_j}|Q_k)/π_old(o_i^{Q_j}|Q_k), but the response was actually sampled from π_old(·|Q_j). A correct importance-sampling surrogate for target Q_k with behavior policy π_old(·|Q_j) would use π_θ(o|Q_k)/π_old(o|Q_j), or equivalently include the density-ratio correction π_old(o|Q_k)/π_old(o|Q_j). As written, the k ≠ j terms are not unbiased estimators of the GRPO objective for either Q_j or Q_k. The bias is governed by the discrepancy between π_old(·|Q_j) and π_old(·|Q_k), which is generically nonzero because the variants are different prompts, and this discrepancy can grow during training. This flaw is independent of whether the SCT variants are semantically consistent: even perfect semantic consistency does not make the two old-policy distributions identical. The paper supplies no derivation, no off-policy correction, and no diagnostic (e.g., KL divergence between variant-conditional old-policy distributions) showing the bias is small. Consequently, the claim that Eq. 8 shares trajectories in a principled way is unsupported, and the ablations in Table 2 cannot be attributed to a well-founded sharing mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Share-GRPO, an online reinforcement learning method for multimodal large language models (MLLMs) that aims to mitigate sparse reward and advantage vanishing in GRPO-style training. Each input question is expanded into m semantically consistent variants via offline GPT-4o text rewriting and online image transformations (rotation, noise), and the policy generates n responses per variant. The method then optimizes a shared policy objective (Eq. 8) that evaluates responses from every variant against every variant, using a hierarchical advantage that combines global (across all variants) and local (within one variant) reward normalization. Training with Qwen2.5-VL-7B and 32B on 52K samples from MM-Eureka, without cold-start SFT, reportedly improves MathVista, MMStar, MMMU, MathVerse, MathVision, and AI2D over the base model, GRPO, and several prior RL-based reasoning MLLMs. Ablations attribute the gains to shared policy optimization and shared advantage estimation.","tokens_in":161,"tokens_out":5158,"duration_ms":80009,"significance":"If the method is sound, the paper would make a useful empirical contribution: the sparse-reward and advantage-vanishing problems are real, the proposed recipe is simple and does not require a reward model, and the evaluation spans six benchmarks. The ablations are incremental and clearly show that each added component contributes to the final MathVista score. The main weaknesses are that the core shared-policy objective is not derived as an unbiased estimator, the semantic-consistency assumption underlying the whole pipeline is not validated, and all headline numbers are single-run results with hyperparameters selected on a test benchmark. These issues currently prevent the paper from establishing Share-GRPO as a principled method; however, they appear addressable with additional derivations, diagnostics, and experiments.","major_comments":[{"comment":"The shared policy objective in Eq. (8) is not a valid surrogate for the GRPO objective when k≠j. In those terms the response o_i^{Q_j} was sampled from π_old(·|Q_j), but the ratio is written as π_θ(o_i^{Q_j}|Q_k)/π_old(o_i^{Q_j}|Q_k). An unbiased importance-sampling surrogate for target question Q_k with behavior policy π_old(·|Q_j) would require π_θ(o|Q_k)/π_old(o|Q_j), or equivalently an additional density-ratio correction π_old(o|Q_k)/π_old(o|Q_j); the form in Eq. (8) is biased by the discrepancy between π_old(·|Q_k) and π_old(·|Q_j). This bias is independent of whether the SCT variants are semantically consistent, because different prompts generally induce different response distributions, and it can change during training. The paper provides no derivation, no off-policy correction, and no diagnostic such as the KL divergence between variant-conditional old-policy distributions. Consequently, the central mechanism of Share-GRPO is not established as a principled policy-gradient update. Please re-derive the objective with the correct importance weights, bound the bias, or explicitly reposition Eq. (8) as a heuristic and supply diagnostics showing that the bias is small in practice.","section":"§3.2.3, Eq. (8)"},{"comment":"The assumption that SCT preserves the correct answer is asserted but never validated. The offline textual rewrites in Eq. (3) are generated by GPT-4o with no consistency check, and the online visual transformations in Eq. (4) (rotation, noise injection) are described only qualitatively, with the accompanying textual transformation τ not specified concretely. If a variant has a different answer than the seed question, the rule-based reward computed in §3.2.2 is wrong, and Eqs. (5)–(8) train the model toward incorrect solutions while the hierarchical advantage can amplify that error. Please report consistency checks (e.g., answer audits on a sample, automatic verification of variant answers on rule-checkable tasks, or human evaluation) and exclude or correct inconsistent variants during training.","section":"§3.2.1"},{"comment":"The empirical comparison is based on single runs with no standard errors or significance tests, and the hyperparameters m, n, p plus the design choices in Tables 2, 4, and 5 are tuned on MathVista, which is also one of the headline benchmarks. This makes it hard to separate the method's intrinsic effect from selection effects and run-to-run variance. Please provide multiple seeds or error bars for the main comparisons and ablations, and state clearly which hyperparameters are fixed before evaluation and which are selected on test benchmarks.","section":"§4.2–§4.4, Tables 1–5"},{"comment":"The hierarchical advantage in Eq. (7) combines global and local advantages only for j=k and uses the global advantage alone for j≠k. The paper gives no justification for excluding the local term for cross-variant evaluations, and no sensitivity analysis for the relative weighting of global and local terms. Please clarify the intended estimator and test alternative weightings or inclusion rules, since this design choice directly affects the training signal.","section":"§3.2.2, Eq. (7)"}],"minor_comments":[{"comment":"The notation in Eq. (5) uses RQj_i in the numerator but rQj_i inside the mean and std expressions; please unify the reward notation across the equations.","section":"§3.2.2, Eq. (5)"},{"comment":"The summation structure in Eq. (8) is ambiguous: it averages over i=1..n, while responses are indexed as o_i^{Q_j} for each variant j, and the total number of responses is m·n. Please clarify what i indexes and why the normalization is 1/n rather than 1/(m·n).","section":"§3.2.3, Eq. (8)"},{"comment":"Figure 2 contains informal Chinese annotations and very small text that is hard to read; please replace them with clean English labels and enlarge the diagram.","section":"Figure 2"},{"comment":"The dynamic sampling baseline is introduced only by reference to [58]; please describe briefly how it is applied and whether it uses the same reward functions and rollout settings as Share-GRPO.","section":"§4.4, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a potentially useful empirical study, but the validity of the central estimator and the unvalidated semantic-consistency assumption need to be addressed before publication. The novelty claim of being the first to introduce information sharing into MLLM reasoning RL should be checked carefully in light of prior work from the same group (R1-VL, Mulberry) and the broader RL literature. No ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical recipe in search of a correct derivation. The idea — expand each question into semantically consistent variants, sample responses across variants, then share both the responses and a hierarchical local/global advantage in the GRPO update — is a new combination for MLLM RL, and the ablations in Table 2 give it some support: each added piece (offline variants, online visual transforms, local advantage) buys about a point on MathVista, and the 7B and 32B results beat the GRPO baseline without cold-start SFT. That is a real, if incremental, contribution.\n\nThe soft spot is in the math that supposedly justifies the sharing. Equation 8 scores a response o_i^{Q_j} — sampled from the old policy under variant Q_j — with the ratio π_θ(o|Q_k)/π_old(o|Q_k) for every other variant Q_k. That ratio is not a valid importance weight: the behavior policy is π_old(·|Q_j), not π_old(·|Q_k), so the k≠j terms are biased estimators of the GRPO objective for Q_k (and for Q_j). The stress-test note is right, and the issue is independent of whether the SCT variants are semantically consistent. The paper offers no off-policy correction, no density-ratio term, no KL diagnostic. So the central claim that trajectory sharing is principled is unsupported, even though the empirical gains might still be real.\n\nTwo more concerns, both secondary. First, semantic consistency is asserted, not checked: no audit that GPT-4o paraphrases or rotations/noise actually keep the answer unchanged. If any variant's answer flips, the shared objective actively trains toward a wrong solution. Second, all results appear to be single-run with no error bars, and the hyperparameters m and n are tuned on MathVista, which is also the headline comparison benchmark — a mild selection leak.\n\nWho this is for: people working on RL for MLLMs, especially on the sparse-reward and advantage-vanishing problem. The recipe is plausible and the paper is worth a serious referee, but the theory section needs to be fixed before publication — either derive a proper off-policy surrogate (e.g., include the correction term π_old(o|Q_k)/π_old(o|Q_j)) or present the sharing as a heuristic with empirical justification, not as a principled objective. I'd send it to review with that expectation.","headline":"Useful empirical recipe for MLLM RL — expand question space and share trajectories/advantages — but Eq. 8's importance ratio is not a valid estimator, so the paper's central mechanism lacks a sound derivation.","tokens_in":15594,"tokens_out":2374,"would_cite":true,"duration_ms":18928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Share-GRPO fixes the two main failure modes of GRPO on multimodal reasoning by sharing responses and rewards across question variants.","keywords":["multimodal large language models","reinforcement learning","GRPO","reasoning","semantically consistent transformation","sparse reward","advantage vanishing","question space expansion"],"falsifier":"Compute the answer-consistency rate of the transformations: take a few hundred training questions, apply the same GPT-4o paraphrases and image rotations or noise used in training, and check with a strong judge or human whether the variant's correct answer matches the original. If a non-negligible fraction of variants flip the answer, the shared reward labels in Eq. (8) are corrupted; a direct test would compare MathVista scores with and without consistency-filtered variants.","tokens_in":14519,"feed_emoji":"🧠","tokens_out":8081,"duration_ms":61190,"temperature":0.7,"pith_summary":"The paper proposes Share-GRPO, a reinforcement-learning algorithm for multimodal large language models that treats each training question not as a single prompt but as a family of semantically equivalent variants. By rewriting the text with GPT-4o and applying safe image transformations such as rotation and noise, it creates an expanded question space, then lets the model generate responses for every variant and shares both the responses and their rewards across the whole family during policy updates. The paper's claim is that this sharing fixes two failures of plain GRPO on multimodal models: rewards are too sparse early in training, and advantages collapse toward zero when all responses to a question become homogeneous. With the shared objective and a hierarchical advantage estimate, R1-ShareVL-7B reaches 75.4 on MathVista, 67.0 on MMStar, and 58.1 on MMMU without any cold-start supervised fine-tuning, outperforming the GRPO baseline and prior RL-trained reasoning MLLMs. A sympathetic reader would care because the method offers a way to get more usable reward signal out of the same questions, rather than more data or a separate reward model.","feed_headline":"Share answers across question variants to lift MLLM reasoning scores","feed_subtitle":"Paraphrased and rotated versions of each question push MathVista from 72.8 to 75.4 without cold-start SFT.","key_machinery":"The load-bearing object is the expanded question family produced by semantically consistent transformation (SCT). An SCT is a text or image rewrite that is supposed to preserve the question's meaning and correct answer while changing its surface form: GPT-4o paraphrases of the text, and visual operations such as rotation or noise injection with an appended textual hint. The second mechanism is hierarchical shared advantage estimation, which computes a global normalized advantage across all variants' responses and a local normalized advantage within each variant, combining the two only for responses generated from the same variant. The third is the shared policy objective in the paper's Eq. (8), which evaluates each response's probability under every variant of the same seed question, so trajectories discovered while solving one version of the question can improve the model on all versions. Together these make the reward signal denser and the advantages non-degenerate.","core_discovery":"Share-GRPO's central claim is that the two known failures of GRPO when applied to multimodal models, sparse reward and advantage vanishing, are information problems rather than data or architecture problems. For a given image-text question, it generates $m$ semantically consistent variants: GPT-4o paraphrases offline, and randomly chosen visual changes with a matching textual hint online. The policy samples $n$ responses per variant, and then the responses are shared: every response is scored under every variant of the same seed question in the policy objective, and its advantage is computed from a global normalization across all m variant responses plus a local normalization within its own variant. The claimed effect is that the reward signal becomes denser, the valid-advantage ratio stays higher during training, and the trained model generalizes to both mathematical and general reasoning benchmarks. The reported numbers are 75.4 on MathVista for the 7B model against 72.8 for GRPO, 67.0 on MMStar, 58.1 on MMMU, and a 32B model that reaches 77.6, 70.2, 70.1, 59.0, 40.3, and 86.2 on the six benchmarks, all without cold-start supervised fine-tuning.","pith_inferences":["Editorial inference: because the paper never checks answer consistency of the transformed variants, a testable extension is to filter variants by comparing their predicted answers with the original before reward assignment; if consistency filtering further improves training, it would confirm that semantic preservation, not mere diversity, is the active ingredient.","Editorial inference: the same sharing principle should transfer to text-only LLM reasoning, where semantic paraphrases are cheaper and visual augmentations are absent; if it transfers, question-space expansion could replace part of the data scaling that current RL-for-LLM pipelines rely on.","Editorial inference: the hierarchical advantage can also be read as a variance-reduction device; comparing Share-GRPO against GRPO with matched total samples and matched compute would separate the benefit of sharing from the benefit of simply seeing more responses."],"forward_implications":["If Share-GRPO is correct, RL for MLLM reasoning does not require a cold-start SFT stage: starting from Qwen2.5-VL-7B, the 75.4 MathVista score is reached by online RL that shares responses and rewards across variants.","Sharing makes sampling more efficient: R1-ShareVL-7B with six responses per question beats GRPO with twenty-four responses, so the method converts extra sampling into extra signal rather than extra cost.","Generalization to non-mathematical reasoning follows: gains on MMStar and MMMU indicate that shared RL training does not sacrifice multi-discipline accuracy the way some reasoning-only RL runs do.","Combining Share-GRPO with dynamic sampling gives a further boost, reaching 75.9 on MathVista and 59.0 on MMMU, so the two mechanisms are complementary rather than redundant.","Scaling holds: the 32B model improves on average by +2.7 over its base model and +3.4 over MM-Eureka-32B."],"supporting_citations":[{"why":"Defines GRPO, the group-relative baseline that Share-GRPO starts from and compares against.","marker":"[3]"},{"why":"Shows that rule-based RL can incentivize long-chain reasoning in LLMs, the motivation for extending this to MLLMs.","marker":"[2]"},{"why":"Identifies sparse rewards in MLLM RL and proposes step-wise rewards, the problem Share-GRPO targets.","marker":"[4]"},{"why":"Provides the 52K training sample and the online-filtering RL baseline whose scores Share-GRPO exceeds.","marker":"[5]"},{"why":"Documents advantage vanishing and motivates the need to reuse high-advantage samples, the issue Share-GRPO addresses.","marker":"[6]"},{"why":"A representative cold-start-SFT RL reasoning MLLM used as a comparison baseline.","marker":"[30]"},{"why":"Supplies the Qwen2.5-VL-7B and 32B base models that R1-ShareVL is trained from.","marker":"[49]"},{"why":"Provides the dynamic-sampling baseline shown to be complementary to Share-GRPO.","marker":"[58]"},{"why":"Toolkit used for official-weight evaluation of comparison models.","marker":"[51]"}],"fun_headline_variants":["Share-GRPO reuses reasoning paths across question variants","Variant sharing fixes GRPO sparse reward and advantage vanish","Paraphrased question variants share rewards to boost MLLM","Share trajectories across variants to lift MLLM reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every semantically consistent transformation, every GPT-4o paraphrase and every rotation or noise injection, leaves the question's meaning and its correct answer unchanged; the paper reports no check of this, and if any variant's answer differs, the shared reward and the policy update in Eq. (8) are computed against a wrong label.","fun_headline_variants_meta":{"raw":{"variants":["Share-GRPO reuses reasoning paths across question variants","Variant sharing fixes GRPO sparse reward and advantage vanish","Paraphrased question variants share rewards to boost MLLM","Share trajectories across variants to lift MLLM reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1753,"prompt_tokens":1000,"completion_tokens":753,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":685}},"tokens_in":616,"tokens_out":753,"duration_ms":6364,"temperature":1.0,"reasoning_tokens":685,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:56:39.949367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the answer-consistency rate of the transformations: take a few hundred training questions, apply the same GPT-4o paraphrases and image rotations or noise used in training, and check with a strong judge or human whether the variant's correct answer matches the original. If a non-negligible fraction of variants flip the answer, the shared reward labels in Eq. (8) are corrupted; a direct test would compare MathVista scores with and without consistency-filtered variants.","supporting_citations":[],"review_version":1}