{"id":"ff858587-df9c-458f-838e-b510f5f0fef3","arxiv_id":"2505.24164","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Mixed-R1 uses four reward types under GRPO, including a new bidirectional max-average token similarity (BMAS) reward, and lifts MLLM reasoning benchmarks by 2-5%.","lead":"Researchers trained multimodal AI models with a mix of reward functions, one for each task type, plus a new similarity reward for captions. The approach, Mixed-R1, improved math, chart, and grounding benchmarks on Qwen2.5-VL and InternVL by 2-5%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mixed-45K is assembled from public benchmarks and there is no stated deduplication against the evaluation suites; if MathVista/MathVision/MMMU/MMStar contain even a small fraction of Mixed-45K images/questions, the reported 2–5% gains could be memorization rather than generalizable RL.","rationale":"The paper's stated contribution is that a mixed reward design plus Mixed-45K improves general MLLM reasoning, with the benchmark deltas in Tables 1 and 3b as evidence. For that evidence to support the causal claim, the test items must not have been in training. The paper never reports deduplication, and the training sources overlap with the benchmark construction: MathVista is assembled from many public visual-QA sets (including MapQA/CLEVR-like and chart sets), and MathVision and MMMU similarly contain public questions. Because GRPO optimizes exact matching and format on those very sources, an overlap of even a few percent of evaluation examples could account for the whole reported gain. This is a concrete, checkable threat and is more load-bearing than the Sec. 3.1 filter assumption identified by the reader. I credit the release of code/data/models and the incremental ablation in Table 2, but these do not resolve leakage. A secondary issue is Eq. (5): as printed, r_format is inverted, so the final reward would penalize well-formatted answers; this is likely a typo but must be corrected for reproducibility. Conditional acceptance is appropriate until the overlap audit is supplied.","tokens_in":16013,"tokens_out":4792,"duration_ms":65000,"concrete_test":"Run an overlap audit: for each example in Mixed-45K and each item in MathVista, MathVision, MMMU, and MMStar, compare images by perceptual hash (pHash/dHash) and questions by normalized exact match and 6-gram overlap. Report the overlap matrix. Then rerun the Qwen2.5-VL-3B/7B evaluations excluding any overlapping items (or retrain on the deduplicated data). If the average gains drop to at most 1 point on the disjoint subset, the central claim fails as stated; if gains persist on the fully disjoint subset, contamination is ruled out.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that Mixed-45K training examples are disjoint from the evaluation benchmarks. The paper states only that Mixed-45K is sampled from open datasets: MapQA, CLEVR, FigureQA, GeoQA, UniGeo, ScienceQA, ChartQA, DocVQA, InfographicVQA, RefCOCO, and ALLaVA. Several of these are known source components of the evaluation benchmarks: MathVista was constructed from existing visual QA datasets (including chart and geometry subsets), and MathVision/MMMU also draw on public question banks. No overlap analysis or deduplication is reported. Under GRPO, the model is explicitly rewarded for exact matching and format on this training distribution; if even a few hundred benchmark items (or their source images) appear in Mixed-45K, the 2–5% average improvement could be driven by memorized answers rather than by the mixed reward design. The Sec. 3.1 filter makes this worse: it intentionally keeps examples on which Qwen2.5-VL-7B has partial success, which may preferentially retain easy/medium public benchmark items and exacerbate overlap relevance. This is more load-bearing than the filter's 'identical rewards' assumption, because the benchmark numbers are the entire quantitative support for the unified-RL claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mixed-R1, an RL post-training framework for multimodal LLMs that combines four task-specific reward functions (matching, chart, IoU, and a new tokenizer-based open-ended reward BMAS) with a filtered mixed dataset Mixed-45K. The authors apply GRPO to Qwen2.5-VL-3B/7B and InternVL2.5-4B and report average improvements of 2–5% on MathVista, MathVision, MMMU, MMStar, MMBench, and AI2D. The paper also ablates reward components, data scale, and open-ended reward designs, and releases code, data, and model weights.","tokens_in":16296,"tokens_out":6937,"duration_ms":74140,"significance":"If the reported gains are robust and reflect generalization rather than benchmark overlap, the paper would make a useful contribution: it is among the first to combine heterogeneous MLLM tasks in a single GRPO run, and its BMAS reward avoids the cost of an external judge model. The dataset and code release would facilitate reproducibility. However, the current evidence is weakened by the absence of statistical error bars, a possible training/evaluation overlap, and an inverted format-reward equation, so the significance hinges on these being resolved.","major_comments":[{"comment":"Mixed-45K is built from MapQA, CLEVR, FigureQA, GeoQA, UniGeo, ScienceQA, ChartQA, DocVQA, InfographicVQA, RefCOCO, and ALLaVA, but the paper never reports a deduplication or overlap analysis against the evaluation suites (MathVista, MathVision, MMMU, MMStar). Because MathVista in particular was constructed in part from existing chart/geometry/QA datasets (including ChartQA and FigureQA), and the §3.1 filter deliberately keeps examples on which the base model has partial success, even a small overlap could let GRPO memorize benchmark items and inflate the 2–5% average gains. The authors should quantify exact-duplicate and image-overlap rates between Mixed-45K and each evaluation benchmark, and either exclude overlapping items or report results on a cleaned subset.","section":"§3.1, Table 1"},{"comment":"All results are single runs with no error bars, multiple seeds, or significance tests. The headline improvements are 2–3% on the average, which is within the typical run-to-run variability of GRPO on small datasets; e.g., Table 3(a) shows a 3.4-point swing on MathVista between the 45K and 90K conditions. The authors should provide at least three seeds for the main comparisons (or otherwise quantify variance) before the claimed improvements can be taken as reliable.","section":"Tables 1–4"},{"comment":"Equation (5) defines r_format = 0 if output matches format and 1 if it doesn't, which is the opposite of the surrounding prose ('assigning a reward value of 1 if the response conforms to the specified format and 0 otherwise'). This is a direct contradiction in a load-bearing definition, since the format reward is applied to every training example. The authors must state the correct definition and clarify which variant was used in the experiments.","section":"Equation (5)"},{"comment":"The BMAS reward is computed using 'the input embedding space of the policy model,' but the paper does not state whether those embeddings are detached or frozen during GRPO. Since the policy's embedding weights are updated during training, the reward signal would change as training progresses, making it non-stationary and potentially explaining why alternative embedding-based rewards (ModernBert, Sentence-BERT) underperform. Please specify how the embeddings are obtained (frozen vs. trainable, detached vs. attached) and justify the stability of the reward.","section":"§3.2, BMAS Reward"},{"comment":"The data filter drops an example when all g=8 model samples receive identical rewards under the exact same reward functions that later define the GRPO training signal. This conflates 'useful for training' with 'easy to score under the chosen rewards,' and the filter's preference for partial-success examples may systematically select for base-model-friendly benchmark-like items. The paper should ablate the filter (e.g., train on unfiltered 45K or a random subset) and report whether the gains in Table 1 persist.","section":"§3.1"}],"minor_comments":[{"comment":"The text contains a typo: 'GPRO' should be 'GRPO' (it appears twice, once near the description of the data filter and once in §3.2).","section":"§3.1"},{"comment":"The standard deviation is written as std({ri, r2, ...}); the first index should be r1 (or the notation should be made consistent) to avoid confusion with the group index i.","section":"Equation (2)"},{"comment":"Reference [8] is incomplete: the title ends with 'with less than $3' and the intended model size is missing.","section":"Reference [8]"},{"comment":"Figure 4's caption does not define what the plotted curves represent, and the x-axis label 'Steps' is ambiguous; the comparison would be easier to interpret if the training reward and completion length were shown on shared axes with clear legends.","section":"Figure 4"},{"comment":"The training setup for Table 4 is not described; in particular, it is unclear whether the comparison trains only on the open-ended portion of Mixed-45K, which would explain the lower absolute numbers relative to Table 2. Please state the training data and hyperparameters used for this ablation.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The most serious risk is benchmark contamination: the authors should be asked to provide overlap statistics and results on a clean subset before acceptance. The other concerns (error bars, the inverted format-reward equation, and the non-stationary BMAS reward) are fixable with additional analysis or clarification. I see no sign of deliberate misconduct; the omissions are consistent with a preprint that needs a revision pass."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is worth a serious look. The BMAS reward is a genuinely useful idea—giving an open-ended reward from the policy model's own tokenizer embeddings, with no external judge model—and the mixed-reward design for matching, chart, IoU, and open-ended data is a sensible unification even if the 'first' in the abstract overstates it. The ablations are honest: each reward adds something, and the comparison of BMAS against BERT, Sentence-BERT, and an LLM judge is well designed. Two model families are trained, and the gains, while modest, are consistent.\n\nThe soft spots are real. The most load-bearing is the one the stress-test flags: Mixed-45K is drawn from ChartQA, CLEVR, MapQA, GeoQA, ScienceQA, FigureQA and other public sets, and MathVista is substantially constructed from those same sources. The paper gives no deduplication or overlap analysis against the evaluation benchmarks. If any of those items (or images from the same sources) show up in the training set, part of the 2–5% average improvement could be memorization rather than generalization. This is not a knock-down flaw by itself—the gains also appear on MMBench and AI2D, which are less clearly overlapping—but a referee should demand an explicit analysis before the headline numbers are taken at face value.\n\nSecond, there are no error bars or multiple seeds; on small benchmark splits, 2–3% differences are not obviously significant. Third, the format reward equation (Eq. 5) is inverted relative to its prose—surely a typo, but it should be fixed. The data filter that uses the same reward functions to select training data is a mild selection bias, not a circularity, because evaluation is on external benchmarks. Also, the 'no works can leverage multi-source MLLM tasks' claim is contradicted by the paper's own citations to MM-Eureka and R1-Onevision, though those focus on math/science rather than grounding or open-ended.\n\nWho this is for: anyone doing GRPO-style RL post-training on MLLMs, especially with open-ended or heterogeneous task types. It would be a useful reading-group paper, and I would cite the BMAS reward if I were working in this area.\n\nRecommendation: send to peer review. The paper deserves referee time. Reviewers should ask for deduplication against MathVista and the other eval suites, error bars or multiple seeds, and a corrected Eq. 5. If the overlap analysis comes back clean, this is a solid contribution.","headline":"Genuinely useful mixed-reward RL post-training paper, but the missing deduplication check against evaluation suites keeps me from fully trusting the headline gains.","tokens_in":16827,"tokens_out":6204,"would_cite":true,"duration_ms":67944,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single reinforcement-learning reward design, matched to each answer format, improves Qwen2.5-VL and InternVL models on four visual-reasoning benchmarks by 2-5 percentage points.","keywords":["multimodal large language models","reinforcement learning","GRPO","reward design","visual reasoning","BMAS","open-ended reward","post-training"],"falsifier":"Train the same GRPO and Mixed-Reward setup on a random 45K sample drawn from the same unfiltered source pools; if the random sample matches Mixed-45K's gains on MathVista, MathVision, MMMU, and MMStar, then the all-same-reward filtering rule is not doing the work the paper assigns to it.","tokens_in":15846,"feed_emoji":"🧠","tokens_out":8135,"duration_ms":73381,"temperature":0.7,"pith_summary":"Mixed-R1 claims that a single reinforcement-learning post-training recipe can improve a multimodal large language model across several different visual reasoning tasks at once, rather than requiring one training run for math, another for charts, another for grounding, and another for captions. The authors assemble a 45K-sample training set from existing open datasets and filter it with a simple rule: generate eight responses per question with Qwen2.5-VL-7B, and drop any question whose eight responses all receive the same reward, on the grounds that such questions are too easy or too hard for GRPO to learn from. They then train with four reward functions matched to the answer format, including a newly proposed open-ended reward called BMAS that uses the model's own tokenizer embeddings instead of an external judge model. The reported result is a 2-5 percentage point average improvement on MathVista, MathVision, MMMU, and MMStar for Qwen2.5-VL-3B, Qwen2.5-VL-7B, and InternVL2.5-4B.","feed_headline":"Mixed rewards lift multimodal reasoning by 2-5 points","feed_subtitle":"Qwen2.5-VL and InternVL improve on math, chart, grounding, and caption tasks in a single RL run.","key_machinery":"The load-bearing object is the mixed reward design running under Group Relative Policy Optimization (GRPO), a rule-based RL method that computes advantages from rewards within a group of sampled responses: four content rewards selected by data type, plus a format reward that enforces <think> and <answer> tags. For open-ended answers, BMAS is the central new mechanism: it computes pairwise cosine similarity between the policy model's input-token embeddings of the generated text and the ground-truth text, then takes the average of the row-wise maxima and column-wise maxima of that similarity matrix. This gives a reward that can tell long-form captions apart without an external judge LLM, and the maximum-based aggregation is what the paper credits for stable training.","core_discovery":"The paper's central claim is that the bottleneck for R1-style reinforcement learning in multimodal models is reward design, not the RL algorithm: each task family needs a reward matched to its output format, and open-ended text needs a stable dense reward that does not require a second model. Mixed-R1 sorts data into yes/no, multiple-choice, chart, grounding, and open-ended categories; assigns matching, chart, IoU, and BMAS rewards; and combines them with a format reward inside GRPO. The distinctive piece is BMAS, which embeds the generated response and the ground truth with the policy model's own tokenizer embeddings, computes pairwise cosine similarities, and scores the response by averaging the maximum similarity over each row and each column of the similarity matrix. In the paper's experiments this design improves Qwen2.5-VL-3B, Qwen2.5-VL-7B, and InternVL2.5-4B on MathVista, MathVision, MMMU, and MMStar, with average gains of 2-5 percentage points over the base models.","pith_inferences":["Inference: the reward-format taxonomy is probably portable to video and multi-image inputs, since the four reward types depend on output form (choice, number, box, text) rather than on the visual modality; the paper only tests still images.","Inference: BMAS may owe its stability to using the policy model's own embedding space, so a testable prediction is that freezing the reward embeddings from an earlier checkpoint or swapping in a different tokenizer will reduce its advantage.","Inference: the all-same-reward filter could be selecting for examples where the GRPO advantage is nonzero rather than for examples that generalize; a direct check is comparing Mixed-45K against a random 45K sample from the same sources."],"forward_implications":["One GRPO run with mixed rewards can lift several distinct capabilities at once, so task-specific post-training pipelines are not required for the five data types covered.","Open-ended caption and VQA data can be rewarded from the model's own tokenizer embeddings, removing the extra model cost of LLM-as-judge rewards.","The ablation that adds rewards in easy-to-hard order shows each added data family contributes, with open-ended data delivering the final and largest jump.","Dataset scale has a sweet spot: 20K underperforms, 90K improves MathVision and MMStar but hurts MathVista, so 45K is the reported best balance of gain and cost.","Format rewards for <think>/<answer> can be combined with content rewards at weight 0.5 without collapsing accuracy."],"supporting_citations":[{"why":"Supplies the GRPO objective and the group-relative advantage formula that the mixed rewards plug into.","marker":"[55]"},{"why":"DeepSeek-R1 provides the R1-style rule-based RL pipeline and the <think>/<answer> reasoning format that Mixed-R1 adopts.","marker":"[20]"},{"why":"Qwen2.5-VL is the main base model, the generator used in the data filter, and the primary evaluation baseline.","marker":"[3]"},{"why":"InternVL2.5 is the second model family used to test whether the mixed reward design transfers across architectures.","marker":"[10]"},{"why":"VideoChat-R1 shows the LLM-as-judge reward for open-ended responses that BMAS is designed to replace.","marker":"[30]"},{"why":"R1-V represents the task-specific counting RL work whose narrow scope Mixed-R1 aims to generalize.","marker":"[8]"},{"why":"VLM-R1 represents the task-specific grounding RL work whose narrow scope Mixed-R1 aims to generalize.","marker":"[56]"}],"fun_headline_variants":["Rethinking R1: Mixed rewards for multimodal reasoning","BMAS: A stable reward for long-form AI answers","Mixed-R1: One RL run, many tasks, +2-5 points","Why reward design is the real lever for reasoning","New reward mix boosts multimodal reasoning by 2-5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The filter's rule that a question is worth keeping only if its eight sampled responses earn different rewards is the load-bearing premise; if that heuristic selects examples for training stability rather than for generalization, the Mixed-45K dataset may not be the cause of the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Rethinking R1: Mixed rewards for multimodal reasoning","BMAS: A stable reward for long-form AI answers","Mixed-R1: One RL run, many tasks, +2-5 points","Why reward design is the real lever for reasoning","New reward mix boosts multimodal reasoning by 2-5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1462,"prompt_tokens":1060,"completion_tokens":402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":676,"tokens_out":402,"duration_ms":4247,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:31:47.996991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same GRPO and Mixed-Reward setup on a random 45K sample drawn from the same unfiltered source pools; if the random sample matches Mixed-45K's gains on MathVista, MathVision, MMMU, and MMStar, then the all-same-reward filtering rule is not doing the work the paper assigns to it.","supporting_citations":[{"cited_title":"R1-v: Reinforcing super generalization ability in vision-language models with less than $3.https://github.com/Deep-Agent/R1-V, 2025","cited_arxiv_id":null,"evidence_quote":"R1-V represents the task-specific counting RL work whose narrow scope Mixed-R1 aims to generalize."}],"review_version":1}