{"id":"54e37364-dfd7-4097-ae59-29d5edb77b18","arxiv_id":"2505.24871","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Multi-domain RLVR data mixing, guided by a quadratic surrogate fitted to 11 pilot runs, improves a Qwen2-VL-2B model's out-of-distribution accuracy by about 5 points over uniform mixing.","lead":"A team tested how to mix five visual-question datasets when fine-tuning a 2B vision-language model with reinforcement learning. They fit a simple mathematical model to a few pilot runs and found mixtures that beat both single-dataset and uniform training on new benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 5.24% OOD gain is selected on the same four benchmarks used for evaluation, so the headline improvement is not an unbiased estimate of out-of-distribution generalization.","rationale":"The reader's weakest assumption is essentially this selection-bias concern, and I agree it is load-bearing. The paper's central claim is that optimizing the RLVR data mixture improves accuracy on out-of-distribution benchmarks by 5.24% over uniform. For that claim to be true, the mixture chosen by the surrogate must generalize to benchmarks not used in the selection objective. Currently, the four benchmarks are used both to fit bLtest and to compute the reported gain, so the optimization procedure has direct access to the evaluation metric. This is the classic selection-on-the-test-set problem: even a purely noisy surrogate will produce a selected mixture whose evaluation-score is biased upward, and the bias grows with the number of candidate mixtures explored. The paper's own Figure 6 shows the model-based strategy has lower variance, but that does not address the bias. The heuristic Norm result (0.5133) being the best row in Table 2 further weakens the attribution of the gain to the model-based pipeline, though it does not by itself falsify the claim that mixtures matter. The proposed leave-one-benchmark-out test is decisive: if the optimization genuinely finds mixtures that transfer to unseen benchmarks, the selected mixture should win on the held-out benchmark in most folds; if not, the headline number is an artifact of selection. I would keep the reader's CONDITIONAL verdict: the phenomenon of mixture sensitivity is empirically grounded, but the quantitative OOD claim needs this validation before it can be accepted. The concern does not warrant rejection because the test is feasible and the authors may already have the compute to run it.","tokens_in":20297,"tokens_out":5132,"duration_ms":51276,"concrete_test":"Leave-one-benchmark-out evaluation. For each of the four OOD benchmarks in turn, exclude it from the Out-Score used to fit the surrogate and to select the mixture (e.g., fit bLtest on ChartQA+InfoVQA+MathVista, select w* via Algorithm 4, run RLVR, evaluate only on MMMU). Repeat for the other three benchmarks. Compare each selected mixture's held-out benchmark score against the uniform All mixture on the same benchmark. If the model-based mixture does not beat uniform in at least three of the four folds, the 5.24% headline gain is attributable to selection bias rather than to genuine OOD generalization. Report per-benchmark scores and the number of folds in which the selected mixture wins.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on Out-Score, the weighted average over ChartQA, InfoVQA, MathVista, and MMMU (Table 1). The quadratic surrogate bLtest(w) in Eq. (3) is fit by ERM to these Out-Scores from seed and heuristic runs; Algorithm 4 (QuadSurface) then selects the top-k mixtures by predicted Out-Score on those same four benchmarks. Consequently, the four 'out-of-distribution' benchmarks are not held out from the mixture-selection procedure: they are the objective used to choose the mixture. Any noise or idiosyncratic behavior in the four benchmarks is exploited by the argmax, inflating the measured improvement over the uniform All mixture. The paper reports no benchmark-level holdout, no repeated-seed variance, and no evaluation on a benchmark not used in fitting. Table 2 also shows that the best Out-Score in the entire table belongs to the heuristic Norm row (0.5133 at 2000 steps), not to any model-based run (max 0.5104 for run 010), so the abstract's 'best mixture' is not the output of the proposed method unless one arbitrarily includes heuristics. If the selection loop is biased, the claimed 5.24% advantage over uniform could shrink or disappear when evaluated on a truly held-out benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoDoMoDo, a framework for optimizing data mixture weights in multimodal RLVR post-training. The authors curate five verifiable image-text training datasets (COCO, LISA, GeoQA V, SAT, ScienceQA), train Qwen2-VL-2B-Instruct with GRPO under different mixture recipes, and introduce a model-based strategy that fits a quadratic response surface to mixture-to-performance data, samples candidate mixtures, and selects the top candidates. The claimed headline result is that the best found mixture improves out-of-distribution accuracy by 5.24% over uniform mixing. The paper also reports heuristic mixture baselines and analyzes the effect of single, exclude-one, and all-dataset mixtures.","tokens_in":20577,"tokens_out":5268,"duration_ms":63163,"significance":"The qualitative finding that mixture composition affects OOD performance in multimodal RLVR is useful and is supported by Table 2: different seed and heuristic mixtures produce markedly different Out-Scores, and the authors provide a reasonably comprehensive set of training and evaluation benchmarks. The dataset curation and the formal mixture-optimization framing are potentially valuable for future work on multimodal RL post-training. However, the model-based contribution is not validated by the reported experiments. The evaluation protocol uses the same four OOD benchmarks both to fit the surrogate and to select and report the final mixture, and the experimental scale (11 pilot runs for a 21-parameter surrogate, single seed, no held-out benchmark) is too limited to support the claimed quantitative advantage of the proposed method. If a clean held-out evaluation and repeated-seed analysis were provided, the empirical contribution could be substantial; in its current form the central quantitative claim is not supported.","major_comments":[{"comment":"The four 'Out' benchmarks (ChartQA, InfoVQA, MathVista, MMMU) are used to compute Out-Score, which is the target variable y in Eq. (3) and in Algorithm 4. The same Out-Score is then used to rank candidate mixtures and to report the final winner in Table 2. Consequently, the abstract's 5.24% improvement is an in-sample maximum of a response surface fitted to the evaluation benchmarks, not an unbiased estimate of out-of-distribution generalization. A benchmark held out from the mixture-fitting and mixture-selection procedure is needed before the headline claim can be assessed.","section":"§3.1, Table 1; Algorithm 4"},{"comment":"The quadratic surrogate has 21 free parameters (a, b, and symmetric C for m=5), while the seed set described in §3.2 consists of 11 runs (5 Single, 5 Exclude-One, All). Even if the heuristic runs are added to the fitting set, fitting 21 parameters to this many points and then selecting the best of 5 random CV splits (Figure 7) does not provide evidence of generalization. The held-out R² in Figure 7 is computed on random splits of the same points and cannot validate extrapolation to the simplex region from which the GMM samples candidate mixtures; the claim that linear models fail while quadratic models generalize is therefore not established.","section":"§2.2, Eq. (3); Algorithm 4; Figure 7"},{"comment":"The best Out-Score in Table 2 is achieved by the heuristic Norm row (0.5133 at 2000 steps), not by any model-based run (maximum 0.5104 for run 010). The abstract's 'best mixture' therefore appears to refer to the leave-one-out heuristic rather than to the surrogate-based method, unless the authors intend to claim the best across all strategies, in which case the model-based optimizer is not the source of the headline gain. This ambiguity needs to be resolved and the reported 5.24% needs to be attributed to the correct method.","section":"Table 2; Abstract"},{"comment":"The comparison reports single runs without repeated-seed variance; Section 3.1 states that training uses deterministic seed 42. The claim that the model-based strategy 'reduces variance' refers to variance across mixtures within one seed, not across training stochasticity. The Out-Score differences among the top model-based runs (e.g., 0.5104, 0.5067, 0.5048) are small and no confidence intervals or significance tests are provided, so it is unclear whether the observed ranking is meaningful.","section":"§3.3, Figure 6; Table 2"}],"minor_comments":[{"comment":"The 5.24% and 20.74% figures appear to be absolute percentage-point differences (e.g., 0.5133 - 0.4609 = 0.0524), not relative improvements; the paper should state this explicitly to avoid misinterpretation.","section":"Abstract and Table 2"},{"comment":"Section 3.2 describes the model-based strategy as using 'Covariance Matrix Adaptation Evolution Strategy (CMA-ES)', but Algorithm 4 fits a quadratic surface by ordinary least squares and samples candidates from a GMM; the relationship between CMA-ES and the implemented algorithm is not explained.","section":"§3.2, Algorithm 4"},{"comment":"The labels 'Step-averaged' and '2000-step' are not defined in the main text; clarify whether step-averaged scores average over multiple checkpoints and over which steps.","section":"Table 2"},{"comment":"The mixture weight vectors for the model-based runs 001-010 are not reported, so the experiments cannot be reproduced; please include the selected mixtures in a supplementary table.","section":"Table 2, Model-based rows"},{"comment":"The label '1@5 Splits' is ambiguous; it appears to mean the best of five random train-test splits, which is a form of selection and should be described as such in the text.","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical setup and a plausible qualitative message, but the evaluation protocol for the model-based method is circular and the reported headline number is not attributable to the proposed method. A proper revision would need to add held-out benchmarks not used in mixture fitting, repeated seeds, and honest reporting of the selection procedure; otherwise the central claim should be substantially weakened. I would not recommend rejection if the authors can provide such evidence, but the current version is not ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is showing that data mixture composition materially changes RLVR post-training outcomes for a 2B multimodal model across five task domains. The dataset curation and the head-to-head comparison of single, uniform, heuristic, and model-based mixtures are useful and mostly reproducible in spirit. The qualitative finding that the uniform \"All\" mixture is not optimal, and that simple heuristics like Norm can beat it, is well supported by Table 2.\n\nThe soft spots are real. The quadratic surrogate has 21 parameters fitted to 11 pilot runs (plus a few heuristic runs, but still small). Selecting the best of 5 random splits in Figure 7 inflates apparent fit. More importantly, the four \"out-of-distribution\" benchmarks—ChartQA, InfoVQA, MathVista, MMMU—are exactly the benchmarks used to compute Out-Score, which is the target of the surrogate fitting and the argmax in Algorithm 4. So the reported 5.24% improvement over uniform is not an unbiased estimate on truly held-out data; it is an in-sample maximum of a fitted surface. The paper also has an inconsistency: the best Out-Score in Table 2 belongs to the heuristic Norm strategy (0.5133), not to any model-based run (max 0.5104 for run 010), yet the abstract calls the best mixture \"our best mixture\" without clarifying that it may be heuristic. That matters because the claim is about the model-based method.\n\nThe math is simple and appropriate for the problem; the citation pattern covers DoReMi, RegMix, data mixing laws, and single-domain MLLM RLVR, so novelty is incremental but real. The paper would benefit from code/data release, repeated-seed error bars, and evaluation on at least one benchmark not used in fitting. If those are added, the practical claim of a 5-point OOD gain could be credible. As it stands, the qualitative story is solid and the quantitative claim is plausible but unverified.\n\nI would send it to peer review rather than desk reject, because the problem is important and the empirical comparison, though flawed, is useful to the community. A competent referee could push for the right experiments. It is not a strong accept in current form.","headline":"Useful qualitative result on mixture effects in multimodal RLVR, but the headline gain is selected on the same benchmarks used to fit the surrogate, so the quantitative claim is provisional.","tokens_in":21135,"tokens_out":1926,"would_cite":false,"duration_ms":19815,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a quadratic surface over data-mixture weights predicts multimodal RLVR out-of-distribution accuracy well enough that the optimized mixture beats uniform mixing by 5.24% and the pre-finetuning base model by 20.74%.","keywords":["multimodal large language models","reinforcement learning with verifiable rewards","data mixture optimization","GRPO","quadratic response surface","out-of-distribution generalization","vision-language reasoning","online RL post-training"],"falsifier":"Reserve at least one of the four out-of-distribution benchmarks entirely from the fitting loop, select a mixture without seeing it, and post-train the same base model on that mixture. If the chosen mixture does not beat uniform mixing on the reserved benchmark, the generalization claim collapses.","tokens_in":20047,"feed_emoji":"🧠","tokens_out":12643,"duration_ms":128496,"temperature":0.7,"pith_summary":"This paper tries to establish that how you mix multiple verifiable-reward vision-language datasets during RLVR post-training is a learnable design choice, not a detail. It curates five image-text datasets with different rule-based rewards, post-trains a 2B vision-language model with GRPO on weighted mixtures, and fits a quadratic surface to the out-of-distribution accuracy of a few pilot mixtures. It then uses that surface to pick a new mixture and reports that this mixture improves out-of-distribution benchmarks by an average of 5.24% over uniform mixing and by 20.74% over the pre-finetuning baseline. If this is right, data-mixture selection becomes a cheap lever for multimodal reasoning that does not require more data or a bigger model.","feed_headline":"A learned data mix beats uniform mixing in multimodal RL by 5.24%","feed_subtitle":"Five verifiable vision-language tasks are mixed by a learned response surface to lift out-of-distribution reasoning.","key_machinery":"The load-bearing object is the quadratic response surface $\\hat{L}_{\\mathrm{test}}(\\mathbf{w}) = a + \\mathbf{b}^{\\top}\\mathbf{w} + \\mathbf{w}^{\\top} C \\mathbf{w}$, a second-order model of out-of-distribution test accuracy as a function of the mixture weights $\\mathbf{w}$. The linear term captures each dataset's independent contribution, while the quadratic term $\\mathbf{w}^{\\top} C \\mathbf{w}$ captures dataset interactions that linear models miss. Fitted by Eq. (3) over a small set of pilot RLVR runs and validated by five-fold cross-validation, this surface turns an exponentially large mixture search into a one-time optimization: candidate mixtures are sampled around observed mixtures, ranked by the fitted surface, and the top one is used for the final GRPO post-training. The machine also includes the five datasets with their per-domain reward functions (format, accuracy, and IoU) and the GRPO training loop that produces the post-trained checkpoint for each mixture.","core_discovery":"The central discovery is that the post-training outcome $L_{\\mathrm{test}}(\\mathbf{w})$ is a predictable function of the mixture distribution $\\mathbf{w}$, and that a multivariate quadratic model $a + \\mathbf{b}^{\\top}\\mathbf{w} + \\mathbf{w}^{\\top} C \\mathbf{w}$ captures the interaction among diverse datasets well enough to rank unseen mixtures. Using a small set of pilot RLVR runs on deliberately chosen seed mixtures, fitting the quadratic by least squares, and then sampling candidate mixtures around the observed ones, the framework selects a data mixture whose post-trained model outperforms a uniformly mixed model on four out-of-distribution benchmarks on average and exceeds the pre-finetuning baseline by a total of 20.74%. The paper presents this as the first systematic treatment of dataset mixtures for multimodal RLVR post-training.","pith_inferences":["Iterating the loop is an untested extension: after a selected mixture is run, add it to the pilot set, refit the quadratic surface, and select again; if the surrogate is accurate near the optimum, the gains should compound until diminishing returns.","The quadratic surrogate is modality-agnostic, so the same recipe could be transferred to text-only RLVR or to video and audio verifiable domains; this extrapolates beyond the paper's image-text experiments.","A stricter validation would reserve one entire out-of-distribution benchmark for final evaluation only, since the paper uses the same four benchmarks both to fit and to judge the surrogate; that would reveal how much of the 5.24% gain is genuinely out-of-sample."],"forward_implications":["With a small set of pilot RLVR runs, a quadratic mixture-to-performance model can select a data mixture that beats uniform mixing on out-of-distribution benchmarks; in these experiments the average gain is 5.24%.","Multi-domain RLVR post-training on verifiable vision-language tasks improves both in-domain scores (LISA, SAT, ScienceQA) and out-of-domain benchmarks (ChartQA, InfoVQA, MathVista, MMMU) over the pre-finetuning base model.","Dataset interactions are nonlinear, so independent-weighting heuristics leave performance on the table; a quadratic model with cross terms captures these interactions better than linear regression.","The strategy is efficient: full search over the mixture simplex is replaced by a few pilot runs plus one optimized post-training run.","Any collection of datasets with verifiable rewards can be plugged into the same mixture-selection procedure, making mixture choice a reusable post-training component."],"supporting_citations":[{"why":"Defines GRPO, the verifiable-reward RL algorithm used for every post-training run in the paper.","marker":"[54]"},{"why":"Provides the model-based idea that a predictive model of an outcome can be learned from the training data distribution, which the quadratic surface instantiates.","marker":"[20]"},{"why":"Shows single-domain visual RLVR with verifiable rewards, the starting point this work extends to multiple mixed domains.","marker":"[35]"},{"why":"The object-recognition dataset with IoU reward, one of the five training domains in the mixture.","marker":"[29]"},{"why":"The referring-expression grounding dataset with IoU reward, one of the five training domains.","marker":"[23]"},{"why":"The math-VQA dataset with multiple-choice accuracy reward, one of the five training domains.","marker":"[26]"},{"why":"The spatial-VQA dataset with natural-language accuracy reward, one of the five training domains.","marker":"[52]"},{"why":"The science-VQA dataset with multiple-choice accuracy reward, one of the five training domains.","marker":"[37]"},{"why":"The base vision-language model whose parameters all RLVR runs start from and post-train.","marker":"[61]"}],"fun_headline_variants":["Learned data mixes beat uniform for multimodal RL post-training","Predicting RL data mixtures boosts multimodal reasoning","Quadratic mixture model optimizes multimodal RL training data","5.24% OOD gain from learned mixing in multimodal RL","Data mixture prediction improves multimodal LLM post-training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a quadratic surface fitted to a handful of pilot RLVR runs ranks unseen mixtures correctly, and that the same out-of-distribution benchmarks used for fitting can also serve as the final evaluation of the chosen mixture without inflating the measured gain.","fun_headline_variants_meta":{"raw":{"variants":["Learned data mixes beat uniform for multimodal RL post-training","Predicting RL data mixtures boosts multimodal reasoning","Quadratic mixture model optimizes multimodal RL training data","5.24% OOD gain from learned mixing in multimodal RL","Data mixture prediction improves multimodal LLM post-training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000906,"raw_usage":{"total_tokens":3924,"prompt_tokens":1000,"completion_tokens":2924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":2847}},"tokens_in":616,"tokens_out":2924,"duration_ms":24852,"temperature":1.0,"reasoning_tokens":2847,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:13:13.554263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reserve at least one of the four out-of-distribution benchmarks entirely from the fitting loop, select a mixture without seeing it, and post-train the same base model on that mixture. If the chosen mixture does not beat uniform mixing on the reserved benchmark, the generalization claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows single-domain visual RLVR with verifiable rewards, the starting point this work extends to multiple mixed domains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The object-recognition dataset with IoU reward, one of the five training domains in the mixture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The referring-expression grounding dataset with IoU reward, one of the five training domains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The math-VQA dataset with multiple-choice accuracy reward, one of the five training domains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The spatial-VQA dataset with natural-language accuracy reward, one of the five training domains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The science-VQA dataset with multiple-choice accuracy reward, one of the five training domains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The base vision-language model whose parameters all RLVR runs start from and post-train."}],"review_version":1}