{"id":"dab0a749-edf3-4e06-a788-2fc8ff211c68","arxiv_id":"2506.14907","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PeRL applies reinforcement learning to a vision-language model with image-order permutation and difficulty-based data filtering, improving multi-image reasoning while keeping single-image performance.","lead":"PeRL trains a vision-language model with reinforcement learning to handle questions that mention several images at once. It improves performance by randomly swapping image order during training and concentrating on hard examples, beating other open 7B models on multi-image benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing risk is the unvalidated GPT-4o semantic-equivalence labels and the unspecified Lambda(y,sigma) used to modify answers after image permutation; if these targets are wrong, PeRL's advantages are computed against corrupted rewards.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: Section 3.3 and Eq. (1) assume GPT-4o can reliably decide whether a permuted image order changes the answer and can produce the correct transformed answer Lambda(y,sigma), while Algorithm 1 applies random swaps and 'modify answers correspondingly' without specifying Lambda or matching offline labels. If these labels are wrong, the advantage estimates in Eqs. (4)-(6) are computed against corrupted answers, and the policy is trained toward incorrect targets. My independent read of the manuscript converges on this as the central risk because it sits directly between the proposed mechanism (permutation-enhanced GRPO) and the reported gains. The concern is not speculative: the paper itself leaves the online answer-modification step underspecified, and no validation of the GPT-4o judgments is reported. I also considered alternative concerns. The SOTA claim is overstated relative to GPT-4o on BLINK, MMIU, Remi, and MV-MATH in Table 1, but the paper's comparisons among open 7B models are internally consistent, so this is a framing issue rather than a correctness risk. The missing hyperparameters (alpha_0, the decay schedule eta, and the exact difficulty filter rule) are reproducibility problems but do not by themselves invalidate the central claim. The shared-baseline advantage computation across permuted groups is nonstandard but defensible because all groups derive from the same underlying example. The strongest independent support is the ablation study (ns=0 vs ns=1 vs ns=2) and the disentanglement ablation in Table 3, which show consistent improvements tied to the proposed components; if the label audit passes, these results substantially support the central claim. Because the identified concern is the same one the reader flagged, and because the reader already issued a CONDITIONAL verdict, I recommend no change to the verdict. A targeted label audit would convert this conditional acceptance into a firmer one.","tokens_in":16182,"tokens_out":4208,"duration_ms":43827,"concrete_test":"Audit the permutation pipeline on a random sample of the final training set. Take 200 instances, run Algorithm 1's permutation module exactly as implemented to produce (xi, yi), and independently obtain the correct modified answer using human annotators or a second strong multimodal model without seeing GPT-4o outputs. Compute (i) agreement on the should_change decision S and (ii) exact-match accuracy of the training-time yi against the independent target, broken down by multiple-choice versus open-ended questions. If the mismatch rate exceeds roughly 2-5%, the advantages in Eqs. (4)-(6) are corrupted for a non-negligible subset of the 22K examples and the reported gains require re-training with corrected labels; if the mismatch rate is near zero, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PeRL's central claim depends on Eq. (1) and Algorithm 1 producing a correct answer yi for every permuted input xi. Section 3.3 obtains S(x, xhat) by prompting GPT-4o, and Eq. (1) defines yhat = S*y + (1-S)*Lambda(y,sigma), but Lambda is never concretely specified beyond 'such as reindexing visual choices.' Algorithm 1's online step says only 'Generate randomly swapped images and modify answers correspondingly,' with no rule for how Lambda is computed at training time and no check that it matches the offline GPT-4o-derived labels. If GPT-4o misclassifies should_change, or if Lambda produces a wrong reindexed or rewritten answer for open-ended or free-form questions, the rule-based reward in Eqs. (4)-(6) will mark correct responses as incorrect and vice versa for a meaningful fraction of the 22K multi-image training examples. The paper provides no label-quality analysis, no human validation, and no consistency check between the offline semantic-variation labels and the online permutation implementation. Since the headline gains are attributed to permutation-enhanced GRPO, corrupted targets would make the policy confidently wrong and could explain part of the reported behavior as an artifact of the data pipeline rather than improved reasoning. This concern is concrete and falsifiable, and it is the single most load-bearing assumption in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PeRL, a GRPO-based reinforcement learning method for interleaved multi-image vision-language reasoning. The method augments each training sample by permuting the order of input images and correspondingly modifying the expected answer, then pools rollouts from the original and permuted variants to compute advantages. A preprocessing pipeline filters Mantis-Instruct by question type, rephrases open-ended questions with GPT-4o, performs rollout-based difficulty filtering using Qwen2.5-VL-7B, and labels semantic variation under image permutation with GPT-4o. The resulting model, initialized from Qwen2.5-VL-7B-Instruct and trained on 22K multi-image and 36K single-image examples, is evaluated on five multi-image and three single-image benchmarks. The paper reports state-of-the-art results on Mantis-Eval, BLINK, and MMIU among compared 7B models, competitive single-image math performance, and ablations on permutation count, data mix, and rollout diversity.","tokens_in":16394,"tokens_out":5548,"duration_ms":53532,"significance":"If the empirical results are reproducible, PeRL is a simple and data-efficient enhancement: it extends R1-style rule-based RL to multi-image positional reasoning without supervised fine-tuning, and the out-of-domain gains on Remi and MV-MATH are encouraging. The paper's strengths include broad evaluation across eight benchmarks, a disentanglement ablation separating single-image and multi-image data (Table 3), a rollout-diversity analysis, and a public code link. However, the central data-generation mechanism -- the semantic-equivalence labeling and the answer transformation Lambda -- is under-specified and unvalidated, and several training hyperparameters needed to reproduce the pipeline are missing. The correctness-risk is concentrated in this mechanism; if the labels or transformed answers are wrong for a meaningful fraction of the 22K multi-image examples, the GRPO rewards in Eqs. (4)-(6) are computed against corrupted targets, which would undermine the reported gains.","major_comments":[{"comment":"The answer transformation Lambda(y, sigma) is never concretely defined. Eq. (1) defines yhat = S(x, xhat)*y + (1 - S(x, xhat))*Lambda(y, sigma), but the text only says that Lambda adapts y 'such as reindexing visual choices,' and Algorithm 1's online generation step says only 'Generate randomly swapped images and modify answers correspondingly.' This is load-bearing because every rule-based reward in Eqs. (4)-(6) is computed against yhat: if GPT-4o's semantic-equivalence judgment S is wrong, or if Lambda produces an incorrect reindexed or rewritten answer for open-ended or free-form questions, then the policy is trained against corrupted targets for a meaningful fraction of the 22K multi-image examples. The paper provides no human validation of S, no consistency check between the offline GPT-4o semantic-variation labels and the online permutation implementation, and no per-question-type specification of Lambda. Please specify Lambda exactly for each question type in Mantis-Instruct, report its coverage and failure modes, and provide an agreement study (human or rule-based) for both S and Lambda.","section":"Section 3.3, Eq. (1); Algorithm 1"},{"comment":"The difficulty-filtering pipeline is not reproducible. The text reports that questions are filtered based on average rollout accuracy, with the mean score falling from 0.78 to 0.38, but it never gives the filtering threshold, the resampling target, or the number of examples retained after each preprocessing stage. In addition, the initial permutation probability alpha_0 in Algorithm 1 is never reported, even though Section 4.2 states that the swap probability decays linearly over training. Without these values, the 22K multi-image training subset cannot be reconstructed, and the paper's data-efficiency claim cannot be checked. Please report alpha_0, the decay schedule, the exact difficulty threshold, the final dataset sizes after each stage, and the number of examples removed by rule-based filtering, format rephrasing, rollout filtering, and semantic-variation checking.","section":"Section 3.3, Fig. 3b; Section 5"},{"comment":"The permutation-intensity ablation and the training-curve comparisons appear to be single runs, and several differences are small relative to expected evaluation noise. For example, ns=1 versus ns=2 changes Mantis-Eval from 76.39 to 77.78, BLINK from 58.53 to 59.50, MathVista from 73.0 to 72.2, and Remi from 41.65 to 39.04; Mantis-Eval has only 217 test samples (Table 4). The claim that 'increasing ns from 0 to 2 improves performance on most benchmarks' and the associated trade-off conclusion need repeated seeds with standard deviations or significance tests. Without these, the recommendation of ns=2 and the interpretation of the permutation gain over naive GRPO are not statistically supported.","section":"Table 2 and Fig. 5"}],"minor_comments":[{"comment":"The heading 'Formuation' in Section 3 should be 'Formulation', and the heading 'Data Prepocessing' in Section 3.3 should be 'Data Preprocessing'.","section":"Section 3 and Section 3.3"},{"comment":"The main text states a batch size of 256 in Section 5, while Appendix A.1 states a batch size of 128; please reconcile this inconsistency.","section":"Section 5 vs. Appendix A.1"},{"comment":"The role of alpha_t is ambiguous: if alpha_t is the probability that a swap is applied, then some of the ns+1 samples may be identical to the original input; please define P((x1, y1), alpha_t) explicitly and state what happens when no swap is sampled for a given i.","section":"Algorithm 1"},{"comment":"The reference list contains malformed entries: [6] includes 'less than $3' in the title and an incomplete URL, and [1] lacks a complete author list and standard bibliographic information.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is a practical RL recipe rather than a conceptual breakthrough, and it overlaps closely with NoisyRollout and VL-Rethinker. That overlap is not itself disqualifying, but the missing operational details of the permutation-answer transformation and the difficulty filter are exactly what an editor should require before accepting the reported gains. The authors should be asked for the full preprocessing specification and a validation study of the GPT-4o labels; without those, the method's central mechanism is not assessable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: PeRL is a genuinely useful RL recipe for multi-image VLMs—permuting image order as a training-time augmentation, with text and answers adjusted to stay consistent, plus a difficulty-filtered data pipeline. On the evidence here, it works: the gains over naive GRPO and other 7B baselines are consistent (+3 to +7 points across multi-image benchmarks), the ablations show the permutation and data-mix components matter, and the OOD improvements on Remi and MV-MATH are a good sign. This is the first paper I've seen that directly targets positional coreference with RL rather than test-time bias mitigation or pure noise augmentation.\n\nThe main soft spot is not the core idea but the missing details around the target labels. Eq. (1) defines the answer after permutation as a mix of the original answer and a transformation Lambda(y,sigma), where Lambda is only described as 'such as reindexing visual choices.' Algorithm 1 says 'modify answers correspondingly' without specifying the rule. That's load-bearing because the advantage estimates in Eqs. (4)-(6) are computed against these targets. If GPT-4o mislabels whether the answer should change, or if Lambda produces a wrong reindexed or rewritten answer for open-ended questions, a meaningful fraction of the 22K multi-image examples get corrupted rewards, and the policy learns to be confidently wrong. The paper provides no validation of GPT-4o's semantic-equivalence judgments, no human agreement, and no consistency check between the offline labels and the online permutation implementation. This is a concrete, falsifiable risk, and it should be addressed.\n\nAdditional smaller issues: alpha_0 (the initial swapping probability) and the difficulty-filter threshold/resampling target are never reported, so reproducibility suffers. And the abstract's 'state-of-the-art on multi-image benchmarks' overreaches: GPT-4o in the same table beats PeRL on four of the five multi-image benchmarks. The correct claim is SOTA among open 7B models.\n\nThe paper is otherwise coherent and the empirical evidence is broad. It deserves a serious referee, but I'd ask for a revision that reports the missing hyperparameters, specifies Lambda, and adds a GPT-4o label-quality analysis (e.g., human sample, consistency with the online oracle). If those hold up, this becomes a solid contribution.","headline":"PeRL is a useful, believable RL recipe for multi-image VLMs, but the authors must pin down the permutation answer-modification rule and validate GPT-4o labels before I'd trust the headliner gains.","tokens_in":17051,"tokens_out":2771,"would_cite":true,"duration_ms":24683,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PeRL shows that permuting image order during RL training makes a 7B VLM reason reliably across multiple images and beat prior baselines.","keywords":["reinforcement learning","vision-language models","multi-image reasoning","positional bias","GRPO","permutation augmentation","rollout filtering","interleaved multimodal reasoning"],"falsifier":"Take 200 questions from the 22K training set, relabel semantic equivalence with human annotators, and compare the GPT-4o labels for both the change decision and the transformed answer; if disagreement is more than a small fraction, the advantage estimates are computed against wrong answers. Alternatively, run the final model on every permutation of a held-out set of 100 multi-image questions and check whether accuracy and the predicted option stay invariant.","tokens_in":15906,"feed_emoji":"🖼️","tokens_out":5597,"duration_ms":50786,"temperature":0.7,"pith_summary":"The paper aims to make vision-language models reason reliably across multiple interleaved images, not just within a single image. It claims that a reinforcement-learning recipe called PeRL, which randomly permutes the order of input images, rewrites the question when the image order changes the meaning, and filters rollouts to concentrate on informative samples, mitigates the positional bias that makes multi-image models flip answers when image order changes. If this is right, a model trained this way scores 76.39 on Mantis-Eval, 58.53 on BLINK, and 54.23 on MMIU, beating existing 7B instruction-tuned and R1-style baselines, while staying competitive on single-image math benchmarks. The central message is that multi-image reasoning can be improved with a small RL-only dataset plus an order-diversifying exploration mechanism.","feed_headline":"Permutation RL lifts multi-image reasoning across benchmarks","feed_subtitle":"Qwen-PeRL tops Mantis-Eval, BLINK, and MMIU among 7B models while keeping single-image math scores.","key_machinery":"The load-bearing object is the permutation-augmented GRPO group: for each training sample, the pipeline creates $n_s = 1$ randomly swapped image order, updates the answer to match via $\\Lambda(y,\\sigma)$, and pools rollouts from all orders into one baseline so the policy receives gradient signal from the difference between orders. The second mechanism is rollout filtering: questions are scored by average accuracy over 10 policy rollouts, and only low-scoring questions are kept, shifting the difficulty mean from 0.78 to 0.38. The third is the semantic-variation prompt that produces the indicator $S$ and the answer-transformation function $\\Lambda$, introduced in Equation (1) to prevent augmented samples from teaching stale answers after a permutation changes the meaning.","core_discovery":"PeRL establishes that permutation-enhanced GRPO is an effective way to train interleaved vision-language models to reason across images. Starting from a Qwen2.5-VL-7B-Instruct policy and only 22K multi-image examples plus 36K single-image examples, the method computes semantic-variation labels with GPT-4o, then in each training step generates one randomly permuted copy of each sample and merges all rollouts from all orders into a single advantage baseline. The shared baseline makes permuted and unpermuted trajectories compete, which spreads the advantage distribution and prevents the near-zero advantages that naive GRPO produces on multi-image inputs. The paper reports consistent gains across five multi-image benchmarks and no degradation on three single-image math benchmarks.","pith_inferences":["If the GPT-4o semantic-equivalence labels are noisy on a nontrivial fraction of the 22K samples, the advantage estimator normalizes against wrong answers; auditing a few hundred of these labels against human judgment is the cheapest check of the whole pipeline.","The permutation operation enforces invariance under the symmetric group on image positions, making PeRL a group-invariance data augmentation for order-sensitive inputs; the same idea should transfer to video frame order or multi-view reasoning.","The observed advantage-spread effect suggests a testable hypothesis: any augmentation that creates input diversity within a GRPO group may improve learning efficiency, so image corruptions, crops, or question rephrasing could substitute when order is semantically fixed."],"forward_implications":["Multi-image positional reasoning can be improved with RL alone on a small curated subset of roughly 22K examples, without large-scale supervised multi-image instruction tuning.","The permutation trick generalizes across benchmark families: gains appear on natural-scene tasks (Mantis-Eval, BLINK, MMIU) and on out-of-domain multi-image math benchmarks (Remi, MV-MATH).","Permuting more than once per sample ($n_s = 2$) helps spatially oriented tasks like BLINK and Mantis-Eval but slightly hurts reasoning-centric benchmarks, so permutation intensity is a tuning knob rather than a free lunch.","Because the method is a plug-in to GRPO, the same recipe could be transferred to other VLM backbones and other interleaved multimodal settings."],"supporting_citations":[{"why":"Supplies the group-relative policy optimization framework that PeRL modifies with permutations and a shared baseline.","marker":"[30]"},{"why":"Provides the 721K Mantis-Instruct multi-image instruction corpus from which the 22K training set is curated, plus the Mantis-Eval benchmark.","marker":"[15]"},{"why":"Defines the Qwen2.5-VL model used for difficulty scoring, rollout filtering, and as the policy initialization.","marker":"[5]"},{"why":"The closest RL-only baseline, NoisyRollout, which PeRL contrasts with on how policy updates condition on augmented inputs.","marker":"[23]"},{"why":"Documents the position-bias problem in multi-image VLMs that motivates the permutation mechanism.","marker":"[36]"},{"why":"The RL-only single-image math baseline MM-Eureka that PeRL must match on MathVista and MathVerse.","marker":"[26]"},{"why":"R1-style step-wise GRPO baseline that PeRL compares against on multi-image and single-image benchmarks.","marker":"[46]"},{"why":"R1-OneVision, an R1-style multi-image reasoning baseline that PeRL reports outperforming.","marker":"[43]"},{"why":"GPT-4o is the model that produces the semantic equivalence indicator and answer-transformation labels in data preprocessing.","marker":"[1]"}],"fun_headline_variants":["Permutation RL lifts multi-image VLM reasoning","PeRL: RL with image permutations for interleaved tasks","Shuffling images boosts multimodal RL","Permutation-enhanced RL advances multi-image reasoning","PeRL improves multi-image reasoning without losing single-image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on GPT-4o's semantic-equivalence judgments being right: if it cannot reliably decide whether permuting the image order changes the intended answer, or cannot rewrite the answer correctly, then the permutation-augmented rewards teach the policy to be confidently wrong.","fun_headline_variants_meta":{"raw":{"variants":["Permutation RL lifts multi-image VLM reasoning","PeRL: RL with image permutations for interleaved tasks","Shuffling images boosts multimodal RL","Permutation-enhanced RL advances multi-image reasoning","PeRL improves multi-image reasoning without losing single-image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1749,"prompt_tokens":926,"completion_tokens":823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":751}},"tokens_in":542,"tokens_out":823,"duration_ms":10940,"temperature":1.0,"reasoning_tokens":751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:09:39.687867+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 200 questions from the 22K training set, relabel semantic equivalence with human annotators, and compare the GPT-4o labels for both the change decision and the transformed answer; if disagreement is more than a small fraction, the advantage estimates are computed against wrong answers. Alternatively, run the final model on every permutation of a held-out set of 100 multi-image questions and check whether accuracy and the predicted option stay invariant.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4o is the model that produces the semantic equivalence indicator and answer-transformation labels in data preprocessing."}],"review_version":1}