{"id":"e3b48369-7e68-4735-9252-b7f4c948df46","arxiv_id":"2608.08802","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PIRL, a trinary reward plus embedding-space adversarial consistency regularizer, reduces prompt-rewording degradation of RLVR-trained multimodal LLMs to about 1% average, versus about 3% for GRPO.","lead":"Reinforcement learning with verifiable rewards makes multimodal AI models more accurate on the exact prompts used in training, but they break when the same question is rephrased. This paper introduces a training method that adds a three-level reward and an adversarial consistency penalty, which keeps accuracy nearly flat under reworded prompts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DTR and the embedding adversary are not isolated in the ablation; the reported gains over multi-template GRPO are within the paper's own noise threshold, so the novel components' load-bearing role is unverified.","rationale":"The reader's weakest assumption correctly identifies the unverified bridge between embedding-space perturbations and literal semantic rewordings. My stress-test pass found a more immediately decisive gap: even granting that bridge, the experiments do not establish that the two novel components are load-bearing. Table 3's ablation is the paper's own instrument for this, and its outcome is that full PIRL and MT+DTR are within noise, with ADV+DTR without MT worse. The paper honestly states this limitation, which is credit to the authors, but it means the central claim 'PIRL, consisting of DTR and the consistency regularizer, reduces degradation' is not separately supported; only the full recipe is. I do not see an internal inconsistency in the theory: Theorem B.1, Proposition B.5, and Theorem B.8 are derived carefully, and Proposition B.11 even notes that invariance does not imply optimality. The empirical tables are plausible and the authors flag the places where their wins are or are not conclusive. A CONDITIONAL verdict remains right: the method may work, but the specific contribution of the proposed mechanism needs either a cleaner factorial ablation or released code to check the missing cells. I would not move to REJECT because the full PIRL recipe does beat GRPO and mostly GRPO-MT, and the theoretical story is coherent; the missing attribution is a reproducibility-and-analysis gap rather than a demonstrated failure.","tokens_in":33590,"tokens_out":8456,"duration_ms":95712,"concrete_test":"Run a complete 2x2x2 ablation (MT, DTR, ADV) with a fifth random-perturbation arm that samples deltas uniformly from the same epsilon-ball without the ascent in Eq. (9), keeping lambda, alpha_rew, seeds, and evaluation identical; report ΔT/ΔD on the seven benchmarks, and include GRPO-MT as the MT-only cell. If MT-only or MT+random is within 1.5pp of full PIRL on Exam and Medical blocks, the novel components are not load-bearing and the central claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.5 (Table 3) is the only place the proposed components are separated, and the paper itself states the ablation does not establish that DTR or the adversary is individually necessary beyond MT. The controlled rows are consistent with that concession: full PIRL has T-Stress ΔT=+0.6±0.5 on Exam VQA Avg, MT+DTR gives 0.0±0.5, and ADV+DTR without MT gives -0.9±0.5; under the paper's own ~1.5pp inconclusive threshold these are not separated. Against GRPO-MT (Table 1), PIRL's mean T-Stress advantage is about +1.0pp on Qwen2.5-VL-7B but only +0.4pp on Qwen3-VL-8B, within the stated noise band for most benchmarks. Meanwhile the mechanism that is supposed to make the adversary valuable, surrogate transfer from the l2 instruction-embedding ball to literal rewordings, is explicitly left open in Appendix A, and the per-token KL penalty is not a certificate of the answer-level C2 objective. The headline therefore rests on the full recipe beating GRPO, not on evidence that DTR or the adversarial consistency regularizer contributes; if multi-template sampling, or even random embedding noise, is the active ingredient, the claimed contribution is not supported by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies why RLVR-trained multimodal LLMs are brittle to prompt paraphrases and template changes, and proposes Prompt-Invariant RLVR (PIRL) as a remedy. The authors identify two causes: binary verifiers entangle format compliance with semantic correctness, and training over a narrow prompt distribution cannot identify behavior on unseen formats. They formalize these with a reward-variance share theorem and a non-identifiability proposition, and derive two design conditions, C1 (reward decomposition) and C2 (invariance control). PIRL instantiates C1 with a Dynamic Trinary Reward (DTR) plus decoupled advantage normalization, and C2 with an embedding-space adversarial prompt sampler plus a per-token KL consistency regularizer. Experiments on exam, medical, and legal VQA benchmarks with Qwen2.5-VL-7B and Qwen3-VL-8B compare PIRL with GRPO and multi-template GRPO-MT under template stress and dynamic evaluation. The main empirical claim is that PIRL has the smallest mean degradation among trained methods, with abstract-level summary that PIRL drops only ≤1% on average while GRPO drops about 3%. The paper explicitly concedes in Section 4.5 that the ablation does not isolate DTR or the adversary, and in Appendix A that the per-token surrogate's effect on answer-level divergence remains open.","tokens_in":33830,"tokens_out":4212,"duration_ms":45712,"significance":"If the aggregate empirical claims hold, PIRL is a useful contribution: it targets a real deployment risk for high-stakes multimodal VQA, and the paper ships a sizable evaluation suite with two backbones and seven benchmarks. The theoretical propositions (Theorem B.1, Propositions B.5, B.10, B.11) are derived from stated definitions and are not fitted to the experiments, and the manuscript is unusually transparent about the limits of its adversarial surrogate and its component ablation. These strengths are genuine. The significance is tempered by two facts: the advantage of full PIRL over GRPO-MT is, on one backbone, within the paper's own noise band, and the specific mechanisms claimed as contributions—DTR and the embedding adversary—are not cleanly isolated by the reported ablation. As a result, the paper presently supports the full recipe as a heuristic combination, but not the attribution of its gains to the proposed components.","major_comments":[{"comment":"The component ablation does not support the claim that DTR or the embedding adversary contributes to the leading-order robustness gain. On Exam VQA Avg, full PIRL has ΔT=+0.6±0.5, MT+DTR has 0.0±0.5, and ADV+DTR without MT has -0.9±0.5; the paper's own inconclusive threshold is ±1.5pp, so the novel components are not separated from multi-template sampling. Since the abstract and introduction present DTR and the adversary as the method's two measures, the central attribution needs evidence: add a MT+ADV (no DTR) row, run paired comparisons across the per-benchmark T-Stress gaps, and ideally include a control with random embedding noise to show the adversary's adversarial selection, not mere perturbation, is the active ingredient.","section":"Section 4.5, Table 3"},{"comment":"The mechanism connecting the per-token KL consistency penalty to the answer-level C2 objective is explicitly left open: the paper states that whether optimizing the per-token surrogate decreases answer-level divergence remains an open empirical question. This is load-bearing because the consistency regularizer is one of the two named components of PIRL, and without evidence that training reduces answer-level divergence between clean and adversarial prompts, the regularizer could be acting as a generic smoothing term. I recommend measuring the answer-level KL or total-variation distance on held-out prompt pairs before and after PIRL training, or otherwise providing a direct test of the surrogate transfer.","section":"Appendix A, Eq. (8)"},{"comment":"The claimed advantage over GRPO-MT is not consistently outside the noise band. Computing mean template-stress gaps from Table 1 gives about -0.1pp for PIRL versus -1.1pp for GRPO-MT on Qwen2.5-VL-7B, but only about -0.5pp versus -0.9pp on Qwen3-VL-8B, a difference of roughly 0.4pp against per-benchmark uncertainties of ±0.4 to ±0.7pp. The abstract's 'GRPO drops ~3%' matches only the Qwen2.5 backbone, not the Qwen3 result (~1.3%). To support the headline claim, the authors should report a statistical comparison across benchmarks and seeds (e.g., a paired test or bootstrap over the seven benchmarks), rather than relying on mean gaps that are partly within the inconclusive threshold.","section":"Table 1, Qwen3-VL-8B"}],"minor_comments":[{"comment":"The text says PIRL restores template-stress robustness to base level; this is accurate for Qwen2.5-VL-7B (mean ΔT ≈ -0.1 vs. base ≈ -0.1) but less so for Qwen3-VL-8B, where PIRL's mean is -0.5 vs. the base's +0.3.","section":"Section 4.6 and Figure 5"},{"comment":"The hyperparameter table does not list the advantage-mixing coefficient α_adv used in DAN, which is needed for exact reproduction; either add it to Table 5 or state explicitly that it equals 1.0.","section":"Table 5, Eq. (7)"},{"comment":"The description 'sample one from n∈{1,...,15} templates' is ambiguous: n could be the number of sampled templates or an index over templates; Figure 3 makes the intended meaning clearer, and the text would benefit from a definition such as 'evaluate with a pool of n templates.'","section":"Section 4.1"},{"comment":"The notation τ̄_φ^{x0}(x0) is hard to parse; define the frozen adversarial prompt with a lighter notation, e.g., x̃0, and use it consistently in the policy update step.","section":"Appendix C, Algorithm 2"},{"comment":"Theorem B.1 is stated under an independence assumption that the main text later relaxes in Appendix B.2; a forward pointer to the relaxation when the formula is first introduced would reduce the risk of misreading the independence assumption as essential to the argument.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent and well-written, and the aggregate template-stress results are suggestive. The key risk for the editors is attribution: Table 3, together with the explicit concession in Section 4.5, means the community cannot currently tell whether the proposed DTR and adversarial consistency components are responsible for the gains, or whether multi-template sampling alone would reproduce them. The Appendix A admission that the per-token surrogate's transfer to answer-level invariance is open further weakens the mechanism story. I do not see this as a rejection issue: the direction is sound, the evaluation is reasonably broad, and the required experiments (MT+ADV row, answer-divergence measurements, paired statistics) are within the manuscript's scope. I would therefore send it for a major revision rather than accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nThe paper's main empirical claim—PIRL degrades less than GRPO under template stress—is supported by the reported tables, and the authors deserve credit for stating plainly that their own ablation does not isolate the new components. What the paper does not establish is that those components, the dynamic trinary reward and the embedding-space adversary, are the ones doing the work.\n\nWhat's genuinely new: framing prompt brittleness as reward entanglement and objective non-identifiability, the C1/C2 design conditions, and the PIRL combination itself. The theoretical propositions (B.1, B.5, B.8) are internally consistent; B.5 is essentially a formalized underdetermination result, modest but fine. The empirical case against plain GRPO is fairly strong: on both backbones, PIRL's template-stress drops are near zero where GRPO drops a few points, and it usually beats GRPO-MT too. The paper is unusually honest—it explicitly says the ablation doesn't show DTR or the adversary is necessary beyond multi-template training, and Appendix A leaves open whether the per-token KL surrogate actually reduces answer-level divergence.\n\nThe soft spots are real but proportionate. Table 3's full PIRL (ΔT=+0.6) and MT+DTR (0.0) are inside the paper's own ±1.5pp inconclusive band, so the novel pieces' load-bearing role is unverified. Against GRPO-MT, the advantages are about 1.0pp on Qwen2.5 and 0.4pp on Qwen3, within noise on several benchmarks. The core mechanism—perturbing instruction embeddings in an l2 ball as a surrogate for semantics-preserving rewordings—is assumed, not proved; the paper concedes the perturbations need not decode to literal prompts. Code and checkpoints are not released despite a stated intention, which makes the evaluations hard to probe.\n\nWho is this for: people working on RLVR robustness and reward design. It deserves peer review: the experiments are well-described, the limitations are candid, and the formal framing will be useful even if the components later turn out to be replaceable by simpler MT. I'd want a revision that either releases artifacts and a cleaner component attribution, or softens the contribution claim to the full recipe rather than DTR/adversary.\n\nRecommendation: send to review with encouragement; the paper is honest and probably right about the overall effect, but the component-level story needs either more evidence or less assertive framing.","headline":"A useful, honest robustness method whose novel components are not cleanly isolated; the effect over GRPO-MT is small but the formal framing and candid limitations make it worth review.","tokens_in":34433,"tokens_out":2511,"would_cite":true,"duration_ms":28708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PIRL is a training recipe that keeps RLVR-trained multimodal models accurate when prompts are paraphrased or reformatted.","keywords":["prompt robustness","RLVR","multimodal large language models","reward decomposition","semantic equivalence class","adversarial consistency","visual question answering","template stress"],"falsifier":"Take a PIRL-trained policy and a hand-built set of literal paraphrases and template swaps that lie outside the $\\ell_2$ ball used in training; if answer-level accuracy under those rewordings drops as much as GRPO's while clean accuracy stays high, the per-token KL surrogate did not transfer to answer-level invariance. A cheaper diagnostic: track $\\Phi_{wf}$ during GRPO training, and if it does not rise toward 1 as format compliance increases, the variance-entanglement mechanism is not the operative cause of the robustness gap.","tokens_in":1742,"feed_emoji":"🛡️","tokens_out":2206,"duration_ms":91455,"temperature":0.7,"pith_summary":"The paper's claim is that RLVR-trained multimodal models are brittle to prompt rephrasing because the standard objective gives no way to distinguish a wrong answer from a badly formatted one, and because it trains only on a thin slice of the prompts the model will meet. It argues that both problems can be addressed by optimizing over the semantic equivalence class of a prompt, and it proposes Prompt-Invariant RLVR (PIRL), which decomposes the reward into a format-aware trinary term and an entailment-based semantic term, and regularizes the policy to be consistent under an embedding-space adversary that perturbs the instruction. On exam, medical, and legal visual question answering, template-stress accuracy of PIRL-trained policies drops by at most about 1 percent on average, compared with roughly 3 percent for standard GRPO, while in-distribution accuracy is largely preserved. The method's theoretical results identify why this works: reward decomposition restores useful variation within format buckets, and invariance control with small $\\varepsilon$ excludes the format-fixed shortcut policies that the training objective cannot distinguish.","feed_headline":"Prompt rephrasing accuracy loss drops from ~3% to ≤1% with PIRL","feed_subtitle":"A trinary reward plus embedding adversarial training keeps RLVR gains from vanishing under paraphrased prompts.","key_machinery":"The load-bearing objects are: the semantic equivalence class $\\llbracket\\boldsymbol{x}\\rrbracket$, the set of transformations that preserve the task answer; the Dynamic Trinary Reward $R_{\\mathrm{DTR}} = R_t + \\alpha_{\\mathrm{rew}} R_s$, where $R_t$ distinguishes correct, wrong-but-formatted, and format-failure and $R_s$ is a bounded entailment score; and the adversarial consistency objective, which perturbs instruction-token embeddings within an $\\ell_2$ ball to drive down DTR reward and adds a token-level KL penalty between clean and perturbed policy distributions. The theory supplies two population-level conditions, (C1) Reward Decomposition and (C2) Invariance Control; Proposition B.10 shows that under small $\\varepsilon$ the format-fixed shortcut branch is excluded while the correct-answer branch is feasible, and Proposition B.11 gives a coverage-transfer bound. The per-token KL is an acknowledged tractable surrogate, not a certificate of answer-level invariance.","core_discovery":"The central claim is that prompt-template brittleness in RLVR-trained multimodal LLMs is not an inevitable side effect of reinforcement learning but can be addressed by two design conditions: reward decomposition and invariance control. The paper formalizes a worst-case objective over the semantic equivalence class of a prompt and shows that binary verifiers entangle format with semantics (Theorem B.1), and that the training objective cannot identify the prompt-invariant policy when unseen formats carry test mass (Proposition B.5). PIRL instantiates the two conditions with a Dynamic Trinary Reward, which assigns $+1$, $-1$, and $0$ to correct, format-failure, and wrong-but-formatted cases alongside an NLI-based semantic score, and with a projected-gradient adversary on the instruction embedding plus a per-token KL consistency penalty. The paper's empirical claim is that under template-stress evaluation PIRL's mean accuracy drops by at most about 1 percentage point versus roughly 3 percentage points for GRPO, and that PIRL has the smallest mean drop among trained methods under dynamic evaluation.","pith_inferences":["If the surrogate-transfer assumption holds, the same consistency-regularizer design could be applied to image-embedding perturbations; the paper's own modality ablation already suggests that image robustness requires image-side adversarial training, a direct extension of the same mechanism.","The theory predicts a diagnostic that could be measured in any existing GRPO run: the within-format reward-variance share $\\Phi_{wf}$ should climb toward 1 as format compliance rises, while PIRL's floor stays bounded by $4\\alpha_{\\mathrm{rew}}^2 m/(\\alpha_{\\mathrm{rew}}+2)^2$; the paper's instrumentation in Appendix C.3 checks exactly this without retraining.","Because the coverage-transfer bound transfers a policy's own reward, a natural follow-up is to choose the training transformation pool adaptively from the deployment prompt distribution rather than fixing five templates, making the small stress-drop number a design target rather than an evaluation artifact."],"forward_implications":["On seven public VQA benchmarks and two base models of 7–8B parameters, template-stress accuracy of PIRL-trained policies drops by about 1 percentage point on average while GRPO drops about 3 percentage points (Table 1).","In-distribution accuracy is not sacrificed for robustness: PIRL usually matches GRPO on standard-format benchmarks and on some benchmarks improves both standard and stress accuracy.","The robustness gain concentrates where the non-identifiability theory says it should: as more unseen templates are introduced, GRPO's in-distribution accuracy falls while PIRL stays flat (Figure 3).","Under dynamic evaluation PIRL has the smallest mean performance drop among trained methods, but on task-altering mutations (multiple-choice converted to free-form, document layout changes) every method degrades substantially, so the claim is scoped to semantic rewording rather than all input changes."],"supporting_citations":[{"why":"Supplies the GRPO algorithm and standard RLVR setup whose prompt brittleness the paper targets and which serves as the main baseline.","marker":"[24]"},{"why":"Provides the multimodal reasoning benchmark used for training and exam VQA evaluation, where RLVR gains and prompt-stress losses are measured.","marker":"[20]"},{"why":"Defines the dynamic evaluation operator pool (text, image, layout mutations) used for the D-Test stress protocol.","marker":"[30]"},{"why":"Grounds the worst-case-over-semantic-equivalence-class objective in distributionally robust optimization, motivating conditions (C1) and (C2).","marker":"[22]"},{"why":"Supplies the fixed NLI entailment scorer used as the semantic reward component $R_s$ of the Dynamic Trinary Reward.","marker":"[3]"},{"why":"Provides the medical VQA training set used for the high-stakes medical robustness experiments.","marker":"[11]"},{"why":"Provides the legal/document VQA training set used for the legal VQA stress experiments.","marker":"[26]"}],"fun_headline_variants":["PIRL cuts prompt-paraphrase accuracy drop from 3% to 1%","Trinary reward plus adversarial training makes RLVR robust to rephrased prompts","Prompt-invariant RLVR: accuracy drop shrinks from ~3% to ≤1% under stress","Embedding adversary and trinary reward halve RLVR's prompt-sensitivity"],"cache_read_input_tokens":36480,"weakest_assumption_plain":"The load-bearing premise is that small bounded perturbations of the instruction-token embedding are a workable stand-in for worst-case semantics-preserving rewordings, and that optimizing a per-token KL consistency penalty transfers to answer-level invariance on literal rephrased prompts.","fun_headline_variants_meta":{"raw":{"variants":["PIRL cuts prompt-paraphrase accuracy drop from 3% to 1%","Trinary reward plus adversarial training makes RLVR robust to rephrased prompts","Prompt-invariant RLVR: accuracy drop shrinks from ~3% to ≤1% under stress","Embedding adversary and trinary reward halve RLVR's prompt-sensitivity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000935,"raw_usage":{"total_tokens":4029,"prompt_tokens":1001,"completion_tokens":3028,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":2935}},"tokens_in":617,"tokens_out":3028,"duration_ms":19855,"temperature":1.0,"reasoning_tokens":2935,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:23:36.690357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a PIRL-trained policy and a hand-built set of literal paraphrases and template swaps that lie outside the $\\ell_2$ ball used in training; if answer-level accuracy under those rewordings drops as much as GRPO's while clean accuracy stays high, the per-token KL surrogate did not transfer to answer-level invariance. A cheaper diagnostic: track $\\Phi_{wf}$ during GRPO training, and if it does not rise toward 1 as format compliance increases, the variance-entanglement mechanism is not the operative cause of the robustness gap.","supporting_citations":[{"cited_title":"Dynamic multimodal evaluation with flexible complexity by vision-language bootstrapping","cited_arxiv_id":null,"evidence_quote":"Defines the dynamic evaluation operator pool (text, image, layout mutations) used for the D-Test stress protocol."},{"cited_title":"Hashimoto, and Percy Liang","cited_arxiv_id":null,"evidence_quote":"Grounds the worst-case-over-semantic-equivalence-class objective in distributionally robust optimization, motivating conditions (C1) and (C2)."},{"cited_title":"e-SNLI:Naturallanguage inference with natural language explanations.Advances in Neural Information Processing Systems, 31, 2018","cited_arxiv_id":null,"evidence_quote":"Supplies the fixed NLI entailment scorer used as the semantic reward component $R_s$ of the Dynamic Trinary Reward."},{"cited_title":"Blaschko, Sien Moens, and Tomasz Stanisławek","cited_arxiv_id":null,"evidence_quote":"Provides the legal/document VQA training set used for the legal VQA stress experiments."}],"review_version":1}