{"id":"c181ecef-ba0a-4030-b754-d032d3db33fb","arxiv_id":"2512.10414","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Selectively attacking the visual input with a gradient of response entropy during GRPO sampling increases policy entropy and improves visual reasoning accuracy by roughly 1–2 points.","lead":"This paper proposes a tweak to reinforcement learning for vision-language models: during training, it slightly perturbs images with an adversarial, entropy-based signal so the model explores more diverse answers. The result is a small but consistent accuracy gain on several visual math and hallucination benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Entropy-boosting mechanism is asserted but never directly measured; the adversarial gradient in Eq. (4) may not be an unbiased entropy gradient.","rationale":"Reader's identified weak point (Eq. 6) is not the load-bearing one. The importance ratio in Eq. (6) is exactly what is needed to estimate the clean-context PPO/GRPO surrogate from samples drawn under q_adv; the π_old(y|q) factor cancels. I therefore disagree with the reader's choice of weakest assumption. The actual risk is that the paper's named mechanism — entropy-guided adversarial sampling — is never directly tested. All support is indirect: final accuracy, an ablation that swaps token selection, and a training-time entropy curve whose relation to I_adv is not established. Because the gradient in Eq. (4) is underspecified and the standard naive estimator is not an unbiased entropy gradient, the causal arrow from entropy to gains is the least secure link. This does not overturn the empirical result: the method may still work as off-policy augmentation, and the reported comparisons are reasonable for a 7B model with 3 seeds. But until H(I_adv)>H(I) is demonstrated, the paper's central claim should remain conditional. I keep the reader's CONDITIONAL verdict (UNCHANGED). Minor flag: the appendix containing prompts is promised but absent; this should be fixed in revision.","tokens_in":14281,"tokens_out":20549,"duration_ms":224741,"concrete_test":"Freeze the policy at 2-3 checkpoints during SaEI training; take 200 questions from the training set; for each, sample 64 responses under clean I and under the one-step I_adv computed by Eq. (4). Compare the empirical per-token entropy (Eq. 3) and the distinct-n response diversity. Report the mean difference and the fraction of questions with H(I_adv)>H(I). In parallel, for a random subset of 20 questions, compute the unbiased entropy-gradient estimator g = -(1/n)Σ_i (1+logπ(y_i|q))∇_q logπ(y_i|q) and the paper's implemented gradient h = -(1/n)Σ_i ∇_q logπ(y_i|q); report the cosine similarity between sign(g) and sign(h). If H(I_adv) is not consistently larger, or sign(g) does not align with sign(h), the entropy-exploration mechanism is not supported and the gains should be attributed to Eq. (6)'s off-policy regularization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SaEI improves exploration because adversarial images increase the entropy of sampled responses (Eq. (4)). I do not think the reader's Eq. (6) objection is the main risk: for the clean-context surrogate, E_{π_old(·|q)}[(π_θ/π_old(·|q))A] = E_{π_old(·|q_adv)}[(π_θ/π_old(·|q_adv))A], so the ratio in Eq. (6) is a legitimate importance weight and the off-policy form is defensible. The load-bearing gap is that the paper never verifies the causal step I_adv actually raises response entropy. All reported evidence is downstream (final accuracy); the entropy dynamics in Fig. 2 are training-time policy entropy, not the entropy of the rollout distribution under I_adv. Moreover, the gradient computation is underspecified. If Eq. (3) is implemented by differentiating the sampled -logπ(y|q), its expectation over y~π is zero (not the true ∇H = -E[(1+logπ)∇logπ]); if it is implemented by computing full conditional entropies over a sampled prefix, the dependence of the prefix on the image is ignored. The paper does not state which or provide the standard score-function correction. The sign operation in Eq. (4) might make the heuristic work, but that is asserted, not shown. Consequently, the reported gains (Tables 1-2) could be due to Eq. (6) acting as off-policy data augmentation rather than to entropy-guided exploration. A direct measurement of H(I_adv) vs H(I) would separate these. (Minor: the appendix promised for prompts in §5.3 is missing.)","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SaEI, a method that intervenes in policy entropy during the RL sampling phase of GRPO rather than during policy optimization. SaEI first computes the entropy of responses sampled from the current policy on clean visual inputs, then uses a PGD-style attack to perturb the image so as to increase this entropy, and finally mixes responses sampled from the clean and adversarial images to form the GRPO group. A token-selective entropy computation (TsEC) restricts the entropy objective to tokens of moderate entropy, motivated by prior work on the roles of high- and low-entropy tokens. Experiments on Geometry3K and MM-Eureka report consistent in-domain gains over vanilla GRPO and several baselines, together with improved out-of-domain averages, and ablations examine the contribution of each component and compare with random noise.","tokens_in":14685,"tokens_out":4415,"duration_ms":51506,"significance":"If the causal mechanism claimed by the paper were verified, the contribution would be an interesting and simple sampling-side intervention that complements existing optimization-side entropy-control methods. The empirical package is in many ways solid: three seeds with standard deviations, ablations of both components, a random-noise control, out-of-domain evaluation on four benchmarks, and a check at a smaller group size. The reported gains are modest but consistent. However, the central claim — that adversarial entropy maximization is what drives the gains — is not directly supported by the measurements in the paper. The reported evidence is downstream accuracy; no experiment shows that the adversarial image actually increases the entropy of the rollout distribution. Moreover, the gradient computation in Eq. (4) is underspecified, and the objective in Eq. (6) is an off-policy mismatch that is neither derived nor tested separately. The paper is therefore best read at present as an empirical data-augmentation heuristic whose mechanism remains unsubstantiated.","major_comments":[{"comment":"","section":"§4.2, Eq. (4), Fig. 2"},{"comment":"","section":"§4.2, Eq. (3)–(4)"},{"comment":"","section":"§4.2, Eq. (6)"},{"comment":"","section":"§5.3, Fig. 4"}],"minor_comments":[{"comment":"","section":"§5.3"},{"comment":"","section":"§5.5, Fig. 6"},{"comment":"","section":"Eq. (2)"},{"comment":"","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict is reasonable. I agree that the Eq. (6) objection alone is not the central problem: one can view the clean-context surrogate as a defensible off-policy choice. The more serious gap is that the paper never verifies that the adversarial perturbation actually increases sampled-response entropy, and the gradient estimator in Eq. (4) is ambiguous. These are fixable with an additional measurement and a clarification/derivation, so I do not recommend rejection. If the authors cannot produce direct entropy measurements, the contribution should be reframed as an off-policy data-augmentation method, and the exploration narrative should be softened. The empirical work is otherwise carefully presented, and the random-noise control is a good baseline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something new: during GRPO sampling, it attacks the image with the gradient of the sampled-response entropy, then mixes those adversarial rollouts into the update batch. That is a genuinely different intervention from KL-Cov (which constrains token updates during optimization) and NoisyRollout (random Gaussian noise). The empirical package is decent: three seeds with standard deviations, ablations, a random-noise control, and a smaller group-size comparison. The gains are modest but consistent—about 2 points in-domain and around 1 point on OOD average across two training sets. The writing is clear and the related work is placed honestly.\n\nThe soft spots are real, though. The load-bearing claim is that adversarial images increase response entropy and thereby improve exploration. The paper never measures that. Fig. 2 shows training-time policy entropy, not the entropy of the rollout distribution under the perturbed image. The gradient in Eq. (4) is also underspecified: if it differentiates the sampled −logπ, the expectation is zero, not the entropy gradient; if it computes full conditional entropies, the dependence of the sampled prefix on the image is ignored. Either way, the causal step is asserted, not shown. The gains could plausibly come from off-policy data augmentation rather than entropy-driven exploration. A direct comparison of H(·|I_adv) vs H(·|I) would settle this.\n\nOn Eq. (6): the ratio is actually a legitimate importance-weighted estimator of the clean-context objective, so the reader's worry about behavior cloning is not quite on target—but the paper doesn't frame it that way, and 'keeping the optimization objective unchanged' is misleading. That needs a derivation or a reference to standard off-policy RL.\n\nThe per-dataset tuning of α, the best-checkpoint reporting, and the missing code/prompts all add friction. None of these refute the empirical result, but they leave the central mechanism unverified. This deserves a serious referee, not a desk reject. The referee should ask for a direct entropy measurement, a precise description of the gradient computation, and final-checkpoint numbers.","headline":"Entropy-guided adversarial sampling is a new and plausible way to improve GRPO for visual reasoning, but the paper never demonstrates the attack actually raises response entropy.","tokens_in":15132,"tokens_out":4991,"would_cite":false,"duration_ms":52007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Perturbing images with entropy-guided adversarial noise during RL sampling improves visual reasoning in VLMs by around 2% on math benchmarks.","keywords":["reinforcement learning","vision-language models","entropy intervention","adversarial attack","GRPO","policy exploration","visual reasoning","token selection"],"falsifier":"Train two GRPO policies on the same data—one with SaEI and one with a control that adds random Gaussian noise matched in L2 norm to the adversarial perturbation. If the random-noise control achieves the same or greater improvement in response diversity and accuracy, then SamEI's gains do not come from entropy-guided exploration but merely from perturbation, undermining the claimed mechanism.","tokens_in":14116,"feed_emoji":"🎯","tokens_out":1706,"duration_ms":20716,"temperature":0.7,"pith_summary":"The paper argues that the standard reinforcement-learning fine-tuning method GRPO for vision-language models suffers from collapsing policy entropy, which limits exploration. Instead of intervening during policy optimization as prior work does, the authors intervene during RL sampling by adding small adversarial perturbations to the input image, chosen to increase the entropy of the model's sampled responses. They show that this entropy-guided adversarial sampling boosts accuracy on both in-domain math datasets and out-of-domain visual reasoning benchmarks. A careful reader would care because it suggests a lightweight, sampling-side alternative to entropy-controlling objectives that leaves the policy loss unchanged.","feed_headline":"Adversarial entropy boost lifts VLM reasoning by ~2%","feed_subtitle":"Perturbing images to raise response entropy during sampling improves visual-reasoning RL without changing the policy loss.","key_machinery":"EgAS (Entropy-Guided Adversarial Sampling) formulates the entropy of sampled responses as an adversarial objective and uses projected gradient descent to perturb the image, thereby increasing policy entropy during sampling. TsEC (Token-Selective Entropy Computation) filters tokens by entropy rank, keeping only the middle third, so the adversarial attack targets moderately uncertain tokens that blend continuation and direction. Together they intervene in entropy at the sampling stage rather than the policy-update stage, which is the paper's claimed novelty.","core_discovery":"The paper introduces SaEI (Selective-adversarial Entropy Intervention), a method that maintains exploratory behavior in GRPO by attacking the visual input with the gradient of the average token-wise entropy of sampled responses. Clean-image responses and adversarial-image responses are mixed into one group for advantage estimation and policy update. A token-selective variant recomputes the entropy only on tokens with moderate entropy, avoiding both low-entropy factual tokens and high-entropy already-explorative tokens. Experiments report SaEI outperforms vanilla GRPO by 2.00% on MM-Eureka (64.45 vs 62.45) and 2.16% on Geometry3K (56.18 vs 54.02), with the best out-of-domain average across th","pith_inferences":["A broader implication is that any policy-gradient method could benefit from entropy-guided input perturbation, not just GRPO, since the entropy objective is defined on responses and does not depend on the advantage estimation scheme.","The paper's logic suggests an easy testable extension: applying the same adversarial entropy attack to other modalities (e.g., audio or structured inputs) and measuring whether exploration increases there too.","If the off-policy ratio issue is real, a corrected formulation that treats adversarial responses as off-policy data could reveal whether the gains come from entropy-driven exploration or from implicit behavior cloning of diverse rollouts.","The reported training-stability advantage over KL-Cov hints that sampling-side intervention may avoid the variance spikes seen with token-update restriction, but this claim needs direct variance analysis beyond the reported standard deviations."],"forward_implications":["If SaEI is correct, entropy collapse in GRPO can be mitigated without altering the policy optimization objective, decoupling exploration-promotion from the loss.","A single-step adversarial attack (T=1) suffices for the gains, making the method computationally affordable, while two steps already cause instability.","The token-selective principle implies that not all tokens should be equally targeted for entropy intervention; moderate-entropy tokens carry the useful exploratory signal.","Better out-of-domain generalization suggests that sampling-time input perturbation can transfer reasoning improvements beyond the training distribution."],"fun_headline_variants":["Entropy-gradient attacks on images boost RL reasoning 2%","Perturb images to raise entropy, lift VLM reasoning","Selective entropy attack sharpens visual reasoning RL","Visual entropy attack improves GRPO exploration by 2%","Attack images with entropy gradient to boost reasoning by 2%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The objective in Equation (6) evaluates the current policy's likelihood on clean images for both clean- and adversarial-image responses, while the old policy's denominator uses the actual sampling input; this ratio is not a true importance weight, and the paper does not formally justify that the surrogate is valid for optimizing on the mixed sampling distribution.","fun_headline_variants_meta":{"raw":{"variants":["Entropy-gradient attacks on images boost RL reasoning 2%","Perturb images to raise entropy, lift VLM reasoning","Selective entropy attack sharpens visual reasoning RL","Visual entropy attack improves GRPO exploration by 2%","Attack images with entropy gradient to boost reasoning by 2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000891,"raw_usage":{"total_tokens":3701,"prompt_tokens":785,"completion_tokens":2916,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2834}},"tokens_in":529,"tokens_out":2916,"duration_ms":19081,"temperature":1.0,"reasoning_tokens":2834,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T17:07:52.669880+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two GRPO policies on the same data—one with SaEI and one with a control that adds random Gaussian noise matched in L2 norm to the adversarial perturbation. If the random-noise control achieves the same or greater improvement in response diversity and accuracy, then SamEI's gains do not come from entropy-guided exploration but merely from perturbation, undermining the claimed mechanism.","supporting_citations":[],"review_version":1}