{"id":"8ffc0be8-6feb-46c9-98c1-0d6ee204cbdd","arxiv_id":"2502.08336","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SCPL adds saliency-guided value consistency, a dynamics module, and KL-based policy consistency to SAC, claiming large zero-shot generalization gains in visual RL benchmarks.","lead":"This paper introduces SCPL, a reinforcement learning method that keeps an agent's attention and decisions consistent across visually perturbed observations, using saliency maps and a KL constraint. It reports large gains over prior methods on three visual RL generalization benchmarks, though the evaluation has some caveats.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DMC-GB evaluation selects a different data augmentation per test setting (conv for Color hard, overlay for Video), which inflates the reported zero-shot margins; the central claim of a 14% video-hard improvement may be an artifact of this test-set-dependent choice.","rationale":"The reader's verdict identified the saliency-map assumption as the weakest assumption, but their rationale also flagged the DMC-GB augmentation selection and the false theorem. I consider the augmentation selection to be the single most load-bearing concern because the central claim is the empirical SOTA margin (14%, 39%, 69%). If the margin disappears under a fixed augmentation policy, the entire empirical contribution is undermined. The saliency concern is real but secondary: even if the saliency maps are imperfect, the value-consistency loss might still act as a useful regularizer, and the ablation shows a large contribution. The false theorem is also secondary to the empirical claim; it damages the theoretical narrative but does not by itself falsify the performance numbers. The proposed concrete test—re-running with one augmentation—would settle whether the headline improvement is genuine or a selection artifact. Since the reader already conditioned the verdict on this issue and I agree the paper can be repaired, I recommend keeping the verdict unchanged.","tokens_in":22208,"tokens_out":7951,"duration_ms":80706,"concrete_test":"Run SCPL on all DMC-GB settings with a single fixed augmentation (e.g., random overlay only), keeping all other hyperparameters and seeds identical to Table 1. If the Video hard average drops from 853 toward the SGQN baseline (747) or the relative improvement falls below about 5%, the reported zero-shot margin is an artifact of augmentation selection. For a stronger check, also give the best baseline (SGQN) the same per-setting augmentation search and compare margins.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the per-setting augmentation selection in the DMC-GB experiments (Section 5.2). The paper states that SCPL results are based on random convolution for Color hard and random overlay for Video easy/hard, and Appendix D explicitly justifies this by noting that random convolution performs better in Color hard while random overlay performs better in Video settings. This is a form of test-set selection: the augmentation policy is chosen with knowledge of the test distribution, which contradicts the zero-shot generalization claim. Baselines are evaluated with their standard augmentations, so SCPL effectively gets an extra hyperparameter tuned to the test setting. The reported +14% on Video hard, +1% on Color hard, and +2% on Video easy may therefore overstate the algorithm's inherent generalization advantage. This concern directly bears on the headline numbers. The false Theorem 1 (the bound should be linear in total variation, not quadratic) is a secondary issue affecting the theoretical contribution, but the empirical claim is the central one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCPL, a SAC-based method for zero-shot visual generalization. It adds a value consistency loss that matches Q-values on saliency-masked and original observations (Eqs. 3-4), a dynamics module that predicts next latents and rewards from original and augmented observations (Eqs. 6-8), and a policy-consistency KL loss between policies on original and augmented observations (Eqs. 12-14). The authors claim average improvements of 14% on DMC-GB video hard, 39% on Robotic hard, and 69% on CARLA, and provide Theorem 1 to motivate policy consistency by bounding the return gap with the KL divergence of the two policies.","tokens_in":22393,"tokens_out":5418,"duration_ms":59441,"significance":"If the empirical results were obtained under a fair protocol, SCPL would be a useful contribution: it combines saliency-guided Q-consistency with policy regularization and is evaluated on three benchmarks with ablations. The paper also gives a theoretical motivation for policy consistency. However, two load-bearing issues—test-set-dependent augmentation selection and an incorrect theorem—currently prevent the results from being accepted at face value. The implementation details are relatively complete, and the ablation study gives useful per-module information.","major_comments":[{"comment":"The DMC-GB evaluation selects a different data augmentation per test setting: Table 1 states that SCPL uses random convolution for Color hard and random overlay for Video easy and Video hard, and Appendix D explicitly justifies this by saying each augmentation performs better in its assigned setting. This is a test-set-dependent choice: the augmentation hyperparameter is selected after observing the test distribution, whereas baselines are evaluated with fixed standard augmentations. The reported +14% on video hard, and the DMC-GB margins in general, are therefore not a clean zero-shot comparison. Please rerun with a single augmentation chosen before seeing test results, or report both augmentations on all settings, and adjust the claims accordingly.","section":"Section 5.2 and Appendix D"},{"comment":"The asserted O(alpha^2) bound is not correct as stated. The standard performance-difference bound is linear in the total-variation distance alpha (Kakade-Langford), not quadratic. In the proof, the step leading to Eq. (18) is invalid: the expectation over a coupled pair does not equal P(disagreement) times the expected difference, and the subsequent use of P(n_t > 0) to multiply each advantage term by alpha double-counts the coupling failure probability. Consequently Eq. (19) and the derived KL bound in Eq. (11) are not justified. Please correct the theorem to the standard linear bound, or remove the quantitative bound and present policy consistency only as a heuristic motivation.","section":"Appendix B, Theorem 1 and Eq. (19)"},{"comment":"The method relies on vanilla-gradient saliency maps to identify task-relevant pixels, but the paper never defines a ground truth for 'task-relevant' regions. Table 2 reports ACC, AUC, and F1 without explaining what labels are used for these metrics. Please specify the ground-truth generation procedure and validate the saliency maps against it (for example, using known distractor masks or human annotations), because Eqs. (3)-(4) will actively hurt if the saliency maps are noisy or misaligned with task-relevant pixels.","section":"Section 4.1 and Table 2"},{"comment":"The Robotic and CARLA comparisons include only SAC, SVEA, and SGQN, yet the text concludes that SCPL 'outperforms the best prior methods' and claims state-of-the-art performance. Recent visual-RL generalization methods discussed earlier in the paper, such as MaDi, CG2A, and CNSN, are absent from these benchmarks. Please add those baselines, or limit the claim to the methods actually compared.","section":"Section 5.4, Tables 4 and 5"}],"minor_comments":[{"comment":"The symbol alpha is used both for the SAC temperature in Eq. (12) and for the maximal total-variation distance in Theorem 1; please rename one of them to avoid ambiguity.","section":"Eq. (12) and Theorem 1"},{"comment":"In the pseudocode, L_Q2 is formatted without the subscript style used in the main text; please make the notation consistent.","section":"Algorithm 1, line 6"},{"comment":"The t-SNE plots are qualitative; a quantitative clustering metric, such as silhouette score computed on the embeddings and actions, would better support the claim of consistent task-relevant representations.","section":"Figure 6"},{"comment":"The sentence 'let s_t is the state' contains a grammar error; please revise.","section":"Section 3"},{"comment":"The phrase '15 visual perturbed control tasks' could be clearer as five DMC tasks evaluated under three perturbation settings.","section":"Section 5.2"},{"comment":"The text says rho=0.95 'slightly outperforms' rho=0.9, but some entries are lower for 0.95 (e.g., Ball in cup Color hard); please hedge the claim accordingly.","section":"Appendix J, Table 8"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader's conditional verdict. The augmentation-selection issue is the most serious because it directly affects the headline numbers. If the authors can provide a fair comparison with a fixed augmentation and correct or remove the flawed theorem, the paper may become publishable. The saliency ground-truth issue also needs to be addressed before the mechanism can be considered validated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nSCPL is a competently assembled SAC variant: saliency-guided value consistency, a dynamics head on augmented observations, and a KL term pinning the policy across original and augmented observations. The combination is new, the ablations are thorough, and the empirical story across three benchmarks is coherent. The paper is worth engaging with.\n\nWhere it earns its keep: the policy-consistency KL is a simple, sensible idea that previous methods mostly ignore, and the ablation table shows each module buys something. The Robotic and CARLA results are strong and not obviously methodologically compromised.\n\nThe soft spots are real. First, the DMC-GB evaluation picks random convolution for Color hard and random overlay for Video easy/hard, based on what works better on each test setting. That is test-set-dependent hyperparameter selection, and it breaks the zero-shot framing. A fair comparison needs one augmentation policy across all DMC settings, or at least a report of both augmentations per setting. The reported +14% on Video hard almost certainly shrinks under a single augmentation. Second, Theorem 1 is false as stated: the bound should be linear in total variation distance, not quadratic. The proof couples the two policies and then multiplies the per-step advantage bound by the probability of divergence earlier in the trajectory, which double-counts the divergence. The standard coupling argument gives O(alpha), not O(alpha^2); the error propagates to the KL bound in Eq. (11). The theory is not load-bearing for the experiments, but it should be corrected or removed.\n\nWorth noting: the saliency maps are never validated against ground truth; Table 2's ACC/AUC/F1 need a defined label source. That's a minor point relative to the two above.\n\nThis paper is for researchers in visual RL generalization. As it stands, I would not rely on the headline numbers. But the algorithm idea is clean and the ablations are informative. A serious referee could extract a valuable paper if the authors fix the evaluation protocol and either repair or drop the theory.\n\nSend it to review, but with the expectation of a major revision. The empirical core is plausible enough to deserve referee time; the current form is not acceptable.","headline":"Clean empirical recipe with a sensible policy-consistency idea, but the DMC-GB numbers are inflated by test-set-dependent augmentation choice and the theory is wrong; worth a serious referee if both are fixed.","tokens_in":22958,"tokens_out":5557,"would_cite":false,"duration_ms":56562,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Saliency-masked value consistency plus policy KL-regularization improves zero-shot visual RL generalization, giving 14%, 39%, and 69% average gains over prior methods on DMC-GB, robotic manipulation, and CARLA.","keywords":["visual reinforcement learning","zero-shot generalization","saliency","data augmentation","policy consistency","value consistency","dynamics model","KL divergence"],"falsifier":"On the DMC-GB video-hard benchmark, replace SCPL's gradient saliency masks with random binary masks of the same density; if the test return stays near 853 instead of dropping, saliency guidance is not carrying the claimed load. Alternatively, evaluate SCPL with human-annotated or segmentation-based oracle masks: if oracle-masked SCPL does not beat gradient-masked SCPL, the gradient saliency map is not the active ingredient.","tokens_in":21989,"feed_emoji":"🤖","tokens_out":10150,"duration_ms":93100,"temperature":0.7,"pith_summary":"SCPL is a soft actor-critic variant that trains the encoder and value function to assign equal Q-values to an observation and its saliency-masked version, for both original and augmented views, so the agent's attention stays on task-relevant pixels. A dynamics module predicts next-state latents and rewards from augmented data, and a KL-divergence constraint keeps the policy's action distribution unchanged across perturbations. A theorem bounds the gap in expected return between two policies by a constant times their maximum KL divergence, which motivates the policy-consistency term. The paper reports state-of-the-art zero-shot generalization: average gains of 14% on the DMC-GB video-hard setting, 39% on the robotic manipulation hard setting, and 69% on CARLA. If true, this matters because agents would transfer to unseen visual environments without any adaptation data.","feed_headline":"Zero-shot visual RL gains 14-69% with saliency-guided consistency","feed_subtitle":"A saliency-stable value loss plus policy KL-regularization outperforms prior methods on three visual RL benchmarks.","key_machinery":"The load-bearing object is the saliency attribute masked map: the gradient of the Q-network with respect to the input image, binarized with a $\\rho$-quantile threshold so that a pixel is set to 1 only if its gradient belongs to the top $1-\\rho$ quantile, and then multiplied elementwise with the observation. Equations (3)-(4) force $Q_\\zeta(f_\\theta(\\hat{s}),a)$ to match $Q_\\zeta(f_\\theta(s),a)$ for both the original and the augmented observation, which drives the encoder and value function to ignore pixels outside the saliency mask. The dynamics module (Eqs. (6)-(8)) predicts the next latent and the reward from each view, and the policy module adds $\\beta\\,D_{\\mathrm{KL}}(\\pi_\\phi(\\cdot|e_s)\\,\\|\\,\\pi_\\phi(\\cdot|e_{s_\\alpha}))$ to the SAC policy loss. Theorem 1, proved with the performance-difference lemma and the $\\alpha$-coupling advantage bound, converts total-variation distance between training and test policies into a return-gap bound, and the standard inequality $D_{\\mathrm{TV}}^2 \\le D_{\\mathrm{KL}}$ converts that into the KL bound that justifies the policy-consistency term.","core_discovery":"The paper's central claim is that visual overfitting in RL comes from two coupled failures, attention to task-irrelevant pixels and policy drift under perturbation, and that both can be corrected by three coordinated objectives on top of SAC: value consistency guided by saliency attribute maps, a dynamics objective over augmented observations, and policy consistency via KL divergence. It proves Theorem 1, which states that with $\\alpha = D^{\\max}_{\\mathrm{TV}}(\\pi_o,\\pi_p)$ and $\\epsilon = \\max_{s,a}|A^\\pi(s,a)|$, the return gap obeys $\\eta(\\pi_o)-\\eta(\\pi_p) \\le \\frac{2\\epsilon\\gamma}{(1-\\gamma)^2}\\alpha^2$; using $D_{\\mathrm{TV}}(p\\|q)^2 \\le D_{\\mathrm{KL}}(p\\|q)$, the paper concludes the return gap is bounded by $C\\,D^{\\max}_{\\mathrm{KL}}(\\pi_o,\\pi_p)$ with $C=\\frac{2\\epsilon\\gamma}{(1-\\gamma)^2}$. The empirical claim is that SCPL reaches 853 average return on DMC-GB video hard (14% above the best baseline), 65.1 on robotic manipulation hard (+39%), and 352 on CARLA (+69%).","pith_inferences":["Editorial extension: the saliency-masked value-consistency loss could be applied to offline RL, where augmented or perturbed observations are cheap; the KL policy term would then act as a conservative regularizer against distribution shift.","Editorial extension: the paper fixes the $\\rho$-quantile per task (0.9 or 0.95) and reports sensitivity to it; an adaptive quantile chosen by a saliency-uncertainty or return-based criterion could remove that per-task tuning.","Editorial extension: the theory bounds the return gap by the maximum KL over states, but the experiments estimate consistency only on sampled observations; a state-covering worst-case KL estimator would make the bound actionable during deployment.","Editorial extension: the method's reliance on gradient saliency could be tested by replacing the learned masks with oracle masks from segmentation or human annotation; if oracle masks do not outperform learned masks, the saliency map is not the active ingredient."],"forward_implications":["Agents trained with SCPL should transfer to unseen backgrounds, colors, and video distractors without any test-environment data, because all objectives are computed from training observations and augmentations.","The KL bound gives a deployment diagnostic: if an agent's action distribution on a new observation stays close to its training distribution, the expected return gap is guaranteed small up to constants.","The shared encoder also supports next-latent and reward prediction, so the same representation can serve both model-free control and model-based planning or world-model objectives.","Because SCPL is built on top of SAC, the three auxiliary losses can be added to existing value-based or actor-critic pipelines without changing the base algorithm's architecture.","On the three benchmarks, the biggest reported gains occur in the hardest settings (video hard, robotic hard, and rainy or dark CARLA weather), suggesting the method's advantage grows with perturbation strength."],"supporting_citations":[{"why":"Provides the saliency-guided Q-network (SGQN) baseline and the idea of guiding attention with saliency; SCPL extends this to augmented observations and adds value and policy consistency.","marker":"[2]"},{"why":"Provides the SVEA baseline that updates the value function with augmented data; SCPL's value consistency module builds directly on this mechanism.","marker":"[11]"},{"why":"Provides the DMC-GB benchmark, the random-overlay augmentation used for video settings, and the SODA baseline compared in the appendix.","marker":"[12]"},{"why":"Provides the performance-difference lemma (Lemma 1) used to decompose the return gap in Theorem 1.","marker":"[15]"},{"why":"Provides the random-convolution augmentation used for the color-hard DMC setting.","marker":"[17]"},{"why":"Supplies the inequality that total-variation distance squared is at most KL divergence, which turns the total-variation bound into the KL bound.","marker":"[26]"},{"why":"Provides the alpha-coupling advantage bound (Lemma 2) used in the proof of Theorem 1.","marker":"[29]"},{"why":"Defines the vanilla-gradient saliency map from which the rho-quantile attribute maps are computed.","marker":"[31]"},{"why":"Provides the DBC invariant-representation baseline and the CARLA reward setting used for the autonomous-driving experiments.","marker":"[45]"}],"fun_headline_variants":["Saliency-guided RL beats baselines by 14-69% on visual tasks","SCPL: zero-shot visual RL with saliency and policy consistency","Ignore distractors, generalize: SCPL for visual RL","Visual RL zero-shot gains via saliency-stable policies","New algorithm SCPL improves visual RL generalization by up to 69%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that the Q-network's gradient saliency maps point at exactly the task-relevant pixels in both original and augmented images, so that masking and enforcing value equality on those pixels preserves task information instead of deleting it.","fun_headline_variants_meta":{"raw":{"variants":["Saliency-guided RL beats baselines by 14-69% on visual tasks","SCPL: zero-shot visual RL with saliency and policy consistency","Ignore distractors, generalize: SCPL for visual RL","Visual RL zero-shot gains via saliency-stable policies","New algorithm SCPL improves visual RL generalization by up to 69%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3729,"prompt_tokens":1080,"completion_tokens":2649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":696,"completion_tokens_details":{"reasoning_tokens":2556}},"tokens_in":696,"tokens_out":2649,"duration_ms":19197,"temperature":1.0,"reasoning_tokens":2556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T05:32:11.879591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the DMC-GB video-hard benchmark, replace SCPL's gradient saliency masks with random binary masks of the same density; if the test return stays near 853 instead of dropping, saliency guidance is not carrying the claimed load. Alternatively, evaluate SCPL with human-annotated or segmentation-based oracle masks: if oracle-masked SCPL does not beat gradient-masked SCPL, the gradient saliency map is not the active ingredient.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the saliency-guided Q-network (SGQN) baseline and the idea of guiding attention with saliency; SCPL extends this to augmented observations and adds value and policy consistency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SVEA baseline that updates the value function with augmented data; SCPL's value consistency module builds directly on this mechanism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DMC-GB benchmark, the random-overlay augmentation used for video settings, and the SODA baseline compared in the appendix."},{"cited_title":"Kakade and John Langford","cited_arxiv_id":null,"evidence_quote":"Provides the performance-difference lemma (Lemma 1) used to decompose the return gap in Theorem 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the random-convolution augmentation used for the color-hard DMC setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the inequality that total-variation distance squared is at most KL divergence, which turns the total-variation bound into the KL bound."},{"cited_title":"Jordan, and Philipp Moritz","cited_arxiv_id":null,"evidence_quote":"Provides the alpha-coupling advantage bound (Lemma 2) used in the proof of Theorem 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the vanilla-gradient saliency map from which the rho-quantile attribute maps are computed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DBC invariant-representation baseline and the CARLA reward setting used for the autonomous-driving experiments."}],"review_version":1}