{"id":"ec8edb72-343e-4ed4-be3c-297c05217dce","arxiv_id":"2607.23835","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Submodular region-search attributions plus a ranking/truncation loss regularize models toward spatially corresponding evidence under geometric transforms, improving attribution metrics and transformed accuracy with small clean-accuracy cost.","lead":"The paper trains vision models so their decision evidence stays consistent under flips, rotations, and other geometry-preserving transforms, using search-based attributions rather than Grad-CAM-style maps. It matters because many “explanation consistency” methods may only align unfaithful heatmaps while the model still relies on unstable cues.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The headline attribution gains are measured with the same LIMA search engine that produced the training supervision, so Stability/Insertion/Deletion improvements are partly self-confirming; the independent evidence (transformed accuracy, Grad-ECLIP transfer) is small or mixed.","rationale":"The reader's weakest assumption concerned the faithfulness of LIMA trajectories as teaching targets (whether the ranking loss regularizes the decision process or only the search procedure). My concern is the mirror image on the evaluation side: the flagship metrics are produced by the same search engine that was regularized, so they cannot by themselves distinguish \"evidence reliance became consistent\" from \"the LIMA objective became easier to satisfy equivariantly.\" These overlap in spirit — both question whether the method moves the underlying decision process — hence partial agreement. The reader's rationale already noted the LIMA metrics are \"partly optimized-for,\" and correctly treated transformed/OOD accuracy as the least circular support; my reading sharpens that into the single load-bearing issue and notes the independent support is small (sub-1-pp means, some negative per-transform cells) and lacks seeds/error bars. I do not recommend changing the verdict: CONDITIONAL with MODERATE confidence already prices in this circularity, and the proposed independent-attribution-engine evaluation is exactly the kind of condition that should accompany acceptance. The method itself is well-motivated, the efficiency and ablation sections are reasonably careful, and the mixed Grad-ECLIP transfer in Table V is honestly reported — that mixed result is itself the paper's internal evidence that the concern is live rather than hypothetical.","tokens_in":19784,"tokens_out":1452,"duration_ms":57576,"concrete_test":"Re-evaluate the Table IV metrics on the regularized vs. baseline ViT-B/16 using an attribution engine that shares no machinery with LIMA — e.g., RISE-style random-mask Insertion/Deletion AUCs and a perturbation-based stability measure (mask the top-k RISE region, flip, re-measure) — plus Grad-ECLIP Insertion/Deletion. If the LIMA-measured gains (Insertion +52%, Deletion −62%, Stability 0.14→0.27) shrink to small or null effects under the independent engine, the headline attribution claims are metric-specific. Additionally, run 3 seeds of the Table III fine-tuning to test whether the +0.55/+0.33 pp transformed-accuracy means exceed seed variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that search-derived supervision regularizes the underlying decision process, not just the search procedure. But the strongest quantitative support (Table IV: Stability 0.14→0.27, Insertion +52.1%, Deletion −62.2%; Table V: PG_LIMA/EPG_LIMA gains) is computed with LIMA itself — the same greedy submodular search whose trajectories and stopping criterion the Submodular Ranking Loss (Eqs. 7–9) explicitly trains the model to make equivariant. Training makes Ψ-trajectories on transformed inputs follow the transformed query path; evaluation then re-runs Ψ-search and measures path/region agreement. This is metric-level train/test overlap: even a model that merely made the LIMA objective surface more symmetric (without any change in true evidence reliance) would post large gains on these metrics. The paper's own independent checks are noticeably weaker: (a) Grad-ECLIP localization transfer in Table V is mixed — on ViT-B/16 the method does not get best PG, and on ViT-L/16 it is below CGC on both PG and EPG; (b) the transformed-input accuracy gains on ImageNet-1K (Table III) are +0.55/+0.33 pp means with per-transform regressions on ConvNeXt-B (crop −0.13, rotate −0.16), and no seeds or error bars are reported anywhere, so these deltas are within typical fine-tuning noise; (c) the Stability metric is a custom modified IoU with an ad-hoc area penalty, also LIMA-derived. The concern is not that the method does nothing — the corruption/OOD columns in Table II are consistently favorable — but that the magnitude of the flagship attribution improvements cannot be attributed to \"consistent evidence reliance\" as opposed to \"the LIMA objective is now easier to satisfy equivariantly\" without an attribution-engine-independent evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper proposes an annotation-free attribution regularization framework in which a LIMA-style greedy submodular region search, run on the pretrained model itself, produces ordered evidence sequences that serve as self-generated supervision. A Submodular Ranking Loss (selection-ranking term, Eq. 7, plus selection-truncation term, Eq. 8) trains the model so that the search trajectory on a geometrically transformed input follows the spatially transformed query trajectory and terminates at the corresponding step. The authors argue this regularizes the underlying decision process rather than merely aligning gradient-based saliency maps, and report large gains in LIMA-based attribution stability/faithfulness on ImageNet-100 (Stability 0.14→0.27, Insertion +52.1%, Deletion −62.2% on ViT-B/16 at −0.28 pp accuracy), modest transformed-input accuracy gains on ImageNet-1K for ResNet-50/ConvNeXt-B, and improvements across corruption and OOD benchmarks (Table II).","tokens_in":20297,"tokens_out":3612,"duration_ms":139594,"significance":"The core idea — turning the ordered, path-dependent output of a search-based attribution method into a differentiable training signal via a pairwise ranking loss — is, to my knowledge, new and well-motivated; prior consistency regularizers (GC, CGC) align gradient maps whose faithfulness is questionable, and the authors correctly identify this gap. The formulation is clean, the paper is honest about the lack of a formal submodularity guarantee (§III-B, App. A), the path-constraint ablation (Table VI) is a genuinely informative control, and the empirical sweep is broad (two ViT scales, two CNNs, corruption/OOD suites). Some independent evidence is encouraging: the ObjectNet (+4.3 pp over the fine-tuned control) and ImageNet-C/P gains in Table II are not LIMA-derived and suggest real behavioral change. However, the headline attribution claims rest largely on metrics computed with the same search engine used to generate the training supervision, so the significance of the central claim is currently underdetermined; if confirmed on independent evaluators, this would be a useful contribution to explanation-guided learning.","major_comments":[{"comment":"The headline attribution results are measured with the same procedure that generates the training supervision. The Submodular Ranking Loss (Eqs. 7–9) explicitly trains the model so that LIMA's greedy Ψ-trajectory on a transformed input follows the transformed query path and so that Ψ_cons reaches τ_cons at the target terminal step; Stability, Insertion, Deletion, PG_LIMA, EPG_LIMA, and Area are then all computed by re-running LIMA (App. B: Insertion AUC is literally the AUC of Ψ_cons along the LIMA path, which L_SelT directly optimizes). A model that merely made the Ψ objective surface more symmetric — without any change in true evidence reliance — would post large gains on these metrics, so the 0.14→0.27 Stability and +52.1% Insertion numbers are partly self-confirming. The paper's own independent check supports this concern: Grad-ECLIP localization transfer in Table V is mixed (ViT-B/1","section":"§IV-C, Tables IV–V, Appendix B"},{"comment":"No seeds, error bars, or significance estimates are reported anywhere, and several of the load-bearing accuracy deltas are within typical fine-tuning noise: Table III's mean transformed-input gains are +0.55/+0.33 pp with per-transform regressions on ConvNeXt-B (crop −0.13, rotate −0.16), and Table II's clean-accuracy margins over the strongest baseline are 0.07–0.17 pp. Given that fine-tuning alone already moves these numbers substantially (the '+finetune' row shifts Sketch by −6 pp and ImageNet-A by +3.6 pp relative to pretrained), single-run deltas of this size cannot support the claim of 'improved transformed-input accuracy' without replication. Please report mean ± std over at least 3 seeds for Tables II–IV, or restrict claims to effects that are large relative to run-to-run variance (the ObjectNet and ImageNet-C/P gains may well survive this test).","section":"§IV-B, Tables II–III"},{"comment":"The transformation distribution T used in training is under-specified relative to the evaluation. §III-C describes the loss 'using horizontal flipping as the geometric transformation,' and Algorithm 1 samples t ~ T without defining T, while evaluation uses translation, rotation, affine, and scaling. If training uses only horizontal flip, then the Table III results measure zero-shot transfer to unseen transformation families — which would strengthen the paper and should be stated explicitly; if T includes the evaluated transforms, the equivariance training directly targets the test conditions and this must be disclosed. Either way, the ambiguity is load-bearing for interpreting the robustness claims and must be resolved.","section":"§III-C–III-D, Algorithm 1 vs. §IV-B-2"}],"minor_comments":[{"comment":"A leftover editorial placeholder remains in the text: 'The ImageNet-P metric name must be inserted after it is verified.' This must be fixed; it also leaves the ImageNet-P column of Table II without a defined metric.","section":"§IV-B-3"},{"comment":"Key hyperparameters are not reported: τ_conf, τ_a, and the ranking margin δ appear in Eqs. 7–8 and 11 but no values are given in the main text or the training-details appendix. Since the filtering indicator g (Eq. 11) determines which samples receive regularization, the fraction of training samples passing the filter should also be reported.","section":"§III-D / Appendix B"},{"comment":"The Pointing Game metric is cited to a G-CAME paper (Nguyen et al.); the standard citation is Zhang et al., 'Top-Down Neural Attention by Excitation Backprop' (or the original pointing-game source). Please correct.","section":"Reference [53]"},{"comment":"Bolding and cross-table consistency need a pass: in Table VII the best PG (0.9220, Conf.+Cons.) is bolded while the text argues the full objective is the best trade-off — please state the bolding convention; and Area for the full-objective ViT-B/16 configuration differs between Table V (31.04%, ImageNet-1K) and Table VII (37.03%, ImageNet-100), which is presumably a dataset difference but should be flagged in the captions to avoid confusion.","section":"Tables V, VII, VIII"},{"comment":"Notation: q_i(˜r) is defined but the loss is written with Ψ directly; S̃_0 as 'the all-zero starting image' is unclear (all-zero pixel values? empty mask?). Also, p. 2, 'question of this work:how' is missing a space, and Fig. 2's caption refers to L_SelR formulas that should be cross-referenced to Eqs. 7–8.","section":"§III-C, Eq. (7)"},{"comment":"The efficiency claims (regularization every 20 iterations 'empirically preserves performance') are asserted without supporting data; a small ablation on the sparsity interval would strengthen this, since it affects how the reported results should be reproduced.","section":"§III-E, Table I"}],"recommendation":"major_revision","confidential_remarks":"The citation pattern leans heavily on the authors' own prior work ([1], [2], [9], [10], [27], [30] are from the same group), which is natural given that the method builds directly on LIMA, but the editor may wish to note it. The circularity concern in my first major comment is, in my view, addressable within the paper's scope — the authors already have Grad-ECLIP evaluation infrastructure and the Table II behavioral evidence is encouraging — so I do not regard it as disqualifying, but the headline numbers should not appear in the abstract in their current form without an independent evaluator."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit here is not another Grad-CAM consistency loss. They take ordered LIMA search paths and stop conditions as annotation-free targets, then train with a pairwise ranking loss on corresponding candidates plus a truncation term so the transformed trajectory hits the same stop. That is a concrete, new training recipe on top of existing GC/CGC and LIMA lines.\n\nWhat they do well: the motivation is clean (map agreement ≠ evidence agreement), the two-stage filter (correct, confident, compact area) is sensible, and the empirics are broad—ImageNet-100/1K, ViT-B/L, ResNet-50, ConvNeXt-B, plus corruption/OOD tables. Clean accuracy barely moves (−0.28 pp on ViT-B/16; ≤0.30 on the conv nets). Ablations on path constraint, Ψ terms, and λ are present. Table II’s OOD/corruption columns are consistently favorable versus the listed baselines, and compute is honest (no second-order grads, time comparable to CGC).\n\nThe soft spot is real but not fatal. Stability, Insertion, Deletion, and LIMA PG/EPG are measured with the same search engine the loss is built to make equivariant, so those big deltas (0.14→0.27 stability, +52% Insertion, −62% Deletion) partly self-confirm. Grad-ECLIP transfer is mixed; ImageNet-1K transform gains are +0.55/+0.33 pp means with a couple of small regressions and no seeds/error bars. Code is “soon.” None of that kills the central idea—it just means the independent evidence for “decision process, not just LIMA surface” is the modest transform/OOD accuracy story, not the headline attribution numbers.\n\nMath is standard ranking/ReLU margins; citations cover the right prior art without weird gaps. For people working on explanation-guided training or transform robustness this is worth a careful read. I would send it to referees; they should demand engine-independent metrics, seeds, and code, not a rewrite of the method.","headline":"Solid methods paper: LIMA trajectories as self-supervision plus a ranking/truncation loss; flagship attribution gains are partly circular, independent robustness gains are real but small.","tokens_in":21004,"tokens_out":522,"would_cite":true,"duration_ms":11097,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Faithful region-search attributions can train vision models to keep using the same evidence after flips, rotations, and other label-preserving transforms.","keywords":["attribution regularization","attribution equivariance","submodular search","transformation robustness","explainable AI","faithfulness","geometric transformations"],"falsifier":"If after this training, models still flip their relied-on regions under the same geometric transforms while the search-based maps look more stable—or Insertion/Deletion and transformed-input accuracy fail to improve relative to gradient-map consistency baselines—the claim that faithful search supervision fixes evidence reliance would fail.","tokens_in":20711,"feed_emoji":"🔍","tokens_out":834,"duration_ms":20225,"temperature":0.7,"pith_summary":"Vision models often keep the right class label after a flip or rotation while silently switching which image regions they rely on. That shift is a sign of brittle evidence use. This paper argues that you can fix it only if the training signal truly reflects decision evidence, not a convenient but unfaithful heatmap. It builds an annotation-free loop: greedily search image regions by how they change the model’s output, keep only compact high-confidence sequences, map them to transformed views, and train with a ranking loss that forces the transformed search to pick matching regions in order and stop at the same point. On ImageNet-scale tests the method raises attribution stability and faithfulness sharply and modestly improves accuracy under geometric transforms, with only a small clean-accuracy cost. A sympathetic reader cares because it turns explanations into a practical way to harden what the model actually uses, not just how the maps look.","feed_headline":"Search-based attributions teach models steadier evidence under transforms","feed_subtitle":"Path-and-stop ranking lifts stability and transform accuracy with little clean drop","key_machinery":"Submodular Ranking Loss: a differentiable surrogate with a path-consistency (selection-ranking) term that ranks each transformed target region above remaining candidates at every search step, plus a termination-alignment (selection-truncation) term that pushes the terminal subset over the same consistency stop threshold—regularizing both final attributions and the discrete evidence-selection trajectory.","core_discovery":"When compact, class-discriminative region sequences from submodular output-based search supervise training, and a Submodular Ranking Loss aligns both the ordered selection path and the stopping step on transformed inputs, models learn more consistent decision evidence under label-preserving geometric transforms—improving attribution stability and faithfulness and modestly raising transformed-input accuracy while barely hurting clean accuracy.","pith_inferences":["Any post-hoc explainer that scores region subsets by output change could plug into the same two-stage query-and-rank loop.","If path alignment is doing real work, similar ranking losses might stabilize evidence under photometric or style shifts, not only geometry.","Deployment settings that already run multi-crop or flip test-time augmentation are natural places to harvest these paired trajectories at low extra label cost."],"forward_implications":["Attribution consistency objectives are only trustworthy when the attribution is decision-linked; gradient-map agreement alone need not improve transform robustness.","Self-supervised geometric equivariance of evidence can be trained without human relevance labels by recycling search trajectories as targets.","Regularizing the full selection path and stop condition beats matching only a final mask for both explanation quality and transform accuracy.","The same recipe transfers across ViTs and convolutional backbones with only small clean-accuracy cost on ImageNet-scale data."],"fun_headline_variants":["Submodular search teaches models steadier evidence under transforms","Path-and-stop ranking aligns attributions for transform-robust decisions","Output-based region search regularizes consistent evidence reliance","Search-derived supervision steadies attributions with little accuracy cost","Ranking loss ties selection paths so evidence holds under geometric shifts"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The filtered greedy region-search sequences are faithful enough teaching targets that matching their order and stop point on transformed images really rewires the model’s decision evidence, not just the search procedure.","fun_headline_variants_meta":{"raw":{"variants":["Submodular search teaches models steadier evidence under transforms","Path-and-stop ranking aligns attributions for transform-robust decisions","Output-based region search regularizes consistent evidence reliance","Search-derived supervision steadies attributions with little accuracy cost","Ranking loss ties selection paths so evidence holds under geometric shifts"]},"model":"grok-4.5","effort":"low","cost_usd":0.002819,"raw_usage":{"total_tokens":1093,"prompt_tokens":826,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":28188000,"prompt_tokens_details":{"text_tokens":826,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":184,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":826,"tokens_out":83,"duration_ms":5408,"temperature":1.0,"reasoning_tokens":184,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T10:55:30.775954+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If after this training, models still flip their relied-on regions under the same geometric transforms while the search-based maps look more stable—or Insertion/Deletion and transformed-input accuracy fail to improve relative to gradient-map consistency baselines—the claim that faithful search supervision fixes evidence reliance would fail.","supporting_citations":[],"review_version":1}