REVIEW 3 major objections 53 references
A fixed-size top-k evidence mask found by black-box search is enough for state-of-the-art faithful visual attribution and single-shot MLLM repair.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 22:38 UTC pith:M3CTJI7R
load-bearing objection Solid mask-first search paper with real empirical gains and honest cost accounting; the SOTA/repair claims are useful but over-precise given single-seed TRACE. the 3 major comments →
A Good Initialization is All You Need for Faithful Visual Attribution
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that a good fixed-cardinality fine-region evidence mask—found by TRACE’s cross-entropy search or seeded by COPAIR’s coarse pairs—is the right primary output for faithful visual attribution: it either stands alone as a compact, actionable mask or initializes Greedy/PhaseWin so that full insertion–deletion rankings set a new search-based state of the art, including single-point RePOPE repair rates of 94.44% and 96.00%.
What carries the argument
TRACE (Top-k Region Attribution via Cross-Entropy): a forward-only sampler that draws exactly-k fine-region masks with Gumbel-top-k, keeps elite high-scoring masks, and updates region logits toward those elites; the best mask is returned directly or partially released to warm-start Greedy/PhaseWin. COPAIR is the complementary deterministic coarse singleton/pair initializer derived from the PhaseWin–Greedy gap.
Load-bearing premise
The recovery argument assumes high-value evidence is concentrated in a small number of semantic region blocks whose score is mostly monotone when blocks are added, plus only a small residual from fine fragments.
What would settle it
On the same ImageNet-derived and RePOPE splits, replace TRACE’s elite cross-entropy updates with non-adaptive fixed-k random sampling under the same total forward budget; if insertion AUC, High, and single-point repair rates match TRACE, the claimed advantage of distribution updates and the block-basin recovery story fail.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reframes faithful visual attribution around a mask-first contract: return a compact exactly-k evidence set rather than only a full region ranking. It proposes two forward-only methods. COPAIR is a deterministic coarse singleton/pair initializer derived from a PhaseWin–Greedy gap diagnosis, with internal greedy ordering and partial release into Greedy or PhaseWin. TRACE directly searches fixed-cardinality fine-region masks via Gumbel-top-k cross-entropy sampling, elite retention, and logit updates, with a finite-budget recovery analysis under a PhaseWin-style block model. The same TRACE mask can be used as a standalone attribution/repair object or as an initializer for full-ordering search. Empirically, initialized search improves insertion/deletion faithfulness on ImageNet-derived splits with CLIP ViT-L/14, CLIP RN101, and ResNet-101 under inclusive MEC accounting; on POPE/RePOPE with Qwen2.5-VL-3B and LLaVA-v1.5-7B, TRACE+Greedy is the strongest search-based method, and direct TRACE masks report single-point RePOPE repair rates of 94.44% and 96.00%.
Significance. If the results hold under stronger statistical reporting, the contribution is solid and practically useful rather than purely incremental. The mask-first formulation matches real MLLM attribution/repair needs better than full-curve ranking alone, and the dual-use design (direct mask or initializer) is clean. Inclusive forward-call accounting, multi-backbone classification, MLLM decision and repair settings, ASN negative controls, and k/M/R ablations are strengths relative to typical saliency papers. The single-point RePOPE repair rates are the most actionable empirical result: they show compact evidence masks can be interventions, not only ranking prefixes. The recovery analysis is secondary and assumption-dependent, but the empirical protocol is careful enough that the paper would be a useful reference for black-box search-based attribution if the stochastic SOTA claims are better supported.
major comments (3)
- TRACE is stochastic (Gumbel-top-k sampling, elite updates; Table 7 fixes random seed 0; Limitations admits run-to-run variance), yet the central SOTA claims in Tables 1–4 and 13–17 are single-seed point estimates with no multi-seed means, standard deviations, or confidence intervals. Several headline deltas are small relative to that uncertainty (e.g., RePOPE High 0.9245→0.9273 for Qwen TRACE+G; CLIP RN101 Correct where COPAIR+G beats TRACE+G). The abstract’s “new state-of-the-art frontier” and precise repair rates (94.44%/96.00%) are therefore over-specified relative to the reported evidence. Multi-seed evaluation on at least the main TRACE rows (classification Correct/Cause/Repair and RePOPE High/repair rate), or a clear statement that all TRACE numbers are single-seed with seed 0, is needed before those claims can be accepted at face value.
- The finite-budget recovery analysis (Appendix A.4, Eqs. 23–29, 36–43) is presented as support for TRACE’s advantage, but it inherits PhaseWin block partition, block-level monotonicity of Φ, and bounded residual assumptions. The paper does not empirically validate that high-value evidence concentrates in one or two semantic blocks for the ImageNet/POPE proxies, nor does it report how often the certificate-mass conditions hold. This is acceptable as a conditional analysis, but the main text currently over-sells it as explaining TRACE’s gains. Either add a simple empirical check (e.g., fraction of TRACE elites covered by few coarse groups / object parts) or demote the analysis to a clearly conditional appendix note that does not underwrite the SOTA claim.
- The backbone-dependent winner pattern is under-discussed relative to the uniform SOTA language. On CLIP ViT-L/14 and ResNet-101, TRACE+G is strongest on the emphasized metrics (Tables 1–2), but on CLIP RN101 COPAIR+G wins Correct/Cause/Repair emphasized scores, and Appendix D.2 shows TRACE needs larger k to become competitive. The abstract and conclusion still present “initialized search methods” as establishing a single new frontier. Please qualify the claim by backbone/output contract: when coarse pairing helps (RN101), when fixed-k TRACE helps (ViT/MLLM/direct repair), and what default (k, initializer) a practitioner should use.
Circularity Check
No derivation-by-construction circularity; only minor non-load-bearing self-citation of the authors' PhaseWin block model in the TRACE recovery appendix.
specific steps
-
self citation load bearing
[Appendix A.4, Eqs. (23)–(29); also §3.2 COPAIR motivation]
"The structural assumptions used in this subsection are not new modeling claims of this paper. They are inherited from the extended PhaseWin analysis (Gu et al., 2026b), which interprets the hard tail of search-based visual attribution through partition-sensitive semantic evidence blocks. This inheritance includes the block-level monotonicity assumption used below... We derive our initial design strategy by contrasting the PhaseWin algorithm with the greedy approach. As detailed in (Gu et al., 2026b, Appendix), the greedy method's apparent advantage in handling the long tail can be reinterprete"
COPAIR's design story and TRACE's recovery bound both rest on the authors' own PhaseWin extended analysis (same first author and senior author). The block partition, monotone Φ, and residual Δ are imported rather than independently established here. This is self-citation of an unverified modeling abstraction, but it is not load-bearing for the SOTA/repair claims: those rest on external insertion/deletion and RePOPE measurements, and the paper does not use the PhaseWin citation as a uniqueness theorem that forces TRACE or forbids alternatives.
full rationale
The paper's central claims are empirical: initialized Greedy/PhaseWin improve insertion–deletion faithfulness under inclusive MEC, and direct TRACE masks achieve high single-point RePOPE repair. Those quantities are measured by external behavioral protocols (insertion/deletion replay, discrete repair success) that are not algebraically forced by TRACE's sampling objective. TRACE optimizes the same black-box sufficiency proxy F(S)=fy(x_S) used by prior search methods and returns the best observed k-mask under a finite budget; the leaderboard metrics evaluate full orderings or repair outcomes, not a quantity defined to equal the optimizer's score. COPAIR is a new coarse-pair initializer motivated by a PhaseWin–Greedy gap diagnosis, then validated on the same external tables—not a redefinition of that gap. Appendix A.4 honestly inherits PhaseWin's semantic-block model for a conditional finite-budget recovery bound; that analysis is interpretive scaffolding, not a uniqueness theorem that forbids alternatives or a fitted parameter renamed as a prediction. No self-definitional loop, fitted-input-as-prediction, uniqueness import, or renaming of a known result appears in the load-bearing chain. Score 1 only for the non-load-bearing PhaseWin self-citation in the recovery narrative.
Axiom & Free-Parameter Ledger
free parameters (5)
- TRACE mask size k =
8 (classification main); 10 (POPE/RePOPE)
- TRACE rounds R and samples M =
R=5, M=32
- TRACE elite ratio, update rate, temperature, smoothing =
ρe=0.2, η=0.7, T=1.0, λsm=0.05
- COPAIR coarse count and pair thresholds =
β-scale 2.0; δ=0.05; τ=0.1
- partial-release ratio γ =
0.8
axioms (5)
- domain assumption Faithfulness is operationalized by insertion/deletion of region masks and by answer-token or class-score changes under a fixed baseline image.
- domain assumption SLICO superpixels (64 regions) are adequate discrete units for evidence search.
- ad hoc to paper Early coarse-group value is dominated by singleton and pairwise terms, with higher-order remainder small for initializer candidates.
- domain assumption PhaseWin-style semantic blocks have a monotone dominant component Φ plus bounded residual Δ.
- ad hoc to paper Gumbel-top-k sampling with inclusion-moment logit updates is a valid practical surrogate for elite fixed-k search.
invented entities (2)
-
COPAIR initializer
no independent evidence
-
TRACE fixed-k cross-entropy mask search
no independent evidence
read the original abstract
Faithful visual attribution identifies which image regions support a model prediction. Search-based perturbation methods lead the insertion--deletion faithfulness frontier by masking regions and measuring score changes, but they usually output a complete ordering of all regions. Many applications, especially MLLM attribution and repair, only need a compact top-\(k\) evidence mask. We study this mask-first attribution problem. An exactly \(k\)-region mask is combinatorial: useful evidence can depend on interactions among fine regions. Coarse grouping can stabilize early search but aggregates redundant content, whereas one-step scoring can miss high-value combinations. We introduce two forward-only methods. \textsc{CoPAIR} uses a PhaseWin--Greedy gap diagnosis to construct coarse singleton/pair candidates that warm-start full-ordering search. \textsc{TRACE} directly searches fixed-cardinality fine-region masks with cross-entropy sampling, elite retention, and distribution updates, with a finite-budget recovery analysis. The resulting evidence set can be returned as a compact attribution mask or used to initialize Greedy or PhaseWin when a complete ranking is required. Across ImageNet classification with CLIP ViT-L/14, CLIP RN101, and ResNet-101, our initialized search methods establish a new state-of-the-art frontier for faithful full-ordering attribution under inclusive forward-call accounting. On POPE and RePOPE with Qwen2.5-VL-3B-Instruct and LLaVA-v1.5-7B, \textsc{TRACE}+Greedy gives the strongest search-based MLLM attribution results. Direct \textsc{TRACE} masks further achieve single-point RePOPE repair rates of \(94.44\%\) and \(96.00\%\), showing that compact evidence masks can be actionable attribution outputs, not merely prefixes of full rankings.
Figures
Reference graph
Works this paper leans on
-
[1]
Selvaraju, Ramprasaath R. and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , journal=IJCV, volume=
-
[2]
Axiomatic attribution for deep networks , author=
-
[3]
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , author=. ICLR Workshop , year=
-
[4]
Visualizing and Understanding Convolutional Networks , author=
-
[5]
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation , author=. PLOS ONE , volume=
-
[6]
Striving for Simplicity: The All Convolutional Net , author=. ICLR Workshop , year=
-
[7]
Smilkov, Daniel and Thorat, Nikhil and Kim, Been and Vi. 2017 , eprint=
work page 2017
-
[8]
Learning Deep Features for Discriminative Localization , author=
-
[9]
Chattopadhyay, Aditya and Sarkar, Anirban and Howlader, Prantik and Balasubramanian, Vineeth N , booktitle=
-
[10]
Wang, Haofan and Wang, Zifan and Du, Mengnan and Yang, Fan and Zhang, Zijian and Ding, Sirui and Mardziel, Piotr and Hu, Xia , booktitle=
-
[11]
Sanity Checks for Saliency Maps , author=
-
[12]
Xie, Weiyan and Li, Xiao-Hui and Cao, Caleb Chen and Zhang, Nevin L. , booktitle=
- [13]
- [14]
-
[15]
Khorram, Saeed and Lawson, Tyler and Li, Fuxin , booktitle=
-
[16]
Petsiuk, Vitali and Das, Abir and Saenko, Kate , booktitle=BMVC, year=
-
[17]
``Why Should I Trust You?'': Explaining the Predictions of Any Classifier , author=. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages=
-
[18]
Interpretable Explanations of Black Boxes by Meaningful Perturbation , author=
-
[19]
Understanding Deep Networks via Extremal Perturbations and Smooth Masks , author=
-
[20]
Black-Box Explanation of Object Detectors via Saliency Maps , author=
-
[21]
Making sense of dependence: Efficient black-box explanations using dependence measure , author=
-
[22]
Gradient-based Visual Explanation for Transformer-based
Zhao, Chenyang and Wang, Kun and Zeng, Xingyu and Zhao, Rui and Chan, Antoni B , booktitle=ICML, pages=. Gradient-based Visual Explanation for Transformer-based
-
[23]
A unified approach to interpreting model predictions , author=
-
[24]
Contributions to the Theory of Games , volume=
A value for n -person games , author=. Contributions to the Theory of Games , volume=
-
[25]
Kumar, I. Elizabeth and Venkatasubramanian, Suresh and Scheidegger, Carlos and Friedler, Sorelle , booktitle=ICML, pages=. Problems with
-
[26]
Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions
Gudovskiy, Denis and Hodgkinson, Alec and Yamaguchi, Takuya and Ishii, Yasunori and Tsukizawa, Sotaro , year=. Explain to Fix: A Framework to Interpret and Correct. 1811.08011 , archivePrefix=
work page internal anchor Pith review Pith/arXiv arXiv
-
[27]
Explain Any Concept: Segment Anything Meets Concept-Based Explanation , author=
-
[28]
Less is More: Fewer Interpretable Region via Submodular Subset Selection , author=
-
[29]
Interpreting Object-level Foundation Models via Visual Precision Search , author=
-
[30]
Gu, Zihan and Chen, Ruoyu and Zhang, Junchi and Liu, Li and Cao, Xiaochun and Zhang, Hua , booktitle=CVPR, year=
-
[31]
PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution
Gu, Zihan and Chen, Ruoyu and Zhang, Junchi and Liu, Li and Cao, Xiaochun and Zhang, Hua , year=. 2606.18008 , archivePrefix=
work page internal anchor Pith review Pith/arXiv arXiv
- [32]
-
[33]
Methodology and Computing in Applied Probability , volume=
The cross-entropy method for combinatorial and continuous optimization , author=. Methodology and Computing in Applied Probability , volume=
-
[34]
Stochastic beams and where to find them: The
Kool, Wouter and van Hoof, Herke and Welling, Max , booktitle=ICML, pages=. Stochastic beams and where to find them: The
-
[35]
Achanta, Radhakrishna and Shaji, Appu and Smith, Kevin and Lucchi, Aurelien and Fua, Pascal and S
-
[36]
Deep residual learning for image recognition , author=
-
[37]
An image is worth 16x16 words: Transformers for image recognition at scale , author=
-
[38]
Learning transferable visual models from natural language supervision , author=
-
[39]
Li, Junnan and Li, Dongxu and Savarese, Silvio and Hoi, Steven , booktitle=ICML, year=
-
[40]
Liu, Shilong and Zeng, Zhaoyang and Ren, Tianhe and Li, Feng and Zhang, Hao and Yang, Jie and Jiang, Qing and Li, Chunyuan and Yang, Jianwei and Su, Hang and Zhu, Jun and Zhang, Lei , booktitle=ECCV, year=
-
[41]
Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu , booktitle=CVPR, pages=
-
[42]
Visual Instruction Tuning , author=
-
[43]
Improved baselines with visual instruction tuning , author=
-
[44]
Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others , journal=
-
[45]
Bai, Shuai and others , year=. Qwen2.5-. 2502.13923 , archivePrefix=
work page internal anchor Pith review Pith/arXiv arXiv
-
[46]
From Redundancy to Relevance: Information Flow in
Zhang, Xiaofeng and Quan, Yihao and Shen, Chen and Yuan, Xiaosong and Yan, Shaotian and Xie, Liang and Wang, Wenxiao and Gu, Chaochen and Tang, Hao and Ye, Jieping , booktitle=NAACL, year=. From Redundancy to Relevance: Information Flow in
-
[47]
Zhang, Jiarui and Khayatkhoei, Mahyar and Chhikara, Prateek and Ilievski, Filip , year=
-
[48]
Token Activation Map to Visually Explain Multimodal
Li, Yi and Wang, Hualiang and Ding, Xinpeng and Wang, Haonan and Li, Xiaomeng , booktitle=ICCV, year=. Token Activation Map to Visually Explain Multimodal
-
[49]
Where do Large Vision-Language Models Look at when Answering Questions? , author=. 2025 , eprint=
work page 2025
-
[50]
Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li , booktitle=CVPR, pages=
-
[51]
Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll
-
[52]
Evaluating object hallucination in large vision-language models , author=
-
[53]
RePOPE: Impact of Annotation Errors on the POPE Benchmark
Neuhaus, Yannic and Hein, Matthias , year=. 2504.15707 , archivePrefix=
work page internal anchor Pith review Pith/arXiv arXiv
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.