{"id":"bb7dbd91-7468-44ae-bb56-74891f0e6f98","arxiv_id":"2509.22415","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MSEA+ARC, a multi-scale and ranking-based residualization method, claims consistent F1-IoU gains over TAM for token-level MLLM visual attribution.","lead":"This paper presents two techniques to improve visual-attribution maps for multimodal language models: averaging logit-lens attributions over several image scales, and subtracting an estimated noise component from preceding text tokens. On three MLLM families it reports higher F1-IoU than the previous best method, but the manuscript has inconsistencies and the evaluation has methodological caveats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ARC residualization assumes \\hat{A}_t is pure noise; if it overlaps the target's true mask, Eq. 9 removes real evidence and the F1-IoU gain may reflect noise suppression rather than improved fidelity.","rationale":"The reader's weakest_assumption identifies the linear-additive noise model in §3.3, and I agree it is the most load-bearing condition for the central claim. The paper's quantitative claims are extensive and the multi-scale aggregation (MSEA) is well-motivated, but ARC's residualization is not independently validated: there is no evidence that the aggregated map \\hat{A}_t is free of target-related signal. The per-token least-squares fit of β in Eq. 9 ensures that the component of A_t aligned with \\hat{A}_t is always removed; if \\hat{A}_t overlaps the target's mask, that removal is evidence loss. The ablation shows ARC increases Func-IoU by ~48 points, which could result from lowering diffuse activations without any ranking-based selectivity. The proposed overlap test would directly measure whether the subtracted map is orthogonal to the target evidence. Without such a diagnostic, the claim that ARC yields 'more faithful' maps is not fully supported. I do not reject the numerical results, but they remain conditional on validating the mechanism. The unused A0 and unspecified RBO p are secondary completeness issues that reinforce the conditional status. The verdict should remain CONDITIONAL, i.e., UNCHANGED relative to the reader, until the test is run or equivalent evidence is provided.","tokens_in":12351,"tokens_out":12094,"duration_ms":93450,"concrete_test":"Sample 500 target noun tokens from the COCO Caption minival for Qwen2-VL-2B. For each, compute \\hat{A}_t (Eq. 8) and the ground-truth mask of T_t, then compute the mean IoU of the top-50% activation region of \\hat{A}_t against that mask. If the mean IoU is below 5%, the additive-noise assumption is plausible; if above, Eq. 9 removes real target evidence, and the reported F1-IoU improvement must be re-interpreted as a trade-off rather than a pure fidelity gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ARC's residualization improves attribution fidelity rests on the untested additive-noise assumption in §3.3: preceding-token interference is modeled as a weighted sum of individual attribution maps (Eq. 8), with weights (1−r_j) from RBO, and the component along \\hat{A}_t is then subtracted in Eq. 9 via a least-squares fit β. This removes the projection of A_t onto \\hat{A}_t, so any overlap between \\hat{A}_t and the true evidence region for T_t is subtracted away. No diagnostic shows that \\hat{A}_t is uncorrelated with the target's ground-truth mask. The ablation's large Func-IoU gains are consistent with either genuine noise suppression or with a generic reduction of diffuse activations; the two are not distinguished. The defined-but-unused A0 preceding Eq. 9 and the missing RBO persistence parameter are symptoms that the mechanism is only partially specified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two post-hoc modifications to logit-lens visual attribution in multimodal LLMs. MSEA computes attribution maps at several input scales and fuses them, aiming to reduce fragmentation caused by the limited receptive field of a single visual token. ARC models interference from preceding text tokens by computing RBO rank similarity between top-k logit rankings, building a weighted map of preceding-token attributions, and subtracting a fitted component of this map from the current token's map. The method is evaluated on Qwen2-VL, LLaVA-1.5, and InternVL2.5 over COCO Caption, GranDf, and OpenPSG, using Obj-IoU, Func-IoU, and their harmonic mean F1-IoU. The main numerical claims are consistent improvements over TAM, e.g., F1-IoU rising from 39.10 to 44.45 on COCO Caption for Qwen2-VL-2B.","tokens_in":12640,"tokens_out":4103,"duration_ms":32380,"significance":"If the claims hold, the paper would offer a practical, training-free refinement for token-level visual attribution in MLLMs, with convincing breadth across architectures and scales. The strengths are the clear motivation, the wide empirical coverage (seven models, three datasets), and the fact that the method is purely post-hoc. However, the central improvement is carried by the ARC residualization step, whose underlying model is not independently validated, and by a metric (F1-IoU) that is tightly aligned with what ARC is designed to suppress. The absence of error bars and the selection of hyperparameters on the evaluation sets weaken the empirical backing. These issues are fixable but currently leave the main claim less secure than the tables suggest.","major_comments":[{"comment":"The ARC mechanism rests on an unvalidated additive-noise assumption: preceding-token interference is modeled as a weighted sum of individual attribution maps, with weights (1−r_j), and the component of A_t along \\hat{A}_t is then removed by the least-squares fit β. If \\hat{A}_t overlaps the true visual evidence for T_t — which is likely when preceding tokens refer to the same object or scene — Eq. (9) subtracts real evidence. The paper provides no diagnostic showing that \\hat{A}_t is uncorrelated with the target ground-truth mask. I ask for a controlled experiment that distinguishes genuine noise suppression from a generic reduction of diffuse activations: e.g., compare against subtracting a random or constant map matched in total activation, and report Obj-IoU on target tokens as well as Func-IoU.","section":"§3.3, Eq. (8)–(9)"},{"comment":"Func-IoU is defined only verbally as 'activations on non-semantic tokens (e.g., punctuation), where higher values indicate fewer false positives.' No formula is given, and no specification of which tokens are treated as non-semantic is provided. Since Func-IoU contributes to the primary metric F1-IoU, and since ARC is explicitly designed to suppress diffuse activations on non-semantic tokens, the reported gains in Func-IoU may be partly by construction. The definition must be made precise, and the authors should report whether the improvement persists when non-semantic tokens are excluded or when a metric less aligned with the method's objective is used.","section":"§4.1, Eq. (10)"},{"comment":"The hyperparameters appear to be selected on the same evaluation sets used for the headline results. Figure 3 reports that four scales, e.g., [0.5,0.75,1.0,1.25], yield peak performance, yet the main configuration in §4.1 and Table 1 uses three scales [0.5,0.75,1.0]. No error bars, variance estimates, or significance tests are reported. The authors should either justify the choice of three scales, or adopt the empirically better four-scale setting; in either case, reporting multiple random/order replicates with confidence intervals would make the improvements credible.","section":"§4.2, Fig. 3 and Table 1"},{"comment":"The ARC algorithm is incompletely specified. A base attribution A_0 is defined just before Eq. (9) as the raw attribution for the vocabulary token with minimal visual activation, but it never appears in the refinement formula. The RBO metric in Eq. (7) requires a persistence parameter, which is not stated. It is also unclear whether the maps A_j are normalized before the weighted sum, and whether β in Eq. (9) is estimated per token or globally. These details are necessary to reproduce the method and to understand what exactly is subtracted.","section":"§3.3, Eq. (8)"},{"comment":"The ablation table is unreadable as printed. The header row mixes method labels ('Mean', 'Max', 'ARC', 'TAM') with the metric names, and the body rows do not clearly identify which configuration each row corresponds to. The parenthetical deltas such as '+10.21' and '+12.88' are not tied to a baseline. Since the claim that MSEA and ARC are complementary rests entirely on this table, it must be reformatted with explicit row and column labels.","section":"Table 3"}],"minor_comments":[{"comment":"The paper title and submitted abstract describe 'Evidence Recomposition' and 'Predictive Context Residualization' (ERCR/PCR), but the method sections and experiments present MSEA and ARC. This inconsistency suggests a version-control problem and must be fixed.","section":"Title/Abstract vs. §3"},{"comment":"The caption says 'MSEA (a) and SAC (b),' but the method is called ARC. Please correct the acronym.","section":"Figure 1 caption"},{"comment":"The resize/aggregation step in Eq. (5) uses 1/α_s, but for models with fixed input size the padded image in Eq. (3) has already been embedded in a canvas of size H×W. It should be clarified how the padded regions are handled during resizing so that spatial coordinates remain consistent.","section":"§3.2, Eq. (5)"},{"comment":"The notation G(⌊A_t − β\\hat{A}_t⌋_+) is confusing: the floor symbol is nonstandard for a ReLU-like operation, and 'Rank Gaussian Filter' G is never formally defined. Please define the post-processing and the positive-part operation explicitly.","section":"§3.3, Eq. (9)"},{"comment":"The baseline 'Archi.-Surgery 2025b' has no corresponding full reference in the bibliography, and the year/venue is unclear. Please add the missing reference.","section":"Table 1"},{"comment":"There are numerous spacing and typographical issues in the tables (e.g., 'Obj-IoUFunc-IoUF1-IoU') and in the text ('LLaV A-1.5' vs. 'LLaVA'). A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially useful, but the central empirical claim currently rests on an incompletely specified residualization step and a metric that is closely tied to the method's design. I would also flag that the abstract/title mismatch raises a concern about the manuscript's readiness. If the authors can provide diagnostics separating genuine evidence improvement from systematic suppression, and add statistical rigor, the paper could become suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick read on arXiv:2509.22415. The paper's real content is MSEA (multi-scale aggregation of logit-lens attribution maps) plus ARC (weighting preceding-token attribution maps by RBO rank alignment and subtracting a fitted component). Multi-scale ensembling is old hat from CAM literature, and context residualization is the core of TAM. The genuinely new bit is using RBO rank similarity to weight which preceding tokens to subtract, instead of raw logit scores. That is a small but defensible idea, and the paper deserves credit for testing it across three model families and three benchmarks with consistent F1-IoU gains.\n\nThe main soft spot is exactly what the stress-test flagged: ARC's additive-noise assumption is unvalidated. Equation 9 subtracts the projection of the target map onto the weighted preceding-token map. If that map overlaps the true evidence region, you remove real signal. No diagnostic shows this doesn't happen. The Func-IoU gains are consistent with genuine noise suppression, but also with just shrinking diffuse activations. That is a real ambiguity, not a manufactured one. Also, the defined-but-unused A0 before Eq. 9 and the missing RBO persistence parameter say the mechanism is under-specified. The hyperparameter selection is another issue: Figure 3 shows four scales peak, yet the default is three, and there are no error bars or significance tests. Since scales and k were tuned on the same evaluation sets, the reported gains are probably optimistic. The abstract/manuscript name mismatch (ERCR vs MSEA/ARC) is sloppy but minor.\n\nAll that said, the central claim is not circular or dishonest. The method is a post-hoc refinement, and the improvement direction is plausible. I don't think the quantitative strengths hold up at face value, but the idea is worth refereeing. A good referee could push for validation of the additive-noise model and a more careful selection protocol. I'd bring this to a reading group as a useful example of incremental interpretability work, and I'd likely cite it for the RBO trick if I were working on MLLM attribution. For peer review: yes, send it out. It is not desk-reject material; it needs a revision with additional diagnostics.","headline":"A plausible incremental refinement for token-level attribution, with consistent gains but tuning-selection and mechanism-specification issues that likely inflate the headline numbers.","tokens_in":13091,"tokens_out":978,"would_cite":true,"duration_ms":10234,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the two main sources of noise in token-level visual attributions for multimodal LLMs—fragmented patches and contamination from preceding text tokens—can be corrected post-hoc by multi-scale evidence fusion and rank-bas","keywords":["visual attribution","multimodal large language models","logit lens","multi-scale aggregation","token interaction","interpretability","Rank-Biased Overlap","context residualization"],"falsifier":"Take a fixed image and a generated caption, compute ERCR's residualized map, then recompute it with the caption's context tokens shuffled or deleted before the target token. If the map barely changes, the residualization is not needed. Also compare a version of ARC with permuted (random) RBO weights: if F1-IoU does not drop back to the un-residualized baseline, then the ranking signal is not the operative part of the method.","tokens_in":12248,"feed_emoji":"🔍","tokens_out":5662,"duration_ms":90110,"temperature":0.7,"pith_summary":"This paper tries to establish that visual attribution maps for multimodal large language models are noisy for two specific, fixable reasons: each visual token is decoded in isolation, losing spatial context, and the autoregressive context inherits spurious activations from preceding tokens. It introduces ERCR, which aggregates logit-lens maps over several input resolutions (MSEA) and subtracts a fitted combination of preceding-token maps weighted by rank-biased overlap (ARC). On Qwen2-VL-2B it reports F1-IoU rising from 39.10 to 44.45 on COCO Caption and from 30.83 to 37.20 on GranDf, with consistent gains across LLaVA, Qwen2-VL, and InternVL variants of 2B–13B. A sympathetic reader would care because the method is training-free and post-hoc, so it can be dropped into existing models without retraining.","feed_headline":"MLLM attribution gains 5.35 points via two post-hoc fixes","feed_subtitle":"ERCR fuses multi-scale maps and subtracts interfering context; consistent gains across seven models.","key_machinery":"Two complementary mechanisms: Multi-Scale Explanation Aggregation (MSEA), which resizes the input image, re-tokenizes it, reads out logit-lens scores at each scale, and fuses the rescaled maps by averaging; and Activation Ranking Correlation (ARC), which defines a context token's interference weight as one minus the Rank-Biased Overlap between its top-50 predicted token rankings and the target token's rankings, forms a context map by weighting each preceding token's attribution map accordingly, and subtracts a fitted multiple (β) of that map from the target map before positive clipping and a rank Gaussian filter. MSEA supplies spatial context; ARC suppresses context-token interference.","core_discovery":"The central claim is that existing logit-lens attribution—decoding each visual token's hidden state through the unembedding layer—fails in two systematic ways: token-level decoding ignores spatial context, and the target token's activation is contaminated by preceding text tokens. ERCR addresses the first by re-reading the image at several scales and averaging the resulting attribution maps, giving each visual location a dynamic receptive field; it addresses the second by computing a context map from the preceding tokens' own attribution maps, weighting each such map by (1−r_j) where r_j is the Rank-Biased Overlap between the top-k token predictions of that context token and the target, then","pith_inferences":["The core hypothesis—that preceding-token interference enters additively and can be estimated from the preceding tokens' own attribution maps—could be tested directly by deleting or permuting context tokens and checking whether the residualized map changes exactly as ARC predicts; this would separate the mechanism from the aggregation gain.","Because ARC only uses output logits and top-k rankings, the same residualization idea could transfer to other autoregressive decoders (e.g., text-only or speech models), where the 'visual tokens' would be replaced by any parallel input stream.","The paper defines a base attribution A0 but never uses it in the final formula, so the residualization as written is under-specified; reproducing the exact algorithm may require additional assumptions about which tokens are treated as the irrelevant set.","A synthetic test with one controlled known noise source (e.g., injecting a fixed activation pattern from a distractor token) would tell whether the RBO weights are identifying the true interferer or merely fitting noise; if random weights work as well, the ranking information is not the active ingredient."],"forward_implications":["F1-IoU improves by 5.35 points on COCO Caption and 6.37 points on GranDf for Qwen2-VL-2B relative to the prior logit-lens method, driven mainly by a 23-point reduction in functional false positives.","The method is post-hoc and training-free, so it can be applied to any off-the-shelf MLLM without altering weights or requiring ground-truth masks at inference time.","Across seven model variants, gains are consistent and tend to grow with model scale (from 11.53 to 14.52 percentage points in InternVL's F1-IoU), suggesting the technique leverages robust statistical properties of learned representations.","MSEA and ARC are complementary: the multi-scale fusion raises object-level IoU while the ranking-based residualization suppresses noise on non-semantic tokens, so using both gives more than either alone."],"fun_headline_variants":["ERCR fixes two visual-attribution flaws, lifting F1-IoU by 5.35","New post-hoc method sharpens MLLM visual evidence, beats logit lens","Two simple fixes give multimodal models cleaner token-level attention","5.35-point jump: ERCR recomposes evidence and strips token interference"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that interference from preceding text tokens contaminates the current token's attribution additively, as a weighted sum of those tokens' own attribution maps with weights given by rank-biased overlap, and that subtracting a single fitted multiple of this sum removes noise without removing real evidence.","fun_headline_variants_meta":{"raw":{"variants":["ERCR fixes two visual-attribution flaws, lifting F1-IoU by 5.35","New post-hoc method sharpens MLLM visual evidence, beats logit lens","Two simple fixes give multimodal models cleaner token-level attention","5.35-point jump: ERCR recomposes evidence and strips token interference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000577,"raw_usage":{"total_tokens":2589,"prompt_tokens":807,"completion_tokens":1782,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1698}},"tokens_in":551,"tokens_out":1782,"duration_ms":91958,"temperature":1.0,"reasoning_tokens":1698,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T14:52:15.454535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed image and a generated caption, compute ERCR's residualized map, then recompute it with the caption's context tokens shuffled or deleted before the target token. If the map barely changes, the residualization is not needed. Also compare a version of ARC with permuted (random) RBO weights: if F1-IoU does not drop back to the un-residualized baseline, then the ranking signal is not the operative part of the method.","supporting_citations":[],"review_version":1}