{"id":"c4960119-6ffa-4873-abf6-53b4b44f388f","arxiv_id":"2607.25467","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Current attention is not a safe-forgetting signal for visual KV memory; damage from eviction concentrates in visually dependent turns, and only explicitly verbalized facts are reliably rescued by assistant text.","lead":"This paper tests when a multimodal AI assistant can safely delete stored image memory during a conversation. It finds that attention to the current question is a poor guide to what will be needed later, and only facts the assistant already said are reliably recoverable from text memory once the image is gone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Drop as 'unreadable positions' may not mimic physical token removal; the negative attention–utility correlation could be an artifact of the intervention.","rationale":"The reader’s weakest assumption—that teacher-forced NLL on a frozen trajectory under 'drop' interventions faithfully captures real losses—is well founded. My stress-test sharpens this: the specific 'drop' operationalization (unreadable positions with fixed RoPE indices) is not equivalent to physical token removal, and the paper’s own position-contamination analysis makes this gap concretely threatening. However, the paper already partially mitigates the proxy concern with free-running exact-match checks on the high-dependence quartile, and the central claim is also supported by the controlled fact experiment showing substitution only for stated facts—an independent, less metric-dependent result. The conditional verdict is therefore appropriate; my concern reinforces the need for the promised artifact release and an additional operationalization check, but it does not move the verdict to reject or to a stronger condition. I agree with the reader that the primary metric is the weakest point, and I recommend UNCHANGED.","tokens_in":12460,"tokens_out":5487,"duration_ms":59921,"concrete_test":"On a random 100-dialog subset of the frozen 200-dialog regional audit, recompute the 4×4 attention–future-utility Spearman and the 25/50/75% joint-retention regret under an alternative intervention that physically removes selected visual tokens and re-indexes the remaining positions/RoPE (with teacher forcing retained), instead of masking positions as unreadable. If the correlation stays negative and regret remains worse than random, the drop operationalization is validated; if the correlation flips or regret reverses, the primary proxy-failure claim is intervention-specific and the practical conclusion is not supported by the regional audit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central proxy-failure result—attention–future-utility Spearman −0.103 and joint-retention regret worse than seeded random (Section 3, Figure 2)—is computed under a specific 'drop' intervention defined in Section 2: 'Drop makes the selected cache positions unreadable, leaving cache length, positions, and RoPE indices unchanged.' However, deployed visual-KV eviction physically removes tokens, which shifts subsequent positions and RoPE indices. The paper’s causal estimand is therefore for a masked-cache trajectory, not for the actual deletion operation that the practical conclusion ('compressors should not treat low current attention as sufficient for irreversible deletion') targets. Because attention patterns are position-sensitive (the paper itself shows position contamination in Figure 2c,d), making a high-attention position unreadable while keeping its positional index could produce out-of-distribution inputs that inflate NLL for reasons unrelated to the semantic utility of the discarded token. The marginal-utility and random baselines are computed under the same masking, so the headroom comparison is internally consistent but may not transfer to real eviction. The free-running exact-match check (Figure 4e,f) uses existing selectors that physically alter the cache, but it does not test the regional attention ranking that underpins the headline correlation. Thus the single most load-bearing assumption is that 'unreadable-position masking' is faithful to physical deletion; if that fails, the quantitative proxy-failure evidence could reverse.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CVMA, a paired single-prefill causal audit framework for deciding when visual KV cache entries can be safely evicted in multi-turn multimodal dialogs. It defines four intervention-based estimands: regional future utility (Eq. 3), whole-image visual dependence (Eq. 4), retention regret against seeded random and a marginal-utility control (Eq. 5), and image–text factorial substitution (Eq. 6). On Qwen2.5-VL-7B and Idefics3-8B, it reports that current attention is negatively correlated with future regional utility (Spearman −0.103 at the primary 4×4 grain), that attention-based joint retention is worse than seeded random, and that apparent safety arises only when later turns do not need the image or when the needed fact was explicitly verbalized in assistant text. The paper also includes full-dataset comparisons of five compressors, a matched non-deletion coverage control across bit budgets, and a free-running exact-match bridge on high-dependence dialogs.","tokens_in":12749,"tokens_out":6557,"duration_ms":73629,"significance":"If the central claims hold, this is a valuable methodological contribution: it converts a vague concern about unknown future queries into falsifiable, paired causal estimands, and it provides a clean separation between proxy validity, dependence-conditioned damage, and image–text substitution. The strengths are substantial: a single-prefill paired design with runtime invariants, bootstrap CIs, cross-model replication, explicit scope bounding, an executable audit manifest, and a free-running exact-match check that partially bridges teacher-forced NLL to observable generation quality. The paper does not overclaim a new compression policy; its practical recommendation—that low current attention is not a certificate for irreversible visual-KV deletion—is important and, if confirmed, should influence how multimodal KV eviction is evaluated.","major_comments":[{"comment":"The central proxy-failure estimates define 'Drop' as making selected cache positions unreadable while keeping cache length, positions, and RoPE indices unchanged. Physical KV eviction removes tokens and shifts positions/RoPE. Because the paper itself shows that top-attention positions are position-contaminated (Fig. 2c,d), the negative attention–future-utility Spearman (−0.103 at 4×4) and the 'worse than seeded random' retention regret could be artifacts of feeding an out-of-distribution masked cache rather than evidence about real deletion. The free-running exact-match check (Fig. 4e,f) uses deployed selectors with their own rankings, so it does not validate the regional attention ranking under physical deletion. I ask for a direct comparison between unreadable-position masking and actual token removal/re-indexing for the same selected regions, or alternatively a clear restriction of th","section":"Section 2, Drop definition; Section 3, Fig. 2"},{"comment":"The headline correlation and the joint-retention comparisons are computed only on 200 frozen Qwen-7B dialogs, after excluding 72 pilot/confirmation IDs and freezing the first 200 after a single shuffle. Since this subset is the sole basis for the regional proxy-failure claim, the paper should demonstrate representativeness with respect to the full 2,064-dialog validation set—for example, compare all-drop dependence Dd, question type mix, and answer-length distributions—or repeat the correlation on additional random subsets. Without this, selection bias from the exclusion or the arbitrary freeze cannot be fully ruled out.","section":"Section 3, first paragraph"}],"minor_comments":[{"comment":"The definition 'wt = |H| −1' appears to be a typesetting error for 1/|H|. The current reading would assign weight 207 to every turn for a ten-turn dialog, which contradicts the text. Please correct.","section":"Eq. (2)"},{"comment":"The caption contains a garbled numeric block ('76 3 · 2 · 4 2 · ...') that looks like a table artifact. Please reformat or move to a table.","section":"Figure 2 caption"},{"comment":"The exclusion of 72 pilot/confirmation IDs is mentioned only briefly. Even if the remaining 200 are frozen, please clarify whether the 72 were excluded for reasons that could correlate with dialog properties, and report the size of the pool from which the 200 were drawn.","section":"Section 2, sample description"},{"comment":"The statement '25.3% have negative Dd' is useful but could be clarified as the fraction of dialogs where all-image-drop improves NLL. Also, consider reporting a histogram or quantile table beyond decile means, since the decile plot compresses a wide range.","section":"Section 4, dependence reporting"},{"comment":"The robustness variants (last-four, last-layer, mass-weighted) are presented only for the 4×4 grain in Fig. 4a,b. Please state explicitly whether the 8×8 and 2×2 grains were also checked under these aggregations, or note in the text that they were not.","section":"Section 3, aggregation robustness"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically strong and the empirical design is unusually careful, with paired interventions, bootstrap CIs, cross-model replication, and a free-running bridge. The main risk is the gap between the unreadable-position masking intervention and physical token deletion; this gap sits exactly on the practical conclusion about 'irreversible deletion.' If the authors can add a regional validation of masking versus physical deletion—or clearly narrow the claim—I would be willing to support acceptance. The representativeness of the 200-dialog regional sample is a second, smaller concern. The work is within scope for a serious CV/ML venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper does something genuinely useful: it stops treating aggregate NLL as the only outcome and decomposes “safe forgetting” into three testable questions — does current attention predict future utility, when does a wrong eviction actually hurt, and can assistant text substitute for image memory. The CVMA framework is the real contribution: paired single-prefill interventions with causal estimands, image–text factorial tests, and explicit scope bounding. On the evidence here, the central claim holds up: attention–future-utility Spearman around −0.1 at the primary grain, attention-based retention worse than seeded random, while a same-dialog marginal-utility oracle shows selection headroom. The dependence-conditioned results (Q1 vs. Q4, turn progression) explain why aggregate numbers look benign. And the fact-level substitution result — stated facts transfer to text KV, unstated ones don't — is a clean, useful boundary. The paper is honest about its limits: teacher-forced NLL as primary metric, limited model families, no universal claims.\n\nSoft spots: the biggest is the gap between the “drop” intervention and physical eviction. Making positions unreadable while keeping RoPE indices fixed is not the same as removing tokens, and position-sensitive attention patterns could make the masked state OOD in a way that biases the measured utility. The free-running exact-match check on the high-dependence quartile helps, but it tests existing selectors, not the regional ranking at the heart of the proxy-failure claim. So the quantitative regret numbers should be read as internal to the masking definition. This is a real but bounded limitation — the paper explicitly states the estimand is for the frozen trajectory. Also, no code or data released, so the audit manifest is currently unverifiable.\n\nThe citation pattern looks fine; related work is recent and relevant. Self-citations are not excessive. Statistical treatment is careful (bootstrap CIs, hierarchical intervals). The 200-dialog regional audit with 72 pilot IDs excluded is a bit odd, but the paper discloses it and the robustness checks are broad.\n\nVerdict: this deserves a serious referee. The masking-vs-deletion concern should be addressed either by a rebuttal or by an extra experiment (e.g., actually shifting positions after deletion), but the paper is not broken. I'd take it to a reading group and would cite it if I worked on multimodal KV cache compression.","headline":"A careful causal audit showing that current attention is a poor proxy for future visual-KV utility — solid, bounded, and worth refereeing, with a real but nonfatal gap between masking and physical eviction.","tokens_in":13235,"tokens_out":3415,"would_cite":true,"duration_ms":36485,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that in stateful vision-language assistants, low current attention is not a valid certificate for irreversibly deleting visual memory, and that safe forgetting requires either low future visual dependence or explicit verba","keywords":["visual KV cache","attention-guided eviction","causal audit","safe forgetting","multimodal dialog","teacher-forced NLL","visual memory substitution","verbalization"],"falsifier":"Free-run the audited dialogs without teacher forcing on the high-dependence subset: if attention-based retention matches full-cache answer accuracy within noise on the very dialogs where the all-image drop causes the largest likelihood jump, the claim that low attention is unsafe for irreversible deletion would be contradicted. Alternatively, compute the attention–future-utility correlation using future-question-conditional relevance labels; a positive correlation would indicate the negative association is an artifact of anchoring to the first question.","tokens_in":12356,"feed_emoji":"🖼️","tokens_out":5276,"duration_ms":52260,"temperature":0.7,"pith_summary":"This paper asks when a vision-language assistant is actually safe to forget a seen image during a long conversation. It argues that current attention scores, which many cache-compression methods use to decide which visual regions to evict, are a mis-specified proxy for future usefulness: on held dialog data, attention ranks future-useful visual regions worse than random, and keeping regions by attention hurts later answers more than keeping them by chance. The paper does not stop at proxy failure; it shows that the damage only surfaces when later turns truly need the discarded evidence, and that assistant-written text can substitute for image memory only for facts that were explicitly stated, not for facts merely seen. The practical upshot is that safe forgetting should be decided by measured future visual dependence or fact-level verbalization, not by current attention. The contribution is a reusable paired-intervention methodology, the Causal Visual Memory Audit (CVMA), that separates these explanations within a single dialog trajectory.","feed_headline":"Attention ranks future-useful visual regions worse than random","feed_subtitle":"Forgetting by attention loses facts later turns need; only stated facts stay recoverable.","key_machinery":"The load-bearing object is the Causal Visual Memory Audit (CVMA), a paired single-prefill framework that treats each dialog as one persistent memory trajectory and applies controlled 'drop' interventions to the key-value cache. Its three linked estimators are: regional future utility (the causal target attention is supposed to predict, defined as the likelihood change from removing a region), all-image-drop dependence (the likelihood change from removing the whole image span, used as an effect modifier), and a factorial image–text interaction that asks whether completed assistant-output KV can substitute for image KV. A marginal-utility control ranks regions by their single-drop utility to w","core_discovery":"The central discovery is that 'what the model is looking at now' does not license 'what it can forget safely later'. Under CVMA's paired single-prefill protocol, regional attention and future causal utility are negatively correlated (Spearman −0.103 at 4×4 resolution on the primary model set), attention-based joint retention is worse than seeded random retention at 25/50/75% budgets, and a same-dialog marginal-utility control shows selection headroom exists. The failure is masked in aggregate because many dialogs do not need vision later; when conditioned on an all-image-drop dependence measure, damage concentrates in high-dependence dialogs and later turns. A factorial image–text interventi","pith_inferences":["The same mis-specification likely applies to text-only long-context eviction, where attention is used to predict future relevance; CVMA's intervention logic could be transplanted to audit that setting.","A practical extension would be a cheap pre-estimate of future visual dependence (e.g., from question type or dialog structure) to gate when eviction is allowed.","Because the marginal-utility control shows headroom, a selector trained on paired drop-out utility labels rather than attention might substantially close the gap to full cache."],"forward_implications":["Compression policies for multimodal assistants should stop treating low current attention as sufficient evidence for irreversible visual-token deletion.","Aggregate evaluation scores can hide real loss; reports should condition on visual dependence (all-image-drop effect) and on which facts were verbalized.","The measured damage from eviction rises sharply on later turns for dialogs that depend on vision, so an early one-shot compression decision is the risky point.","Assistant-text KV provides a second memory route, but only for facts explicitly stated; unsaid facts are not reliably recoverable after image KV is gone.","Non-deletion compression (low-bit coverage) stays close to the full cache only while its representation remains decodable; below a precision floor even 'non-deletion' collapses."],"fun_headline_variants":["Attention misranks visual facts needed later","Safe to forget? Attention says no, future says yes","Current attention worse than random for visual memory","Forgetting by attention loses future visual recall","Visual memory audit: attention fails future turns"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The quantitative regret numbers rest on teacher-forced next-token likelihood under a 'drop' that makes cache positions unreadable while keeping positions and RoPE indices fixed being a faithful proxy for what real free-running deployments lose when tokens are truly removed.","fun_headline_variants_meta":{"raw":{"variants":["Attention misranks visual facts needed later","Safe to forget? Attention says no, future says yes","Current attention worse than random for visual memory","Forgetting by attention loses future visual recall","Visual memory audit: attention fails future turns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00014,"raw_usage":{"total_tokens":973,"prompt_tokens":698,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":442,"tokens_out":275,"duration_ms":3997,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:19:02.173110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Free-run the audited dialogs without teacher forcing on the high-dependence subset: if attention-based retention matches full-cache answer accuracy within noise on the very dialogs where the all-image drop causes the largest likelihood jump, the claim that low attention is unsafe for irreversible deletion would be contradicted. Alternatively, compute the attention–future-utility correlation using future-question-conditional relevance labels; a positive correlation would indicate the negative association is an artifact of anchoring to the first question.","supporting_citations":[],"review_version":1}