Pith. sign in

REVIEW 2 major objections 5 minor 35 references

This paper claims that in stateful vision-language assistants, low current attention is not a valid certificate for irreversibly deleting visual memory, and that safe forgetting requires either low future visual dependence or explicit verba

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 02:19 UTC pith:DRYLRA5H

load-bearing objection A careful causal audit showing that current attention is a poor proxy for future visual-KV utility — solid, bounded, and worth refereeing, with a real but nonfatal gap between masking and physical eviction. the 2 major comments →

arxiv 2607.25467 v1 pith:DRYLRA5H submitted 2026-07-28 cs.CV cs.AI

Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns

classification cs.CV cs.AI
keywords visual KV cacheattention-guided evictioncausal auditsafe forgettingmultimodal dialogteacher-forced NLLvisual memory substitutionverbalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks when a vision-language assistant is actually safe to forget a seen image during a long conversation. It argues that current attention scores, which many cache-compression methods use to decide which visual regions to evict, are a mis-specified proxy for future usefulness: on held dialog data, attention ranks future-useful visual regions worse than random, and keeping regions by attention hurts later answers more than keeping them by chance. The paper does not stop at proxy failure; it shows that the damage only surfaces when later turns truly need the discarded evidence, and that assistant-written text can substitute for image memory only for facts that were explicitly stated, not for facts merely seen. The practical upshot is that safe forgetting should be decided by measured future visual dependence or fact-level verbalization, not by current attention. The contribution is a reusable paired-intervention methodology, the Causal Visual Memory Audit (CVMA), that separates these explanations within a single dialog trajectory.

Core claim

The central discovery is that 'what the model is looking at now' does not license 'what it can forget safely later'. Under CVMA's paired single-prefill protocol, regional attention and future causal utility are negatively correlated (Spearman −0.103 at 4×4 resolution on the primary model set), attention-based joint retention is worse than seeded random retention at 25/50/75% budgets, and a same-dialog marginal-utility control shows selection headroom exists. The failure is masked in aggregate because many dialogs do not need vision later; when conditioned on an all-image-drop dependence measure, damage concentrates in high-dependence dialogs and later turns. A factorial image–text interventi

What carries the argument

The load-bearing object is the Causal Visual Memory Audit (CVMA), a paired single-prefill framework that treats each dialog as one persistent memory trajectory and applies controlled 'drop' interventions to the key-value cache. Its three linked estimators are: regional future utility (the causal target attention is supposed to predict, defined as the likelihood change from removing a region), all-image-drop dependence (the likelihood change from removing the whole image span, used as an effect modifier), and a factorial image–text interaction that asks whether completed assistant-output KV can substitute for image KV. A marginal-utility control ranks regions by their single-drop utility to w

Load-bearing premise

The quantitative regret numbers rest on teacher-forced next-token likelihood under a 'drop' that makes cache positions unreadable while keeping positions and RoPE indices fixed being a faithful proxy for what real free-running deployments lose when tokens are truly removed.

What would settle it

Free-run the audited dialogs without teacher forcing on the high-dependence subset: if attention-based retention matches full-cache answer accuracy within noise on the very dialogs where the all-image drop causes the largest likelihood jump, the claim that low attention is unsafe for irreversible deletion would be contradicted. Alternatively, compute the attention–future-utility correlation using future-question-conditional relevance labels; a positive correlation would indicate the negative association is an artifact of anchoring to the first question.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Compression policies for multimodal assistants should stop treating low current attention as sufficient evidence for irreversible visual-token deletion.
  • Aggregate evaluation scores can hide real loss; reports should condition on visual dependence (all-image-drop effect) and on which facts were verbalized.
  • The measured damage from eviction rises sharply on later turns for dialogs that depend on vision, so an early one-shot compression decision is the risky point.
  • Assistant-text KV provides a second memory route, but only for facts explicitly stated; unsaid facts are not reliably recoverable after image KV is gone.
  • Non-deletion compression (low-bit coverage) stays close to the full cache only while its representation remains decodable; below a precision floor even 'non-deletion' collapses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same mis-specification likely applies to text-only long-context eviction, where attention is used to predict future relevance; CVMA's intervention logic could be transplanted to audit that setting.
  • A practical extension would be a cheap pre-estimate of future visual dependence (e.g., from question type or dialog structure) to gate when eviction is allowed.
  • Because the marginal-utility control shows headroom, a selector trained on paired drop-out utility labels rather than attention might substantially close the gap to full cache.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes CVMA, a paired single-prefill causal audit framework for deciding when visual KV cache entries can be safely evicted in multi-turn multimodal dialogs. It defines four intervention-based estimands: regional future utility (Eq. 3), whole-image visual dependence (Eq. 4), retention regret against seeded random and a marginal-utility control (Eq. 5), and image–text factorial substitution (Eq. 6). On Qwen2.5-VL-7B and Idefics3-8B, it reports that current attention is negatively correlated with future regional utility (Spearman −0.103 at the primary 4×4 grain), that attention-based joint retention is worse than seeded random, and that apparent safety arises only when later turns do not need the image or when the needed fact was explicitly verbalized in assistant text. The paper also includes full-dataset comparisons of five compressors, a matched non-deletion coverage control across bit budgets, and a free-running exact-match bridge on high-dependence dialogs.

Significance. If the central claims hold, this is a valuable methodological contribution: it converts a vague concern about unknown future queries into falsifiable, paired causal estimands, and it provides a clean separation between proxy validity, dependence-conditioned damage, and image–text substitution. The strengths are substantial: a single-prefill paired design with runtime invariants, bootstrap CIs, cross-model replication, explicit scope bounding, an executable audit manifest, and a free-running exact-match check that partially bridges teacher-forced NLL to observable generation quality. The paper does not overclaim a new compression policy; its practical recommendation—that low current attention is not a certificate for irreversible visual-KV deletion—is important and, if confirmed, should influence how multimodal KV eviction is evaluated.

major comments (2)
  1. [Section 2, Drop definition; Section 3, Fig. 2] The central proxy-failure estimates define 'Drop' as making selected cache positions unreadable while keeping cache length, positions, and RoPE indices unchanged. Physical KV eviction removes tokens and shifts positions/RoPE. Because the paper itself shows that top-attention positions are position-contaminated (Fig. 2c,d), the negative attention–future-utility Spearman (−0.103 at 4×4) and the 'worse than seeded random' retention regret could be artifacts of feeding an out-of-distribution masked cache rather than evidence about real deletion. The free-running exact-match check (Fig. 4e,f) uses deployed selectors with their own rankings, so it does not validate the regional attention ranking under physical deletion. I ask for a direct comparison between unreadable-position masking and actual token removal/re-indexing for the same selected regions, or alternatively a clear restriction of th
  2. [Section 3, first paragraph] The headline correlation and the joint-retention comparisons are computed only on 200 frozen Qwen-7B dialogs, after excluding 72 pilot/confirmation IDs and freezing the first 200 after a single shuffle. Since this subset is the sole basis for the regional proxy-failure claim, the paper should demonstrate representativeness with respect to the full 2,064-dialog validation set—for example, compare all-drop dependence Dd, question type mix, and answer-length distributions—or repeat the correlation on additional random subsets. Without this, selection bias from the exclusion or the arbitrary freeze cannot be fully ruled out.
minor comments (5)
  1. [Eq. (2)] The definition 'wt = |H| −1' appears to be a typesetting error for 1/|H|. The current reading would assign weight 207 to every turn for a ten-turn dialog, which contradicts the text. Please correct.
  2. [Figure 2 caption] The caption contains a garbled numeric block ('76 3 · 2 · 4 2 · ...') that looks like a table artifact. Please reformat or move to a table.
  3. [Section 2, sample description] The exclusion of 72 pilot/confirmation IDs is mentioned only briefly. Even if the remaining 200 are frozen, please clarify whether the 72 were excluded for reasons that could correlate with dialog properties, and report the size of the pool from which the 200 were drawn.
  4. [Section 4, dependence reporting] The statement '25.3% have negative Dd' is useful but could be clarified as the fraction of dialogs where all-image-drop improves NLL. Also, consider reporting a histogram or quantile table beyond decile means, since the decile plot compresses a wide range.
  5. [Section 3, aggregation robustness] The robustness variants (last-four, last-layer, mass-weighted) are presented only for the 4×4 grain in Fig. 4a,b. Please state explicitly whether the 8×8 and 2×2 grains were also checked under these aggregations, or note in the text that they were not.

Circularity Check

0 steps flagged

No significant circularity: the causal estimands are independently defined and tested against external benchmarks.

full rationale

The paper's derivation chain is non-circular. Attention (Eq. 1), regional future utility (Eq. 3), all-image dependence (Eq. 4), proxy regret (Eq. 5), and image-text interaction (Eq. 6) are each defined as distinct interventions on a frozen trajectory; none is defined in terms of the quantity it is used to test. The headline negative attention-utility Spearman (-0.103) and joint-retention regrets are correlations between independently operationalized attention rankings and teacher-forced NLL differences under dropout, with no parameter fitted to the target result. The same-dialog marginal-utility control is explicitly labeled a non-deployable 'witness' and 'not a joint optimum,' so its use to demonstrate selection headroom is an oracle diagnostic, not a disguised prediction or fit. The fact-level substitution result is supported by a separate controlled 80-image experiment and a stock-generated 960/960 factorial audit, not derived from the aggregate interaction alone. Self-citations (e.g., NestedKV, SONIC, FlowKV, Chen et al. 2026a/b/c, Liu et al. 2025) appear only in related-work surveys of prior methods and are not load-bearing for CVMA's claims; no uniqueness theorem or ansatz is imported from same-author work. The paper explicitly bounds its claims to the teacher-forced frozen trajectory and flags the intervention's deviation from physical token removal; these are validity limitations, not circularity. External evaluation on VisDial and ConvBench, plus the free-running exact-match check on the high-dependence quartile, provides independent evidence. No load-bearing step reduces, by definition or self-citation, to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The paper fits no model parameters; all numeric settings (budgets, grid sizes, uniform turn weights) are stated experimental design choices rather than fitted quantities. The central claims rest on the assumptions listed above, especially the validity of teacher-forced NLL and the drop intervention as models of eviction. No new physical or architectural entities are postulated.

axioms (5)
  • domain assumption Teacher-forced reference-answer NLL measures what later answers lose.
    Used to define L_{d,t}(Z), U_{d,r}, D_d (§2). If NLL under a frozen reference diverges from real answer quality, the quantitative causal estimates could misstate harm; the paper partially validates with free-running exact match on the high-dependence quartile (§4).
  • domain assumption The drop intervention (making cache positions unreadable while preserving length/positions/RoPE indices) faithfully models irreversible eviction.
    §2: 'Drop makes the selected cache positions unreadable, leaving cache length, positions, and RoPE indices unchanged.' Real eviction usually removes tokens and shifts positions; the invariant preserves paired comparison but may differ from deployed deletion.
  • domain assumption The four causal identification conditions (consistency, no interference, identical pre-intervention state, no intervention-induced change to future history tokens) hold on the teacher-forced trajectory.
    §2: 'The estimands are causal for the frozen, teacher-forced trajectory under four conditions...' If hidden-state interference occurs despite token-exact extension, counterfactual readings weaken.
  • domain assumption The 200 frozen dialogs (regional audit) and the 80-image fact-relay set are representative of the full benchmarks.
    §2 removes 72 pilot/confirmation IDs, shuffles, and freezes first 200; §5 uses an untouched 80-image confirmation. If these subsets are not representative, the attention–utility correlations and verbalization boundary could be specific to the subset.
  • domain assumption A same-dialog marginal-utility (single-drop) oracle is a valid witness of selection headroom.
    §3: 'aheadroom witness rather than an upper bound, since it ranks single-region effects instead of solving the combinatorial problem.' The oracle is in-sample, so it only shows signal exists, not that a deployable policy can attain it.

pith-pipeline@v1.3.0-alltime-deepseek · 12109 in / 17452 out tokens · 179518 ms · 2026-08-01T02:19:02.173110+00:00 · methodology

0 comments
read the original abstract

Stateful multimodal assistants encode an image once but may answer questions about it many turns later. Attention-guided visual-KV eviction assumes that evidence irrelevant now will remain dispensable, although future questions are unknown. We ask when a visual fact is actually safe to forget and introduce the Causal Visual Memory Audit (CVMA), a paired single-prefill framework that tests what later answers lose when a visual region, the whole image, or prior assistant text becomes unavailable. On VisDial and ConvBench, current attention can rank future-useful regions worse than random even though a diagnostic marginal-utility control shows substantial selection headroom. Aggregate scores hide this failure when later turns do not need vision; controlled and stock-generated histories reveal a second escape route, in which assistant-text KV replaces image KV for facts already stated but not reliably for unstated facts. In the tested stacks, safe forgetting is supported by low future visual dependence or fact-specific verbalization---not by low current attention.

Figures

Figures reproduced from arXiv: 2607.25467 by Bo Wang, Hong Chen, Kang Chen, Xuming Hu, Yuanlin Chu, Yubo Gao, Yuxuan Fan.

Figure 1
Figure 1. Figure 1: When is visual memory safe to forget? The top trajectory exposes the hidden commitment behind current-attention eviction: the clock is seen once, ignored by the current question, left unsaid, and unavailable when a future question needs it. The bottom shows CVMA’s three linked tests of the proposed chain—proxy validity, visual dependence, and fact-specific image–text substitution. alogs, and turns. (3) Con… view at source ↗
Figure 2
Figure 2. Figure 2: Proxy-validity tests for current attention. (a) Mean per-dialog attention–utility Spearman (200 dialogs). (b) Joint￾retention ∆NLL at a matched 4×4 grid for attention, seeded random, and the marginal-utility control (“Oracle”). (c) Top-attention frequency and future-drop effect by cache position, endpoints outlined (percentages and 10−3 NLL). (d) Top-1 persistence after content flips, at the fixed position… view at source ↗
Figure 3
Figure 3. Figure 3: Dependence- and turn-conditioned damage at exact common budgets. Full VisDial is binned into ten equal￾frequency deciles by an independent all-image drop. (a,b) Damage versus dependence decile and dialog turn at four accounted bits (five 25% selectors and matched coverage); (c,d) the same at two bits (12.5% selectors and the one-bit coverage endpoint). The lower row uses symmetric-log scaling with 95% boot… view at source ↗
Figure 4
Figure 4. Figure 4: Robustness checks and free-running answer quality. (a) Attention–utility correlation under four aggregation rules. (b) Joint-retention ∆NLL versus seeded random; positive is worse. (c) Idefics all-image-drop dependence with VisDial’s caption kept versus removed. (d) All-drop dependence versus visual-token count, answer length, and question length. (e) Strict exact match on turns 2–10 of all 516 dependence-… view at source ↗
Figure 5
Figure 5. Figure 5: Matched-storage damage across the accounted￾bit ladder. The aligned full-VisDial sweep gives every se￾lector 50/25/12.5% retention and covers coverage’s full mea￾sured precision range; the shaded slice marks four-bit ac￾counting. The symmetric-log axis has a linear zone of ±10−3 NLL and retains the failed two-bit endpoint. 516 dialogs and 4,644 later-turn cells—and generate all ten turns under Full, all fi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 11 linked inside Pith

  1. [1]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Visual dialog , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  2. [2]

    Advances in Neural Information Processing Systems , volume=

    Convbench: A multi-turn conversation evaluation benchmark with hierarchical ablation capability for large vision-language models , author=. Advances in Neural Information Processing Systems , volume=

  3. [3]

    2025 , eprint=

    Qwen2.5-VL Technical Report , author=. 2025 , eprint=

  4. [4]

    arXiv preprint arXiv:2402.02750 , year=

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache , author=. arXiv preprint arXiv:2402.02750 , year=

  5. [5]

    arXiv preprint arXiv:2402.18096 , year=

    No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization , author=. arXiv preprint arXiv:2402.18096 , year=

  6. [6]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

  7. [7]

    Advances in Neural Information Processing Systems , volume=

    Prefixkv: Adaptive prefix kv cache is what vision instruction-following models need for efficient generation , author=. Advances in Neural Information Processing Systems , volume=

  8. [8]

    Advances in Neural Information Processing Systems , volume=

    Snapkv: Llm knows what you are looking for before generation , author=. Advances in Neural Information Processing Systems , volume=

  9. [9]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Nacl: A general and effective kv cache eviction framework for llm at inference time , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  10. [10]

    International Conference on Learning Representations , volume=

    Efficient streaming language models with attention sinks , author=. International Conference on Learning Representations , volume=

  11. [11]

    Attention is not explanation , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=

  12. [12]

    arXiv preprint arXiv:2602.08585 , year=

    Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction , author=. arXiv preprint arXiv:2602.08585 , year=

  13. [13]

    arXiv preprint arXiv:2605.09649 , year=

    Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction , author=. arXiv preprint arXiv:2605.09649 , year=

  14. [14]

    International Conference on Learning Representations , volume=

    VL-cache: Sparsity and modality-aware KV cache compression for vision-language model inference acceleration , author=. International Conference on Learning Representations , volume=

  15. [15]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  16. [16]

    Meda: Dynamic kv cache allocation for efficient multimodal long-context inference , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

  17. [17]

    arXiv preprint arXiv:2606.14782 , year=

    Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression , author=. arXiv preprint arXiv:2606.14782 , year=

  18. [18]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

  19. [19]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Hybridkv: Hybrid kv cache compression for efficient multimodal large language model inference , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  20. [20]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Myopia Rectification: KV Cache Pruning for MLLMs Via Dynamic Attention Subsidy and Token Reclamation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  21. [21]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  22. [22]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Rethinking Token Reduction for Large Vision-Language Models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  23. [23]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Visionzip: Longer is better but not necessary in vision language models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    Zipcache: Accurate and efficient kv cache quantization with salient token identification , author=. Advances in Neural Information Processing Systems , volume=

  25. [25]

    2025 IEEE International Conference on Multimedia and Expo (ICME) , pages=

    Akvq-vl: Attention-aware kv cache adaptive 2-bit quantization for vision-language models , author=. 2025 IEEE International Conference on Multimedia and Expo (ICME) , pages=. 2025 , organization=

  26. [26]

    arXiv preprint arXiv:2605.02262 , year=

    WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimization , author=. arXiv preprint arXiv:2605.02262 , year=

  27. [27]

    arXiv preprint arXiv:2605.17170 , year=

    TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks , author=. arXiv preprint arXiv:2605.17170 , year=

  28. [28]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  29. [29]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Boosting multimodal large language models with visual tokens withdrawal for rapid inference , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  30. [30]

    arXiv preprint arXiv:2504.13915 , year=

    Memory-efficient streaming videollms for real-time procedural video understanding , author=. arXiv preprint arXiv:2504.13915 , year=

  31. [31]

    arXiv preprint arXiv:2605.15128 , year=

    MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory , author=. arXiv preprint arXiv:2605.15128 , year=

  32. [32]

    arXiv preprint arXiv:2502.14882 , year=

    CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs , author=. arXiv preprint arXiv:2502.14882 , year=

  33. [33]

    arXiv preprint arXiv:2605.26678 , year=

    NestedKV: Nested Memory Routing for Long-Context KV Cache Compression , author=. arXiv preprint arXiv:2605.26678 , year=

  34. [34]

    arXiv preprint arXiv:2601.21927 , year=

    SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Caching , author=. arXiv preprint arXiv:2601.21927 , year=

  35. [35]

    arXiv preprint arXiv:2505.15347 , year=

    FlowKV: Enhancing multi-turn conversational coherence in LLMs via isolated key-value cache management , author=. arXiv preprint arXiv:2505.15347 , year=