REVIEW 2 major objections 5 minor 35 references
This paper claims that in stateful vision-language assistants, low current attention is not a valid certificate for irreversibly deleting visual memory, and that safe forgetting requires either low future visual dependence or explicit verba
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 02:19 UTC pith:DRYLRA5H
load-bearing objection A careful causal audit showing that current attention is a poor proxy for future visual-KV utility — solid, bounded, and worth refereeing, with a real but nonfatal gap between masking and physical eviction. the 2 major comments →
Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that 'what the model is looking at now' does not license 'what it can forget safely later'. Under CVMA's paired single-prefill protocol, regional attention and future causal utility are negatively correlated (Spearman −0.103 at 4×4 resolution on the primary model set), attention-based joint retention is worse than seeded random retention at 25/50/75% budgets, and a same-dialog marginal-utility control shows selection headroom exists. The failure is masked in aggregate because many dialogs do not need vision later; when conditioned on an all-image-drop dependence measure, damage concentrates in high-dependence dialogs and later turns. A factorial image–text interventi
What carries the argument
The load-bearing object is the Causal Visual Memory Audit (CVMA), a paired single-prefill framework that treats each dialog as one persistent memory trajectory and applies controlled 'drop' interventions to the key-value cache. Its three linked estimators are: regional future utility (the causal target attention is supposed to predict, defined as the likelihood change from removing a region), all-image-drop dependence (the likelihood change from removing the whole image span, used as an effect modifier), and a factorial image–text interaction that asks whether completed assistant-output KV can substitute for image KV. A marginal-utility control ranks regions by their single-drop utility to w
Load-bearing premise
The quantitative regret numbers rest on teacher-forced next-token likelihood under a 'drop' that makes cache positions unreadable while keeping positions and RoPE indices fixed being a faithful proxy for what real free-running deployments lose when tokens are truly removed.
What would settle it
Free-run the audited dialogs without teacher forcing on the high-dependence subset: if attention-based retention matches full-cache answer accuracy within noise on the very dialogs where the all-image drop causes the largest likelihood jump, the claim that low attention is unsafe for irreversible deletion would be contradicted. Alternatively, compute the attention–future-utility correlation using future-question-conditional relevance labels; a positive correlation would indicate the negative association is an artifact of anchoring to the first question.
If this is right
- Compression policies for multimodal assistants should stop treating low current attention as sufficient evidence for irreversible visual-token deletion.
- Aggregate evaluation scores can hide real loss; reports should condition on visual dependence (all-image-drop effect) and on which facts were verbalized.
- The measured damage from eviction rises sharply on later turns for dialogs that depend on vision, so an early one-shot compression decision is the risky point.
- Assistant-text KV provides a second memory route, but only for facts explicitly stated; unsaid facts are not reliably recoverable after image KV is gone.
- Non-deletion compression (low-bit coverage) stays close to the full cache only while its representation remains decodable; below a precision floor even 'non-deletion' collapses.
Where Pith is reading between the lines
- The same mis-specification likely applies to text-only long-context eviction, where attention is used to predict future relevance; CVMA's intervention logic could be transplanted to audit that setting.
- A practical extension would be a cheap pre-estimate of future visual dependence (e.g., from question type or dialog structure) to gate when eviction is allowed.
- Because the marginal-utility control shows headroom, a selector trained on paired drop-out utility labels rather than attention might substantially close the gap to full cache.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CVMA, a paired single-prefill causal audit framework for deciding when visual KV cache entries can be safely evicted in multi-turn multimodal dialogs. It defines four intervention-based estimands: regional future utility (Eq. 3), whole-image visual dependence (Eq. 4), retention regret against seeded random and a marginal-utility control (Eq. 5), and image–text factorial substitution (Eq. 6). On Qwen2.5-VL-7B and Idefics3-8B, it reports that current attention is negatively correlated with future regional utility (Spearman −0.103 at the primary 4×4 grain), that attention-based joint retention is worse than seeded random, and that apparent safety arises only when later turns do not need the image or when the needed fact was explicitly verbalized in assistant text. The paper also includes full-dataset comparisons of five compressors, a matched non-deletion coverage control across bit budgets, and a free-running exact-match bridge on high-dependence dialogs.
Significance. If the central claims hold, this is a valuable methodological contribution: it converts a vague concern about unknown future queries into falsifiable, paired causal estimands, and it provides a clean separation between proxy validity, dependence-conditioned damage, and image–text substitution. The strengths are substantial: a single-prefill paired design with runtime invariants, bootstrap CIs, cross-model replication, explicit scope bounding, an executable audit manifest, and a free-running exact-match check that partially bridges teacher-forced NLL to observable generation quality. The paper does not overclaim a new compression policy; its practical recommendation—that low current attention is not a certificate for irreversible visual-KV deletion—is important and, if confirmed, should influence how multimodal KV eviction is evaluated.
major comments (2)
- [Section 2, Drop definition; Section 3, Fig. 2] The central proxy-failure estimates define 'Drop' as making selected cache positions unreadable while keeping cache length, positions, and RoPE indices unchanged. Physical KV eviction removes tokens and shifts positions/RoPE. Because the paper itself shows that top-attention positions are position-contaminated (Fig. 2c,d), the negative attention–future-utility Spearman (−0.103 at 4×4) and the 'worse than seeded random' retention regret could be artifacts of feeding an out-of-distribution masked cache rather than evidence about real deletion. The free-running exact-match check (Fig. 4e,f) uses deployed selectors with their own rankings, so it does not validate the regional attention ranking under physical deletion. I ask for a direct comparison between unreadable-position masking and actual token removal/re-indexing for the same selected regions, or alternatively a clear restriction of th
- [Section 3, first paragraph] The headline correlation and the joint-retention comparisons are computed only on 200 frozen Qwen-7B dialogs, after excluding 72 pilot/confirmation IDs and freezing the first 200 after a single shuffle. Since this subset is the sole basis for the regional proxy-failure claim, the paper should demonstrate representativeness with respect to the full 2,064-dialog validation set—for example, compare all-drop dependence Dd, question type mix, and answer-length distributions—or repeat the correlation on additional random subsets. Without this, selection bias from the exclusion or the arbitrary freeze cannot be fully ruled out.
minor comments (5)
- [Eq. (2)] The definition 'wt = |H| −1' appears to be a typesetting error for 1/|H|. The current reading would assign weight 207 to every turn for a ten-turn dialog, which contradicts the text. Please correct.
- [Figure 2 caption] The caption contains a garbled numeric block ('76 3 · 2 · 4 2 · ...') that looks like a table artifact. Please reformat or move to a table.
- [Section 2, sample description] The exclusion of 72 pilot/confirmation IDs is mentioned only briefly. Even if the remaining 200 are frozen, please clarify whether the 72 were excluded for reasons that could correlate with dialog properties, and report the size of the pool from which the 200 were drawn.
- [Section 4, dependence reporting] The statement '25.3% have negative Dd' is useful but could be clarified as the fraction of dialogs where all-image-drop improves NLL. Also, consider reporting a histogram or quantile table beyond decile means, since the decile plot compresses a wide range.
- [Section 3, aggregation robustness] The robustness variants (last-four, last-layer, mass-weighted) are presented only for the 4×4 grain in Fig. 4a,b. Please state explicitly whether the 8×8 and 2×2 grains were also checked under these aggregations, or note in the text that they were not.
Circularity Check
No significant circularity: the causal estimands are independently defined and tested against external benchmarks.
full rationale
The paper's derivation chain is non-circular. Attention (Eq. 1), regional future utility (Eq. 3), all-image dependence (Eq. 4), proxy regret (Eq. 5), and image-text interaction (Eq. 6) are each defined as distinct interventions on a frozen trajectory; none is defined in terms of the quantity it is used to test. The headline negative attention-utility Spearman (-0.103) and joint-retention regrets are correlations between independently operationalized attention rankings and teacher-forced NLL differences under dropout, with no parameter fitted to the target result. The same-dialog marginal-utility control is explicitly labeled a non-deployable 'witness' and 'not a joint optimum,' so its use to demonstrate selection headroom is an oracle diagnostic, not a disguised prediction or fit. The fact-level substitution result is supported by a separate controlled 80-image experiment and a stock-generated 960/960 factorial audit, not derived from the aggregate interaction alone. Self-citations (e.g., NestedKV, SONIC, FlowKV, Chen et al. 2026a/b/c, Liu et al. 2025) appear only in related-work surveys of prior methods and are not load-bearing for CVMA's claims; no uniqueness theorem or ansatz is imported from same-author work. The paper explicitly bounds its claims to the teacher-forced frozen trajectory and flags the intervention's deviation from physical token removal; these are validity limitations, not circularity. External evaluation on VisDial and ConvBench, plus the free-running exact-match check on the high-dependence quartile, provides independent evidence. No load-bearing step reduces, by definition or self-citation, to its own inputs.
Axiom & Free-Parameter Ledger
axioms (5)
- domain assumption Teacher-forced reference-answer NLL measures what later answers lose.
- domain assumption The drop intervention (making cache positions unreadable while preserving length/positions/RoPE indices) faithfully models irreversible eviction.
- domain assumption The four causal identification conditions (consistency, no interference, identical pre-intervention state, no intervention-induced change to future history tokens) hold on the teacher-forced trajectory.
- domain assumption The 200 frozen dialogs (regional audit) and the 80-image fact-relay set are representative of the full benchmarks.
- domain assumption A same-dialog marginal-utility (single-drop) oracle is a valid witness of selection headroom.
read the original abstract
Stateful multimodal assistants encode an image once but may answer questions about it many turns later. Attention-guided visual-KV eviction assumes that evidence irrelevant now will remain dispensable, although future questions are unknown. We ask when a visual fact is actually safe to forget and introduce the Causal Visual Memory Audit (CVMA), a paired single-prefill framework that tests what later answers lose when a visual region, the whole image, or prior assistant text becomes unavailable. On VisDial and ConvBench, current attention can rank future-useful regions worse than random even though a diagnostic marginal-utility control shows substantial selection headroom. Aggregate scores hide this failure when later turns do not need vision; controlled and stock-generated histories reveal a second escape route, in which assistant-text KV replaces image KV for facts already stated but not reliably for unstated facts. In the tested stacks, safe forgetting is supported by low future visual dependence or fact-specific verbalization---not by low current attention.
Figures
Reference graph
Works this paper leans on
-
[1]
Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
Visual dialog , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
-
[2]
Advances in Neural Information Processing Systems , volume=
Convbench: A multi-turn conversation evaluation benchmark with hierarchical ablation capability for large vision-language models , author=. Advances in Neural Information Processing Systems , volume=
-
[3]
2025 , eprint=
Qwen2.5-VL Technical Report , author=. 2025 , eprint=
2025
-
[4]
arXiv preprint arXiv:2402.02750 , year=
Kivi: A tuning-free asymmetric 2bit quantization for kv cache , author=. arXiv preprint arXiv:2402.02750 , year=
-
[5]
arXiv preprint arXiv:2402.18096 , year=
No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization , author=. arXiv preprint arXiv:2402.18096 , year=
-
[6]
Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=
Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=
2024
-
[7]
Advances in Neural Information Processing Systems , volume=
Prefixkv: Adaptive prefix kv cache is what vision instruction-following models need for efficient generation , author=. Advances in Neural Information Processing Systems , volume=
-
[8]
Advances in Neural Information Processing Systems , volume=
Snapkv: Llm knows what you are looking for before generation , author=. Advances in Neural Information Processing Systems , volume=
-
[9]
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Nacl: A general and effective kv cache eviction framework for llm at inference time , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[10]
International Conference on Learning Representations , volume=
Efficient streaming language models with attention sinks , author=. International Conference on Learning Representations , volume=
-
[11]
Attention is not explanation , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=
2019
-
[12]
arXiv preprint arXiv:2602.08585 , year=
Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction , author=. arXiv preprint arXiv:2602.08585 , year=
-
[13]
arXiv preprint arXiv:2605.09649 , year=
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction , author=. arXiv preprint arXiv:2605.09649 , year=
-
[14]
International Conference on Learning Representations , volume=
VL-cache: Sparsity and modality-aware KV cache compression for vision-language model inference acceleration , author=. International Conference on Learning Representations , volume=
-
[15]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
-
[16]
Meda: Dynamic kv cache allocation for efficient multimodal long-context inference , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
2025
-
[17]
arXiv preprint arXiv:2606.14782 , year=
Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression , author=. arXiv preprint arXiv:2606.14782 , year=
-
[18]
Findings of the Association for Computational Linguistics: ACL 2026 , pages=
RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=
2026
-
[19]
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Hybridkv: Hybrid kv cache compression for efficient multimodal large language model inference , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[20]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Myopia Rectification: KV Cache Pruning for MLLMs Via Dynamic Attention Subsidy and Token Reclamation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[21]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
-
[22]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Rethinking Token Reduction for Large Vision-Language Models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[23]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Visionzip: Longer is better but not necessary in vision language models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[24]
Advances in Neural Information Processing Systems , volume=
Zipcache: Accurate and efficient kv cache quantization with salient token identification , author=. Advances in Neural Information Processing Systems , volume=
-
[25]
2025 IEEE International Conference on Multimedia and Expo (ICME) , pages=
Akvq-vl: Attention-aware kv cache adaptive 2-bit quantization for vision-language models , author=. 2025 IEEE International Conference on Multimedia and Expo (ICME) , pages=. 2025 , organization=
2025
-
[26]
arXiv preprint arXiv:2605.02262 , year=
WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimization , author=. arXiv preprint arXiv:2605.02262 , year=
-
[27]
arXiv preprint arXiv:2605.17170 , year=
TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks , author=. arXiv preprint arXiv:2605.17170 , year=
-
[28]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[29]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Boosting multimodal large language models with visual tokens withdrawal for rapid inference , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[30]
arXiv preprint arXiv:2504.13915 , year=
Memory-efficient streaming videollms for real-time procedural video understanding , author=. arXiv preprint arXiv:2504.13915 , year=
-
[31]
arXiv preprint arXiv:2605.15128 , year=
MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory , author=. arXiv preprint arXiv:2605.15128 , year=
-
[32]
arXiv preprint arXiv:2502.14882 , year=
CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs , author=. arXiv preprint arXiv:2502.14882 , year=
-
[33]
arXiv preprint arXiv:2605.26678 , year=
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression , author=. arXiv preprint arXiv:2605.26678 , year=
-
[34]
arXiv preprint arXiv:2601.21927 , year=
SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Caching , author=. arXiv preprint arXiv:2601.21927 , year=
-
[35]
arXiv preprint arXiv:2505.15347 , year=
FlowKV: Enhancing multi-turn conversational coherence in LLMs via isolated key-value cache management , author=. arXiv preprint arXiv:2505.15347 , year=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.