Artificial Intelligence · arXiv:2610.23290 v1 · submitted 16 Oct 2026 · 88 references
CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective
The paper says
“Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key–value cache. Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps.”
more of the abstract
Such caching, however, often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure we term the Repeat Curse. Tracing the context-token information flow reveals three disruptions. Within a step, caching disperses the attention that anchors a position on nearby context tokens. Across steps, similarity-based refresh is not staleness-aware, so some positions are decoded against stale key–value states. Across layers, the deep-layer entropy of the context fails to converge where repetition arises. We accordingly propose CoTA++, a training-free, plug-and-play framework that counteracts each disruption where it originates: Context-Token Attention Re-anchoring re-forms the attention routing against the current context, Decode-Aware Refresh reserves part of the cache's refresh budget for the positions about to be committed, and Context-Token Entropy-Guided Voting discounts the confidence of a position by the deep-layer entropy of its context, which reorders the unmasking. The three components act at non-overlapping stages of a decoding step and compose freely with existing dMLLMs and caches. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6% to at most 17.0%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15% overhead, making approximate caching reliable for dMLLMs.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 19 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A well-executed diagnostic-and-fix paper that makes caching reliable for diffusion MLLMs — the empirical work is thorough and the claims are supported. This line is wrong
1 major objection · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 88 references say.
The paper says, at §VI-A, Table VII
The review did not quote this passage.
Pith says: Objection 1 of 1 · major
The 'Judge score' for response quality relies on a proprietary model (gpt-5.6-luna). While the paper provides the protocol reference to supplementary material, the judge is not a publicly available, fixed checkpoint. This raises a limited reproducibility concern for the quality metric.
more of objection 1
The central claims about repetition reduction (SRR, ARR) do not depend on the judge, so this does not undermine the main thesis, but for the journal version the authors should either (a) confirm that the judge model is documented with a specific version identifier that allows exact replication, or (b) supplement with a human evaluation on a subset or with a widely-used open model (e.g., GPT-4o) to increase confidence.
6 minor objections
The paper says, at Abstract and §I
Pith says: minor objection 1 of 6
The phrase 'at most 17.0%' is precise and supported by Table VI (LaViDa under SlowFast+CoTA++ at L=512 gives 17.0%). However, the claim 'makes approximate caching reliable for dMLLMs' is broad; consider softening to 'makes approximate caching substantially more reliable for the evaluated dMLLMs' to match the scope acknowledged in §VIII.
The paper says, at §IV-A, Eq. (5–6)
Pith says: minor objection 2 of 6
The definition of ARR over content tokens (truncated at first EOS) is clear. For completeness, specify how responses without any content token (e.g., empty) are handled; presumably they are excluded from ARR but counted in SRR only if ARR>0? Clarify in the supplementary material.
This objection may describe the copy Pith gave the reviewer, not the paper.
show 4 more
The paper says, at §V-A, Eq. (9–10) and Algorithm 1
Pith says: minor objection 3 of 6
CTAR recomputes attention for positions selected by the monitor. The monitor counts 'context tokens within ±w committed since its last re-anchoring'. It may be helpful to explicitly state that the monitor resets ni to zero for re-anchored positions (as Algorithm 1 line 3 states). In the main text, consider adding a one-sentence note that the monitor is cheap and independent of the cache policy.
The paper says, at §VII-B, Table XII
Pith says: minor objection 4 of 6
The design alternatives table is informative. The row 'Dispersed anchoring (F1)' for DAR slots seems to test filling reserved slots by the F1 signal rather than confidence. The label could be more explicit in the caption: 'DAR slots filled by the positions of most dispersed anchoring (F1)' to avoid confusion.
The paper says, at Fig. 5 and Fig. 6
Pith says: minor objection 5 of 6
The figures are dense but generally legible. Consider enlarging the axis labels in Fig. 5(b–d) and Fig. 6(a) for the print version. The color scheme (red vs. blue) is consistent and accessible.
The paper says, at §IV-E, Joint Diagnosis
Pith says: minor objection 6 of 6
The sentence 'the three findings describe one failure along the three axes of the information flow' is clear. However, the compound-repetition statistic (55% vs. 2.4%) is relegated to supplementary material. Including it in the main text would strengthen the case for the joint diagnosis; consider moving it or a summary to §IV-E.
What would settle it
Verify that the mechanistic patterns reported in Fig. 5 (attention dispersion), Fig. 6 (KV staleness at decode moment), and Fig. 7 (entropy consolidation) hold under the SlowFast caching backend with its default settings, using the same LLaDA-V model at L=128 on the 94 MS-COCO images that repeat under SlowFast. If the same three signatures are found at repeat positions under SlowFast, the claim that the disruptions are generic across caching backends is confirmed; if not, the diagnostic generality is weakened.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 19 Oct 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 1 major objection, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper does something rare: it identifies a concrete failure mode in deployed systems, traces it to specific mechanistic disruptions, and designs a training-free fix that works across models and backends. The core claim — that the Repeat Curse in cached dMLLMs is the convergent symptom of three distinct information-flow disruptions (dispersed anchoring, stale KV states, failed entropy consolidation) — is supported by controlled experiments, ablation studies, and statistical tests. The three components (CTAR, DAR, CTEV) are cleanly motivated, each targeting one disruption, and the ablation grid (Table X) shows that each contributes and that the full combination is the most consistent. The evaluation is unusually broad for this literature: three dMLLMs, two caching backends, three response lengths, nine benchmarks. The results are clear: from up to 99.6% sample repetition rate down to at most 17.0%, while preserving quality and adding under 15% latency overhead. The code is released. The paper is a genuine advance over the conference version [1], both in analysis (the cross-step F2 axis and DAR are new) and in evaluation scale.
The soft spots are proportionate. The independence of the three disruptions is the weakest assumption; the authors acknowledge it and partially test it via ablations, but a full factorial interaction analysis would strengthen the diagnosis. The hyperparameters (κ, w, r, λ) are fixed and ablated, but one could ask whether they generalize across caches without re-tuning. The evaluation covers three models and two caches; the authors honestly note that open-weight dMLLMs and caching methods are still few, so the scope is reasonable but not exhaustive. These are not fatal flaws — they are the usual limits of an empirical paper that chooses depth over breadth. The central argument holds.
Who is this for? Anyone working on dMLLM deployment, inference acceleration, or text degeneration in generation. The paper shows that approximate caching can be made reliable with targeted, lightweight interventions.
Verdict: ACCEPT. I would send this to review without hesitation. It is a solid, honest, reproducible piece of work.
The strongest objection
The paper provides a thorough, multi-faceted argument for both the diagnostic claim (three information-flow disruptions cause the Repeat Curse) and the practical claim (CoTA++ counteracts them). Mechanistic evidence (attention maps, staleness histograms, entropy profiles) is presented for the default cache configuration, and ablation studies (Table X) empirically test the independence and composition of the components. Across three dMLLMs, two caching backends, three response lengths, and nine benchmarks, the central claim holds: CoTA++ reduces the sample repetition rate from up to 99.6% to at most 17.0% while preserving quality and speed. The weakest assumption—that the three disruptions are independent and additive—is partially addressed by component ablations, and the full method's strength across varied settings further validates the design. No single load-bearing concern undermines the central argument; the limitations stated by the authors (few models, backends, and compute) are transparent and do not invalidate the findings within the reported scope.
The reader identified the independence of disruptions/components as the weakest assumption. While the paper does rely on additive decomposition, the ablation study (Table X) and the compound analysis (Section IV-E) provide strong empirical support that the components work in concert without destructive interference. Moreover, the central claim does not hinge on strict independence; even if interactions exist, the full method robustly solves the curse. The paper acknowledges its limitations (few models, backends, compute) and does not overclaim generality beyond the tested configurations. Given the controlled experiments, statistical tests, and reproducible code, the verdict of ACCEPT remains well-supported. I find no load-bearing concern that would warrant a change.
The referee's report
The paper makes a significant contribution to the practical deployment of dMLLMs by providing a mechanistic understanding of a critical failure mode (the Repeat Curse) and offering a lightweight, training-free mitigation. The three-axis information-flow analysis is novel and actionable. The method is validated extensively across multiple models, backends, and benchmarks, and the experimental design includes controlled ablations, statistical tests, and component-wise efficiency analyses. The results are practically impactful: they enable approximate caching to be used without catastrophic repetition, preserving the inference speedup that is essential for dMLLM deployment.
Pith's reading of the paper (a machine's summary, not the authors')
Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, precluding standard autoregressive caching and forcing reliance on approximate caching that reuses states across steps. This caching often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure termed the Repeat Curse. Tracing the information flow of context tokens reveals three disruptions: within a step caching disperses the attention anchoring on nearby context tokens (F1), across steps similarity-based refresh leaves some positions decoded against stale key–value states (F2), and across layers the deep-layer entropy of the context fails to consolidate where repetition arises (F3). The paper proposes CoTA++, a training-free, plug-and-play framework whose three components—Context-Token Attention Re-anchoring, Decode-Aware Refresh, and Context-Token Entropy-Guided Voting—each counteract one disruption at its origin. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate from up to 99.6% to at most 17.0%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15% overhead.
What carries the argument. The central machinery is the three-component framework CoTA++ that operates on the objects corrupted by each disruption. CTAR (Context-Token Attention Re-anchoring) recomputes the attention distribution for positions in deep layers (25–32) when a context token within ±5 has been committed since the last re-anchoring, restoring local anchoring. DAR (Decode-Aware Refresh) reserves r slots in the cache's refresh set for the r masked positions with highest confidence, ensuring they are decoded against fresh KV states. CTEV (Context-Token Entropy-Guided Voting) computes the average deep-layer entropy (layers 26–30) of the committed context tokens within ±5 and subtracts a scaled version from the confidence, producing a voting score that ranks candidates; this causes tokens in consolidated contexts to be committed before those in unconsolidated ones. The three components act on non-overlapping stages of a decoding step (attention output, refresh set, voting score) and compose freely with existing dMLLMs and caching backends.
Core claim. The central claim is that the Repeat Curse is the convergent symptom of three distinct disruptions that approximate caching causes along the three axes of dMLLM inference: routing within a step (dispersed attention anchoring), timeliness across steps (stale KV states due to similarity-based refresh), and consolidation across layers (failed deep-layer entropy convergence). Each disruption has its own signature, object, and mechanism, and the paper demonstrates that they are not redundant by showing that staleness accounts for most of the anchoring loss in the shallow band but almost none in the deep band, and that failed consolidation persists even at equal staleness. The paper further claims that addressing each disruption independently with a targeted intervention suffices to mitigate the curse: CTAR re-forms the attention routing against current keys, DAR reserves cache refresh slots for positions about to be committed, and CTEV discounts the confidence by the context's deep-layer entropy to reorder commitment. Empirical validation across three dMLLMs, two caching backends, and three response lengths shows that the combined framework brings repetition rates close to the uncached baseline while preserving the latency gains of caching.
The premise it rests on. The three disruptions (dispersed anchoring, stale KV states, and failed entropy consolidation) are each independent and additive, and the three proposed components (CTAR, DAR, CTEV) each address exactly one disruption without interfering with the others.
If the paper is right (Pith's inferences, not the authors')
Approximate caching for dMLLMs becomes reliable for long-form generation once the Repeat Curse is removed, preserving both speed and output quality.
The training-free nature of CoTA++ means it can be deployed on existing systems without retraining or fine-tuning, requiring only access to the cache states and confidence ordering.
The framework's component design allows individual parts to be enabled or tuned per deployment, e.g., using only DAR and CTAR for maximum speed while still eliminating long runs.
Because the disruptions are diagnosed along orthogonal axes, the information-flow analysis method may transfer to other artifacts in diffusion inference, such as position-specific quality degradation.
The paper's results imply that future caching designs for dMLLMs should incorporate decode-aware refresh policies and preserve local attention anchoring, not just average state freshness.
The assumption of independent and additive disruptions is challenged by the ablation results: pairs of components sometimes leave residual repetition that the full method removes, and DAR with CTEV at L=128 leaves a run of 25 tokens, suggesting interaction effects that future work could disentangle.
The fixed hyperparameters (κ=1, w=5, r=4, λ=0.25, deep window 26–30) were validated on LLaDA-V and may need adjustment for other dMLLMs or languages; the paper does not provide a sensitivity analysis across model families.
The Repeat Curse is shown to be caching-induced, but recent work (cited as [76]) notes that cache-free decoding can also repeat under aggressive step reduction, hinting at a possible shared root cause (mask-prior drift) that the paper does not address; further isolation experiments could clarify whether CoTA++ also handles that scenario.
The context-entropy penalty (CTEV) could be extended to a continuous risk score that adaptively allocates computational budgets across components, as the paper mentions but leaves to future work.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key–value cache. Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps. Such caching, however, often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure we term the Repeat Curse. Tracing the context-token information flow reveals three disruptions. Within a step, caching disperses the attention that anchors a position on nearby context tokens. Across steps, similarity-based refresh is not staleness-aware, so some positions are decoded against stale key–value states. Across layers, the deep-layer entropy of the context fails to converge where repetition arises. We accordingly propose CoTA++, a training-free, plug-and-play framework that counteracts each disruption where it originates: Context-Token Attention Re-anchoring re-forms the attention routing against the current context, Decode-Aware Refresh reserves part of the cache's refresh budget for the positions about to be committed, and Context-Token Entropy-Guided Voting discounts the confidence of a position by the deep-layer entropy of its context, which reorders the unmasking. The three components act at non-overlapping stages of a decoding step and compose freely with existing dMLLMs and caches. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6% to at most 17.0%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15% overhead, making approximate caching reliable for dMLLMs.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-20
The authors' main claim, as the classifier picked it out 1.00
“Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6\% to at most 17.0\%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15\% overhead, making approximate caching reliable for dMLLMs.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.26 · human participants 0.03 · controlled comparison 0.77 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.78 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key--value cache. | background 1.00 | 0.09 | 0.02 | asserted 1.00 |
| Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps. | background 1.00 | 0.06 | 0.02 | asserted 1.00 |
| Such caching, however, often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure we term the \emph{Repeat Curse}. | background 0.87 | 0.34 | 0.09 | asserted 0.84 |
| Tracing the context-token information flow reveals three disruptions. | contribution 0.48 | 0.87 | 0.05 | asserted 0.99 |
| Within a step, caching disperses the attention that anchors a position on nearby context tokens. | interpretation 0.59 | 0.76 | 0.03 | asserted 1.00 |
| Across steps, similarity-based refresh is not staleness-aware, so some positions are decoded against stale key--value states. | result 0.47 | 0.59 | 0.03 | asserted 0.97 |
| Across layers, the deep-layer entropy of the context fails to converge where repetition arises. | result 0.75 | 0.86 | 0.04 | asserted 1.00 |
| We accordingly propose CoTA++, a training-free, plug-and-play framework that counteracts each disruption where it originates: Context-Token Attention Re-anchoring re-forms the attention routing against the current context, Decode-Aware Refresh reserves part of the cache's refresh budget for the positions about to be committed, and Context-Token Entropy-Guided Voting discounts the confidence of a position by the deep-layer entropy of its context, which reorders the unmasking. | method 0.67 | 0.92 | 0.21 | asserted 1.00 |
| The three components act at non-overlapping stages of a decoding step and compose freely with existing dMLLMs and caches. | method 0.72 | 0.65 | 0.06 | asserted 1.00 |
| Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6\% to at most 17.0\%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15\% overhead, making approximate caching reliable for dMLLMs. | result 1.00 | 0.95 | 0.88 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,729 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (13)
Showing 12 of the paper's 13 figures. Figures are the authors', reproduced from the paper.
References (88)
- [1] Q. Zhao, X. Zhang, S. Chang, Q. Chen, X. Yuan, X. Chen, L. Liu, J. Zhang, X.-Y . Zhang, and D.-H. Wang, “Context tokens are anchors: Understanding the repeat curse in dMLLMs from an information flow perspective,” inThe Fourteenth International Conference on Learning Representa... on Pith
- [2] S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. ZHOU, Y . Lin, J.-R. Wen, and C. Li, “Large language diffusion models,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025 on Pith
- [3] F. Zhu, R. Wang, S. Nie, X. Zhang, C. Wu, J. Hu, J. Zhou, J. Chen, Y . Lin, J.-R. Wen, and C. Li, “Llada 1.5: Variance-reduced preference optimization for large language diffusion models,” 2025 on Pith
- [4] J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong, “Dream 7b,” 2025. [Online]. Available: https: //hkunlp.github.io/blog/2025/dream on Pith
- [5] T. Bie, M. Cao, K. Chen, L. Du, M. Gong, Z. Gong, Y . Gu, J. Hu, Z. Huang, Z. Lanet al., “Llada2. 0: Scaling up diffusion language models to 100b,”arXiv preprint arXiv:2512.15745, 2025 on PithPith's reviewarXiv
- [6] H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,”2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26 286–26 296, 2023 on Pith
- [7] Z. You, S. Nie, X. Zhang, J. Hu, J. Zhou, Z. Lu, J.-R. Wen, and C. Li, “Llada-v: Large language diffusion models with visual instruction tuning,”arXiv preprint arXiv:2505.16933, 2025 on PithPith's reviewarXiv
- [8] L. Yang, Y . Tian, B. Li, X. Zhang, K. Shen, Y . Tong, and M. Wang, “MMaDA: Multimodal large diffusion language models,” inThe Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025 on Pith
- [9] S. Li, K. Kallidromitis, H. Bansal, A. Gokul, Y . Kato, K. Kozuka, J. Kuen, Z. Lin, K.-W. Chang, and A. Grover, “Lavida: A large diffusion model for vision-language understanding,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025 on Pith
- [10] S. Li, J. Gu, K. Liu, Z. Lin, Z. Wei, A. Grover, and J. Kuen, “Lavida-o: Elastic large masked diffusion models for unified multimodal understanding and generation,” inInternational Conference on Learning Representations, vol. 2026, 2026, pp. 50 571–50 603 on Pith
- [11] L. Zeng, J. Yao, B. Liao, H. Tao, W. Liu, and X. Wang, “Diffusionvl: Translating any autoregressive models into diffusion vision language models,”arXiv preprint arXiv:2512.15713, 2025 on PitharXiv
- [12] I. AI, T. Bie, H. Chen, T. Chen, Z. Cheng, L. Cui, K. Gan, Z. Huang, Z. Lan, H. Liet al., “Llada2. 0-uni: Unifying multimodal understanding and generation with diffusion large language model,”arXiv preprint arXiv:2604.20796, 2026 on PithPith's reviewarXiv
- [13] Y . Xin, Q. Qin, S. Luo, K. Zhu, J. Yan, Y . Tai, J. Lei, Y . Cao, K. Wang, Y . Wanget al., “Lumina-dimoo: An omni diffusion large language model for multi-modal generation and understanding,”arXiv preprint arXiv:2510.06308, 2025 on PitharXiv
- [14] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024 on PithPith's reviewarXiv
- [15] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Liet al., “Qwen2.5 technical report,”CoRR, vol. abs/2412.15115, 2024 on PithPith's reviewarXiv
- [16] R. Xu, J. Xu, Q. Zhao, and X. Zhang, “Fixing semantic blind spots in anchor tokens of dmllms,” inFindings of the Association for Computa- tional Linguistics: ACL 2026, 2026, pp. 19 359–19 374 on Pith
- [17] S. Chang, X. Zhang, Q. Liu, and L. Niu, “D3tom: Decider-guided dy- namic token merging for accelerating diffusion mllms,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 24, 2026, pp. 19 961–19 969 on Pith
- [18] Z. Liu, Y . Yang, Y . Zhang, J. Chen, C. Zou, Q. Wei, S. Wang, Y . Zhu, and L. Zhang, “dLLM-cache: Accelerating diffusion large language models with adaptive caching,” inForty-third International Conference on Machine Learning, 2026 on Pith
- [19] Q. Wei, Y . Zhang, Z. Liu, P. Zeng, Y . Wang, B. Qi, D. Liu, and L. Zhang, “Accelerating diffusion large language models with slowfast sampling: The three golden principles,” inInternational Conference on Learning Representations, vol. 2026, 2026, pp. 4340–4354 on Pith
- [20] C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie, “Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding,” inInternational Conference on Learning Representations, vol. 2026, 2026, pp. 57 027–57 051 on Pith
show 68 more
- [21] T.-Y . Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” inEuropean Conference on Computer Vision, 2014 on Pith
- [22] Z. Yu, Z. Wang, Y . Fu, H. Shi, K. Shaikh, and Y . C. Lin, “Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration,” inForty-first Interna- tional Conference on Machine Learning, 2024 on Pith
- [23] L. Wang, L. Li, D. Dai, D. Chen, H. Zhou, F. Meng, J. Zhou, and X. Sun, “Label words are anchors: An information flow perspective for understanding in-context learning,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 9840–9855 on Pith
- [24] L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang, “An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models,” inEuropean Conference on Computer Vision, 2024 on Pith
- [25] I. Yona, I. Shumailov, J. Hayes, and Y . Gandelsman, “Interpreting the repeated token phenomenon in large language models,” inForty-second International Conference on Machine Learning, 2025 on Pith
- [26] X. Zhang, Y . Quan, C. Shen, C. Gu, X. Yuan, S. Yan, J. Cao, H. Cheng, K. Wu, and J. Ye, “Shallow focus, deep fixes: Enhancing shallow layers vision attention sinks to alleviate hallucination in lvlms,” inProceedings of the 2025 Conference on Empirical Methods in Natural Langu... on Pith
- [27] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. M ¨uller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 1862–1874 on Pith
- [28] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. M ¨uller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boeselet al., “Scaling rectified flow transformers for high-resolution image synthesis,” inForty-first international conference on machine learning, 2024 on Pith
- [29] R. ZHANG, S. Zhai, Y . Zhang, J. Thornton, Z. Ou, J. M. Susskind, and N. Jaitly, “Target concrete score matching: A holistic framework for discrete diffusion,” inForty-second International Conference on Machine Learning, 2025 on Pith
- [30] S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong, “Diffuseq: Sequence to sequence text generation with diffusion models,” inThe Eleventh International Conference on Learning Representations, 2023 on Pith
- [31] K. Zheng, Y . Chen, H. Mao, M.-Y . Liu, J. Zhu, and Q. Zhang, “Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 63 186–63 227 on Pith
- [32] J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li, “Your absorbing discrete diffusion secretly models the conditional distributions of clean IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, 202X 16 data,” inInternational Conference on Learn... on Pith
- [33] S. Nie, F. Zhu, C. Du, T. Pang, Q. Liu, G. Zeng, M. Lin, and C. Li, “Scaling up masked diffusion models on text,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 82 974– 82 997 on Pith
- [34] Z. Xie, J. Ye, L. Zheng, J. Gao, J. Dong, Z. Wu, X. Zhao, S. Gong, X. Jiang, Z. Liet al., “Dream-coder 7b: An open diffusion language model for code,”arXiv preprint arXiv:2509.01142, 2025 on PithPith's reviewarXiv
- [35] F. Zhu, Z. You, Y . Xing, Z. Huang, L. Liu, Y . Zhuang, G. Lu, K. Wang, X. Wang, L. Weiet al., “Llada-moe: A sparse moe diffusion language model,”arXiv preprint arXiv:2509.24389, 2025 on PitharXiv
- [36] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdul- mohsin, N. Parthasarathy, T. Evans, L. Beyer, Y . Xia, B. Mustafa et al., “Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,”arXiv preprint arXiv... on PithPith's reviewarXiv
- [37] S. Bai, K. Chen, X. Liu, J. Wanget al., “Qwen2.5-vl technical report,” 2025 on Pith
- [38] X. Zhang, Y . Zhu, C. Gu, X. Yuan, Q. Zhao, J. Cao, F. Tang, S. Fan, Y . Shen, C. Shenet al., “Hallucination begins where saliency drops,” arXiv preprint arXiv:2601.20279, 2026 on PitharXiv
- [39] Q. Zhao, X. Zhang, B. Liu, M. Chen, W. Xiong, J. Chen, G. Ye, W. Yu, X. Yuan, S. Hanet al., “Coalition-aware skill reliability for self-evolving agents,”arXiv preprint arXiv:2608.22610, 2026 on PithPith's reviewarXiv
- [40] Y . Zhao, Q. Zhao, J. Zhang, X. Zhang, X. Yuan, and P. Cheng, “Context- aware cluster decoding: Semantic anchor-driven coherence in dmllms,” arXiv preprint arXiv:2608.22367, 2026 on PithPith's reviewarXiv
- [41] X. Zhang, Y . Zhu, Q. Zhao, X. Yuan, J. Cao, and X. Chen, “Token- Penalty: Alleviating attention sinks and positional decay in LVLMs,” in Findings of the Association for Computational Linguistics: ACL 2026. San Diego, California, United States: Association for Computational Li... on Pith
- [42] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” inProceedings of Machine Learning and Systems, vol. 5, 2023, pp. 606–624 on Pith
- [43] J. Huang, Y . Zhang, Y . Yang, B. Huang, and L. Zhang, “Mask tokens as prophet: Fine-grained cache eviction for efficient dllm inference,” in Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 3456–3479 on Pith
- [44] Z. Hu, J. Meng, Y . Akhauri, M. Abdelfattah, J.-s. Seo, Z. Zhang, and U. Gupta, “Flashdlm: Accelerating diffusion language model inference via efficient kv caching and guided diffusion,” inInternational Con- ference on Learning Representations, vol. 2026, 2026, pp. 153 529– 153 544 on Pith
- [45] Y . Song, Z. Zhang, C. Luo, P. Gao, F. Xia, H. Luo, Z. Li, Y . Yang, H. Yu, X. Quet al., “Seed diffusion: A large-scale diffusion language model with high-speed inference,”arXiv preprint arXiv:2508.02193, 2025 on PithPith's reviewarXiv
- [46] W. Wang, B. Fang, C. Jing, Y . Shen, Y . Shen, Q. Wang, H. Ouyang, H. Chen, and C. Shen, “Time is a feature: Exploiting temporal dynamics in diffusion language models,” inInternational Conference on Learning Representations, vol. 2026, 2026, pp. 72 909–72 935 on Pith
- [47] X. Ma, R. Yu, G. Fang, and X. Wang, “dkv-cache: The cache for diffusion language models,”Advances in Neural Information Processing Systems, vol. 38, pp. 149 009–149 033, 2026 on Pith
- [48] S. Abnar and W. Zuidema, “Quantifying attention flow in transformers,” inProceedings of the 58th annual meeting of the association for computational linguistics, 2020, pp. 4190–4197 on Pith
- [49] J. Ferrando and E. V oita, “Information flow routes: Automatically interpreting language models at scale,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 17 432–17 445 on Pith
- [50] Z. Zhang, S. Yadav, F. Han, and E. Shutova, “Cross-modal information flow in multimodal large language models,” in2025 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025, pp. 19 781–19 791 on Pith
- [51] Z. Qiu, Z. Wang, B. Zheng, Z. Huang, K. Wen, S. Yang, R. Men, L. Yu, F. Huang, S. Huanget al., “Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free,”Advances in Neural Information Processing Systems, vol. 38, pp. 100 092–100 118, 2026 on Pith
- [52] Y . Song, X. Liu, R. Li, Z. Liu, Z. Huang, Q. Guo, Z. He, and X. Qiu, “Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 39, 2026, pp. 33 038–33 046 on Pith
- [53] X. Chen, S. Huang, C. Guo, C. Wei, Y . He, J. Zhang, H. Li, and Y . Chen, “Dpad: Efficient diffusion language models with suffix dropout,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 57 286–57 316 on Pith
- [54] J. Yao, S. Yang, J. Xu, L. Hu, M. Li, and D. Wang, “Understanding the repeat curse in large language models from a feature perspective,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 7787–7815 on Pith
- [55] D. Nguyen, A. Payani, and B. Mirzasoleiman, “Beyond semantic en- tropy: Boosting llm uncertainty quantification with pairwise semantic similarity,” inFindings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 4530–4540 on Pith
- [56] Y . Zhang, C. Wang, Y . Chen, C. Fan, J. Jia, and S. Liu, “The fragile truth of saliency: Improving llm input attribution via attention bias optimization,”Advances in Neural Information Processing Systems, vol. 38, pp. 154 395–154 421, 2026 on Pith
- [57] X. Zhang, Y . Quan, C. Shen, X. Yuan, S. Yan, L. Xie, W. Wang, C. Gu, H. Tang, and J. Ye, “From redundancy to relevance: Information flow in lvlms across reasoning tasks,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Comput... on Pith
- [58] Y . Li, H. Wang, X. Ding, H. Wang, and X. Li, “Token activation map to visually explain multimodal llms,” in2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025, pp. 48–58 on Pith
- [59] C. Gan, Y . Tu, X. Chen, T. Chen, Y . Li, M. Harandi, and W. Lin, “Unleashing diffusion transformers for visual correspondence by modu- lating massive activations,”Advances in Neural Information Processing Systems, vol. 38, pp. 114 432–114 462, 2026 on Pith
- [60] C. Gan, Z. Zhao, Y . Tu, X. Chen, Z. Qin, T. Chen, M. Harandi, and W. Lin, “Massive activations are the key to local detail synthesis in diffusion transformers,” inInternational Conference on Learning Representations, vol. 2026, 2026, pp. 104 855–104 883 on Pith
- [61] Q. Huang, X. wen Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. H. Yu, “Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection- allocation,”2024 IEEE/CVF Conference on Computer Vision and Pat- tern Rec... on Pith
- [62] Q. Zhao, X. Zhang, Y . Li, Y . Xing, X. Yuan, F. Tang, S. Fan, X. Chen, D.-H. Wang, and X.-Y . Zhang, “Mca-llava: Manhattan causal attention for reducing hallucination in large vision-language models,” in Proceedings of the 33rd ACM International Conference on Multimedia, 2025... on Pith
- [63] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y . Bai, A. Chenet al., “In-context learning and induction heads,”arXiv preprint arXiv:2209.11895, 2022 on PithPith's reviewarXiv
- [64] M. E. Rulli, S. Petruzzi, E. Michielon, F. Silvestri, S. Scardapane, and A. Devoto, “Attention sinks in diffusion language models,” inFindings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 43 244–43 256 on Pith
- [65] Z. Zhang, Z. Xie, L. Zhong, H. Liu, and S. Cao, “One token is enough: Improving diffusion language models with a sink token,” inFindings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 6479–6490 on Pith
- [66] H. Zuo, Y . Li, L. Yang, and X. Zhang, “Diffusion-cam: Faithful visual explanations for dmllms,” inProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, pp. 12 085–12 101 on Pith
- [67] W. Zhu, H. Hao, and R. Wang, “Penalty decoding: Well suppress the self-reinforcement effect in open-ended text generation,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 1218–1228 on Pith
- [68] Y . Su, T. Lan, Y . Wang, D. Yogatama, L. Kong, and N. Collier, “A contrastive framework for neural text generation,”Advances in neural information processing systems, vol. 35, pp. 21 548–21 561, 2022 on Pith
- [69] Z. Fu, W. Lam, A. M.-C. So, and B. Shi, “A theoretical analysis of the repetition problem in text generation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 14, 2021, pp. 12 848– 12 856 on Pith
- [70] E. Lagutin, D. Gavrilov, and P. Kalaidin, “Implicit unlikelihood train- ing: Improving neural text generation with reinforcement learning,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021, pp. 1432–1441 on Pith
- [71] J. Xu, X. Liu, J. Yan, D. Cai, H. Li, and J. Li, “Learning to break the loop: Analyzing and mitigating repetitions for neural text generation,” IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, 202X 17 Advances in Neural Information Processing Sys... on Pith
- [72] H. Li, T. Lan, Z. Fu, D. Cai, L. Liu, N. Collier, T. Watanabe, and Y . Su, “Repetition in repetition out: Towards understanding neural text degeneration from the data perspective,”Advances in Neural Information Processing Systems, vol. 36, pp. 72 888–72 903, 2023 on Pith
- [73] H. Yang, D. Cai, H. Li, W. Lam, S. Shiet al., “A frustratingly simple decoding method for neural text generation,” inProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 536–557 on Pith
- [74] A. A. Ginart, N. Kodali, J. Lee, C. Xiong, S. Savarese, and J. R. Emmons, “Lz penalty: An information-theoretic repetition penalty for autoregressive language models,”arXiv preprint arXiv:2504.20131, 2025 on PithPith's reviewarXiv
- [75] W. Wang, Z. Li, D. Lian, C. Ma, L. Song, and Y . Wei, “Mitigating the language mismatch and repetition issues in llm-based machine translation via model editing,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 15 681– 15 700 on Pith
- [76] S. Hong, C. Yoon, and S. J. Hwang, “Mitigating mask prior drift and positional attention collapse in large diffusion vision-language models,” inForty-third International Conference on Machine Learning, 2026 on Pith
- [77] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston, “Neural text generation with unlikelihood training,”arXiv preprint arXiv:1908.04319, 2019 on PithPith's reviewarXiv
- [78] G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 21 875–21 895 on Pith
- [79] R. Ali, F. Caso, C. Irwin, and P. Li `o, “Entropy-lens: Uncovering decision strategies in llms,”arXiv preprint arXiv:2502.16570, 2025 on PitharXiv
- [80] M. Mathew, D. Karatzas, and C. Jawahar, “Docvqa: A dataset for vqa on document images,” in2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021, pp. 2199–2208 on Pith
- [81] A. Masry, J. Q. Tan, S. Joty, E. Hoqueet al., “Chartqa: A benchmark for question answering about charts with visual and logical reasoning,” inFindings of the association for computational linguistics: ACL 2022, 2022, pp. 2263–2279 on Pith
- [82] L. Chen, J. Li, X. Dong, P. Zhang, Y . Zang, Z. Chen, H. Duan, J. Wang, Y . Qiao, D. Linet al., “Are we on the right way for evaluating large vision-language models?”Advances in Neural Information Processing Systems, vol. 37, pp. 27 056–27 087, 2024 on Pith
- [83] C. Fu, P. Chen, Y . Shen, Y . Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sunet al., “Mme: A comprehensive evaluation benchmark for multimodal large language models,”Advances in Neural Information Processing Systems, vol. 38, 2026 on Pith
- [84] B. Li, R. Wang, G. Wang, Y . Ge, Y . Ge, and Y . Shan, “Seed-bench: Benchmarking multimodal llms with generative comprehension,”arXiv preprint arXiv:2307.16125, 2023 on PithPith's reviewarXiv
- [85] Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liuet al., “Mmbench: Is your multi-modal model an all-around player?” inEuropean conference on computer vision. Springer, 2024, pp. 216–233 on Pith
- [86] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 23 439– 23 554 on Pith
- [87] R. Zhang, D. Jiang, Y . Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.- W. Chang, Y . Qiaoet al., “Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?” inEuropean Conference on Computer Vision. Springer, 2024, pp. 169–186 on Pith
- [88] R. Paulus, C. Xiong, and R. Socher, “A deep reinforced model for abstractive summarization,”arXiv preprint arXiv:1705.04304, 2017 on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment