Pith. sign in

REVIEW 4 major objections 5 minor 45 references

VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VLZip claims that compressing both images and text into layer-specific soft prefixes lets a 3B pure Transformer stay accurate past 128K tokens, scoring 61.9 on LongVLBench—a 31.4% relative gain.

desk verdict VLZip's compression framework is a genuinely fresh combination and the efficiency work is solid, but the LongVLBench SOTA rests on an unreleased 140-item benchmark with an unvalidated Gemini judge and at least one leaked-looking example, so the headline reasoning claim needs verification. read the letter →

arxiv 2608.08630 v1 pith:YIZFMA3W submitted 2026-08-09 cs.CV

Yuqi Zhang , Cheng Chen , Yuyu Guo , Wenjie Yang , Lingchen Meng , Peng Di , Hang Yu , Zuxuan Wu
show 1 more author
Yu-Gang Jiang
This is my paper · ORCID
classification cs.CV
keywords long-contextvision-languagemodelsmultimodaltokencompressioninterleavedimage-textreasoningsoftprefixinjectionnarrativebenchmarkhierarchicalcontextdistillationQ-Formercompressor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to dissolve a trade-off that has shaped long-context vision-language work: either prune tokens and lose information, or swap Transformers for cheaper architectures and lose precision. It argues that both sacrifices stem from treating the problem as a visual one, when in interleaved inputs the text segments are equally heavy. VLZip therefore compresses images and text in tandem into small soft prefixes and feeds those prefixes into every decoder layer, so attention runs over a short sequence while each layer still sees distilled global context. On LongVLBench, a new video-narrative benchmark the paper builds, VLZip reports 61.9 average, 31.4% better than the next-best model, and holds up at extreme lengths where the 3B baseline scores 0.16 and VLZip itself reaches 7.60. If the results replicate, the paper would establish that high-fidelity, narrative-level reasoning over ultra-long multimodal sequences is possible inside an ordinary Transformer.

What carries the argument

The load-bearing mechanism is hierarchical context distillation with multi-layer injection. A visual compressor (a Q-Former with layer-specific queries) encodes each image chunk into $M_v$ vectors per decoder layer; a textual compressor runs each text chunk through a lightweight encoder and a second Q-Former into $M_t$ vectors per layer; and before self-attention, those vectors are added element-wise to the hidden states of placeholder tokens representing each chunk. In the default configuration, each 100-token chunk is compressed to $M=4$ tokens per layer, so the attention sequence is drastically shortened while every layer receives a fine-grained distilled summary of the entire context; the ablations attribute the strongest extreme-length and in-context-learning gains to this full-layer injection.

What would settle it

Two checks would settle it: an oracle answering each LongVLBench question from a single keyframe or a single caption segment should score far below VLZip's 61.9, and ablating the injected soft-prefix features to random vectors should collapse the extreme-length score; if either fails, the benchmark or the mechanism is not doing the work the paper claims.

Watch

Extended reading notes

Core claim

The central claim is that unified compression, not better pruning and not a new architecture, is what unlocks long-context multimodal reasoning. Concretely, each image and text segment is cut into chunks, every chunk is reduced by a Q-Former into a handful of layer-specific token vectors, and those vectors are added into the hidden states of placeholder tokens at every decoder layer before self-attention. The attention sequence is thereby short, while every layer is continuously conditioned on the distilled whole. The paper's evidence is that this scheme scores 61.9 on LongVLBench against 47.1 for the next-best model, keeps a 100% inference success rate at every tested length including beyond 128K tokens, extends training to 120K tokens (6x over the 20K uncompressed limit), and processes over 280K tokens on one 80GB accelerator.

Load-bearing premise

The claim rests on LongVLBench's 140 questions genuinely requiring narrative-level reasoning and on its automated judge faithfully scoring semantic equivalence.

Editorial extensions

If this is right

  • Training on 120K-token interleaved sequences becomes feasible on a 32-accelerator setup, where the uncompressed backbone runs out of memory at 20K tokens, so long multimodal documents can be fine-tuned directly rather than chunked.
  • A single 80GB accelerator can handle inputs beyond 280K tokens, and prefilling memory stays under 50GB even at 2M tokens, which is the regime of hour-long video and large GUI trace logs.
  • In-context learning over interleaved examples stays strong at 128K tokens (58.8), beating a 7B reference model (44.0) and most alternatives, which suggests the multi-layer injection preserves reusable patterns across many examples.
  • Compressing only one modality is insufficient: image-only compression collapses text-retrieval tasks and text-only compression collapses in-context learning, so the unified design is what balances both.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if LongVLBench's judging protocol holds up, the same soft-prefix recipe should port to other interleaved long-horizon domains, such as GUI operation logs, illustrated manuals, and visual medical histories, where the bottleneck is sustained context rather than a single image.
  • Editorial inference: freeing the tied chunk-size and token-count parameters, so image and text compression can differ, is the most direct lever for reclaiming short-context retrieval accuracy without giving up the 128K gains the paper reports.
  • Editorial inference: the benchmark-construction pipeline, with semantic keyframe selection, hierarchical captioning, and constrained human-and-model refinement, is reusable as a scaffold for generating larger narrative test sets, which would give the 61.9 result more statistical footing than 140 samples provide.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VLZip, a compression framework for long interleaved image-text sequences. VLZip partitions each image and text segment into chunks, compresses each chunk into layer-specific soft-prefix features using shared Q-Former compressors, and injects those features additively into the hidden states of placeholder tokens at every decoder layer. The model is trained in four stages, and the paper introduces LongVLBench, a 140-sample video-derived benchmark for narrative-level reasoning. Experiments report a state-of-the-art LongVLBench average of 61.9 for a 3B model, efficiency and memory measurements up to hundreds of thousands of tokens, ablations over compression and injection design choices, and results on MMLongBench and short-context VQA benchmarks.

Significance. If the central claims are supportable, VLZip is a significant contribution: it unifies visual and textual compression inside a pure Transformer decoder, and the reported memory and time scaling (Figures 5-6) is striking, including a claimed path to 2M-token contexts. The paper has real strengths: substantial ablations (Tables 4-8 and Appendix C), a single-modality isolation experiment (Table 12), a zero-effort 7B scale transfer (Table 11), and unusually complete training details. However, the headline state-of-the-art claim rests on a new, unreleased, self-constructed 140-sample benchmark whose QA-pair generation is not described and whose LLM-based judge is not validated. Until that evidence is supplied, the narrative-level reasoning claim is not established.

major comments (4)
  1. [Section 4 / Appendix D.1 / Figure 11] LongVLBench internal validity is not established, and the displayed example raises a serious concern. The QA-pair generation step is described only at a high level ('we generate QA pairs'), and the claimed human review is mentioned without any procedure, number of reviewers, or validation results. Figure 11 asks why 'the right most man in black' stands at a distance from 'the man in white', with answer 'To throw waterball'; none of the displayed cleaned captions mentions black clothing, a man in white, or a waterball. If this sample is representative, the ground-truth answers may derive from video-level metadata or from content outside the interleaved context that the models actually see. The authors should release the benchmark, describe the QA-generation procedure in detail, and provide a human answerability study demonstrating that each question is answerable from the supplied context and requires synthesis across multiple frames or text segments.
  2. [Appendix D.2 / Table 1] The Gemini semantic-equivalence judge is unvalidated. There is no human-judge agreement measurement, no report of judge calibration on a development set, and no check that the judge does not reward fluent but imprecise answers. Table 1 reports averages across seven length bins without per-bin sample counts or confidence intervals; the >128k bin could contain very few items, so the headline gap of 7.60 vs. 0.16 may not be stable. The paper should report per-bin sample counts and item-level scores, and should calibrate the judge against human scores, ideally with multiple evaluation passes or seeds.
  3. [Table 2 / Section 5.2] The MMLongBench evidence is selective. At 8k, VLZip scores 25.2 on VRAG versus 43.4 for Qwen2.5-VL-3B and 48.3 for Qwen2.5-VL-7B; on NIAH at 8k the gap is 35.7 versus 56.2 and 56.1. Describing this as a 'modest performance trade-off' understates the short-context degradation. The long-context advantage is concentrated in ICL and in the 128k rows of VRAG and NIAH. The paper should present per-task profiles and make clear that the general long-context reasoning claim is carried primarily by the LongVLBench result and by ICL, not uniformly across MMLongBench.
  4. [Table 4 / Figures 5-6 / Section 5.2] The 'training up to 120K tokens' and 'inference beyond 280K tokens' claims are supported by memory-fitting measurements, not by correctness measurements at those lengths. Table 4 reports the maximum sequence length before out-of-memory, and Figures 5-6 report memory and time; no experiment evaluates VLZip's accuracy on a 120K or 280K-token reasoning sample. The LongVLBench >128k bin is not further specified, so it is unclear whether the model is actually reasoning at 280K tokens. The paper should explicitly distinguish memory/scalability capacity from demonstrated reasoning accuracy at those lengths, or add accuracy measurements at the claimed lengths.
minor comments (5)
  1. [Figure 4] The histogram shows the token-length distribution of 26,164 generated documents, but the test set is only 140 samples; the paper should report the per-bin counts for the actual 140 test samples and explain the selection criteria.
  2. [Table 2] The caption does not explain the '-' entries, and several baselines are missing at many length bins; a short note in the caption would prevent over-interpretation of absent values.
  3. [Table 1] The header 'A vg' contains a formatting typo and should read 'Avg'.
  4. [Appendix D.3] The classifier is referred to as 'Gemini-2.5' here but 'Gemini 2.5 Pro' elsewhere; the naming should be consistent.
  5. [General] The model name is spelled both 'LongLLaVA' and 'LongLLAVA' in the text, tables, and appendix; unify the spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: VLZip's compression mechanism and reported evaluations are empirical and not forced by construction.

full rationale

VLZip is an empirical compression architecture, and its derivation is self-contained. The compression and injection equations (Eqs. 1-6) define a trainable Q-Former-based compressor with additive layer-wise injection; the reported efficiency, memory, and MMLongBench results are measured quantities, not identities that follow from the definitions. The LongVLBench SOTA is an evaluation outcome rather than a derived consequence: VLZip is not trained on LongVLBench, and the benchmark's construction and Gemini-based judging raise validity concerns (same-team benchmark, unvalidated semantic-equivalence judge, possible selection on the benchmark through the ablation in Table 7), but these are correctness and generalizability risks, not circular reductions of the claimed result to its own inputs. The only self-citations, such as DeepStack as related work, are contextual and not load-bearing. No circular step was found.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on learnable compression modules and a self-built, LLM-judged benchmark. The main free parameters are the chunk size and tokens-per-chunk (both tied across modalities), selected by ablation. The assumptions are domain-level: that 4-token-per-layer compression preserves reasoning-critical information, that layer-wise addition conveys that information to attention, and that the Gemini-judged 140-sample LongVLBench measures true narrative reasoning.

free parameters (3)
  • chunk size C_v = C_t = 100 tokens
    Tied image and text chunk sizes, chosen via ablation (Table 6) to optimize LongVLBench >128k and ICL-128k; C=100 wins over 25 and 50 for ultra-long tasks.
  • tokens per chunk per layer M_v = M_t = 4
    Chosen via ablation (Table 6); M=4 gives best ICL-128k and LongVLBench average, while M=2 loses capacity and M=8 hurts ICL.
  • injection strategy = full-layer
    Selected from Table 7 ablation; full-layer injection gives best ICL and LongVLBench average, though first-layer gives better VRAG/NIAH.
assumptions (5)
  • domain assumption A Q-Former with L*M queries can distill a 100-token chunk into 4 tokens per layer without losing the information needed for downstream reasoning.
    This is the core learnability assumption of the compression modules; it is not proven and is only validated by downstream task scores.
  • domain assumption Element-wise addition of compressed features into placeholder hidden states at every layer preserves global context sufficiently for self-attention to reason over it.
    The injection mechanism (Eq. 6) is assumed to encode context into the placeholder positions; the paper provides no analysis of information flow.
  • domain assumption The Gemini-2.5-judged semantic equivalence rubric yields a valid and consistent measure of reasoning quality for LongVLBench.
    The entire LongVLBench SOTA claim rests on this LLM-as-judge protocol; no human agreement study is reported.
  • domain assumption The 140 LongVLBench QA pairs genuinely require narrative-level reasoning and are not answerable from a single frame or caption.
    Questions were generated by ShareGPT4Video and refined by Gemini 2.5 Pro with human review, but no control experiment (e.g., single-frame baselines) is shown.
  • domain assumption The four-stage training curriculum with frozen and unfrozen modules is a valid way to train the compressors without catastrophic forgetting.
    Assumed by the training pipeline; no comparison against joint end-to-end training from scratch is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling." pith.science (2026). https://pith.science/paper/YIZFMA3W

@misc{pith2026260808630,
  author       = {Pith},
  title        = {Pith review of: VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YIZFMA3W}},
  note         = {Machine review of arXiv:2608.08630}
}
read the original abstract

Vision Language Models (VLMs) face significant challenges with ultra-long, interleaved image-text sequences due to the quadratic complexity of self-attention. Current solutions either resort to aggressive token pruning, risking irreversible information loss, or adopt efficient but less precise architectures, while largely ignoring the equally vital textual component. We introduce VLZip, a framework that unifies visual and textual compression for high-fidelity reasoning within a pure Transformer. At its core, VLZip hierarchically distills visual and textual segments into compact, layer-specific "soft prefixes" and injects them into each decoder layer's hidden states, drastically shortening the attention sequence while preserving fine-grained global context. To address deficient evaluations in the field, we also introduce LongVLBench, a new benchmark derived from video narratives that demands holistic, narrative-level reasoning. Extensive experiments show VLZip achieves leading performance on long-context multimodal reasoning, enabling training up to 120K tokens, a 6x increase over the baseline, and inference beyond 280K tokens with significantly reduced memory, while demonstrating the memory scalability to handle up to 2M tokens. By excelling at extreme context lengths where existing methods collapse, VLZip establishes an efficient and powerful new standard for long-context multimodal AI. Code is available at https://github.com/ShareLab-SII/VLZip.

Figures

Figures reproduced from arXiv: 2608.08630 by the authors.

Figure 1
Figure 1. A comparison of performance and efficiency for VLZip(Ours, red star) and baseline models. Performance across con￾text lengths is indicated by the color in￾tensity of the points. On the architectural side, existing approaches typically address this chal￾lenge by accepting a trade-off between efficiency and fidelity (as shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of VLZip Architecture. VLZip compresses interleaved image￾text sequences via modality-specific hierarchical modules. The visual module partitions each image into chunks and compresses each into Mv layer-specific tokens via a shared Q-Former. The textual module partitions each text segment into chunks, encodes each chunk with a lightweight text encoder, and compresses each into Mt layer-specific tokens via a… view at source ↗
Figure 3
Figure 3. Overview of our four-stage dataset creation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Token length distribution of the interleaved dataset. The histogram shows the frequency of the 26,164 generated documents across seven token length bins. the model to understand plot progression, character inter￾actions, and evolving context. The refined, interleaved n…
Figure 5
Figure 5. Figure 5: Analysis of resource consumption (memory and time) versus input token length on the LongVLBench dataset. design: by continuously providing fine-grained global context to every decoder layer, VLZip retains the ability to leverage long-range in-context examples even at e…
Figure 6
Figure 6. Figure 6: Prefilling peak memory vs. input token length. LongLLaVA reaches the limit around 512K tokens. The uncompressed base￾line (Qwen2.5-VL) similarly runs out of memory beyond 256K tokens. In con￾trast, VLZip’s memory curve flattens af￾ter 512K tokens by batch-processing im…
Figure 7
Figure 7. Figure 7: Detailed results of models on NIAH task of MMLongBench at various lengths. Bench at 128k. We compare the uncompressed Qwen2.5-VL-3B baseline against three compressed variants: Image-only, which compresses only visual segments; Text-only, which compresses only text segm…
Figure 8
Figure 8. Figure 8: Detailed results of models on ICL task of MMLongBench at various lengths [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Detailed results of models on VRAG task of MMLongBench at various lengths. A notable observation from the NIAH breakdown (Fig￾ure 7) is that VLZip exhibits relatively weaker performance on text-centric NIAH sub￾tasks (e.g., MM-NIAH-Ret (T), MM-NIAH-Reason (T)) com￾pare…
Figure 10
Figure 10. Figure 10: Representative GQA failure cases: color (A), spatial relation (B), and semantic granularity (C). cess begins with semantic keyframe extraction from source videos using CLIP￾based similarity to capture significant visual changes. Subsequently, a powerful VLM generates …
Figure 11
Figure 11. Figure 11: A visualization of a single data sample from our constructed dataset. precision as the ground truth. For instance, a full-sentence answer like “There are two people" is correctly judged as a perfect semantic match to the ground-truth keyword “two". This ensures that m…
Figure 12
Figure 12. Figure 12: Distribution analysis of LongVLBench across three dimensions: (a) Topic, (b) Question Intent, and (c) Answer Form. D.3 Dataset Diversity Analysis To quantitatively validate the diversity of LongVLBench, we employ an LLM￾based classifier (Gemini-2.5) to automatically t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 29 canonical work pages

  1. [1]

    Zhang et al

    Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Men- sch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J.L., Borgeaud, S., Brock, A., 16 Y. Zhang et al. Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisser- man, A., Si...

    work page 2022

  2. [2]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  3. [3]

    Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Bin, L., Tang, Z., Yuan, L., Qiao, Y., Lin, D., Zhao, F., Wang, J.: Sharegpt4video: Im- proving video understanding and generation with better captions. In: Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, N...

    work page 2024

  4. [4]

    CoRR abs/2412.05271(2024).https://doi.org/10.48550/ARXIV.2412.05271

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., Gu, L., Wang, X., Li, Q., Ren, Y., Chen, Z., Luo, J., Wang, J., Jiang, T., Wang, B., He, C., Shi, B., Zhang, X., Lv, H., Wang, Y., Shao, W., Chu, P., Tu, Z., He, T., Wu, Z., Deng, H., Ge, J., Chen, K., Dou, M., Lu, L., Zhu, X., Lu, T., Lin, D., Qiao, Y., Dai, J., W...

    arXiv doi:10.48550/arxiv.2412.05271 2024

  5. [5]

    Advances in Neural Information Processing Systems37, 109487–109516 (2024)

    Cheng, X., Wang, X., Zhang, X., Ge, T., Chen, S.Q., Wei, F., Zhang, H., Zhao, D.: xrag: Extreme context compression for retrieval-augmented generation with one token. Advances in Neural Information Processing Systems37, 109487–109516 (2024)

    work page 2024

  6. [6]

    Chevalier, A., Wettig, A., Ajith, A., Chen, D.: Adapting language models to com- presscontexts.In:The2023ConferenceonEmpiricalMethodsinNaturalLanguage Processing (2023)

    work page 2023

  7. [7]

    In: The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Dao, T.: Flashattention-2: Faster attention with better parallelism and work partitioning. In: The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net (2024), https://openreview.net/forum?id=mZn2Xyh9Ec, accessed: 2026-06-25

    work page 2024

  8. [8]

    In: The Twelfth International Conference on Learning Representations (2024)

    Ge, T., Jing, H., Wang, L., Wang, X., Chen, S.Q., Wei, F.: In-context autoencoder for context compression in a large language model. In: The Twelfth International Conference on Learning Representations (2024)

    work page 2024

Show all 45 references
  1. [9]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 6700–6709 (2019)

    2019

  2. [10]

    Jiang, D., He, X., Zeng, H., Wei, C., Ku, M., Liu, Q., Chen, W.: Mantis: Interleaved multi-image instruction tuning. Trans. Mach. Learn. Res.2024(2024)

    2024

  3. [11]

    In: European conference on computer vision

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A di- agram is worth a dozen images. In: European conference on computer vision. pp. 235–251. Springer (2016)

    2016

  4. [12]

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., Li, C.: Llava-onevision: Easy visual task transfer. Trans. Mach. Learn. Res.2025(2025),https://openreview.net/forum?id=zKv8qULV6n, ac- cessed: 2026-06-25

    2025

  5. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., Shan, Y.: Seed- bench: Benchmarking multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13299– 13308 (2024) VLZip: Unified Visual and Textual Compression 17

    2024

  6. [14]

    In: International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA.ProceedingsofMachineLearningResearch,vol.202,pp.19730–19742.PMLR (2023)

    Li, J., Li, D., Savarese, S., Hoi, S.C.H.: BLIP-2: bootstrapping language-image pre- training with frozen image encoders and large language models. In: International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA.ProceedingsofMachineLearningR...

    2023

  7. [15]

    In: Bouamor, H., Pino, J., Bali, K

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.: Evaluating object hallu- cination in large vision-language models. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore,...

    2023 doi

  8. [16]

    Li, Z., Liu, Y., Su, Y., Collier, N.: Prompt compression for large language models: A survey. In: Proceedings of the 2025 Conference of the Nations of the Ameri- cas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). ...

    2025

  9. [17]

    In: Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing

    Liao, Z., Wang, J., Yu, H., Wei, L., Li, J., Zhang, W.: E2llm: Encoder elongated large language models for long-context understanding and reasoning. In: Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 19212–19241 (2025)

    2025

  10. [18]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 26286–26296. IEEE (2024)

    2024

  11. [19]

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 (2023)

    2023

  12. [20]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. pp. 216–233. Springer (2024)

    2024

  13. [21]

    In: The 36th Conference on Neural Information Processing Systems (NeurIPS) (2022)

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. In: The 36th Conference on Neural Information Processing Systems (NeurIPS) (2022)

    2022

  14. [22]

    arXiv:2405.20797 (2024)

    Lu, S., Li, Y., Chen, Q.G., Xu, Z., Luo, W., Zhang, K., Ye, H.J.: Ovis: Struc- tural embedding alignment for multimodal large language model. arXiv:2405.20797 (2024)

    2024 arXiv

  15. [23]

    In: Proceedings of the IEEE/cvf conference on computer vision and pattern recognition

    Marino,K.,Rastegari,M.,Farhadi,A.,Mottaghi,R.:Ok-vqa:Avisualquestionan- swering benchmark requiring external knowledge. In: Proceedings of the IEEE/cvf conference on computer vision and pattern recognition. pp. 3195–3204 (2019)

    2019

  16. [24]

    Ad- vances in Neural Information Processing Systems37, 23464–23487 (2024)

    Meng, L., Yang, J., Tian, R., Dai, X., Wu, Z., Gao, J., Jiang, Y.G.: Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Ad- vances in Neural Information Processing Systems37, 23464–23487 (2024)

    2024

  17. [25]

    Advances in Neural Information Processing Systems36, 19327–19352 (2023)

    Mu, J., Li, X., Goodman, N.: Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems36, 19327–19352 (2023)

    2023

  18. [26]

    In: Proceedings of the 38th In- ternational Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Proceedings of the 38th In- ternational Conference on M...

    2021

  19. [27]

    Shen, Y., Fu, C., Dong, S., Wang, X., Zhang, Y.F., Chen, P., Zhang, M., Cao, H., Li, K., Zheng, X., et al.: Long-vita: Scaling large multi-modal models to 1 million tokenswithleadingshort-contextaccuracy.arXivpreprintarXiv:2502.05177(2025)

    2025

  20. [28]

    CoRRabs/2404.18532(2024)

    Song, D., Chen, S., Chen, G.H., Yu, F., Wan, X., Wang, B.: Milebench: Bench- marking mllms in long context. CoRRabs/2404.18532(2024)

    2024 arXiv

  21. [29]

    In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Sys- tems 2017, December 4-9, 2017, Long Beach, CA, USA

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Sys- tems 2017, December 4-9, 2017, Long Beach, ...

    2017

  22. [30]

    Wang, W., Zhang, S., Ren, Y., Duan, Y., Li, T., Liu, S., Hu, M., Chen, Z., Zhang, K., Lu, L., Zhu, X., Luo, P., Qiao, Y., Dai, J., Shao, W., Wang, W.: Needle in A multimodal haystack. In: Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa...

    2024

  23. [31]

    CoRRabs/2409.02889 (2024)

    Wang, X., Song, D., Chen, S., Zhang, C., Wang, B.: Longllava: Scaling multi-modal llms to 1000 images efficiently via hybrid architecture. CoRRabs/2409.02889 (2024)

    2024

  24. [32]

    Wang, Z., Yu, W., Ren, X., Zhang, J., Zhao, Y., Saxena, R., Cheng, L., Wong, G., See, S., Minervini, P., Song, Y., Steedman, M.: Mmlongbench: Benchmarking long-contextvision-languagemodelseffectivelyandthoroughly.In:The39th(2025) Annual Conference on Neural Information Process...

    2025

  25. [33]

    In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025

    Xu, P., Ping, W., Wu, X., Xu, C., Liu, Z., Shoeybi, M., Catanzaro, B.: Chatqa 2: Bridging the gap to proprietary llms in long context and RAG capabilities. In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRev...

    2025

  26. [34]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025

    Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., Jia, J.: Visionzip: Longer is better but not necessary in vision language models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 19792–19802. Compu...

    2025

  27. [35]

    In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025

    Ye, Z., Xia, K., Fu, Y., Dong, X., Hong, J., Yuan, X., Diao, S., Kautz, J., Molchanov, P., Lin, Y.C.: Longmamba: Enhancing mamba’s long-context capabil- ities via training-free receptive field enlargement. In: The Thirteenth International Conference on Learning Representations...

    2025

  28. [36]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.: Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  29. [37]

    CoRR abs/2508.01548(2025)

    Zeng, Q., Li, Y., Wang, Q., Jiang, P., Wu, Z., Cheng, M., Hou, Q.: A glimpse to compress: Dynamic visual token pruning for large vision-language models. CoRR abs/2508.01548(2025)

    2025 arXiv

  30. [38]

    In: Findings of the Association for Computational Linguis- tics: NAACL 2025

    Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J.A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., Liu, Z.: Lmms-eval: Reality check on the evaluation of large multimodal models. In: Findings of the Association for Computational Linguis- tics: NAACL 2025. p. 881–916. Associati...

    2025 doi

  31. [39]

    In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025

    Zhou, C., Zhang, M., Chen, P., Fu, C., Shen, Y., Zheng, X., Sun, X., Ji, R.: Learning interleaved image-text comprehension in vision-language large models. In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRev...

    2025

  32. [40]

    prune-and-forget

    Zhou, K., Tang, Z., Ming, L., Zhou, G., Chen, Q., Qiao, D., Yang, Z., Qin, L., Qiu, M., Li, J., Zhang, M.: Mmlongcite: A benchmark for evaluating fidelity of long-context vision-language models. CoRRabs/2510.13276(2025) 20 Y. Zhang et al. A Extended Related Work The advancemen...

    2025

  33. [41]

    Analyze the [Question]

  34. [42]

    Identify the core, specific information in the [Ground Truth Answer]

  35. [43]

    Determine if the [Model’s Response] contains this exact same core information, with the same level of precision

  36. [44]

    half right

    Assign aninteger score from 0 to 10based on the followingFine-Grained Semantic Rubric: •10 (Perfect):Semantically identical to the ground truth. Conveys the exact information with the same precision. •9 (Excellent):Correctly includes the entire ground truth but adds minor, non...

  37. [45]

    score" (an integer from 0 to 10) and “justification

    Provide your outputONLYin a valid JSON format with two keys: “score" (an integer from 0 to 10) and “justification" (a concise string explaining your reasoning). Do not add any text before or after the JSON object. — [Question] {question} — [Ground Truth Answer] {answer} — [Mod...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.