Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Vision-text compression's real bottleneck is cross-path inconsistency, and a self-supervised method closes most of the gap.

desk verdict A real diagnosis and a plausible fix, but the paper never proves the aligned model is actually reading the rendered image, so the headline claim is under-evidenced. read the letter →

arxiv 2608.02109 v1 pith:55MCT7OL submitted 2026-08-03 cs.CV

Tianyu Liang , Xiangxi Zheng , Yilin Wang , Dongxing Mao This is my paper · ORCID
classification cs.CV
keywords Vision-TextCompressionCross-pathalignmentinconsistencyOn-policydistillationDirectpreferenceoptimizationLong-contextmodelingMultimodallargelanguagemodelsSelf-improving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-Text Compression (VTC) promises cheap long-context processing by rendering long text into images and encoding it with the vision transformer, but it consistently underperforms native text input. This paper argues that the cause is not compression itself but cross-path inconsistency: the vision encoder reads glyphs, fonts, and layout, while the language model expects linguistic semantics. The paper claims this gap can be closed with only the model's own text-input behavior as supervision, through SPIRAL, which combines token-level on-policy distillation and sequence-level preference optimization. On VTCBench, SPIRAL raises Qwen3-VL-8B from 35.10 to 54.02, close to the native-text score of 55.60, while adding no cost at inference time. If right, representational fidelity rather than compression ratio is the key design target for VTC.

What carries the argument

The load-bearing object is the pair of conditional distributions $p_T$ and $p_S$ produced by the same multimodal backbone from native text and from rendered-image input. The mechanism that carries the argument is self-generated cross-path supervision: because the same model generates both, no external teacher is needed. OPD applies a mode-seeking divergence (reverse KL) at each decoding step on the image path's own rollouts, sharpening $p_S$ toward $p_T$; DPO steers the image-path policy at the sequence level by maximizing the margin between text-derived and image-derived responses. Both operate only during training, leaving the compressed inference path unchanged.

What would settle it

Render a long context at a resolution low enough that a competent OCR system cannot recover the text, keep the query in native text, and run SPIRAL: if the aligned model still approaches native-text scores, the recovery cannot come from reading the rendered content and the premise that no information is lost must be false.

Watch

Extended reading notes

Core claim

The paper's central discovery is that VTC's performance gap is dominated by a mismatch between the two paths the same backbone can take for identical content: native text tokenization and rendered-image encoding. They introduce SPIRAL, which supervises the image path with the text path's own outputs. Token-level OPD distills the image path's next-token distribution toward the text path's distribution along self-generated image-path trajectories, using reverse KL; sequence-level DPO constructs preference pairs where the native-text answer is preferred and the rendered-image answer is dispreferred, both conditioned on the rendered input. Together these close most of the 20.5-point gap to native text, improve robustness to rendering perturbations, and generalize to context lengths and benchmarks not seen in training.

Load-bearing premise

The paper assumes the problem is how the model reads the rendered image, not whether the rendered image still contains the information; if rendering destroys content, then pushing image-path behavior toward text-path behavior could make the model guess from the question instead of reading.

Editorial extensions

If this is right

  • VTC benchmarks should report or control for cross-path alignment, since a large share of the performance gap can be removed without changing the compression pipeline.
  • Models aligned this way preserve the full token-compression and inference-speedup benefits of VTC, because alignment adds nothing at inference time.
  • Alignment trained on contexts up to 8k tokens transfers to 32k contexts, implying the method learns a general semantic grounding rather than length-specific patterns.
  • OPD and DPO offer complementary scaling behavior: OPD is sample-efficient and best for retrieval, while DPO improves reasoning and memory and benefits from more preference data.
  • The gains generalize out of domain (LongBench, 2WikiMultihopQA, GSM8K, TriviaQA) and to a second backbone, InternVL3.5-8B.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if DPO reasoning at 32k surpasses native text, the rendered-image path plus alignment may act as a beneficial regularizer for very long contexts, not merely a lossy approximation.
  • Beyond the paper: the same dual-view alignment recipe could apply to other rendered modalities such as tables, charts, or code listings, where a text-path teacher exists and the vision encoder may again prefer layout over content.
  • Beyond the paper: a hybrid objective that interleaves or schedules OPD and DPO may outperform either alone, since the paper reports complementary strengths but does not combine them.
  • Beyond the paper: the surprising ability to match or beat text-mode reasoning could also mean the model is learning to exploit question priors rather than reading the rendered context; a content-masking control would separate these.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper addresses Vision-Text Compression (VTC), where long text is rendered into an image and encoded through the vision encoder, yielding visual tokens that are much fewer than the native text tokens. The authors identify "cross-path inconsistency": the model's behavior on the rendered-image input diverges from its behavior on the same content given as native text. They propose SPIRAL, a self-supervised framework that aligns the image-path output distribution toward the text-path distribution using token-level on-policy distillation (OPD) and sequence-level direct preference optimization (DPO), with all supervision derived from the model's own text-path outputs. On VTCBench, SPIRAL improves Qwen3-VL-8B from 35.10 to 54.02, approaching the native text-input score of 55.60 and outperforming existing VTC methods; the authors also report gains on out-of-domain benchmarks and on a second backbone, InternVL3.5-8B. The paper concludes that cross-path alignment, rather than compression ratio alone, is the central challenge for effective VTC.

Significance. If the central claim holds, the paper reframes the VTC research agenda from compression efficiency to cross-path representation alignment and offers a practical, fully self-supervised recipe that preserves inference-time efficiency. The strengths are the self-supervised formulation (no external teacher or annotations), the systematic comparison of token-level versus sequence-level alignment, the perturbation-robustness analysis, the reported NLL-gap reduction, and the preliminary cross-backbone evidence. However, the significance is contingent on two issues: the in-domain gains may be inflated by overlap between the training data and the evaluation benchmark families, and the paper does not demonstrate that the aligned model actually reads the rendered image rather than relying on a query-conditioned text prior. Both issues are testable and fixable, but they are load-bearing for the interpretation of the results.

major comments (4)
  1. [Section 3.2, Table 1] The rendering-perturbation experiment does not establish the premise that the cross-path gap is due to style sensitivity rather than information loss. A model that ignores the rendered image and answers from the query and a text prior would also be invariant to rendering perturbations, because it never reads the image. The conclusion stated at the end of Section 3.2, that "the cross-path gap arises because the ViT encoder is sensitive to rendering style rather than to the linguistic content," is therefore not supported by the presented evidence. Please add a content-substitution experiment: hold the query and rendering style fixed while replacing the rendered document with a semantically different document, and measure whether the model's outputs and accuracy track the substituted content. Such an experiment is necessary to distinguish genuine semantic grounding from image blindness.
  2. [Section 4.1, Table 2] The training data overlaps the evaluation benchmark families. Retrieval training uses RULER needles in a PG19 haystack, reasoning training uses NoLiMa needles, and memory training uses LoCoMo-style conversations, while evaluation uses RULER, NoLiMa, and LoCoMo. The headline improvement from 35.10 to 54.02 may therefore be substantially inflated by in-distribution task familiarity. The out-of-domain results in Table 3 are encouraging, but they do not quantify how much of the VTCBench gain is attributable to the overlap. Please report results on an evaluation split whose task families and data sources are disjoint from the training data, or retrain on non-overlapping task families, and explicitly discuss the degree of overlap and its possible effect on the main comparison.
  3. [Section 4.4, Figure 3, Eq. (5)] The DPO reasoning result at 32k (22.26) exceeds the native text-mode score at the same length (7.18). Because the DPO objective uses text-path responses as the preferred samples, a pure imitation mechanism should not systematically outperform the teacher. This result raises the concern that DPO is learning a shortcut: satisfying the preference by increasing dependence on the query and a text prior rather than by reading the rendered image. Please provide diagnostics that test actual use of the rendered content, for example: (a) content substitution (same query, different rendered document), (b) occlusion or removal of the image tokens, and (c) analysis of attention attributed to image tokens. These tests are directly feasible and would resolve whether the observed gains reflect genuine cross-path alignment or an image-blind policy.
  4. [Section 4.2, Tables 2, 4, 5] The manuscript reports no error bars, no number of seeds, and no significance tests for any of the headline numbers. The central comparison, OPD at 54.02 versus text mode at 55.60, is a 1.58-point difference that may fall within the run-to-run variance of fine-tuning an 8B model. Please report mean and standard deviation over at least three seeds for all main configurations (baseline, SFT, OPD, DPO, and text mode) on the primary VTCBench results, and provide a statistical test for the claimed gap-closing effect.
minor comments (5)
  1. [Section 4.1] The phrase "disjoint needle vocabularies" is not explained; please define how disjointness is enforced and clarify why it does not prevent task-format leakage between training and evaluation.
  2. [Tables 2 and 4] Table 4 reports an OPD total of 55.57 at 10k training examples, while Table 2 reports OPD at 54.02 for what appears to be the same method; the relationship between these numbers is unclear. Please state the exact evaluation configuration used in Table 4 (context lengths, task subsets) and explain why the 10k model appears to outperform the 30k model.
  3. [Section 3.3, Eq. (2)] The notation for the divergence is confusing: the text says D(pT, pS) denotes KL(pS||pT). Please use a consistent argument order, e.g., D(pT, pS) with a convention that the first argument is the target, or define D explicitly as D_target(source).
  4. [Figure 3 and Table 2] The native text-mode reasoning score in Figure 3 at 32k (7.18) differs dramatically from the overall text-mode reasoning score in Table 2 (37.26). Please clarify the context-length composition of the Table 2 number and explicitly state that Figure 3 reports a length-conditioned breakdown.
  5. [Section 5.2] The limitations section does not discuss the possibility that the model may learn to rely on the query or a text prior instead of the rendered image. Please add an explicit discussion of this failure mode and how it relates to the reported results.

Circularity Check

1 steps flagged · score 2.0 of 10

SPIRAL's core self-distillation is not circular; only the Section 4.6 NLL-gap analysis restates the training objective by construction.

  1. self definitional [Section 4.6, 'Cross-path NLL gap', Figure 5(a)]
    "Cross-path NLL gap. Figure 5(a) quantifies the cross-path inconsistency via the NLL gap between text-input and image-input predictions. The unaligned baseline exhibits a large reasoning gap (3.15). SPIRAL substantially closes this gap: DPO reduces the average to 0.51, while SFT only reaches 1.24. This confirms that explicit cross-path alignment is more effective than off-policy adaptation at closing the distributional gap."

    The quantity being measured is the divergence between pT and pS, which is exactly what the training objectives are defined to minimize: L_OPD = sum_t D(pS(·|y<t,r(x,s),q), pT(·|y<t,x,q)) in Eq. (2), and L_DPO in Eq. (5) increases the probability of the native-text response y+ under the rendered-image input relative to the image-path response y-. Reporting that SPIRAL reduces the NLL gap therefore confirms that the model moved toward the training target; it is a property of the objective, not an independent verification of semantic grounding. The paper uses this result as evidence of 'genuine alignment rather than task-specific overfitting,' but the reduction is by construction. The headline VTCBench and OOD benchmark gains are external and are not circular.

full rationale

The central derivation of SPIRAL is a legitimate self-distillation scheme rather than equation-level circularity: the same MLLM backbone's text-path distribution pT supervises the image-path distribution pS, and the headline success criteria are VTCBench and out-of-domain benchmark accuracies, which are external to the training loss. The one clearly self-definitional element is the Section 4.6 cross-path NLL-gap analysis, because OPD and DPO are defined to move pS toward pT, so closing that gap is a restatement of the training objective rather than independent confirmation. The perturbation-robustness evidence is suggestive but also consistent with an image-blind model answering from a text-path prior, so it does not by itself prove the ViT reads linguistic semantics; that is an inference gap rather than a circular derivation. Generalization to LongBench, TriviaQA, 2WikiMultihopQA, and GSM8K provides independent external grounding for the central claim, so the paper does not reduce to its own inputs. Score 2 reflects one minor self-confirmatory metric while the main claim retains independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on benchmark tasks, a rendering pipeline, and hyperparameters rather than on a derivation from first principles. The most consequential assumptions are that rendering retains information and that the text path is a safe teacher. These are plausible but not directly verified in the paper.

free parameters (4)
  • DPO beta = 0.1
    Sharpness of the preference margin, set in Section 4.1 with no sensitivity analysis; affects the strength of sequence-level alignment.
  • Rendering parameters (font, line height, resolution, JPEG quality) = Helvetica 12px, line height 1.2, 896x896 at 96 DPI, JPEG quality 85
    Chosen by hand in Section 4.1; determines token compression (2.54x) and readability, and the paper acknowledges rendering choices are not ablated.
  • Training data task mix = 40k retrieval, 21k reasoning, 54k memory
    Hand-chosen composition in Section 4.1; scaling conclusions in Section 4.3 depend on these sizes and the decision not to run OPD at 100k.
  • Divergence choice for OPD = reverse KL (default), forward KL comparable, JSD degrades
    Selected in Section 4.5 as default; the choice affects token-level alignment behavior.
assumptions (4)
  • domain assumption Text path and image path share the same backbone and the text path provides a reliable teacher for the image path.
    Invoked in Section 3.2; if text-path outputs are flawed or too different from what the image path can express, alignment could propagate errors or cause hallucination.
  • domain assumption Rendering preserves the semantic content of the long text sufficiently for alignment to recover it.
    Section 3.2 motivation assumes the gap is representational, not informational; the perturbation experiment does not isolate information loss.
  • ad hoc to paper Filtering out incorrectly answered teacher samples removes bad supervision.
    Section 4.1 applies this filter without evaluating its effect or defining 'incorrectly answered' criteria.
  • domain assumption Standard benchmarks (RULER, NoLiMa, LoCoMo, LongBench) measure the intended long-context abilities.
    Used as training and evaluation targets; benchmark-specific artifacts may drive scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression." pith.science (2026). https://pith.science/paper/55MCT7OL

@misc{pith2026260802109,
  author       = {Pith},
  title        = {Pith review of: Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/55MCT7OL}},
  note         = {Machine review of arXiv:2608.02109}
}
read the original abstract

Vision-Text Compression (VTC) renders long texts into images and encodes them through the vision encoder (ViT), compressing thousands of text tokens into far fewer visual tokens. However, since the ViT is pretrained predominantly on natural images, it captures visual attributes (glyphs, font sizes, layout) rather than linguistic semantics, causing rendered-image representations to diverge from native-text representations. We term this cross-path inconsistency and show, via rendering perturbation experiments, that it is a critical yet overlooked bottleneck of VTC. We propose SPIRAL (Self-improving Path Integration and Realignment), a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision, requiring no external teachers or additional annotations. SPIRAL operates at two complementary granularities: token-level on-policy distillation (OPD) for local faithfulness, and sequence-level preference optimization (DPO) for global coherence. On VTCBench, SPIRAL improves the overall score of Qwen3-VL-8B from 35.10 to 54.02, approaching the native text-input performance (55.60) and outperforming models up to 30x larger. The two granularities exhibit complementary strengths: OPD excels at retrieval and is sample-efficient, while DPO is stronger on reasoning and memory and scales better with data. SPIRAL's benefits also generalize to out-of-domain benchmarks, confirming that effective VTC hinges on aligning rendered-image representations back to native-text semantics.

Figures

Figures reproduced from arXiv: 2608.02109 by the authors.

Figure 1
Figure 1. Overview of cross-path inconsistency in VTC. The [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the SPIRAL framework. The same MLLM backbone processes both native text input and rendered-image [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance vs. context length (1k–32k) for retrieval (left) and reasoning (right). OPD maintains strong retrieval at [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effect of training scale on overall performance. OPD [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: (a) Cross-path NLL gap across retrieval, reasoning, and memory (image NLL [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 8 canonical work pages

  1. [1]

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. 2024. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. arXiv:2306.13649 [cs.LG] https: //arxiv.org/abs/2306.13649

    arXiv 2024

  2. [2]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    arXiv 2022

  3. [3]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. arXiv:2308.14508 [cs.CL] https://arxiv.org/abs/2308.14508

    arXiv 2024

  4. [4]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long- Document Transformer. arXiv:2004.05150 [cs.CL] https://arxiv.org/abs/2004. 05150

    arXiv 2020

  5. [5]

    Rae, Erich Elsen, and Laurent Sifre

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Ruther- ford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, O...

    arXiv 2022

  6. [6]

    Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. 2022. Recurrent Memory Transformer. arXiv:2207.06881 [cs.CL] https://arxiv.org/abs/2207.06881

    arXiv 2022

  7. [7]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending Context Window of Large Language Models via Position Interpolation. arXiv:2306.15595 [cs.CL] https://arxiv.org/abs/2306.15595

    arXiv 2023

  8. [8]

    Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, Yushi Bai, Jie Tang, Hongning Wang, and Minlie Huang. 2025. Glyph: Scaling Context Windows via Visual-Text Compression. arXiv:2510.17800 [cs.CV] https://arxiv.org/abs/2510.17800

    arXiv 2025

Show all 56 references
  1. [9]

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2022. Rethinking Attention with Performers. arXiv:2009.14794 [cs.LG...

    2022 arXiv

  2. [10]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG] https://a...

    2021 arXiv

  3. [11]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAt- tention: Fast and Memory-Efficient Exact Attention with IO-Awareness.Advances in neural information processing systems35 (2022), 16344–16359

    2022

  4. [12]

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. 2023. LongNet: Scaling Transformers to 1,000,000,000 Tokens. arXiv:2307.02486 [cs.CL] https://arxiv.org/abs/2307.02486

    2023 arXiv

  5. [13]

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens. arXiv:2402.13753 [cs.CL] https://arxiv.org/ abs/2402.13753

    2024 arXiv

  6. [14]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. KTO: Model Alignment as Prospect Theoretic Optimization. arXiv:2402.01306 [cs.LG] https://arxiv.org/abs/2402.01306

    2024 arXiv

  7. [15]

    Qianhan Feng, Wenshuo Li, Tong Lin, and Xinghao Chen. 2024. Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model. arXiv:2412.01282 https://arxiv.org/abs/2412.01282

    2024 arXiv

  8. [16]

    Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, and See-Kiong Ng. 2025. CHiP: Cross-modal Hierarchical Direct Preference Optimization. arXiv:2501.16629 [cs.CL] https://arxiv.org/abs/2501.16629

    2025 arXiv

  9. [17]

    Yonghan Gao, Zehong Chen, Lijian Xu, Jingzhi Chen, Jingwei Guan, and Xingyu Zeng. 2026. ZeroSense: How Vision Matters in Long Context Compression. arXiv:2603.11846 [cs.CV] https://arxiv.org/abs/2603.11846

    2026

  10. [18]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reason- ing Steps. InProceedings of the 28th International Conference on Computational Linguistics, Donia Scott, Nuria Bel, and Chengqing ...

    2020 doi

  11. [19]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. ORPO: Monolithic Preference Optimization without Reference Model. arXiv:2403.07691 [cs.CL] https://arxiv. org/abs/2403.07691

    2024 arXiv

  12. [20]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654 [cs.CL] https: //arxiv.org/abs/2404.06654

    2024 arXiv

  13. [21]

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. InProceed- ings of the 30th ACM international conference on multimedia. 4083–4091

    2022

  14. [22]

    Le, Yunhsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. arXiv:2102.05918 [cs.CV] https://arxiv.org/abs/2102.05918

    2021 arXiv

  15. [23]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. arXiv:2310.05736 https://arxiv.org/abs/2310.05736

    2023 arXiv

  16. [24]

    Songtao Jiang, Yan Zhang, Ruizhe Chen, Tianxiang Hu, Yeying Jin, Qinglin He, Yang Feng, Jian Wu, and Zuozhu Liu. 2025. Modality-Fair Preference Optimization for Trustworthy MLLM Alignment. arXiv:2410.15334 [cs.CV] https://arxiv.org/ abs/2410.15334

    2025 arXiv

  17. [25]

    Dian Jiao, Jiaxin Duan, Shuai Zhao, Jiabing Leng, Yiran Zhang, and Feng Huang

  18. [26]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehen- sion. arXiv:1705.03551 [cs.CL] https://arxiv.org/abs/1705.03551

    2017 arXiv

  19. [27]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through Memorization: Nearest Neighbor Language Models. arXiv:1911.00172 https://arxiv.org/abs/1911.00172

    2019 arXiv

  20. [28]

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2021. OCR-free Document Understanding Transformer. arXiv:2111.15664 https://arxiv.org/abs/2111.15664

    2021 arXiv

  21. [29]

    Sanghwan Kim, Rui Xiao, Mariana-Iuliana Georgescu, Stephan Alaniz, and Zeynep Akata. 2024. COSMOS: Cross-Modality Self-Distillation for Vision Lan- guage Pre-training. arXiv:2412.01814 https://arxiv.org/abs/2412.01814

    2024 arXiv

  22. [30]

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer.arXiv preprint arXiv:2001.04451(2020)

    2020 arXiv

  23. [31]

    Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2022. Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding. arXiv:2210.03347 https://arxiv.org/a...

    2022 arXiv

  24. [32]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuettler, Mike Lewis, Wen tau Yih, Tim Rocktaeschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11...

    2020 arXiv

  25. [33]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Boot- strapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597 https://arxiv.org/abs/2301.12597

    2023 arXiv

  26. [34]

    Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2023. TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models. InProceedings of the AAAI conference on artificial intelligence, Vol. 37. 13094–13102

    2023

  27. [35]

    Yanhong Li, Zixuan Lan, and Jiawei Zhou. 2025. Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs. arXiv:2510.18279 https://arxiv.org/abs/2510.18279

    2025

  28. [36]

    Yujie Lu, Xiujun Li, Tsu-Jui Fu, Miguel Eckstein, and William Yang Wang

  29. [37]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. SimPO: Simple Preference Optimization with a Reference-Free Reward.Advances in Neural Information Processing Systems37 (2024), 124198–124235

    2024

  30. [38]

    Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, and Federico Tombari. 2024. SILC: Improving Vision Language Pretraining with Self-Distillation. InComputer Vision – ECCV 2024. doi:10.1007/978-3-031- 72664-4_3

    2024 doi

  31. [39]

    Patil, Ion Stoica, and Joseph E

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560 [cs.AI] https://arxiv.org/abs/2310.08560

    2024 arXiv

  32. [40]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models.arXiv preprint arXiv:2309.00071(2023)

    2023 arXiv

  33. [41]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. 2021. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv:2108.12409 https: //arxiv.org/abs/2108.12409

    2021 arXiv

  34. [42]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.000...

    2021 arXiv

  35. [43]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lilli- crap. 2019. Compressive Transformers for Long-Range Sequence Modelling. arXiv:1911.05507 [cs.LG] https://arxiv.org/abs/1911.05507

    2019 arXiv

  36. [44]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model.Advances in neural information processing systems36 (2023), 53728–53741

    2023

  37. [45]

    Alex Jinpeng Wang, Linjie Li, Yiqi Lin, Min Li, Lijuan Wang, and Mike Zheng Shou. 2024. Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning.NeurIPS(2024)

    2024

  38. [46]

    Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen

    Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. mDPO: Conditional Preference Optimization for Multimodal Large Language Models. arXiv:2406.11839 [cs.CV] https://arxiv.org/ abs/2406.11839

    2024 arXiv

  39. [47]

    Yibo Wang, Yongcheng Jing, Shunyu Liu, Hao Guan, Rong cheng Tu, Chengyu Wang, Jun Huang, and Dacheng Tao. 2026. VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning. arXiv:2601.22069 [cs.CL] https://arxiv. org/abs/2601.22069

    2026

  40. [48]

    Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, Chunrui Han, and Xiangyu Zhang. 2024. General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model. arXiv:2409.01704 [cs.CV] https://arxiv.org/abs/2...

    2024 arXiv

  41. [49]

    Haoran Wei, Yaofeng Sun, and Yukun Li. 2025. DeepSeek-OCR: Contexts Optical Compression. arXiv:2510.18234 [cs.CV] https://arxiv.org/abs/2510.18234

    2025 arXiv

  42. [50]

    Rabe, DeLesley Hutchins, and Christian Szegedy

    Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. 2022. Memorizing Transformers. arXiv:2203.08913 [cs.LG] https://arxiv.org/abs/2203. 08913

    2022 arXiv

  43. [51]

    Ling Xing, Alex Jinpeng Wang, Rui Yan, Xiangbo Shu, and Jinhui Tang. 2025. VIST: Vision-centric Token Compression in LLM. arXiv:2502.00791 [cs.CL] https://arxiv.org/abs/2502.00791

    2025

  44. [52]

    Howard Yen, Tianyu Gao, and Danqi Chen. 2024. Long-Context Language Modeling with Parallel Context Encoding. arXiv:2402.16617 [cs.CL] https:// arxiv.org/abs/2402.16617

    2024 arXiv

  45. [53]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big Bird: Transformers for Longer Sequences.Advances in neural information processing systems33 (2020), 17283–17297

    2020

  46. [54]

    Hongbo Zhao, Meng Wang, Fei Zhu, Wenzhuo Liu, Bolin Ni, Fanhu Zeng, Gaofeng Meng, and Zhaoxiang Zhang. 2025. VTCBench: Can Vision-Language Models Un- derstand Long Context with Vision-Text Compression? arXiv:2512.15649 [cs.CV] https://arxiv.org/abs/2512.15649

    2025

  47. [2024]

    arXiv:2405.14213 [cs.CV] https://arxiv.org/abs/2405.14213

    From Text to Pixel: Advancing Long-Context Understanding in MLLMs. arXiv:2405.14213 [cs.CV] https://arxiv.org/abs/2405.14213

    arXiv

  48. [2026]

    arXiv:2601.10378 [cs.CV] https://arxiv.org/abs/2601.10378

    Global Context Compression with Interleaved Vision-Text Transformation. arXiv:2601.10378 [cs.CV] https://arxiv.org/abs/2601.10378

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.