Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Language models trained to predict the next visual latent on raw document images outperform text-only pretraining on the same corpus at a quarter of the token budget, while also improving image–text alignment without paired supervision.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:28 UTC pith:Z7LZIMIC

load-bearing objection A well-run empirical study whose headline claim—visual pretraining beats text pretraining because it preserves visual structure—is not yet supported: the comparison also changes the loss, token budget, and tuned hyperparameters. the 4 major comments →

arxiv 2607.09657 v2 pith:Z7LZIMIC submitted 2026-07-10 cs.CV cs.AIcs.MM

Scalable Visual Pretraining for Language Intelligence

Yiming Zhang , Zhonghan Zhao , Wenwei Zhang , Haiteng Zhao , Tianyang Lin , Huanze Tang , Yunhua Zhou , Demin Song
show 9 more authors
Kuikun Liu Haochen Ye Haian Huang Yuzhe Gu Haijun Lv Qipeng Guo Bin Liu Gaoang Wang Kai Chen
This is my paper
classification cs.CV cs.AIcs.MM
keywords visual pretrainingdocument understandingnext-latent predictionlanguage model pretrainingcross-modal alignmentscientific reasoningtoken efficiencymultimodal learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to overturn the default that language models must be pretrained on plain text. It claims that training the same backbone to predict the next visual latent on rendered scientific-document pages, with no OCR, no text extraction, and no image–text pair supervision, is a scalable alternative to text-only continued pretraining. In matched comparisons where the only intended difference is whether the additional PDF corpus is parsed into text or kept as page images, visual pretraining outperforms text pretraining across multiple backbones and scientific reasoning benchmarks while using about a quarter of the token budget. The same pretraining also improves multimodal benchmarks and cross-modal embedding alignment, suggesting that the information lost in textualization—equations, figures, tables, page layout—can be recovered directly from the page image. A sympathetic reader would care because, if true, visual pretraining offers an efficient and unlabeled path to stronger foundation-model intelligence.

Core claim

On the paper's own terms, the discovery is that next-visual-latent prediction over sparse page images is a better pretraining signal than next-text-token prediction over the same documents' parsed text. Using a frozen vision encoder, background-patch filtering, a linear projection into the LLM space, and an InfoNCE objective that matches each predicted latent to the next foreground latent in a batch, visual pretraining is trained jointly with text cross-entropy. Across two multimodal and two language-only backbones, it beats matched text pretraining on scientific reasoning benchmarks, with gains up to 3.22 points on GPQA Diamond and larger gains on visually dense pages; it uses roughly 20 bi

What carries the argument

The workhorse is the sparse foreground visual sequence. A frozen vision tower maps each rendered page to patch features; a patch-level variance and luminance mask removes blank regions; surviving features are raster-ordered, linearly projected into the LLM hidden space, and processed under block-causal attention. From each LLM hidden state, an MLP head predicts the next frozen visual latent, scored by InfoNCE with in-batch negatives; this is combined with text cross-entropy through a weighted joint loss, with the vision tower frozen and only the LLM, projection, and prediction head updated. This makes continuous visual latents the native prediction target of the model, in direct analogy to t

Load-bearing premise

The load-bearing premise is that the benchmark gains come from preserving visual document structure rather than from the collinear changes in the training setup—the contrastive loss, the much smaller token budget, and the chosen hyperparameters—since those change together with the representation in the matched comparison.

What would settle it

A decisive experiment: run visual pretraining's exact pipeline with the same 20B token budget and InfoNCE next-latent loss, but replace rendered page features with latents derived from the same pages' parsed text, or randomly permute the spatial layout of page patches; if the GPQA/AIME gains over text pretraining persist, the visual-structure explanation is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Visual pretraining on raw scientific pages is an efficient complement to text continued pretraining: it reaches the same or better reasoning benchmarks with roughly 25% of the parsed-text token budget for the same corpus.
  • The benefit concentrates where text extraction loses most—pages dense in figures, equations, tables, and layout—while text-dominant pages show little gap.
  • Unlabeled visual pretraining transfers to multimodal reasoning benchmarks and measurably aligns image and text representations in the shared backbone.
  • A decoder-free latent-prediction formulation suffices; adding pixel-level reconstruction costs about 1.4x training time without consistent reasoning gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the matched comparison still changes the loss, the token budget, and some hyperparameters together with the representation, the cleanest test of the paper's interpretation is an ablation that keeps visual pretraining's exact training recipe but feeds non-visual inputs; until that is run, the causal role of visual structure is plausible but not fully isolated.
  • Editorial inference: if the mechanism generalizes, the same recipe should transfer to other visually native corpora—textbooks, patents, chart-heavy reports, web pages—and the authors leave that as an open question since only scientific PDFs are tested.
  • Editorial inference: the roughly 4x compression of visual tokens over parsed text raises the possibility of pretraining on compressed visual streams with lighter text alignment; testing this at larger scale would show whether visual corpora can substitute for, not just supplement, text corpora.
  • Editorial inference: the density-stratified analysis suggests a testable prediction—layout-preserving renderings such as typeset math and diagrams should outperform a flattened text reflow of the same content, and perturbing layout structure should shrink the gain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Visual Pretraining (VP), a continued-pretraining method in which a frozen vision encoder turns rendered scientific-document pages into sparse foreground visual-latent sequences, and the LLM is trained to predict the next visual latent with an InfoNCE loss while also continuing standard text next-token prediction. Across four backbones (Qwen3.5, Qwen3, Llama3.2 Vision, Llama3.1), VP is compared with Text Pretraining (TP) on the same underlying scientific-PDF corpus plus a shared text corpus, followed by identical SFT. The authors report that VP outperforms TP on scientific reasoning benchmarks (MMLU-Pro, GPQA, AIME-25, HLE), uses only about 20B visual tokens versus about 80B parsed-text tokens from the same PDFs, improves multimodal benchmarks, and strengthens cross-modal alignment metrics. The central claim is that preserving and modeling native visual document structure provides language-intelligence gains beyond text-only pretraining.

Significance. If the central claim holds, the paper would challenge the default text-only pretraining paradigm for language models and propose a scalable, unlabeled visual pretraining signal derived from raw documents. The empirical design has notable strengths: matched document sources, multiple backbones spanning multimodal and text-only LLMs, identical SFT and evaluation protocols, and detailed supplementary material with a decoder-free ablation and density-stratified analysis. The paper is also appropriately scoped in its limitations section, acknowledging that VP complements rather than replaces language pretraining and that generalization beyond scientific PDFs remains open. However, the central attribution of VP's gains to visual structure is currently under-supported because the VP-versus-TP comparison changes the loss function, token budget, and test-selected hyperparameters alongside the input representation. The absence of error bars on small but headline differences further weakens the quantitative claims. The result is interesting and worth pursuing, but the paper as submitted does not yet isolate the mechanism it claims.

major comments (4)
  1. [§2, Fig. 3 and §D] The statement 'The only controlled difference is the representation of the additional scientific-PDF corpus' is not supported by the presented setup. VP optimizes λ_text·L_CE + λ_vis·L_VP with the InfoNCE next-visual-latent loss (Eq. 3), while TP optimizes only L_CE. The token budgets also differ: approximately 20B visual tokens for VP versus roughly 80B parsed-text tokens for TP, with total budgets of 120B and 180B tokens. Thus the comparison changes the loss form, the number of optimization targets, and the token count simultaneously with the representation. The reported gains could, in principle, arise from auxiliary-loss regularization, different optimization dynamics, or token-count effects rather than from visual structure. A control that keeps the loss form and token budget fixed and varies only the target modality/content — for example, a next-chunk InfoNCE loss on text-token emb
  2. [Table 1] Hyperparameters central to the main comparison appear to be selected on the same benchmarks used as evidence. The 1× visual-token budget (8,192 foreground visual tokens) is chosen from a sweep evaluated on MMLU-Pro, GPQA, AIME, and HLE (Fig. 3, left), which are exactly the benchmarks in Table 1. Similarly, λ_vis=0.1 is fixed in the main Qwen3.5 runs without an equivalent sweep for TP. This is selection on the test metric and inflates the apparent advantage of VP. The authors should either report the sweep protocol explicitly with a separated validation set, or demonstrate that the conclusions are robust across a range of budgets and loss weights.
  3. [§2, Fig. 2(b)] No error bars, confidence intervals, or multiple-seed results are reported. Several headline differences are small — for example, MMLU-Pro +0.42 (81.52 to 81.94) and HLE +0.38 (10.59 to 11.77) on Qwen3 — and pass@1 metrics on a single evaluation run can easily vary by more than this amount. The paper repeatedly describes gains as 'consistent' without statistical support. At least for the main Qwen3.5 and Llama3.2 comparisons, the authors should report variance across seeds or bootstrap estimates, or explicitly state the evaluation protocol's variability.
  4. [§4, Training setup] The scalability/efficiency claim (1.27×, 2.02×, 2.88× normalized gains with 25% of the token budget) is based on comparing VP at roughly 20B visual tokens with TP at roughly 80B text tokens, while total compute also differs. The factor-of-four token reduction is the paper's own representation choice, not an independent result. If the efficiency claim is meant to show that visual tokens carry more information per token, the comparison should control for compute or FLOPs, or at least report total training compute for both runs. Without this, the 'using only 25% of the token budget' framing is an observation about the pipeline, not a controlled efficiency measurement.
minor comments (5)
  1. [Throughout] There are many missing spaces in rendered text: 'VPreaching', 'VPand', 'VPreaches', 'VPs', 'TPand', 'VPtrains'. Please proofread the PDF rendering.
  2. [§2, Table 2] The caption of Table 2 says 'Text pretraining' in panel (a) but the table only shows 'Original' and 'VP(Ours)'. Clarify the baseline naming.
  3. [§2, Table 2(b)] The sentence 'TP yields negligible gains and occasionally regresses' is too strong: TP improves MathVista on Qwen3.5 (84.30 to 85.50) and improves ChartQAPro on Llama3.2 Vision (20.95 to 22.23). The qualitative statement should be adjusted.
  4. [§2, Attention analysis] The attention-map analysis in Fig. 4 is explicitly qualitative. Please add a sentence in the main text clarifying that it is illustrative and not a controlled mechanistic claim, matching the cautious wording in the Discussion.
  5. [References] Reference [39] (Zhao et al., CVPR 2026) is closely related prior work by the same group but is cited only in Supplementary Section C. It should be cited in the main text, especially in the Introduction or Related Work, to give proper context.

Circularity Check

0 steps flagged

No significant circularity: the VP-vs-TP claim rests on external benchmarks and matched corpora; only a minor non-load-bearing self-citation and some benchmark-selected hyperparameters.

full rationale

The paper's central claim is an empirical benchmark comparison (Tables 1 and 2) against independently defined external benchmarks (MMLU-Pro, GPQA, AIME, HLE, ChartQAPro, MathVista, etc.), not a derivation from fitted parameters. VP's objective (Eqs. 3 and 4) predicts frozen visual latents; the evaluation benchmarks are not redefined in terms of L_VP or cosine similarity, so improved scores are not entailed by the training loss by construction. The matched TP baseline and identical SFT stage give the comparison independent content. The only self-citation [39] appears in Supplementary C, where it merely motivates a generative-decoder variant that is then rejected in favor of the decoder-free method; it is not used to establish the main effectiveness/scalability claims, and no uniqueness theorem or ansatz is imported from the authors' prior work. Some design choices (the 8,192-token budget and lambda_vis=0.1) were selected with downstream evaluation, and the VP/TP comparison varies loss and token budget alongside representation, so the statement that 'The only controlled difference is the representation' overstates isolation; however, this is a confound/attribution risk, not a definitional circularity. The paper also explicitly limits its claim by saying VP 'should not be interpreted as a form of visual pretraining that is fully independent of language pretraining.' No predicted result reduces to its own input.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on the assumption that visual-document structure carries recoverable reasoning information and that the matched-corpus experimental design isolates that effect. Several hyperparameters (mask thresholds, loss weight, temperature, token budget, resolution) are free choices that shape the headline token-efficiency number. No new physical or conceptual entities are postulated.

free parameters (6)
  • Foreground mask thresholds = variance < 0.02, high-luminance > 0.95, low-luminance < 0.15
    Defines which patches become visual tokens; directly controls the 20B-vs-80B token-count claim. Set by hand, not derived from theory.
  • InfoNCE temperature τ = 0.07
    Temperature in Eq. 13; selected by hand and affects the scale of the contrastive loss.
  • VP loss weight λ_vis = 0.1
    Weight on the visual loss in Eq. 16; a free hyperparameter that controls the balance between text and visual objectives.
  • Visual token budget (1×) = 8,192 foreground tokens per context
    Chosen after a downstream benchmark sweep (Figure 3, left); this anchors the main efficiency claim.
  • Maximum rendering resolution = sweep at 560/768/1120; main setting not fully specified
    Swept using downstream evaluation (Figure 3, right); affects the visual token count and information content.
  • Text:visual mixing ratio = one VP batch per text step
    Interleaving schedule in Supplementary D; chosen by implementation, not justified by ablations.
axioms (5)
  • domain assumption The frozen ViT + spatial merger provide a stable, information-preserving target space for next-latent prediction.
    The method never reconstructs pixels or directly measures information loss; if the frozen features discard layout or equation information, the objective cannot learn it. Section 4 and Supplementary A.
  • domain assumption Visual documents contain reasoning-relevant information that text extraction weakens or discards.
    Core motivation in the Introduction; not directly measured, only inferred from downstream gains.
  • domain assumption Matched document source isolates the effect of visual representation.
    The paper states this, but loss, token counts, and optimization details also differ. Section 4, Training setup.
  • domain assumption Downstream benchmark scores after SFT are a valid proxy for language intelligence acquired during pretraining.
    Evaluation protocol in Section 4; assumes SFT does not mask or create the differences between VP and TP.
  • domain assumption The 100 held-out image-text pairs for alignment are non-overlapping with VP training data and representative.
    Supplementary D; the sample is small and the selection procedure is not fully specified.

pith-pipeline@v1.3.0-alltime-deepseek · 15551 in / 15856 out tokens · 162508 ms · 2026-08-02T07:28:12.553132+00:00 · methodology

0 comments
read the original abstract

The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 2 canonical work pages

  1. [1]

    Barsalou

    Lawrence W. Barsalou. Perceptual symbol systems.Behavioral and Brain Sciences, 22(4):577–660, 1999. doi: 10.1017/S0140525X99002149. 1

    doi:10.1017/s0140525x99002149 1999

  2. [2]

    Barsalou

    Lawrence W. Barsalou. Grounded cognition.Annual Review of Psychology, 59:617–645, 2008. doi: 10.1146/annurev.psych.59.103006.093639. 1

    arXiv 2008

  3. [3]

    Nougat: Neural optical understanding for academic documents

    Lukas Blecher, Guillem Cucurull Preixens, Thomas Scialom, and Robert Stojnic. Nougat: Neural optical understanding for academic documents. InInternational Conference on Learning Representations, volume 2024, pp. 37646–37663, 2024. 1

    2024

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1

    1901

  5. [5]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185–24198, 2024. 1

    2024

  6. [6]

    Xtuner: A toolkit for efficiently fine-tuning llm

    XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/ InternLM/xtuner, 2023. 4

    2023

  7. [7]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020. 1

    Pith/arXiv arXiv 2020

  8. [8]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 1, 2

    Pith/arXiv arXiv 2024

  9. [9]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 10, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 10, 2022. 1

    Pith/arXiv arXiv 2022

  10. [10]

    The platonic representation hypothesis

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024. 1, 2

    Pith/arXiv arXiv 2024

  11. [11]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  12. [12]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020. 1

    Pith/arXiv arXiv 2001

  13. [13]

    Donut: Document understanding transformer without ocr

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding transformer without ocr. arXiv preprint arXiv:2111.15664, 7(15):2, 2021. 1, 3

    Pith/arXiv arXiv 2021

  14. [14]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. InInternational conference on machine learning, pp. 3519–3529. PMlR, 2019. 2, D

    2019

  15. [15]

    Larkin and Herbert A

    Jill H. Larkin and Herbert A. Simon. Why a diagram is (sometimes) worth ten thousand words.Cognitive Science, 11(1):65–100, 1987. doi: 10.1016/S0364-0213(87)80026-5. 1

    work page doi:10.1016/s0364-0213(87)80026-5 1987

  16. [16]

    Pix2struct: Screenshot parsing as pretraining for visual language understanding

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. InInternational Conference on Machine Learning, pp. 18893–18912. PMLR, 2023. 1, 3 10 Scalable Visual Pretrai...

    2023

  17. [17]

    Tracing the representation geometry of language models from pretraining to post-training

    Melody Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Teru, Adam Santoro, Guillaume Lajoie, and Blake Richards. Tracing the representation geometry of language models from pretraining to post-training. Advances in Neural Information Processing Systems, 38:54691–54724, 2026. 2

    2026

  18. [18]

    Autoregressive image generation without vector quantization.Advances in Neural Information Processing Systems, 37:56424–56445, 2024

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization.Advances in Neural Information Processing Systems, 37:56424–56445, 2024. C

    2024

  19. [19]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.Advances in Neural Information Processing Systems, 35:17612–17625, 2022

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.Advances in Neural Information Processing Systems, 35:17612–17625, 2022. 2

    2022

  20. [20]

    Revisiting the role of language priors in vision-language models.arXiv preprint arXiv:2306.01879, 2023

    Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models.arXiv preprint arXiv:2306.01879, 2023. B

    Pith/arXiv arXiv 2023

  21. [21]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 1

    2023

  22. [22]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InInternational Conference on Learning Representations, volume 2024, pp. 23439–23554, 2024. 2

    2024

  23. [23]

    Chartqapro: A more diverse and challenging benchmark for chart question answering

    Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, et al. Chartqapro: A more diverse and challenging benchmark for chart question answering. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 19123–19151, 2025. 2, 2

    2025

  24. [24]

    American invitational mathematics ex- amination (AIME)

    Mathematical Association of America. American invitational mathematics ex- amination (AIME). 2025. URL https://maa.org/math-competitions/ american-invitational-mathematics-examination-aime. Administered by the MAA as part of the AMC competition series. 2

    2025

  25. [25]

    Llama 3.2 vision model card

    Meta AI. Llama 3.2 vision model card. https://huggingface.co/meta-llama/Llama-3. 2-11B-Vision, 2024. Accessed: 2026-06-02. 1, 2, 2b

    2024

  26. [26]

    Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 4

    Pith/arXiv arXiv 2018

  27. [27]

    Openwebmath: An open dataset of high-quality mathematical web text

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. InInternational Conference on Learning Representations, volume 2024, pp. 20357–20379, 2024. 1

    2024

  28. [28]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Pith/arXiv arXiv

  29. [29]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps://qwen.ai/ blog?id=qwen3.5. 1, 2, 2b

    2026

  30. [30]

    Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023. 2

    Pith/arXiv arXiv 2023

  31. [31]

    Visualizing data using t-sne.Journal of Machine Learning Research, 9(86):2579–2605, 2008

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of Machine Learning Research, 9(86):2579–2605, 2008. URLhttp://jmlr.org/papers/v9/vandermaaten08a.html. 1

    2008

  32. [32]

    Mineru: An open-source solution for precise document content extraction.arXiv preprint arXiv:2409.18839, 2024

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction.arXiv preprint arXiv:2409.18839, 2024. 1, D 11 Scalable Visual Pretraining for Language Intelligence

    Pith/arXiv arXiv 2024

  33. [33]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 1

    Pith/arXiv arXiv 2024

  34. [34]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024. 2

    Pith/arXiv arXiv 2024

  35. [35]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022. 2

    Pith/arXiv arXiv 2022

  36. [36]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 1, 2

    Pith/arXiv arXiv 2025

  37. [37]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15134–15186, 2025. 2

    2025

  38. [38]

    The nature of external representations in problem solving.Cognitive Science, 21(2):179–217,

    Jiajie Zhang. The nature of external representations in problem solving.Cognitive Science, 21(2):179–217,

  39. [39]

    Exploring visual pretraining for learning language intelligence

    ZhonghanZhao, YimingZhang, WenweiZhang, HaitengZhao, XingguangWei, ZhangweiGao, KuikunLiu, Yuzhe Gu, Size Wu, Haian Huang, et al. Exploring visual pretraining for learning language intelligence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 31493–31503,

  40. [40]

    think step by step

    Yuhao Zhou, Yiheng Wang, Xuming He, Ruoyao Xiao, Zhiwei Li, Qiantai Feng, Zijie Guo, Yuejin Yang, Hao Wu, Wenxuan Huang, et al. Scientists’ first exam: Probing cognitive abilities of mllm via perception, understanding, and reasoning.Advances in Neural Information Processing Systems, 38, 2026. 2 12 Scalable Visual Pretraining for Language Intelligence Supp...

    2026

  41. [1997]

    doi: 10.1207/s15516709cog2102_3. 1

    work page doi:10.1207/s15516709cog2102_3