Pith. sign in

REVIEW 3 major objections 5 minor 66 references

FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FALCON's central claim is that 64 learned visual registers per sub-image can replace the full token output of a high-resolution vision encoder, cutting visual tokens 9x while preserving cross-crop continuity through register interaction.

desk verdict A genuinely useful token-compression trick for high-res MLLMs that is buried under a SOTA claim its own Table 1 contradicts. read the letter →

arxiv 2501.16297 v2 pith:76KDTVCS submitted 2025-01-27 cs.CV

Renshan Zhang , Rui Shao , Gongwei Chen , Miao Zhang , Kaiwen Zhou , Weili Guan , Liqiang Nie This is my paper · ORCID
classification cs.CV
keywords visualregistershigh-resolutionmultimodalLLMstokencompressioncropping-basedencodingfragmentationReAttenfine-grainedperceptiondocumentunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that visual redundancy in high-resolution multimodal LLMs can be removed at the encoding stage rather than by a separate compression module. It introduces learned visual registers, a small set of extra tokens fed into a pretrained vision transformer alongside the image patches of each crop; self-attention naturally concentrates the information worth keeping into these registers, so the model sends only 64 register tokens per sub-image to the language model. A second component, Register Interactive Attention (ReAtten), lets registers from different crops attend to one another at every transformer layer, countering the broken words and objects that cropping can cause. FALCON reports large gains across high-resolution benchmarks with a 9x reduction in visual tokens, and its ablations argue that this beats pooling, pixel shuffle, and query-based abstractors while needing far less training data.

What carries the argument

The central object is the visual register: $M=64$ learnable tokens appended to the image-token sequence of each sub-image before it enters every ViT self-attention layer. Because no attention mask constrains them, the registers act as information sinks, able to read every image token while the image tokens mostly exchange local information, so at the output the register states are a compact digest of the whole crop. The same register set is shared across crops, and after each self-attention step the register states from all crops are concatenated and passed through ReAtten, a self-attention layer over registers only, before the FFN; this is the mechanism that stitches sub-images back together. ReAtten is initialized from the ViT attention weights at the corresponding depth, and the model is trained in four stages that progress from low to high resolution, ending with instruction tuning at high resolution with the ViT frozen.

What would settle it

Take a high-resolution document or scene image, crop it, and run FALCON with 64 registers; then identify cases where a small text string or tiny object is fully contained in one crop and compare transcription or detection accuracy against a variant with 144 registers or against the same model receiving uncompressed image tokens. If accuracy degrades on those fine-grained cases even when overall benchmark scores stay high, the 64-register compaction is lossy and the central claim fails for the exact detail regime it targets.

Watch

Extended reading notes

Core claim

FALCON's central claim is that a small set of learned visual registers placed inside a pretrained ViT can serve as the sole visual representation for a high-resolution multimodal LLM. The registers are concatenated with the image tokens of each crop; after self-attention, the register outputs are treated as the complete visual input to the LLM, replacing the roughly 576 tokens a standard 384px patch encoder would emit per crop. The paper calls this Register-based Representation Compacting (ReCompact) and contends that it removes the need for any post-encoder compression module, because a well-pretrained ViT already gathers global information into a few tokens when given register tokens to fill that role. To address fragmentation, Register Interactive Attention (ReAtten) collects the register states of all crops at each ViT layer, runs a self-attention layer over them, and returns the mixed states to their crops; initializing these layers from the ViT's own attention weights lets the mechanism start from learned behavior. With 64 registers per sub-image and a 9x token reduction, FALCON is reported to match or beat models that carry 2-16x more visual tokens across MME-RealWorld, DocVQA, TextVQA, ChartQA, V*, POPE, ScienceQA, MMBench, and SEED-Bench.

Load-bearing premise

The load-bearing premise is that a pretrained vision transformer can, after light adaptation, fold every task-relevant detail of a high-resolution image into a fixed small set of 64 register tokens per crop without dropping small text, fine boundaries, or spatial relations.

Editorial extensions

If this is right

  • A high-resolution MLLM no longer needs a separate query-based compression stage: adapting a pretrained ViT with under 3 million samples teaches the registers to aggregate, where cited abstractor-style modules were pretrained on 129 million to 1.4 billion samples.
  • The token budget drops 9x at 64 registers per sub-image, so a 16-crop high-resolution image enters the LLM as about 1,024 visual tokens rather than around 9,216 image tokens.
  • Cross-crop register interaction recovers visual continuity lost to cropping: FALCON reports higher MME-RealWorld perception and reasoning scores than shifted-window attention or a complementary image pyramid, and higher POPE scores, indicating fewer hallucinations.
  • Register count acts as a direct efficiency-performance dial: raising 36 to 64 registers improves accuracy, while 64 to 144 yields sharply diminishing returns, making 64 the paper's chosen operating point.
  • The combination transfers across diverse tasks, including OCR in the wild, remote sensing, diagrams and tables, monitoring, autonomous driving, document VQA, chart QA, fine-grained attribute recognition, and general reasoning, suggesting the mechanism is not specialized to one image type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is a reconstruction or probing study: feed a crop through ReCompact, then ask a decoder or linear probe to recover small text or object boundaries from the 64 register states; this would quantify exactly how much fine detail the aggregation loses and predict the failure cases.
  • The register recipe looks transferable beyond high-resolution crops: video frames, multi-view images, or any input with many patches could use in-encoder registers as a parameter-efficient alternative to resamplers, provided the training-then-frozen progression in the paper carries over.
  • Because ReAtten is added only in the final instruction-tuning stage with the ViT frozen, a natural control experiment is to add ReAtten to an already-trained cropping-based model without the earlier FALCON stages; if continuity gains persist, the module can be retrofitted to existing systems.
  • The shared-register design also suggests a cheap way to attend over many crops: because interactions happen only over $M \cdot N_c$ register tokens rather than all image patches, the cost of global context grows linearly in the number of crops, which is what makes the 16-crop setting practical.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FALCON, a high-resolution multimodal large language model that introduces a set of learned visual registers inside a pretrained ViT. The ReCompact mechanism aggregates image information into M registers per sub-image, yielding a 9x reduction in visual tokens and eliminating the need for a separate compression module, while a ReAtten module exchanges information across registers of different sub-images to mitigate the fragmentation caused by cropping. The model is trained with a four-stage progressive pipeline and evaluated on MME-RealWorld, DocVQA, TextVQA, ChartQA, V*, and several general benchmarks. The authors claim state-of-the-art performance with 64 tokens per sub-image and argue that the representation is comprehensive and redundancy-free.

Significance. If validated, the idea of performing token compression inside the vision encoder via learnable registers is a useful contribution that could be more parameter- and data-efficient than post-hoc abstractors, and the ReAtten module offers a lightweight mechanism for cross-sub-image information exchange. The manuscript includes several strengths: ablations that control for training data (Table 3), comparisons against alternative compression methods (Fig. 4), and an ablation with and without ReAtten (Table 5). These internal comparisons support the effectiveness of the proposed modules within the authors' pipeline. However, the headline state-of-the-art claim is contradicted by the paper's own Table 1, where LLaVA-OneVision achieves higher average scores, and the 'comprehensive representation' claim is not fully supported by the register-count ablation in Fig. 6, which shows monotonic improvement with more registers. The core idea is defensible as a lossy-compression tradeoff, but the paper currently overstates the results.

major comments (3)
  1. [Sec. 4.1, Table 1] The statement that FALCON 'achieves state-of-the-art performance' is not supported by the numbers in Table 1: LLaVA-OneVision obtains higher weighted average scores in both perception (Avg 59.6 vs 50.3; Avg-C 55.8 vs 48.0) and reasoning (Avg 44.2 vs 40.7; Avg-C 43.4 vs 40.7). The authors either need to restrict the state-of-the-art claim to a clearly defined comparison class (e.g., models with at most 1024 visual tokens) or directly compare with and discuss the tradeoff against higher-token models such as LLaVA-OneVision.
  2. [Sec. 3.1 and Fig. 6] The abstract and Section 3.1 describe the register output as a 'comprehensive' and 'non-redundant' visual representation, but Fig. 6 shows that increasing the register count from 64 to 144 continues to improve both MME-RealWorld and POPE scores. This indicates that 64 registers discard task-relevant information, so the 9x compression is a lossy tradeoff rather than a redundancy-free aggregation. The authors should explicitly acknowledge this tradeoff and soften the wording accordingly, or provide evidence that the residual loss is negligible for downstream tasks.
  3. [Sec. 4.2, Fig. 6] The choice of the register count M = 64 is derived from an ablation on MME-RealWorld and POPE, which are the same benchmarks used in the final evaluation (Tables 1 and 4). This constitutes a form of test-set tuning for a hyperparameter. The authors should either perform the ablation on a held-out validation set or demonstrate that the results are stable for a range of M values and not an artifact of the specific benchmarks chosen for the selection.
minor comments (5)
  1. [Sec. 3.2, paragraph 1] There is a typo: 'effectivly' should be 'effectively'.
  2. [Sec. 4.2, paragraph 4] 'Base on these observations' should be 'Based on these observations'.
  3. [Table 1 caption] The word 'maximun' should be 'maximum'.
  4. [Fig. 1] The label 'Extra Crompression' should be 'Extra Compression'.
  5. [Sec. 4.1, first paragraph] The phrase 'FALCON significantly outperform other advanced MLLMs' is grammatically incorrect; it should be 'FALCON significantly outperforms'.

Circularity Check

1 steps flagged · score 2.0 of 10

No definitional or self-citation circularity; only a mild model-selection circularity from choosing the register count on the same benchmarks used for the headline evaluation.

  1. other [Section 4.2, 'Ablation on the Number of Visual Registers', Fig. 6 caption]
    "Increasing the number of visual registers generally enhances accuracy across both benchmarks. However, the improvement rate noticeably tapers off when the register count exceeds 64. Based on this observation, to better balance the efficiency and performance, we select register counts of 64 for FALCON."

    The register count M=64 is chosen by observing performance on MME-RealWorld and POPE, and the same benchmarks are later reported in Tables 1 and 4 as evidence that 'FALCON achieves state-of-the-art performance' with 64 tokens per sub-image. The headline result is therefore tuned to the evaluation instrument rather than being an independent prediction of the method's performance. This is a mild selection circularity, not a definitional equivalence: the numbers are empirical outcomes, but they are not clean evidence for the claim that 64 registers are 'comprehensive' because the choice of 64 is optimized on the same measurements.

full rationale

The paper is an empirical architecture paper, not a derivation, so there is no formal equivalence between inputs and outputs. The ReCompact mechanism is motivated by an external observation from Darcet et al. about ViT tokens, and ReAtten is an original module; no load-bearing step reduces to a self-citation or to an author-imported uniqueness theorem. The main caveat is the model-selection loop: the number of registers was picked after ablating on MME-RealWorld and POPE, and then the same benchmarks are used to support the state-of-the-art claim. The paper's own Fig. 6 also shows continued improvement up to 144 registers, which weakens the 'comprehensive' wording but is an empirical adequacy concern rather than circularity. Overall the central contribution has independent content, so the score is low.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests primarily on the domain assumption that a pretrained ViT can compress all necessary visual information into a small register set. The only free parameter is the number of registers, tuned on the evaluation benchmarks. No new physical entities are introduced.

free parameters (1)
  • number of visual registers M = 64
    The paper selects 64 registers based on an ablation on MME-RealWorld and POPE (Fig. 6), choosing the point where performance gains taper off. This is a hyperparameter tuned on the evaluation benchmarks.
assumptions (3)
  • domain assumption A pretrained ViT can aggregate global image information into a small set of register tokens through self-attention.
    This is the core premise of ReCompact, stated in Section 3.1 and motivated by Darcet et al. The paper does not prove that 64 tokens are sufficient for all tasks.
  • domain assumption Non-overlapping cropping plus register interaction preserves enough visual continuity for the downstream LLM.
    ReAtten assumes that exchanging information among register tokens across sub-images is sufficient to resolve fragmentation introduced by cropping; this is validated only empirically on selected benchmarks.
  • standard math Standard transformer self-attention and cross-attention definitions.
    Equations (1) to (3) rely on standard attention and feedforward operations from prior literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers." pith.science (2026). https://pith.science/paper/76KDTVCS

@misc{pith2026250116297,
  author       = {Pith},
  title        = {Pith review of: FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/76KDTVCS}},
  note         = {Machine review of arXiv:2501.16297}
}
read the original abstract

The incorporation of high-resolution visual input equips multimodal large language models (MLLMs) with enhanced visual perception capabilities for real-world tasks. However, most existing high-resolution MLLMs rely on a cropping-based approach to process images, which leads to fragmented visual encoding and a sharp increase in redundant tokens. To tackle these issues, we propose the FALCON model. FALCON introduces a novel visual register technique to simultaneously: 1) Eliminate redundant tokens at the stage of visual encoding. To directly address the visual redundancy present in the output of vision encoder, we propose a Register-based Representation Compacting (ReCompact) mechanism. This mechanism introduces a set of learnable visual registers designed to adaptively aggregate essential information while discarding redundancy. It enables the encoder to produce a more compact visual representation with a minimal number of output tokens, thus eliminating the need for an additional compression module. 2) Ensure continuity in visual encoding. To address the potential encoding errors caused by fragmented visual inputs, we develop a Register Interactive Attention (ReAtten) module. This module facilitates effective and efficient information exchange across sub-images by enabling interactions between visual registers. It ensures the continuity of visual semantics throughout the encoding. We conduct comprehensive experiments with FALCON on high-resolution benchmarks across a wide range of scenarios. FALCON demonstrates superior performance with a remarkable 9-fold reduction in visual tokens.

Figures

Figures reproduced from arXiv: 2501.16297 by the authors.

Figure 1
Figure 1. Comparison between existing high-resolution MLLMs [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of the proposed FALCON. The model introduces a set of learnable visual registers, which are fed into the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The progressive training pipeline. Stage 0: Align the frozen vision and language backbones by training only the MLP and registers. Stage 1: Tune all parameters with caption and OCR data on low-resolution for coarse alignment. Stage 2: Integrate high-resolution input along with fine-grained captions and OCR to achieve refined alignment. Stage 3: Froze ViT and introduce ReAtten module for high-resolution instruction t… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of different compression methods. We re [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the register-to-image attention map. We randomly select several visual registers and extract the attention scores [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablation on different visual register counts. Increasing [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Qualitative analysis of FALCON in high-resolution images of natural scenes. The examples illustrate FALCON’s exceptional [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Qualitative analysis of FALCON in high-resolution images of documents. The example demonstrates FALCON’s strong under [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 34 canonical work pages

  1. [1]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 2, 3, 6, 7

    arXiv 2023

  2. [2]

    Lion: Empowering multimodal large language model with dual-level visual knowledge

    Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26540–26550, 2024. 1, 2

    work page 2024

  3. [3]

    Less is more: Empowering gui agent with context- aware simplification

    Gongwei Chen, Xurui Zhou, Rui Shao, Yibo Lyu, Kaiwen Zhou, Wentao Li, Yinchuan Li, Zhongang Qi, and Liqiang Nie. Less is more: Empowering gui agent with context- aware simplification. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , 2025. 3

    work page 2025

  4. [4]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023. 6

    arXiv

  5. [5]

    Spa-bench: A comprehensive benchmark for smartphone agent evalua- tion

    Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gong- wei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye HAO, Jun Wang, and Kun Shao. Spa-bench: A comprehensive benchmark for smartphone agent evalua- tion. In The Thirteenth International Conference on Learn- ing Representations, 2025. 3

    work page 2025

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370–387. Springer, 2024. 6

    2024

  7. [7]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024. 7

    work page 2024

  8. [8]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

Show all 66 references
  1. [9]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty- seventh Conference on Neural Information Processing Sys- tems...

    2023

  2. [10]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representa- tions, 2024. 2, 4

    2024

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  4. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  5. [13]

    Bridging nonlinearities and stochastic regularizers with gaussian error linear units

    Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint arXiv:1606.08415, 2016. 5

    2016 arXiv

  6. [14]

    Cogvlm2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. Cogvlm2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 6

    2024 arXiv

  7. [15]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 3...

    2024

  8. [16]

    Mini-monkey: Alleviating the semantic saw- tooth effect for lightweight MLLMs via complementary im- age pyramid

    Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. Mini-monkey: Alleviating the semantic saw- tooth effect for lightweight MLLMs via complementary im- age pyramid. In The Thirteenth International Conference on Learning Representations, 2025. 3, 7

    2025

  9. [17]

    Hires-llava: Restoring fragmen- tation input in high-resolution large vision-language models

    Runhui Huang, Xinpeng Ding, Chunwei Wang, Jianhua Han, Yulong Liu, Hengshuang Zhao, Hang Xu, Lu Hou, Wei Zhang, and Xiaodan Liang. Hires-llava: Restoring fragmen- tation input in high-resolution large vision-language models. In Proceedings of the Computer Vision and Pattern Re...

    2025

  10. [18]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. In 8th Annual Conference on Robot Learning, 2024. 3

    2024

  11. [19]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 6

    2023 arXiv

  12. [20]

    LLaV A-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. LLaV A-onevision: Easy visual task transfer. Transactions on Machine Learning Research,

  13. [21]

    STAR: Learning diverse robot skill abstractions through rotation-augmented vector quan- tization

    Hao Li, Qi Lv, Rui Shao, Xiang Deng, Yinchuan Li, Jianye HAO, and Liqiang Nie. STAR: Learning diverse robot skill abstractions through rotation-augmented vector quan- tization. In Forty-second International Conference on Ma- chine Learning, 2025. 3

    2025

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with 11 frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with 11 frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  15. [23]

    Flex- attention for efficient high-resolution vision-language mod- els

    Junyan Li, Delin Chen, Tianle Cai, Peihao Chen, Yining Hong, Zhenfang Chen, Yikang Shen, and Chuang Gan. Flex- attention for efficient high-resolution vision-language mod- els. In European Conference on Computer Vision , pages 286–302, 2024. 7

    2024

  16. [24]

    Lion-fs: Fast & slow video-language thinker as online video assistant

    Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. Lion-fs: Fast & slow video-language thinker as online video assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3240–3251, 2025. 3

    2025

  17. [25]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empir- ical Methods in Natural Language Processing, 2023. 6

    2023

  18. [26]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  19. [27]

    Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dong- mei Jiang, and Liqiang Nie. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. In Advances in neural information processing systems, 2024. 3

    2024

  20. [28]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  21. [29]

    Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy. In 2025 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR). IEEE, 2025. 3

    2025

  22. [30]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 3, 6, 7

    2024

  23. [31]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 6, 7

    2024

  24. [32]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 3

    2024

  25. [33]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024. 1, 2, 3, 6, 7

    2024 arXiv

  26. [34]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025. 6

    2025

  27. [35]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 3, 7

    2021

  28. [36]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  29. [37]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  30. [38]

    Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xi- aoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024. 3

    2024 arXiv

  31. [39]

    Spatial-temporal graph diffusion policy with kinematic mod- eling for bimanual robotic manipulation

    Qi Lv, Hao Li, Xiang Deng, Rui Shao, Yinchuan Li, Jianye Hao, Longxiang Gao, Michael Yu Wang, and Liqiang Nie. Spatial-temporal graph diffusion policy with kinematic mod- eling for bimanual robotic manipulation. In Proceedings of the Computer Vision and Pattern Recognition Con...

    2025

  32. [40]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022. 6

    2022

  33. [41]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 6

    2021

  34. [42]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  35. [43]

    Multi-adversarial discriminative deep domain generalization for face presentation attack detection

    Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023–10031, 2019. 3

    2019

  36. [44]

    Detecting and grounding multi-modal media manipulation

    Rui Shao, Tianxing Wu, and Ziwei Liu. Detecting and grounding multi-modal media manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6904–6913, 2023. 1

    2023

  37. [45]

    Detecting and grounding multi-modal media manip- ulation and beyond

    Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Zi- wei Liu. Detecting and grounding multi-modal media manip- ulation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 1, 3

    2024

  38. [46]

    Mome: Mixture of multimodal experts for gen- eralist multimodal large language models

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for gen- eralist multimodal large language models. In Advances in neural information processing systems, 2024. 2

    2024

  39. [47]

    When do we not need larger vision models? In 12 European Conference on Computer Vision, pages 444–462,

    Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? In 12 European Conference on Computer Vision, pages 444–462,

  40. [48]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 6

    2019

  41. [49]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 6

    2023 arXiv

  42. [50]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 3

    2023 arXiv

  43. [51]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  44. [52]

    V*: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024. 6

    2024

  45. [53]

    Gui-explorer: Au- tonomous exploration and mining of transition-aware knowl- edge for gui agent

    Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. Gui-explorer: Au- tonomous exploration and mining of transition-aware knowl- edge for gui agent. In Annual Meeting of the Association for Computational Linguistics (ACL), 2025. 3

    2025

  46. [54]

    Deco: Decoupling token compres- sion from semantic abstraction in multimodal large language models

    Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decoupling token compres- sion from semantic abstraction in multimodal large language models. arXiv preprint arXiv:2405.20985, 2024. 2, 3, 7

    2024 arXiv

  47. [55]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 6

    2024 arXiv

  48. [56]

    mplug-docowl: Modularized multimodal large language model for document understanding

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Jun- feng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499, 2023. 3

    2023 arXiv

  49. [57]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In Findings of the Association for Computational Li...

    2023

  50. [58]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 1, 2

    2023 arXiv

  51. [59]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  52. [60]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016. 1

    2016

  53. [61]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 3, 5

    2023

  54. [62]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 6

    2024 arXiv

  55. [63]

    Token-level correlation-guided com- pression for efficient multimodal document understanding

    Renshan Zhang, Yibo Lyu, Rui Shao, Gongwei Chen, Weili Guan, and Liqiang Nie. Token-level correlation-guided com- pression for efficient multimodal document understanding. arXiv preprint arXiv:2407.14439, 2024. 3

    2024 arXiv

  56. [64]

    From redundancy to relevance: Enhancing explainability in multimodal large language mod- els

    Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Enhancing explainability in multimodal large language mod- els. Annual Conference of the Nations of the Americas Ch...

    2025

  57. [65]

    YiFan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qing- song Wen, Zhang Zhang, Liang Wang, and Rong Jin. MME- realworld: Could your multimodal LLM challenge high- resolution real-world scenarios that are difficult for humans? ...

    2025

  58. [66]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024. 1, 3, 7 13

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.