Pith. sign in

REVIEW 11 cited by

SOLO: A Single Transformer for Scalable Vision-Language Modeling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.06438 v3 pith:ZQ2HYQJJ submitted 2024-07-08 cs.CV cs.CLcs.LG

Yangyi Chen , Xingyao Wang , Hao Peng , Heng Ji This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords visualtrainingsolollmsarchitectureencodersmodelssingle
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present SOLO, a single transformer for Scalable visiOn-Language mOdeling. Current large vision-language models (LVLMs) such as LLaVA mostly employ heterogeneous architectures that connect pre-trained visual encoders with large language models (LLMs) to facilitate visual recognition and complex reasoning. Although achieving remarkable performance with relatively lightweight training, we identify four primary scalability limitations: (1) The visual capacity is constrained by pre-trained visual encoders, which are typically an order of magnitude smaller than LLMs. (2) The heterogeneous architecture complicates the use of established hardware and software infrastructure. (3) Study of scaling laws on such architecture must consider three separate components - visual encoder, connector, and LLMs, which complicates the analysis. (4) The use of existing visual encoders typically requires following a pre-defined specification of image inputs pre-processing, for example, by reshaping inputs to fixed-resolution square images, which presents difficulties in processing and training on high-resolution images or those with unusual aspect ratio. A unified single Transformer architecture, like SOLO, effectively addresses these scalability concerns in LVLMs; however, its limited adoption in the modern context likely stems from the absence of reliable training recipes that balance both modalities and ensure stable training for billion-scale models. In this paper, we introduce the first open-source training recipe for developing SOLO, an open-source 7B LVLM using moderate academic resources. The training recipe involves initializing from LLMs, sequential pre-training on ImageNet and web-scale data, and instruction fine-tuning on our curated high-quality datasets. On extensive evaluation, SOLO demonstrates performance comparable to LLaVA-v1.5-7B, particularly excelling in visual mathematical reasoning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

    cs.CV 2026-01 conditional novelty 6.0 of 10

    NarrativeTrack shows that video AI models, including GPT-4o, falter at tracking a specific person across scene changes, outfit changes, and similar-looking characters in long videos.

  2. Language-Unlocked ViT (LUViT): Empowering Self-Supervised Vision Transformers with LLMs

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LUViT jointly pretrains a ViT with masked auto-encoding and LoRA adapters in a frozen LLM block, reporting +0.4% ImageNet-1K accuracy and up to +2.2% on ImageNet-A over its own MAE baseline.

  3. LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LLaVA-SP adds six spatial tokens produced by multi-scale cropping or pooling and cross-attention to MLLMs, improving 10/11 benchmarks over LLaVA-1.5 with nearly unchanged latency.

  4. HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    HoVLE is a monolithic VLM whose holistic embedding module maps images and text into one shared space, letting a frozen LLM reach near-compositional performance.

  5. LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A hierarchical window transformer that injects image details into upsampled CLIP features and compresses them with cross-scale window attention improves MLLM fine-grained perception by 3.7% on average over LLaVA-UHD.

  6. Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models

    cs.CV 2025-07 conditional novelty 5.5 of 10

    A monolithic multimodal LLM that cuts pre-training data by 58% and first-token latency by up to 69% while matching or beating its predecessor on 15 benchmarks.

  7. Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training

    cs.CV 2025-05 conditional novelty 5.0 of 10

    PRIOR reweights the next-token prediction loss in vision-language pretraining by 1 minus the probability assigned by a text-only reference LLM, and reports consistent benchmark improvements over standard NTP.

  8. EVEv2: Improved Baselines for Encoder-Free Vision-Language Models

    cs.CV 2025-02 conditional novelty 5.0 of 10

    An encoder-free vision-language model using separate attention, normalization, and feed-forward weights for image versus text tokens outperforms earlier encoder-free models and narrows the gap to encoder-based VLMs wi...

  9. LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A frozen-LLM method that steers visual token representations in low-rank subspaces achieves benchmark scores close to LoRA with about 500x fewer trainable parameters.

  10. SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

    cs.CV 2024-12 conditional novelty 5.0 of 10

    SynerGen-VL introduces token folding and vision expert FFNs to train a 2.4B encoder-free MLLM that matches larger unified models like Emu3 on multiple image understanding and generation benchmarks.

  11. Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.

Pith tools