Pith. sign in

REVIEW 13 cited by

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.05935 v3 pith:TLPCO3QQ submitted 2024-02-08 cs.CV cs.AIcs.CLcs.LG

Dongyang Liu , Renrui Zhang , Longtian Qiu , Siyuan Huang , Weifeng Lin , Shitian Zhao , Shijie Geng , Ziyi Lin
show 11 more authors
Peng Jin Kaipeng Zhang Wenqi Shao Chao Xu Conghui He Junjun He Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao
This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords languagetrainingcomprehensivedatalargemllmsmodelsmulti-modal
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX. To improve the architecture and training efficiency, we modify the SPHINX framework by removing redundant visual encoders, bypassing fully-padded sub-images with skip tokens, and simplifying multi-stage training into a one-stage all-in-one paradigm. To fully unleash the potential of MLLMs, we assemble a comprehensive multi-domain and multimodal dataset covering publicly available resources in language, vision, and vision-language tasks. We further enrich this collection with our curated OCR intensive and Set-of-Mark datasets, extending the diversity and generality. By training over different base LLMs including TinyLlama1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in parameter size and multilingual capabilities. Comprehensive benchmarking reveals a strong correlation between the multi-modal performance with the data and parameter scales. Code and models are released at https://github.com/Alpha-VLLM/LLaMA2-Accessory

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    WikiCLIP reaches 28.5% OVEN-unseen accuracy (vs 24.5% AutoVER) at 14.5 ms latency by vision-guided LLM embeddings plus hard-negative text swaps.

  2. DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    DataClaw0 introduces an agentic data-tailoring paradigm, a 9B model trained on a synthetically generated dataset, and a new benchmark, claiming improved downstream adaptation in video generation, VQA, and GUI navigati...

  3. ReMoT: Reinforcement Learning with Motion Contrast Triplets

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Training a 4B vision-language model on rule-generated motion-contrast triplets with GRPO lifts spatio-temporal QA accuracy by about 17 points on the authors' own benchmark and by smaller margins on standard benchmarks.

  4. VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A 1,680-question video benchmark shows leading multimodal models lag humans by ~15 points on visual knowledge, and a See-Think-Answer RL-trained model narrows the gap.

  5. MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark

    cs.CL 2025-01 conditional novelty 6.0 of 10

    MME-Industry is a new 21-domain, bilingual multiple-choice benchmark for measuring multimodal LLMs in industrial settings, with top models scoring around 75 to 79 percent.

  6. LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A hierarchical window transformer that injects image details into upsampled CLIP features and compresses them with cross-scale window attention improves MLLM fine-grained perception by 3.7% on average over LLaVA-UHD.

  7. KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge

    cs.LG 2025-08 conditional novelty 5.0 of 10

    KnowDR-REC is a benchmark that tests image-and-text AI models on object finding that needs real-world knowledge, and on 16 current models most of them fail.

  8. LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    LaVi encodes visual context into LayerNorm affine parameters, bypassing visual token concatenation, and reports LLaVA-comparable accuracy at a 94% FLOP reduction.

  9. Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation

    cs.AI 2025-06 reject novelty 5.0 of 10

    A fixed-image, multi-task evaluation framework aims to detect data contamination in multimodal LLMs, but its judge is unvalidated and possibly self-referential, and the claimed harm to generalization is not supported ...

  10. Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos

    cs.CV 2025-06 reject novelty 5.0 of 10

    PAM extends SAM 2 with a frozen LLM and a Semantic Perceiver to jointly segment and describe regions in images, videos, and streaming video, and contributes a 0.6M-sample region-level streaming video caption dataset.

  11. Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Splitting the vision encoder of an MLLM into domain-specific experts with a lightweight router yields small benchmark improvements at near-zero extra inference cost.

  12. Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding

    cs.CV 2025-01 conditional novelty 5.0 of 10

    PIIP is a multi-branch, parameter-inverted image pyramid that uses smaller pretrained networks for high-resolution inputs and larger networks for low-resolution inputs, improving efficiency across perception and multi...

  13. Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A GUI-controlling visual agent that uses edge-density-based cropping and a self-training loop matches or beats models trained on 10x more data.

Pith tools