pith. sign in

arxiv: 2504.21850 · v3 · pith:G7USP7MRnew · submitted 2025-04-30 · 💻 cs.CV

Visual Compositional Tuning

Xindi Wu , Hee Seung Hwang , Polina Kirichenko , Esin Tureci , Olga Russakovsky This is my paper

Pith reviewed 2026-05-22 17:29 UTC · model grok-4.3

classification 💻 cs.CV
keywords visual instruction tuningdata efficiencycompositional data synthesismultimodal large language modelsdataset curationvision-language taskssynthetic data generation
0
0 comments X

The pith

Compositional synthesis of complex questions from images allows multimodal models to match full-dataset performance using 90% less training data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces COMPACT, a method for creating visual instruction tuning datasets by combining multiple atomic visual capabilities into richer, more complex training examples for each image. This compositional approach to data curation addresses the overlooked issue of sample complexity in large VIT datasets, showing that fewer but denser examples can drive effective finetuning of multimodal large language models. When tested on the LLaVA-665K dataset, the method cuts the data budget by 90 percent while reaching or surpassing the performance of models trained on the complete set across eight benchmarks. It particularly excels on complex reasoning tasks, outperforming full-data training on MM-Vet and MMStar. The work positions synthetic data generation as a scalable path to efficient vision-language model training.

Core claim

COMPACT scales training sample complexity by synthesizing rich and informative text questions for each image that combine multiple atomic visual capabilities into single examples, thereby reducing the number of training instances needed for effective visual instruction tuning while maintaining or improving multimodal benchmark performance.

What carries the argument

COMPACT (COMPositional Atomic-to-complex Visual Compositional Tuning), the recipe that synthesizes rich text questions to merge multiple atomic visual capabilities into high-quality composite training signals.

If this is right

  • Data reduction techniques in visual instruction tuning can prioritize compositional complexity over simple informativeness scoring.
  • Training on COMPACT-generated data yields measurable gains on complex reasoning benchmarks relative to training on the entire original dataset.
  • Synthetic compositional data recipes provide a scalable alternative to collecting ever-larger VIT corpora for multimodal model finetuning.
  • Reducing the data budget by 90 percent while preserving performance lowers the compute and storage costs of vision-language model development.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same atomic-to-complex synthesis principle could be tested on other modalities or tasks where sample density affects learning efficiency.
  • Explicit compositional structure in training data might improve model interpretability by making the learned visual capabilities more traceable.
  • Future experiments could measure whether the performance edge on complex benchmarks persists when COMPACT is applied to different base models or even smaller data fractions.

Load-bearing premise

Synthesized questions accurately combine atomic visual capabilities into informative training signals without introducing artifacts, biases, or loss of quality.

What would settle it

A direct comparison experiment showing that models fine-tuned on the COMPACT 10-percent subset underperform those trained on the full LLaVA-665K dataset across the eight multimodal benchmarks, or fail to exceed full-data results on MM-Vet and MMStar.

Figures

Figures reproduced from arXiv: 2504.21850 by Esin Tureci, Hee Seung Hwang, Olga Russakovsky, Polina Kirichenko, Xindi Wu.

Figure 1
Figure 1. Figure 1: Complexity k. We show that increasing the complexity of LLaVA-665K improves performance. (left) Examples of questions with different k-values, where k is the number of atomic capabilities required. (middle) Distribution of k-value in VIT subset (LLaVA) and VIT subset augmented with 1 additional capability (LLaVAk+1). (right) Performance on downstream tasks (§4.1) for VIT subset (LLaVA), VIT subset regenera… view at source ↗
Figure 2
Figure 2. Figure 2: COMPACT data generation pipeline. (Left): We design a data recipe that can scale the complexity of each training example. We randomly sample kgen ∈ {1, 2, 3} atomic capabilities such as color, object recognition, and spatial relationship. (Center): We generate questions that integrate all kgen sampled capabilities and verify their quality. (Right): We combine the synthetically generated compositional tunin… view at source ↗
Figure 3
Figure 3. Figure 3: Performance across compositional tuning data scales. We show that COMPACT’s compositional tuning data scales more efficiently than conventional VIT. We fix the VIT subset (5% of LLaVA-665K (Liu et al., 2024b)) and scale the compositional tuning data in COMPACT from 2K to 32K. We compare each mix with VIT only datasets with equal data budgets. COMPACT (solid lines) consistently outperforms LLaVA-665K VIT (d… view at source ↗
Figure 5
Figure 5. Figure 5: Impact of instruction tuning data ratio. Relative performance of COMPACT with different amounts instruction tuning data from LLaVA-665K (Liu et al., 2024b). The x-axis is the percentage of LLaVA-665K used as instruc￾tion tuning data, and the y-axis is relative score. The performance improves significantly with a small amount of instruction tuning data and sta￾bilizes around 5%. five settings. As shown in … view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of capability distribution. The bar plots show the frequency of each atomic capability in LLaVA (left) and COMPACT (right) samples. In LLaVA, the distribution is notably imbalanced: object recognition and scene understanding are some of the most frequent, while shape and spatial recognition are less prevalent. In contrast, COMPACT exhibits a more balanced distribution across capability categorie… view at source ↗
Figure 8
Figure 8. Figure 8: Correlation between capabilities. The heatmap shows the correlation between unique capabilities in COMPACT’s composi￾tional tuning data. Object recognition’s correla￾tions with other capabilities are relatively strong. Spatial capabilities are also locally correlated, as spatial questions require some understanding of the scene, depth, and relative position in prac￾tice. Back to Table of Contents 2 Back to… view at source ↗
Figure 9
Figure 9. Figure 9: Limited performance improvements on knowledge-intensive benchmarks. Com￾parison shows modest improvements over ran￾dom baseline on tasks that require substantial world knowledge or domain expertise. Num￾bers reported in accuracy (%) and relative per￾formance to full model (%). Model OK-VQA MMMU MMMU-Pro Rel. Standard Vision (Avg.) Random 49.30 32.89 18.15 11.44 92.0% COMPACT 50.02 33.89 20.23 11.91 96.6% L… view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative comparison of model outputs. Examples showing responses from our compositionally-tuned COMPACT model and LLaVA-665K (Liu et al., 2024b) VIT model on complex queries that require multiple capabilities (k ≥ 3). Our model demonstrates better integration of visual capabilities which leads to more accurate responses. Yellow What color is the vest worn by the train worker? How many people are visibl… view at source ↗
Figure 12
Figure 12. Figure 12: Visualization of COMPACT compositional tuning samples. Back to Table of Contents 10 Back to the First Page [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Visualization of COMPACT compositional tuning samples. Back to Table of Contents 11 Back to the First Page [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
read the original abstract

Visual instruction tuning (VIT) datasets have grown rapidly in scale, yet the informativeness of individual training samples has largely been overlooked. Recent dataset selection methods have shown that a small fraction of such datasets enriched with informative samples can lead to efficient finetuning of Multimodal Large Language Models. In this work, we explore the impact of sample complexity on informative data curation and introduce COMPACT (COMPositional Atomic-to-complex Visual Compositional Tuning), a compositional VIT data recipe that scales training sample complexity by combining multiple atomic visual capabilities in a single training example. Concretely, we synthesize rich and informative text questions for each image, allowing us to significantly reduce the number of training examples required for effective VIT. COMPACT demonstrates superior data efficiency compared to existing data reduction methods. When applied to the LLaVA-665K VIT dataset, COMPACT reduces the data budget by 90% while still achieving 100.2% of the full VIT performance (compared to only 97.5% by the state-of-the-art method) across eight multimodal benchmarks. Furthermore, training on the COMPACT data outperforms training on the full-scale VIT data on particularly complex benchmarks such as MM-Vet (+8.6%) and MMStar (+2.9%). COMPACT offers a scalable and efficient synthetic data generation recipe to improve on vision-language tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript introduces COMPACT, a compositional data synthesis recipe for visual instruction tuning (VIT). It generates rich text questions per image by combining multiple atomic visual capabilities, claiming this allows a 90% reduction in training data from the LLaVA-665K dataset while reaching 100.2% of full-dataset performance across eight multimodal benchmarks and outperforming the full data on MM-Vet (+8.6%) and MMStar (+2.9%).

Significance. If the synthesis procedure produces high-quality, artifact-free questions that genuinely integrate atomic capabilities, the result would demonstrate a scalable path to more efficient VIT that improves both data efficiency and performance on complex tasks relative to full-scale datasets and prior selection methods. The concrete quantitative comparisons to the full LLaVA-665K baseline and to the state-of-the-art reduction method supply a clear, falsifiable benchmark for future work.

major comments (1)
  1. Abstract: The central performance claims (90% data reduction to 100.2% of full VIT performance, +8.6% on MM-Vet) rest on the quality of the synthesized questions that 'combine multiple atomic visual capabilities.' No description is given of the synthesis procedure, the method for identifying or validating atomic capabilities, or any controls for synthetic artifacts, bias, or informativeness. This information is load-bearing; without it the reported gains cannot be attributed to the compositional approach rather than uncontrolled properties of the generated data.
minor comments (1)
  1. Abstract: The acronym VIT is defined as 'visual instruction tuning' on first use; a parenthetical note distinguishing it from Vision Transformer would prevent potential reader confusion.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback and positive assessment of COMPACT's data-efficiency results. We address the single major comment below.

read point-by-point responses

  1. Referee: Abstract: The central performance claims (90% data reduction to 100.2% of full VIT performance, +8.6% on MM-Vet) rest on the quality of the synthesized questions that 'combine multiple atomic visual capabilities.' No description is given of the synthesis procedure, the method for identifying or validating atomic capabilities, or any controls for synthetic artifacts, bias, or informativeness. This information is load-bearing; without it the reported gains cannot be attributed to the compositional approach rather than uncontrolled properties of the generated data.

    Authors: We agree that the abstract, constrained by length, does not elaborate on the synthesis procedure. The full manuscript details the approach in Section 3: atomic capabilities are extracted from existing VIT datasets via capability tagging, then combined through a rule-based compositional generator that produces multi-capability questions per image; quality controls include automated filtering for grammatical coherence and human validation on a subset to check for artifacts and bias. To make this load-bearing information more immediately visible, we will revise the abstract to include a one-sentence summary of the synthesis recipe and validation steps. revision: yes

Circularity Check

0 steps flagged

No significant circularity; forward synthesis procedure only

full rationale

The provided abstract describes a forward data-synthesis recipe that combines atomic visual capabilities into richer training questions to reduce data volume while preserving or improving performance. No equations, fitted parameters, predictions of derived quantities, or self-citations appear in the text. The method is presented as an empirical engineering procedure rather than a derivation chain that reduces to its own inputs by construction, satisfying the criteria for a self-contained non-circular claim.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the unverified effectiveness of the synthetic question generation process that is asserted to preserve or improve training signal quality while reducing volume.

axioms (1)
  • domain assumption Synthesized questions can effectively combine atomic visual capabilities into informative training samples without loss of quality.
    This premise is required for the 90% data reduction to maintain or exceed full-dataset performance.

pith-pipeline@v0.9.0 · 5748 in / 1297 out tokens · 44155 ms · 2026-05-22T17:29:37.091022+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models

    cs.CV 2025-06 unverdicted novelty 7.0

    AVA-Bench evaluates vision foundation models by disentangling 14 atomic visual abilities with aligned training-test distributions to reveal precise ability fingerprints.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · cited by 1 Pith paper · 7 internal anchors

  1. [1]

    SemDeDup: Data-efficient learning at web-scale through semantic deduplication

    Amro Abbas, Kushal Tirumala, D ´aniel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Which of these best describes multiple choice evaluation with llms? a) forced b) flawed c) fixable d) all of the above.arXiv preprint arXiv:2502.14127,

    Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. Which of these best describes multiple choice evaluation with llms? a) forced b) flawed c) fixable d) all of the above.arXiv preprint arXiv:2502.14127,

    work page arXiv

  3. [3]

    Are We on the Right Way for Evaluating Large Vision-Language Models?

    Hyunsik Chae, Seungwoo Yoon, Chloe Yewon Chun, Gyehun Go, Yongin Cho, Gyeongmin Lee, and Ernest K Ryu. Decomposing complex visual comprehension into atomic visual skills for vision language models. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi ...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    A closer look at the limitations of instruction tuning.arXiv preprint arXiv:2402.05119,

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha, et al. A closer look at the limitations of instruction tuning.arXiv preprint arXiv:2402.05119,

    work page arXiv

  5. [5]

    Wang, and Sadid Hasan

    Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. Does prompt formatting have any impact on llm performance?arXiv preprint arXiv:2411.10541,

    work page arXiv

  6. [6]

    Mmcomposition: Revisiting the compositionality of pre-trained vision-language models.arXiv preprint arXiv:2410.09733,

    10 COMPACT Hang Hua, Yunlong Tang, Ziyun Zeng, Liangliang Cao, Zhengyuan Yang, Hangfeng He, Chenliang Xu, and Jiebo Luo. Mmcomposition: Revisiting the compositionality of pre-trained vision-language models.arXiv preprint arXiv:2410.09733,

    work page arXiv

  7. [7]

    Visual instruction tuning towards general- purpose multimodal model: A survey.arXiv preprint arXiv:2312.16602,

    Jiaxing Huang, Jingyi Zhang, Kai Jiang, Han Qiu, and Shijian Lu. Visual instruction tuning towards general- purpose multimodal model: A survey.arXiv preprint arXiv:2312.16602,

    work page arXiv

  8. [8]

    Explain before you answer: A survey on compositional visual reasoning.arXiv preprint arXiv:2508.17298, 2025

    Fucai Ke, Joy Hsu, Zhixi Cai, Zixian Ma, Xin Zheng, Xindi Wu, Sukai Huang, Weiqing Wang, Pari Delir Haghighi, Gholamreza Haffari, et al. Explain before you answer: A survey on compositional visual reasoning. arXiv preprint arXiv:2508.17298,

    work page arXiv

  9. [9]

    Concept-skill transferability-based data selection for large vision-language models.arXiv preprint arXiv:2406.10995,

    Jaewoo Lee, Boyang Li, and Sung Ju Hwang. Concept-skill transferability-based data selection for large vision-language models.arXiv preprint arXiv:2406.10995,

    work page arXiv

  10. [10]

    LLaVA-OneVision: Easy Visual Task Transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024a. Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich...

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Mosaic-it: Free compositional data augmentation improves instruction tuning.arXiv preprint arXiv:2405.13326, 2024c

    Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, Yupeng Hou, Fuxiao Liu, and Tianyi Zhou. Mosaic-it: Free compositional data augmentation improves instruction tuning.arXiv preprint arXiv:2405.13326, 2024c. Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Ea...

    work page arXiv

  12. [12]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306, 2024a. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conf...

    work page arXiv

  13. [13]

    When less is more: Investigating data pruning for pretraining llms at scale.arXiv preprint arXiv:2309.04564,

    Max Marion, Ahmet ¨Ust¨ un, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale.arXiv preprint arXiv:2309.04564,

    work page arXiv

  14. [14]

    Prompting large vision-language models for compositional reasoning.arXiv preprint arXiv:2401.11337,

    Timothy Ossowski, Ming Jiang, and Junjie Hu. Prompting large vision-language models for compositional reasoning.arXiv preprint arXiv:2401.11337,

    work page arXiv

  15. [15]

    None of the others: a general technique to distinguish reasoning from memorization in multiple-choice llm evaluation benchmarks.arXiv preprint arXiv:2502.12896,

    Eva S ´anchez Salido, Julio Gonzalo, and Guillermo Marco. None of the others: a general technique to distinguish reasoning from memorization in multiple-choice llm evaluation benchmarks.arXiv preprint arXiv:2502.12896,

    work page arXiv

  16. [16]

    Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders.arXiv preprint arXiv:2408.15998,

    work page arXiv

  17. [17]

    Gemini: A Family of Highly Capable Multimodal Models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Finetuned Language Models Are Zero-Shot Learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Icons: Influence consensus for vision-language data selection.arXiv preprint arXiv:2501.00654, 2024a

    Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, and Olga Russakovsky. Icons: Influence consensus for vision-language data selection.arXiv preprint arXiv:2501.00654, 2024a. Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty.Advances in ...

    work page arXiv

  20. [20]

    MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Li- juan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    Instruction-Following Evaluation for Large Language Models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    work page internal anchor Pith review Pith/arXiv arXiv