pith. sign in

arxiv: 2510.12184 · v2 · submitted 2025-10-14 · 💻 cs.CV · cs.AI

CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs

Jiwan Kim , Kibum Kim , Sangwoo Seo , Chanyoung Park This is my paper

Pith reviewed 2026-05-18 07:45 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords knowledge distillationmultimodal LLMscompositional reasoningvisual attentionattention alignmentmodel compressionvisual perception
0
0 comments X

The pith

Aligning visual attention during distillation improves compositional reasoning in smaller multimodal LLMs

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing knowledge distillation from large multimodal models to smaller ones falls short at transferring visual perception skills needed for detailed reasoning about images. Analysis shows the root issue is misalignment in where the student and teacher models focus their attention on visual inputs. CompoDistill adds an explicit step to match the student's visual attention maps to the teacher's during training. This change raises accuracy on tasks that combine visual details with reasoning while keeping results on general visual question answering at the level of prior distillation methods. The same gains appear when the approach is applied to stronger backbone models.

Core claim

The paper identifies visual attention misalignment between student and teacher as the main obstacle preventing effective transfer of visual perception abilities in multimodal LLM distillation. CompoDistill is introduced as a framework that explicitly aligns the student's visual attention with the teacher's to overcome this barrier. Experiments demonstrate that the resulting student models perform substantially better on compositional reasoning tasks requiring visual perception while retaining strong performance on visual question answering tasks.

What carries the argument

CompoDistill, a knowledge distillation framework that adds explicit alignment of the student's visual attention maps to those of the teacher model.

If this is right

  • Smaller multimodal models gain improved accuracy on benchmarks that test fine-grained visual understanding combined with reasoning.
  • General visual question answering performance remains comparable to results from existing distillation techniques.
  • The framework continues to deliver gains when paired with more advanced model backbones.
  • Distillation training can be redirected to emphasize visual perception transfer rather than language knowledge alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same attention-matching step could be tested on distillation for other vision-language skills such as spatial or temporal reasoning.
  • If attention alignment proves efficient, it might allow effective distillation even when the teacher model is only modestly larger than the student.
  • Similar internal representation mismatches may limit distillation success in purely language or audio multimodal settings.

Load-bearing premise

Visual attention misalignment is the primary cause of weak transfer of visual perception abilities in existing distillation, and correcting this alignment will directly strengthen the student's compositional reasoning.

What would settle it

Applying the attention alignment step to a new teacher-student pair and observing no gain over standard distillation on compositional reasoning benchmarks would falsify the central claim.

read the original abstract

Recently, efficient Multimodal Large Language Models (MLLMs) have gained significant attention as a solution to their high computational complexity, making them more practical for real-world applications. In this regard, the knowledge distillation (KD) approach has emerged as a promising alternative, which transfers the rich visual and linguistic knowledge from a larger model (teacher) to a smaller model (student). However, we observe that existing KD methods struggle to effectively distill the teacher MLLM's rich visual perception abilities to the student, a challenge that has been largely overlooked in previous studies. Through a systematic analysis, we identify visual attention misalignment between student and teacher as the main cause of this issue. Based on this insight, we propose CompoDistill, a novel KD framework that explicitly aligns the student's visual attention with that of the teacher to enhance the student's visual perception abilities. Our extensive experiments show that CompoDistill significantly improves performance on compositional reasoning tasks that require visual perception abilities while maintaining strong performance on visual question answering tasks, as done in existing studies. Furthermore, CompoDistill demonstrates effectiveness with a more advanced backbone, highlighting its generalizability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces CompoDistill, a knowledge distillation framework for multimodal LLMs that identifies visual attention misalignment between teacher and student models as the primary barrier to transferring visual perception abilities. It proposes an explicit attention alignment loss to address this issue, claiming improved results on compositional reasoning tasks that depend on visual perception while preserving performance on standard VQA tasks and demonstrating applicability to stronger backbones.

Significance. If the core claims are substantiated, the work would provide a mechanistic diagnosis and targeted remedy for a specific weakness in existing KD pipelines for MLLMs, potentially aiding the development of more capable yet efficient vision-language models. The attempt to link attention misalignment directly to compositional deficits is a constructive contribution, though its impact depends on clearer isolation of the proposed mechanism from other training effects.

major comments (2)
  1. [Abstract and systematic analysis section] The abstract and the section describing the systematic analysis assert that visual attention misalignment is the main cause of ineffective visual-perception distillation, yet no quantitative metrics, attention-map comparisons, or controlled comparisons are presented to establish this as the dominant factor over alternatives such as capacity gaps or optimization differences. This causal attribution is load-bearing for the entire framework.
  2. [Experiments section and associated tables] The experimental results claim significant gains on compositional reasoning tasks, but the manuscript lacks ablations that hold all other distillation components (loss weights, data, optimizer) fixed while toggling only the attention-alignment term. Without such controls, observed improvements cannot be confidently attributed to attention alignment rather than ancillary training effects.
minor comments (2)
  1. [Abstract] The abstract states performance improvements without any numerical deltas, baseline names, or dataset identifiers, which limits immediate assessment of the magnitude of gains.
  2. [Method section] The precise formulation of the attention alignment loss would benefit from an explicit equation or pseudocode to clarify how maps are compared and weighted.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and insightful comments. We have carefully reviewed the feedback and believe it highlights important areas where additional evidence can strengthen the presentation of our claims. We address each major comment below and outline the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract and systematic analysis section] The abstract and the section describing the systematic analysis assert that visual attention misalignment is the main cause of ineffective visual-perception distillation, yet no quantitative metrics, attention-map comparisons, or controlled comparisons are presented to establish this as the dominant factor over alternatives such as capacity gaps or optimization differences. This causal attribution is load-bearing for the entire framework.

    Authors: We agree that a more direct and quantitative demonstration of the causal link would improve the manuscript. The systematic analysis currently relies on comparative performance results across task types that differentially require visual perception, together with qualitative attention observations. To address the concern, we will add quantitative metrics (e.g., mean cosine similarity and KL divergence between normalized attention maps of teacher and student) and controlled comparisons that vary model capacity and optimization settings while measuring attention alignment. These results and corresponding attention-map visualizations will be included in a revised systematic analysis section. revision: yes

  2. Referee: [Experiments section and associated tables] The experimental results claim significant gains on compositional reasoning tasks, but the manuscript lacks ablations that hold all other distillation components (loss weights, data, optimizer) fixed while toggling only the attention-alignment term. Without such controls, observed improvements cannot be confidently attributed to attention alignment rather than ancillary training effects.

    Authors: We concur that isolating the contribution of the attention-alignment loss is necessary for a rigorous attribution of gains. The current experiments compare the full CompoDistill framework against prior KD baselines, but do not include the requested single-factor ablation. We will conduct and report new controlled experiments in which all other loss weights, training data, and optimizer settings remain fixed while the attention-alignment term is toggled on or off. The resulting performance deltas on compositional reasoning benchmarks will be added to the experiments section and tables. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper identifies visual attention misalignment via systematic analysis and proposes CompoDistill to align student-teacher attention maps for better distillation of visual perception. No equations, fitted parameters renamed as predictions, self-citations, or ansatzes are shown that reduce the central result to its own inputs by construction. The approach is presented as insight-driven with claimed empirical gains on compositional tasks, remaining self-contained without load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that attention misalignment is the dominant failure mode in visual knowledge transfer for MLLMs; no free parameters or invented entities are detailed in the abstract.

axioms (1)
  • domain assumption Visual attention misalignment is the main cause of ineffective distillation of visual perception abilities from teacher to student MLLMs.
    Identified via systematic analysis as stated in the abstract.

pith-pipeline@v0.9.0 · 5740 in / 1114 out tokens · 26489 ms · 2026-05-18T07:45:00.258666+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs

    cs.LG 2026-04 unverdicted novelty 7.0

    Sink-Token-aware Pruning (SToP) suppresses semantically uninformative sink tokens during visual token pruning in Video LLMs, boosting fine-grained performance even at 90% pruning rates across hallucination, reasoning,...

  2. Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding

    cs.CV 2026-04 unverdicted novelty 7.0

    Visual token pruning in MLLMs fails on complex reasoning due to Relevant Visual Information Shift during decoding, but the DSTP framework fixes it training-free across models.

  3. Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency

    cs.CL 2026-04 conditional novelty 5.0

    Widthwise pruning of LVLM language backbones combined with supervised finetuning and hidden-state distillation recovers over 95% performance using just 5% of data across 3B-7B models.