Pith. sign in

REVIEW 3 major objections 6 minor 21 references

Quizzard@INOVA Challenge 2025 -- Track A: Plug-and-Play Technique in Interleaved Multi-Image Model

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard fine-tuning achieves the highest overall accuracy, while the DCI-enhanced variant wins on semantic-coherence and structured-change datasets.

desk verdict A useful challenge-report numbers table that contradicts its own abstract and Section IV-B, so the headline claim is not trustworthy as written. read the letter →

arxiv 2506.11737 v1 pith:WL46O7QD submitted 2025-06-13 cs.CV cs.CLcs.MM

Dinh Viet Cuong , Hoang-Bao Le , An Pham Ngoc Nguyen , Liting Zhou , Cathal Gurrin This is my paper · ORCID
classification cs.CVcs.CLcs.MM
keywords interleaveLLaVAcomprehensiondenseconnectorChannelIntegrationmulti-imagereasoningplug-and-playtechniquevision-languagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a plug-and-play connector, Dense Channel Integration (DCI), improves an interleaved multi-image model, LLaVA-NeXT-Interleave, across 22 datasets spanning multi-image reasoning, document and knowledge understanding, and interactive multimodal dialogue. Fine-tuning the base model gives the best overall accuracy, while the DCI-enhanced version matches or beats it on tasks that require semantic coherence and structured change, such as MIT-States PropertyCoherence and SlideVQA. The paper reports that both fine-tuned variants far outperform the original model, and identifies a trade-off between general balance and coherence-specific strength.

What carries the argument

The key object is the Dense Channel Integration (DCI) connector, which replaces the vision encoder's final-only output with a hierarchical fusion of all layer features. The connector partitions the $L$ vision-encoder layers into $G$ groups of $M = L/G$ adjacent layers, averages each group to produce fused representations $G^L_i = \frac{1}{M}\sum_{k=(i-1)M+1}^{iM} V_k$, and concatenates these with the final layer feature $V_L$ to form the vision embedding $E_V = \mathrm{Concatenate}([G^L_1, \dots, G^L_G, V_L])$. This adds fewer than 2% trainable parameters and needs only single-stage instruction tuning; the paper's comparison isolates the effect of this connector on top of the LLaVA-NeXT-Interleave base model.

What would settle it

Recompute the comparison with per-example weighting or repeated seeds across the 22 datasets: if the DCI-enhanced model's average reaches or exceeds the standard model's under any plausible reweighting, the claim that standard fine-tuning is best overall fails.

Watch

Extended reading notes

Core claim

The central claim is that standard fine-tuning of LLaVA-NeXT-Interleave achieves the highest overall accuracy, while adding the DCI connector changes the strength profile rather than uniformly improving it. On vision-heavy benchmarks (VISION, NLVR2, Fashion200K) the standard model excels; on datasets demanding semantic coherence or structured change understanding (MIT-States PropertyCoherence, SlideVQA) the DCI version is stronger. The paper also reports training-loss dynamics in which DCI converges faster early but shows higher variance later, and it reports a second-place test result with strong accuracy on majority datasets but low ROUGE-L on free-answer generation datasets.

Load-bearing premise

The overall comparison rests on unweighted averages of one validation-run accuracy or ROUGE-L score per dataset; if datasets were weighted by size or difficulty, or runs repeated with different seeds, the ranking between standard and DCI fine-tuning could change.

Editorial extensions

If this is right

  • Standard fine-tuning of LLaVA-NeXT-Interleave yields the best overall accuracy across the three challenge tasks, so a single model choice is adequate for general use.
  • The DCI connector gives a measurable advantage on semantic-coherence and structured-change benchmarks such as MIT-States PropertyCoherence and SlideVQA.
  • Both fine-tuned variants exceed the original model by wide margins, so task-specific fine-tuning matters more than connector choice for interleaved multi-image models.
  • The low ROUGE-L scores on free-answer datasets (Spot-the-Diff, IEdit, Birds-to-Words) indicate that open-ended generation remains a bottleneck for this training strategy.
  • A hybrid fine-tuning strategy combining standard and DCI approaches is a natural next step suggested by the observed trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper's overall ranking depends on unweighted averages, a task-weighted or example-weighted aggregation might shift the standard-vs-DCI comparison; this is an editorial caveat, not a paper claim.
  • DCI's faster early convergence, if it transfers to low-data regimes, could make it the better choice for few-shot adaptation even though it loses on the full benchmark average.
  • The task-family split (vision-heavy vs coherence-heavy) hints at a simple selection rule: use standard fine-tuning for generic visual QA, DCI for tasks whose questions hinge on property consistency or structured change.
  • Combining the two models in an ensemble or a two-stage router could recover the best of both profiles; this is a testable extension the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper describes the authors' submission to the ICME25 INOVA Challenge Track A. The authors take LLaVA-NeXT-Interleave, fine-tune it on 22 challenge datasets, and compare three configurations: the original model, standard fine-tuning (FT), and fine-tuning with the Dense Channel Integration (DCI) connector. The reported validation results in Table I are used to argue that standard FT gives the best overall accuracy, excelling on VISION, NLVR2, and Fashion200K, while DCI helps on datasets requiring semantic coherence such as MIT-States PropertyCoherence and SlideVQA. The authors also report their second-place test-set score of 74.9 and discuss training-loss behavior.

Significance. If the comparisons were reliable, this would be a useful ablation of a plug-and-play connector in the interleaved multi-image setting, with practical implications for choosing between standard fine-tuning and DCI-enhanced fine-tuning depending on task family. The paper is also valuable as a reproducible challenge report: the code is released, the setup is simple (one A100 GPU, one epoch of fine-tuning), and the test-set result confirms competitiveness. However, the qualitative conclusions as stated are not supported by the printed table, and the single-run nature and unweighted aggregation limit the strength of the quantitative claims; the manuscript therefore requires correction before its central message can be accepted.

major comments (3)
  1. [Section IV-B, Table I] The text's central qualitative claims are contradicted by the numbers in Table I. The abstract and Section IV-B state that FT excels on VISION, NLVR2, and Fashion200K, while DCI is strong on MIT-States PropertyCoherence and SlideVQA. Under the column labels "FT w/ DCI, FT, Original", the table shows VISION 88.90 (DCI) vs. 88.55 (FT), PropertyCoherence 91.50 (DCI) vs. 94.75 (FT), and SlideVQA 51.50 (DCI) vs. 65.25 (FT). Thus DCI beats FT on VISION, which the text attributes to FT, and loses on the two datasets cited as DCI strengths. Swapping the first two columns would make the two DCI examples consistent but would make DCI beat FT on Fashion200K and NLVR2 as well, contradicting the other half of the claim. No consistent labeling of the columns makes all named examples match the text; the headline comparison must be corrected or the table entries must be verified.
  2. [Table I, Document and Knowledge-Based Understanding average] The average reported for the Original model in the Document and Knowledge-Based Understanding block is arithmetically wrong. The six row entries (63.50, 90.50, 26.38, 78.50, 17.90, 11.52) sum to 288.30, so the average is 48.05, not the printed 35.30. The corresponding DCI and FT averages are consistent (49.17 and 56.10), which suggests a localized error, but it undermines confidence in the table's other entries and in the "highest overall accuracy" comparison, since those averages are the basis for the claim.
  3. [Section IV-B and Section III-C] The conclusion that FT "achieves the highest overall accuracy" rests on unweighted averages over 22 datasets from a single validation split, with no error bars or repeated runs. The paper uses a 9:1 split of the provided training data and reports one run per configuration, so it is not possible to assess whether the observed margins (e.g., 1-2 accuracy points in several blocks) are stable. The authors should add confidence intervals or at least report the number of examples per dataset, and should temper the claim accordingly if those are unavailable.
minor comments (6)
  1. [Throughout] Please fix the typos "A VERAGE" in Table I and Figure 4, "ROGUE-L" for ROUGE-L, and the line-break artifacts in "LLaV A" throughout the text.
  2. [References] References [4] and [8] are the same CLIP paper; the reference numbering should be corrected, since [3] is GPT-3 and [4] is currently listed as CLIP.
  3. [Section III-B and Table I] The dataset names are inconsistent: Section III-B writes "MMQA", Table I uses "MultiModalQA" and "ManyModalQA", and Figure 4 uses only "MultiModalQA"; please align the names between text and tables.
  4. [Section IV-B and Figure 4] Figure 4 reports the test-set result but does not state which of the three validation configurations (Original, FT, or FT with DCI) produced that submitted run; please clarify.
  5. [Section IV-B] The test-set scores in Figure 4 differ substantially from the validation scores for several datasets (e.g., SlideVQA 81.40 test vs. 51.50/65.25 validation), but the paper does not explain whether different splits or test-time procedures were used; please provide the protocol.
  6. [Section V] The phrase "we just pre-train the model" is inaccurate for the described procedure; the paper performs fine-tuning, not pre-training, so the wording should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark comparison; the table/text inconsistency is a correctness issue, not a circular derivation.

full rationale

The paper's derivation chain is an empirical comparison of three checkpoints on the INOVA Track A validation and test splits. No equation or fitted parameter is later renamed as a prediction, and the DCI connector is presented as an external implementation adopted from [16]. The only author-overlapping citation is [14], which appears solely in the introduction as related work and is not used to justify any result or to constrain the experimental choices. The central claims are summaries of Table I, and even if those summaries are internally inconsistent (the Document Understanding average for the Original model is arithmetically wrong: the six listed values sum to 288.30, giving 48.05, not the reported 35.30; and Table I shows FT above DCI on both datasets named as DCI strengths, PropertyCoherence and SlideVQA), that inconsistency is a correctness and reproducibility problem, not a circular step. The limitations section also reports only one-GPU and 7B-parameter training, which is an honest scope statement rather than a circular move. Because no prediction reduces to its own input and no self-citation is load-bearing, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on imported components: DCI connector, LLaVA-NeXT-Interleave, and challenge data. No free parameters are fit to the evaluation results and no new entities are introduced. The assumptions listed are operational evaluation choices that the paper does not test.

assumptions (3)
  • domain assumption SigLIP and Qwen-based LLaVA-NeXT-Interleave serve as a suitable base for all three task families.
    Section II-A adopts the model without re-evaluating whether its pretraining matches the challenge distribution.
  • domain assumption Integrating the DCI connector from [16] into the LLaVA-NeXT-Interleave pipeline preserves the connector's stated behavior.
    Section II-B imports Eq. (1)-(2) directly and does not re-tune grouping or projection for this model, assuming compatibility.
  • domain assumption The 9:1 split of the provided data yields validation metrics representative of test performance.
    Section IV-A states the split; no stratification or leakage analysis is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quizzard@INOVA Challenge 2025 -- Track A: Plug-and-Play Technique in Interleaved Multi-Image Model." pith.science (2026). https://pith.science/paper/WL46O7QD

@misc{pith2026250611737,
  author       = {Pith},
  title        = {Pith review of: Quizzard@INOVA Challenge 2025 -- Track A: Plug-and-Play Technique in Interleaved Multi-Image Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WL46O7QD}},
  note         = {Machine review of arXiv:2506.11737}
}
read the original abstract

This paper addresses two main objectives. Firstly, we demonstrate the impressive performance of the LLaVA-NeXT-interleave on 22 datasets across three different tasks: Multi-Image Reasoning, Documents and Knowledge-Based Understanding and Interactive Multi-Modal Communication. Secondly, we add the Dense Channel Integration (DCI) connector to the LLaVA-NeXT-Interleave and compare its performance against the standard model. We find that the standard model achieves the highest overall accuracy, excelling in vision-heavy tasks like VISION, NLVR2, and Fashion200K. Meanwhile, the DCI-enhanced version shows particular strength on datasets requiring deeper semantic coherence or structured change understanding such as MIT-States_PropertyCoherence and SlideVQA. Our results highlight the potential of combining powerful foundation models with plug-and-play techniques for Interleave tasks. The code is available at https://github.com/dinhvietcuong1996/icme25-inova.

Figures

Figures reproduced from arXiv: 2506.11737 by the authors.

Figure 1
Figure 1. LLaVa-NeXT-Interleave with DCI Connector [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Vision Embedding block: CLIP-ViT-based Vision [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. highlights clear differences. Notably, the DCI￾enhanced model converges faster during the early train￾ing phase (up to step 60), consistently showing lower loss values. This indicates that DCI facilitates more efficient multimodal fusion and accelerates learning. However, in the later training stages (after step 100), the DCI variant exhibits higher variance, suggesting a degree of optimization instability likely du… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance on Track A’s Test set. Multi-Image Reasoning Task result contains 2 tables (a) and (b), while [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages

  1. [1]

    Attention is all you need,

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    work page 2017

  2. [2]

    Bert: Pre-training of deep bidirectional transform- ers for language understanding,

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Bert: Pre-training of deep bidirectional transform- ers for language understanding,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , 2019, pp. 4171–4186

    work page 2019

  3. [3]

    Language models are few-shot learners,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al., “Language models are few-shot learners,” Advances in neural information process- ing systems, vol. 33, pp. 1877–1901, 2020

    work page 1901

  4. [4]

    Learning transferable visual models from natural language supervision,

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

    work page 2021

  5. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa De- hghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    arXiv 2010

  6. [6]

    Efficient Classification of Long Documents Using Transformers

    Hyunji Hayley Park, Yogarshi Vyas, and Kashif Shah, “Efficient classification of long documents using transformers,” arXiv preprint arXiv:2203.11258, 2022

    work page Pith review arXiv 2022

  7. [7]

    Is space- time attention all you need for video understanding?,

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani, “Is space- time attention all you need for video understanding?,” in ICML, 2021, vol. 2, p. 4

    work page 2021

  8. [8]

    Learning transferable visual models from natural language supervision,

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    work page 2021

Show all 21 references
  1. [9]

    Improved baselines with visual instruction tuning,

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, “Improved baselines with visual instruction tuning,” 2023

    2023

  2. [10]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024

    2024

  3. [11]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Hao- tian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” Advances in Neural Information Processing Systems, vol. 36...

    2023

  4. [12]

    LLaV A-UHD: an lmm perceiving any aspect ratio and high- resolution images,

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang, “LLaV A-UHD: an lmm perceiving any aspect ratio and high- resolution images,” in ECCV, 2024

    2024

  5. [13]

    Llava- mini: Efficient image and video large multimodal models with one vision token,

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng, “Llava- mini: Efficient image and video large multimodal models with one vision token,” 2025

    2025

  6. [14]

    Logra-med: Long context multi-graph alignment for medical vision-language model,

    Duy M. H. Nguyen, Nghiem T. Diep, Trung Q. Nguyen, Hoang- Bao Le, Tai Nguyen, Tien Nguyen, TrungTin Nguyen, Nhat Ho, Pengtao Xie, Roger Wattenhofer, James Zhou, Daniel Sonntag, and Mathias Niepert, “Logra-med: Long context multi-graph alignment for medical vision-language model,” 2024

    2024

  7. [15]

    Llava needs more knowledge: Retrieval augmented natural language generation with knowledge graph for explaining thoracic pathologies,

    Ameer Hamza, Yong Hyun Ahn, Sungyoung Lee, Seong Tae Kim, et al., “Llava needs more knowledge: Retrieval augmented natural language generation with knowledge graph for explaining thoracic pathologies,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2025, vol...

    2025

  8. [16]

    Dense connector for mllms,

    Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang, “Dense connector for mllms,” Advances in Neural Information Processing Systems , 2024

    2024

  9. [17]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models,

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan, “Llava-prumerge: Adaptive token reduction for efficient large multimodal models,” arXiv preprint arXiv:2403.15388 , 2024

    2024

  10. [18]

    Visual instruction tuning,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, “Visual instruction tuning,” 2023

    2023

  11. [19]

    Sigmoid loss for language image pre-training,

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, “Sigmoid loss for language image pre-training,” in Pro- DATASET ROGUE-L Spot-the-Diff 35.58 CLEVR-Change 60.30 IEdit 32.94 Birds-to-Words 35.09 A VERAGE 40.98 (a) DATASET Accuracy nuscenes 85.80 VISION 93.60 F...

    2023

  12. [20]

    Llama 2: Open foundation and fine-tuned chat models,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  13. [21]

    Yi: Open foundation models by 01. ai,

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jian- qun Chen, et al., “Yi: Open foundation models by 01. ai,” arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.