Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

MSA at ImageCLEF 2025 Multimodal Reasoning: Multilingual Multimodal Reasoning With Ensemble Vision Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An ensemble of two lightweight Gemini models for image captions and a Gemini 2.5 Pro reasoner that outputs only a letter won the ImageCLEF 2025 multilingual multimodal reasoning track at 81.4% accuracy.

desk verdict Solid competition write-up with a trustworthy leaderboard result, but the abstract overclaims a comparison to heavier end-to-end models that the experiments never actually run. read the letter →

arxiv 2507.11114 v1 pith:SLBALEIF submitted 2025-07-15 cs.CL

Seif Ahmed , Mohamed T. Younes , Abdelrahman Moustafa , Abdelrahman Allam , Hamza Moustafa This is my paper · ORCID
classification cs.CL
keywords MultimodalReasoningVision-LanguageModelsLargeLanguageMultilingualQAImageCLEF2025EXAMS-VChallengeEnsemblePromptEngineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a lightweight, two-stage ensemble of vision-language models, coordinated by strict prompt engineering, can beat heavier end-to-end systems on multilingual multimodal exam questions. As evidence, it reports first place on the ImageCLEF 2025 multimodal reasoning leaderboard with 81.4% accuracy over 13 tracks, topping 11 of them. The authors attribute the gain to separating visual description from final answer selection and forcing the reasoner to output only a letter. They also show that translating training questions into English boosts fine-tuned small models substantially, while zero-shot Gemini 2.5 Flash still outperforms them all. If correct, the finding implies that modular, low-cost pipelines are a viable path for high-stakes educational QA.

What carries the argument

The pipeline is a two-stage ensemble. Stage 1 is an OCR–VLM pair: Gemini 2.5 Flash produces a detailed caption of the exam image under a few-shot prompt that preserves math symbols and normalizes answer-option markers, and Gemini 1.5 Pro re-checks the caption against the original image, fixes label mismatches, and flags missing diagrams. Stage 2 passes the caption to Gemini 2.5 Pro with a zero-shot prompt that demands the final answer be exactly one of A–E, with no explanation. The captions are written in the language of the document metadata, which lets the system sidestep OCR failures on non-Latin scripts.

What would settle it

Run a heavy end-to-end multimodal model, such as GPT-4o or a single-pass Gemini 2.5 Pro, on the same EXAMS-V test questions with no two-stage decomposition; if it scores above 81.4%, the claim that the lightweight ensemble outperforms heavier end-to-end models would be contradicted.

Watch

Extended reading notes

Core claim

The central claim is that a modular ensemble—Gemini 2.5 Flash for image captions, Gemini 1.5 Pro for caption verification and normalization, and Gemini 2.5 Pro as a letter-only reasoner—achieves state-of-the-art multilingual multimodal question answering, winning the ImageCLEF 2025 EXAMS-V multilingual track with 81.4% accuracy. The paper further claims that strict prompt constraints (letter-only output, option normalization, language-consistent captions) are a first-order factor, improving English validation accuracy from 55.9% to 61.7%, and that multilingual data augmentation gives large fine-tuned models gains of roughly 20 percentage points while a 7B model gains almost nothing. On this evidence the authors conclude that lightweight OCR–VLM ensembles can outperform heavier end-to-end models in this setting.

Load-bearing premise

The paper's conclusion that lightweight ensembles beat heavier end-to-end models rests on comparing its system only to the organizer-supplied baseline, which it never shows is representative of heavy end-to-end models.

Editorial extensions

If this is right

  • A two-stage 'describe then reason' pipeline with strict output constraints can reach top leaderboard accuracy without training a custom model.
  • Prompt formatting is worth several accuracy points: enforcing letter-only answers and normalized option labels lifted English validation accuracy from 55.9% to 61.7%.
  • Multilingual augmentation transfers reasoning skill across languages for models with enough capacity, producing +19.63 and +19.96 percentage point gains for 14B and 12B models, while a 7B model gains only +0.74 points.
  • Zero-shot Gemini 2.5 Flash (79.65% on the expanded validation set) beats fine-tuned open models, indicating that large-scale pretraining currently matters more than task-specific fine-tuning for this benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The +54-point gap over the organizer baseline suggests the baseline is far below the achievable frontier, so the win is better read as a strong systems result than as evidence about the absolute quality ceiling of current VLMs.
  • A stage-level ablation (removing the Gemini 1.5 Pro verifier) would isolate how much of the gain comes from caption verification versus visual description; the paper does not report this, but the design suggests it is the main novelty.
  • The same normalization tricks could transfer to other noisy document tasks, such as historical OCR or multilingual forms, where option markers and stray scripts break downstream models.
  • If the organizers release per-question scores, one could test whether the ensemble's advantage concentrates on image_text questions vs text-only questions; the paper doesn't break this down.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This working-notes paper describes Team MSA's submission to the ImageCLEF 2025 EXAMS-V Visual Question Answering task. The system is a two-stage ensemble: Gemini 2.5 Flash generates image captions, Gemini 1.5 Pro verifies and normalizes them, and Gemini 2.5 Pro selects the final multiple-choice answer from zero-shot prompts with strict formatting constraints. The paper reports first place on the official multilingual leaderboard with 81.40% accuracy and first place in 10 of the 12 individual language tracks shown in Table 3, alongside ablations on fine-tuned text-only LLMs and on prompt style. The authors conclude that lightweight OCR-VLM ensembles can outperform heavier end-to-end models.

Significance. The paper's main factual contribution is an externally validated leaderboard result: the system ranked first in the multilingual track of ImageCLEF 2025, which is a strong, verifiable outcome for a shared-task system description. The ablation of multilingual data augmentation on several open LLMs, and the prompt-format comparison on the English validation split, provide useful practical evidence for the community. The paper is less strong as a scientific investigation: the headline comparative claim about lightweight versus heavy end-to-end models is not tested by the reported experiments, the ablation results are single-run and lack error bars, and there are inconsistencies in the dataset and rank descriptions. These issues affect the paper's central claims and should be addressed.

major comments (4)
  1. [Abstract, §5.2 (Table 3), §6] The claim that 'lightweight OCR-VLM ensembles ... can outperform heavier end-to-end models' is not supported by any experiment in the paper. Table 3 compares the MSA system only with the organizer-supplied baseline, whose architecture and model identity are never described and whose multilingual accuracy (27.01%) is close to random for five options. Table 4 compares fine-tuned text-only LLMs (Phi-4, Gemma-3, Mistral) with zero-shot Gemini 2.5 Flash, not with any heavier end-to-end VLM. The only heavier model actually used, Gemini 2.5 Pro, is the reasoner inside the proposed pipeline. To keep this conclusion, the authors should compare against at least one heavier end-to-end VLM; otherwise, the conclusion should be restricted to 'achieved first place on this leaderboard and outperformed the organizer baseline'.
  2. [§5.3 (Table 5) and §5.2 (Table 3)] The prompt-ablation results are not connected to the full-pipeline leaderboard results. Table 5 reports accuracy on the English validation split for a 'Reasoner stage' using Gemini 2.5 Flash or Gemini 1.5 Pro, but the deployed system uses Gemini 2.5 Pro as the final reasoner. The paper never reports the full pipeline's English validation accuracy, nor how the 61.7% prompt-ablation result relates to the 86.52% English leaderboard score. Consequently, the statement that prompt engineering 'boosted model accuracy from 55.9% to 61.7%' does not explain the final system performance, and the contribution of the Gemini 1.5 Pro aggregator stage is not isolated by any ablation.
  3. [§3 (Table 2) and Abstract/§5.2 (Table 3)] The dataset statistics are internally inconsistent. The text in §3 says '13 languages' and then enumerates 12 (English, Chinese, German, Spanish, Arabic, Italian, Bulgarian, Croatian, Serbian, Urdu, Polish, Kazakh), while Table 2 lists 11 languages and omits Spanish, Urdu, and Kazakh while including French and Hungarian. Table 3 reports results for 12 individual languages plus the multilingual row. These inconsistencies make it difficult to verify the dataset description and the '13 individual language tracks' claim in the abstract; the counts should be reconciled.
  4. [§5.1 and §5.3] All experimental results are reported from single runs, without repeated trials, confidence intervals, or significance tests. Because the pipeline uses API models at nonzero temperature (e.g., temperature=1.5 for Gemini 2.5 Flash and Gemini 1.5 Pro), the observed differences such as 55.91% versus 57.06% in Table 5 may be within run-to-run noise. At a minimum, the authors should state how many runs were averaged, or provide error bars for the key comparisons; otherwise, the ablation conclusions are not robust.
minor comments (5)
  1. [Abstract and Table 3] The abstract states that the system 'led 11 out of 13 individual language tracks', but Table 3 shows first place in 10 individual languages, second in English, and third in Bulgarian. The count should be corrected to match the table, or the criterion for 'leading' should be clarified.
  2. [References] Several references are duplicated: [5] and [10] are the same M4U paper, [6] and [11] are the same M3Exam paper, and [12] and [13] are the same chain-of-thought paper. These duplicates should be consolidated.
  3. [Table 2] The table caption says 'multilingual dataset under study consists of over 20,000 questions', which is consistent with the sum of the listed counts, but the table itself is incomplete relative to the languages mentioned in the text; please add the missing rows or revise the text.
  4. [§4.2] The few-shot prompt used for Gemini 2.5 Flash is described only in prose ('1 example'). For reproducibility, the exact prompt examples and the number of few-shot examples used in the final system should be included in an appendix.
  5. [§5.3] Table 4 reports results for fine-tuned models, but the fine-tuning setup is not described: no training hyperparameters, number of epochs, learning rate, or evaluation split are given. Adding these details would make the ablation reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: leaderboard results are externally measured and ablations use standard validation splits.

full rationale

The paper's central claim is a measured competition outcome: Team MSA achieved first place on the ImageCLEF 2025 Multimodal Reasoning multilingual track with 81.4% accuracy, as reported on the official leaderboard. Accuracy is defined by the competition's official evaluation protocol and computed against external ground truth; no metric or target quantity is constructed from the system's own outputs. The ablations compare fine-tuned LLMs and prompt variants on English validation splits, and the prompt that improves validation accuracy from 55.9% to 61.7% is standard validation-set tuning, not a parameter fitted to the leaderboard score. The ensemble pipeline (Gemini 2.5 Flash -> Gemini 1.5 Pro -> Gemini 2.5 Pro) is described as a system design, not derived from the leaderboard result. The abstract's generalization that lightweight OCR-VLM ensembles can outperform heavier end-to-end models is not directly tested in the experiments, which we regard as a correctness/evidential weakness rather than circularity: the comparison is unsupported, but it is not an equation or metric that reduces to its own inputs. No load-bearing step relies on a self-citation, a uniqueness theorem, or an ansatz smuggled in from the authors' prior work. The paper is a standard shared-task system description with externally validated results, so the appropriate circularity finding is none.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No invented entities. The reported results rest on external evaluation, commercial API behavior, and machine-generated translations; none are independently verified in the paper. Temperature and shot count are hand-selected without sensitivity analysis.

free parameters (2)
  • API temperature settings = 1.5 (2.5 Flash), 1.5 (1.5 Pro), 0.2 (2.5 Pro)
    Reported in Section 5.1, chosen by hand with no sensitivity analysis; they affect sampling randomness and thus the reported accuracies.
  • Few-shot example count for describer = 1
    Section 4.2 states a few-shot prompt with 1 example is prepended; no search over shot count is reported.
assumptions (3)
  • domain assumption The official leaderboard accuracy and rankings reported are accurate.
    The paper relies on the organizer's evaluation protocol and provides no independent verification.
  • domain assumption Gemini API outputs on the reported runs are representative and stable enough for the stated single-run accuracies.
    No repeated runs, confidence intervals, or API version controls are provided.
  • domain assumption Machine translations generated by Gemini 1.5 Pro preserve the semantic content of original questions.
    The expanded training set is translated automatically without reported human validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MSA at ImageCLEF 2025 Multimodal Reasoning: Multilingual Multimodal Reasoning With Ensemble Vision Language Models." pith.science (2026). https://pith.science/paper/SLBALEIF

@misc{pith2026250711114,
  author       = {Pith},
  title        = {Pith review of: MSA at ImageCLEF 2025 Multimodal Reasoning: Multilingual Multimodal Reasoning With Ensemble Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SLBALEIF}},
  note         = {Machine review of arXiv:2507.11114}
}
read the original abstract

We present a robust ensemble-based system for multilingual multimodal reasoning, designed for the ImageCLEF 2025 EXAMS V challenge. Our approach integrates Gemini 2.5 Flash for visual description, Gemini 1.5 Pro for caption refinement and consistency checks, and Gemini 2.5 Pro as a reasoner which handles final answer selection, all coordinated through carefully engineered few-shot and zero-shot prompts. We conducted an extensive ablation study, training several large language models (Gemini 2.5 Flash, Phi 4, Gemma 3, Mistral) on an English dataset and its multilingual augmented version. Additionally, we evaluated Gemini 2.5 Flash in a zero-shot setting for comparison and found it to substantially outperform the trained models. Prompt design also proved critical: enforcing concise, language-normalized formats and prohibiting explanatory text boosted model accuracy on the English validation set from 55.9% to 61.7%. On the official leaderboard, our system (Team MSA) achieved first place overall in the multilingual track with 81.4% accuracy, and led 11 out of 13 individual language tracks, with top results such as 95.07% for Croatian and 92.12% for Italian. These findings highlight that lightweight OCR-VLM ensembles, when paired with precise prompt strategies and cross-lingual augmentation, can outperform heavier end-to-end models in high-stakes, multilingual educational settings.

Figures

Figures reproduced from arXiv: 2507.11114 by the authors.

Figure 1
Figure 1. Illustrative OCR-related challenges encountered in the dataset. The dataset includes both multiple-choice questions and visual reasoning problems. However, several challenges were observed: OCR-specific Challenges: Some items were printed in a language different from their metadata tag, while others lacked standard option labels (A–E) or used a different script problems that confused OCR and downstream prompts (see … view at source ↗
Figure 2
Figure 2. System pipeline: OCR–VLM ensemble (Gemini 2.5 Flash + Gemini 1.5 Pro) produces text for LLM answer selection (Gemini 2.5 Pro). descriptions from each question image; second, a Reasoner stage maps the cleaned text to a final multiple–choice answer [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

    cs.CL 2026-07 accept novelty 5.5 of 10

    On EXAMS-V, token budget and parseability dominate chain count, PRM-guided search, and selectors for small VLMs; the policy model itself yields the largest gain.

Reference graph

Works this paper leans on

19 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Zhang, J

    J. Zhang, J. Huang, S. Jin, S. Lu, Vision-language models for vision tasks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    work page 2024

  2. [2]

    Y. Wang, W. Chen, X. Han, X. Lin, H. Zhao, Y. Liu, B. Zhai, J. Yuan, Q. You, H. Yang, Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning, arXiv preprint arXiv:2401.06805 (2024)

    arXiv 2024

  3. [3]

    J. Bi, S. Liang, X. Zhou, P. Liu, J. Guo, Y. Tang, L. Song, C. Huang, G. Sun, J. He, et al., Why reasoning matters? a survey of advancements in multimodal reasoning (v1), arXiv preprint arXiv:2504.03151 (2025)

    arXiv 2025

  4. [4]

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, A. Kalyan, Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems 35 (2022) 2507–2521

    work page 2022

  5. [6]

    Huang, et al., M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models, in: NeurIPS Datasets and Benchmarks Track, 2023

    Y. Huang, et al., M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models, in: NeurIPS Datasets and Benchmarks Track, 2023

    work page 2023

  6. [7]

    J. Gao, J. Song, J. Wu, R. Zhu, G. Shen, S. Wang, X. Wei, H. Yang, S. Zhang, W. Li, B. Wang, D. Lin, L. Wu, C. He, Pm4bench: A parallel multilingual multi-modal multi-task benchmark for large vision language model, 2025. URL: https://arxiv.org/abs/2503.18484.arXiv:2503.18484

    work page arXiv 2025

  7. [8]

    Dimitrov, M

    D. Dimitrov, M. S. Hee, Z. Xie, R. Jyoti Das, M. Ahsan, S. Ahmad, N. Paev, I. Koychev, P. Nakov, Overview of imageclef 2025 – multimodal reasoning, in: CLEF 2025 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Madrid, Spain, 2025

    work page 2025

  8. [9]

    Ionescu, H

    B. Ionescu, H. Müller, D.-C. Stanciu, A.-G. Andrei, A. Radzhabov, Y. Prokopchuk, Ştefan, Liviu- Daniel, M.-G. Constantin, M. Dogariu, V. Kovalev, H. Damm, J. Rückert, A. Ben Abacha, A. Garc ’ia Seco de Herrera, C. M. Friedrich, L. Bloch, R. Brüngel, A. Idrissi-Yaghir, H. Schäfer, C. S. Schmidt, T. M. G. Pakull, B. Bracke, O. Pelka, B. Eryilmaz, H. Becker,...

    work page 2025

Show all 19 references
  1. [10]

    H. Wang, J. Xu, S. Xie, R. Wang, J. Li, Z. Xie, B. Zhang, C. Xiong, X. Chen, M4u: Evaluating multilingual understanding and reasoning for large multimodal models, 2025. URL: https://arxiv. org/abs/2405.15638.arXiv:2405.15638

    2025 arXiv

  2. [11]

    Zhang, M

    W. Zhang, M. Aljunied, C. Gao, Y. K. Chia, L. Bing, M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models, in: A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine (Eds.), Advances in Neural Information Processing Systems, volu...

    2023

  3. [13]

    Zhou, et al., Language models are multilingual chain-of-thought reasoners, arXiv preprint arXiv:2210.03057 (2022)

    X. Zhou, et al., Language models are multilingual chain-of-thought reasoners, arXiv preprint arXiv:2210.03057 (2022)

    2022 arXiv

  4. [14]

    R. Das, S. Hristov, H. Li, D. Dimitrov, I. Koychev, P. Nakov, EXAMS-V: A multi-discipline mul- tilingual multimodal exam benchmark for evaluating vision language models, in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for...

    2024 doi

  5. [15]

    K. Zhou, J. Yang, C. C. Loy, Z. Liu, Learning to prompt for vision-language models, International Journal of Computer Vision 130 (2022) 2337–2348

    2022

  6. [16]

    Accessed: 2025-03-15

    Google DeepMind, Gemini 2.5 pro vs flash: Performance comparison and model selection, https: //deepmind.google/technologies/gemini/pro/, 2025. Accessed: 2025-03-15

    2025

  7. [17]

    Accessed: 2025-01-10

    Google AI, Gemini 2.5 pro: Benchmark results and technical specifications, https://blog.google/ technology/ai/google-gemini-ai-update-december-2024/, 2025. Accessed: 2025-01-10

    2024

  8. [18]

    Accessed: 2025-05-28

    Google DeepMind, Gemini 2.5 pro: Our latest advances in reasoning, coding, and multimodal understanding, https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/, 2025. Accessed: 2025-05-28

    2025

  9. [19]

    Accessed: 2024-12-15

    Microsoft, Phi-4 technical report, https://arxiv.org/abs/2412.08905, 2024. Accessed: 2024-12-15

    2024 arXiv

  10. [20]

    DeepMind, Gemma Team, Gemma 3: Advancing open language models, https://blog.google/ technology/developers/gemma-3-google-new-open-model/, 2024

    G. DeepMind, Gemma Team, Gemma 3: Advancing open language models, https://blog.google/ technology/developers/gemma-3-google-new-open-model/, 2024. Accessed: 2024-12-15

    2024

  11. [21]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al., Mistral 7b, https://arxiv.org/abs/2310.06825, 2023. arXiv:2310.06825

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.