Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Automatic captions let text-only language models handle image-based exam questions, more than doubling the number of usable items in the BLUEX benchmark.

desk verdict Useful benchmark increment with a real leakage concern in the context-caption arm; the resource is worth engaging, but the analytic claims need tightening. read the letter →

arxiv 2508.21294 v1 pith:2WINJSKG submitted 2025-08-29 cs.CL cs.AI

João Guilherme Alves Santos , Giovana Kerche Bonás , Thales Sales Almeida This is my paper
classification cs.CLcs.AI
keywords BLUEXPortuguesebenchmarkimagecaptioningLLMevaluationmultimodalreasoningBrazilianuniversityentranceexamsvisualgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that automatically generated image captions can make an image-heavy, Portuguese-language exam benchmark usable by text-only language models. The authors extend BLUEX, a dataset of multiple-choice questions from Brazilian university entrance exams, with 2024 and 2025 exams and with GPT-4o captions for every image, produced under two conditions: blind captions written from the image alone, and context captions written with the question in view. They report 1,422 usable questions, more than double the count previously available to text-only models, and accuracy gains of at least 10 points on image-only-alternative questions for most evaluated models when context captions are supplied. If correct, this gives the Portuguese LLM community a larger, more current benchmark and a controlled way to separate visual grounding from textual reasoning.

What carries the argument

The machinery is the dual captioning pipeline applied to every image in BLUEX via GPT-4o. A blind caption is generated from the image alone; a context caption is generated from the image plus the question and answer options. In evaluation, captions are inserted at the position where the image originally appeared, yielding three conditions: no caption, blind caption, and context caption. This design isolates the effect of visual description on model accuracy and is what makes the image-based questions answerable by text-only models.

What would settle it

Take a random sample of image-based BLUEX questions, and have one group of humans answer them with only the GPT-4o context captions and another group with only the original images. If the caption-only group scores materially worse, or an annotator audit finds a recurring rate of hallucinated details such as wrong labels, invented numbers, or missing diagram elements, then the benchmark's claim that captions make these questions accessible to text-only models is contradicted.

Watch

Extended reading notes

Core claim

The central claim is that for BLUEX, GPT-4o-generated captions are sufficient to make image-dependent questions accessible to text-only LLMs, and that context-aware captions perform as well as blind captions despite being roughly half as long. The expanded set covers exams from 2018 through 2025, with 610 image-based items; converting those items to captions raises the count of text-only-usable questions to 1,422. The paper also reports that captioning improves accuracy on questions whose answer choices are images alone, that larger models benefit most from captions, and that the top evaluated models now score at or above the admission threshold for roughly 90% of undergraduate programs in e

Load-bearing premise

Everything caption-based rests on GPT-4o's captions being faithful, complete substitutes for the original images; if a caption omits or invents a detail the question depends on, the caption-condition accuracy numbers no longer measure the model's reasoning.

Editorial extensions

If this is right

  • Text-only language models can now be benchmarked on the full BLUEX image set instead of only the text-only subset, more than doubling the available items for non-multimodal evaluation.
  • Researchers can compare no-caption, blind-caption, context-caption, and true-image conditions on identical questions to isolate how much of an exam question depends on visual information.
  • The added 2024 and 2025 exams are recent enough that high scores are less likely to come from memorized training data, strengthening their use in contamination and generalization studies.
  • Because context captions are shorter yet match blind captions in accuracy, future evaluation runs can use more economical prompts without sacrificing measured performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the near-tie between blind and context captions suggests task-relevant visual information is usually localizable, and that shorter context captions may be the better default for cost and clarity.
  • Our inference: if caption fidelity holds up under human audit, the same captioning pipeline could be transferred to other image-heavy exams, expanding non-multimodal evaluation beyond this one benchmark.
  • Our inference: the paper does not audit captions for hallucinated details, so the next natural test is an error analysis tagging invented labels, numbers, or diagram elements; systematic hallucination would change how the caption-condition scores should be interpreted.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents BLUEX Revisited, an updated version of the Portuguese university-entrance-exam benchmark BLUEX. It adds 2024 and 2025 exams and automatically generated GPT-4o captions for image-based questions under two conditions: blind captions (image only) and context captions (image plus the associated question). The authors evaluate 18 commercial and open-source models on text-only, blind-caption, and context-caption versions of the dataset, reporting large accuracy gains when captions are provided and comparing model performance with admission cutoffs at Unicamp and USP. The central claims are that captioning increases accessibility to text-only models by more than 40%, yielding 1,422 usable questions, and that context captions, despite being shorter, perform comparably to blind captions.

Significance. If the claims hold, the expanded dataset is a useful resource for Portuguese-language and multilingual model evaluation, especially for text-only models and for contamination-related studies. The paper's explicit strengths are its public dataset and code release, its transparent construction pipeline, and its broad model coverage. However, the central quantitative contribution depends on two assumptions that are not yet validated: that GPT-4o captions faithfully preserve the visual information needed to answer, and that context-caption results are not inflated by answer leakage when the caption generator sees the multiple-choice alternatives. The contamination claim in Section 4.2 is also asserted without probing. These issues affect the headline 'usable questions' and 'caption effectiveness' claims, so the paper is best viewed as a promising resource description with preliminary evaluations rather than a conclusive benchmark validation.

major comments (4)
  1. [Section 3.2; Table 2; Section 4.1] Context captions are generated by GPT-4o with access to 'the associated question' (Section 3.2). Since the question includes the multiple-choice alternatives, the correct answer is visible to the caption generator. The Context Captions condition is therefore not a clean measure of visual understanding: GPT-4o may encode cues that favor the correct alternative, and the similarity between Context and Blind Captions in Table 2 may reflect leakage compensating for shorter captions rather than the 'focusing' hypothesis stated in Section 4.1. Please add a control where context captions are generated without the correct answer (e.g., alternatives removed or replaced) and report whether the Context Captions column changes. This is load-bearing for the conclusion that context-aware captions are effective.
  2. [Section 3.2; Figure 3; Table 2] Caption adequacy is never validated. The paper reports caption length distributions (Figure 3) but no human rating, no alternative captioner comparison, and no check that captions contain the information needed to answer. The entire 'usable questions' claim and all caption-condition accuracies in Table 2 assume GPT-4o captions are faithful and sufficient. A simple fix is to sample, say, 100 image questions, have human raters mark whether each caption preserves the visual information needed to answer, and report agreement; alternatively, compare model accuracy with captions vs original images. Without this, the dataset resource may contain systematic caption errors.
  3. [Section 4.2; Section 5] The paper motivates the update by contamination studies and asserts that high 2025 scores indicate genuine reasoning because 2025 exams are recent. This is asserted, not probed. No exact-match or near-duplicate search, no temporal leakage analysis, and no comparison with held-out questions are reported. Since contamination is a stated central motivation, either add a contamination probe or soften the claim. As written, the conclusion in Section 5 that performance stems from reasoning rather than memorization is unsupported.
  4. [Table 2; Section 4] All accuracies are single-run point estimates without error bars, confidence intervals, or significance tests. Many comparisons are within 1-2 points (e.g., Sabia-3 0.750 vs GPT-4o 0.754 on all questions). The model rankings and the claim that captioning improves accuracy by 'at least 10 accuracy points' in Figure 4 need repeated runs with different random seeds or bootstrap intervals. This is important because the headline contribution is quantitative.
minor comments (5)
  1. [Abstract; Table 1] The abstract says 'more than 40%' and 'more than doubling the number' but the manuscript never states the original BLUEX total or the exact percentage increase. Please reconcile these numbers with Table 1's counts (812 without images, 610 with images, 1,422 total).
  2. [Figure 3] The caption of Figure 3 refers to 'GPT-4V' while the text consistently uses GPT-4o. Please make the terminology consistent.
  3. [Table 2] The header 'Comercial models' is a typo for 'Commercial models'. Also, the table is very dense; consider separating the three partitions into sub-tables or adding clearer column separators.
  4. [Section 2.2] The phrase 'In parallel, Importantly,' is ungrammatical and should be rephrased.
  5. [Figure 4] The y-axis label 'Accuracy gain' should specify 'percentage points' and clarify that the gain is computed relative to the no-caption condition. The x-axis label 'Model Size' is unclear because models are ordered by size but the axis is categorical.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's results are empirical measurements and dataset construction, not derivation-from-inputs.

full rationale

The paper's central claims are (1) that adding GPT-4o captions makes more BLUEX questions accessible to text-only models, and (2) that measured accuracies show models can use these captions. Neither claim reduces to its own inputs by construction. The '1,422 usable questions' figure is a direct count after captioning, not a fitted parameter or a prediction derived from a fitted value; it is a dataset-description fact. The accuracy numbers in Table 2 are obtained by running models on fixed prompts, not by optimizing any parameter to match the reported accuracies. The prior BLUEX paper is cited as the source of the original dataset, which is legitimate foundation rather than an unverified authority invoked to force the new conclusions. The affiliation of the authors with Maritaca and the evaluation of Sabia-3 is a conflict-of-interest consideration but not circular reasoning. The use of GPT-4o both as caption generator and as an evaluated model, and the fact that context captions are generated with access to the question (and possibly the alternatives), are validity concerns about answer leakage or same-model bootstrap, but they do not make any stated result equivalent to its inputs by definition. No equation is fitted, no uniqueness theorem is imported from self-citation, and no ansatz is smuggled in via citation. The paper is self-contained against external benchmarks; its contribution is empirical and its reported results could in principle be falsified by re-running the evaluations. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper imposes no free parameters. Its central empirical claims rest on domain assumptions about caption fidelity, score comparability, and contamination exposure, listed above.

assumptions (3)
  • domain assumption Captions generated by GPT-4o faithfully and sufficiently encode the visual content needed to answer each question.
    The entire caption-based evaluation rests on this. The paper does not validate caption adequacy against human judgments or against vision-model performance on the same questions. Introduced in Section 3.2 caption generation.
  • domain assumption Model accuracy on the multiple-choice subset can be compared directly to official admission cutoff scores.
    Section 4.2 and Figure 5 compare 'accuracy' to cutoff scores for 90% of courses and the highest cutoff. University entrance scores are not simple percentage accuracy; this comparison is unvalidated.
  • domain assumption 2024/2025 exam questions are unlikely to appear in LLM training data.
    Section 4.2 uses recency to argue against memorization, but no contamination probing is performed and training corpora are not disclosed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning." pith.science (2026). https://pith.science/paper/2WINJSKG

@misc{pith2026250821294,
  author       = {Pith},
  title        = {Pith review of: BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WINJSKG}},
  note         = {Machine review of arXiv:2508.21294}
}
read the original abstract

With the growing capabilities of Large Language Models (LLMs), there is an increasing need for robust evaluation methods, especially in multilingual and non-English contexts. We present an updated version of the BLUEX dataset, now including 2024-2025 exams and automatically generated image captions using state-of-the-art models, enhancing its relevance for data contamination studies in LLM pretraining. Captioning strategies increase accessibility to text-only models by more than 40%, producing 1,422 usable questions, more than doubling the number in the original BLUEX. We evaluated commercial and open-source LLMs and their ability to leverage visual context through captions.

Figures

Figures reproduced from arXiv: 2508.21294 by the authors.

Figure 1
Figure 1. Overview of the benchmark construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparative example of blind and context captioning. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Distribution of caption lengths for blind and context captions. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Accuracy gain in BLUEX questions that have images as alternatives. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance of the top 5 tested models in each year, alongside the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MARCA: A Checklist-Based Benchmark for Multilingual Web Search

    cs.CL 2026-04 accept novelty 6.0 of 10

    MARCA is a bilingual benchmark using 52 questions and validated checklists to evaluate LLM web-search completeness and correctness in English and Portuguese.

Reference graph

Works this paper leans on

39 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abonizio, H. et al. (2024). Sabi 'a-3 technical report. arXiv preprint arXiv:2410.12049

    arXiv 2024

  2. [2]

    Almeida, T. S. (2025). Revisited bluex benchmark - code repository. https://github.com/ZanezZephyrs/bluex_eval. Accessed: 2025-08-07

    work page 2025

  3. [3]

    Almeida, T. S. et al. (2025). Tiebe: Tracking language model recall of notable worldwide events through time. arXiv preprint arXiv:2501.07482

    arXiv 2025

  4. [4]

    Almeida, T. S. et al. (2023). Bluex: A benchmark based on brazilian leading universities entrance exams. In Brazilian Conference on Intelligent Systems , pages 337--347. Springer

    work page 2023

  5. [5]

    Bianco, S. et al. (2023). Improving image captioning descriptiveness by ranking and llm-based fusion. arXiv preprint arXiv:2306.11593

    arXiv 2023

  6. [6]

    Chang, Y. et al. (2023). Booookscore: A systematic exploration of book-length summarization in the era of llms. arXiv preprint arXiv:2310.00785

    arXiv 2023

  7. [7]

    Chen, M. et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  8. [8]

    Datasets, P. B. (2025). Bluex: Brazilian undergraduate entrance exams benchmark. https://huggingface.co/datasets/portuguese-benchmark-datasets/BLUEX. Accessed: 2025-08-07

    work page 2025

Show all 39 references
  1. [9]

    Delfino, P. et al. (2017). Passing the brazilian oab exam: data preparation and some experiments. In Legal knowledge and information systems , pages 89--94. IOS Press

    2017

  2. [11]

    Gao, L. et al. (2024). The language model evaluation harness

    2024

  3. [12]

    Grattafiori, A. et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  4. [13]

    Hendrycks, D. et al. (2021). Measuring massive multitask language understanding

    2021

  5. [14]

    Hu, Y. et al. (2023). Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 20406--20417

    2023

  6. [15]

    Lazaridou, A. et al. (2022). Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115

    2022 arXiv

  7. [16]

    Li, B. et al. (2024). Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790

    2024 arXiv

  8. [17]

    Li, B. et al. (2023). Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125

    2023 arXiv

  9. [18]

    Liu, A. et al. (2024a). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    arXiv

  10. [19]

    Liu, J. et al. (2023). Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems , 36:21558--21572

    2023

  11. [20]

    Liu, X. et al. (2024b). Mm-safetybench: A benchmark for safety evaluation of multimodal large language models

  12. [21]

    Nam, D. et al. (2024). Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , pages 1--13

    2024

  13. [22]

    Gpt-4o system card

    OpenAI (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  14. [23]

    OpenAI et al. (2024). Gpt-4 technical report

    2024

  15. [24]

    Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730--27744

    2022

  16. [25]

    Patraucean, V. et al. (2023). Perception test: A diagnostic benchmark for multimodal video models. In Oh, A. et al., editors, Advances in Neural Information Processing Systems , volume 36, pages 42748--42761. Curran Associates, Inc

    2023

  17. [26]

    Petroni, F. et al. (2019). Language models as knowledge bases? arXiv preprint arXiv:1909.01066

    2019 arXiv

  18. [27]

    Pi, R. et al. (2024). Image textualization: An automatic framework for generating rich and detailed image descriptions. Advances in Neural Information Processing Systems , 37:108116--108139

    2024

  19. [28]

    Pires, R. et al. (2023). Evaluating gpt-4's vision capabilities on brazilian university admission exams. arXiv preprint arXiv:2311.14169

    2023 arXiv

  20. [29]

    Rein, D. et al. (2024). Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  21. [30]

    Academic ranking of world universities 2024

    ShanghaiRanking Consultancy (2024). Academic ranking of world universities 2024. Accessed: 2025-04-25

    2024

  22. [31]

    Silveira, I. C. and Mau \'a , D. D. (2017). University entrance exam as a guiding test for artificial intelligence. In 2017 Brazilian Conference on Intelligent Systems (BRACIS) , pages 426--431. IEEE

    2017

  23. [32]

    Singhal, K. et al. (2025). Toward expert-level medical question answering with large language models. Nature Medicine , pages 1--8

    2025

  24. [33]

    Team, T. (2024). Falcon 3 family of open foundation models

    2024

  25. [34]

    World university rankings 2024

    Times Higher Education (2024). World university rankings 2024. Accessed: 2025-04-25

    2024

  26. [35]

    Yang, A. et al. (2024). Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  27. [36]

    Zhang, S. et al. (2023). Planning with large language models for code generation. arXiv preprint arXiv:2303.05510

    2023 arXiv

  28. [37]

    Zhang, T. et al. (2024a). Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics , 12:39--57

  29. [38]

    Zhang, Y. et al. (2024b). A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. arXiv preprint arXiv:2403.02901

  30. [39]

    Zhong, W. et al. (2023). Agieval: A human-centric benchmark for evaluating foundation models

    2023

  31. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.