Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AutoConverter, a multi-agent GPT-4o pipeline, converts open-ended visual questions into multiple-choice items with machine-written distractors at least as challenging as human-crafted ones, yielding the 9,018-question VMCBench benchmark.

desk verdict Useful benchmark, solid engineering, but the 'challenging' claim needs a matched human-difficulty control before it earns the headline. read the letter →

arxiv 2501.03225 v2 pith:M7VCQ5V3 submitted 2025-01-06 cs.CV cs.AIcs.CLcs.CYcs.LG

Yuhui Zhang , Yuchang Su , Yiming Liu , Xiaohan Wang , James Burgess , Elaine Sui , Chenyu Wang , Josiah Aklilu
show 4 more authors
Alejandro Lozano Anjiang Wei Ludwig Schmidt Serena Yeung-Levy
This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.CYcs.LG
keywords visionlanguagemodelsVQAevaluationmultiple-choicequestiongenerationdistractoragenticframeworkVMCBenchbenchmarkconstructionGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-ended visual question answering is hard to score: the paper shows that word-overlap rules barely agree with human judgment, and that LLM-based graders shift scores when the grading model is updated. Its proposed fix is AutoConverter, a multi-agent system built on GPT-4o that rewrites open-ended questions as four-option multiple-choice items with automatically generated plausible wrong answers, called distractors. The central claim is that these machine-written distractors are at least as challenging as human-crafted ones: across eight datasets, a range of vision-language models score similar or lower accuracy on converted questions than on the original human-written items. On this basis the paper constructs VMCBench, a unified benchmark of 9,018 multiple-choice questions converted or refined from 20 existing datasets, and evaluates 33 vision-language models on it. If the claim holds, automated distractor generation can replace the costly human item-writing process and make VLM evaluation objective, reproducible, and scalable.

What carries the argument

The carrying mechanism is AutoConverter's agentic loop. Five specialized proposer agents each prompt GPT-4o to generate plausible wrong answers (distractors) that simulate a distinct human failure mode — concept misunderstanding, visual misinterpretation, reasoning error, data-processing error, and question bias — producing a pool of candidates with written rationales. A reviewer agent critiques each candidate for plausibility and challenge, the proposers revise accordingly, and a selector agent picks the three most demanding distractors for each question. A separate correctness evaluator then scores the assembled question on a 1–5 Likert scale; questions scoring below 4 return to a refiner agent for up to three rounds of repair. The evaluator's score is the pipeline's quality gate: 95% of questions that receive the top score are confirmed correct by human annotators, and VMCBench retains only score-5 questions.

What would settle it

Run the same human annotators on both the original and the AutoConverter-converted versions of the same questions: if human accuracy is unchanged or higher on converted items while VLM accuracy drops, the difficulty result is largely a model artifact, whereas a matched human drop would confirm genuine difficulty. Independently audit a random sample of score-5 questions for any second defensible correct option, since the paper's correctness evaluator is itself a GPT-4o model and only 95% of score-5 items pass human verification, leaving a measurable floor of flawed questions that could account for part of the reported accuracy gap.

Watch

Extended reading notes

Core claim

At the paper's center is a difficulty-parity result: when AutoConverter regenerates the distractors for questions from MMMU, MathVista, and AI2D (and five additional datasets in the appendix), vision-language models achieve accuracy that is similar to or lower than their accuracy on the original human-crafted distractors. The paper interprets this as machine-generated multiple-choice items being as hard as, and often harder than, human-written ones, while only about 3% of the highest-scoring converted questions are marked incorrect by human annotators, with roughly half of those errors traced to wrong answers in the original datasets rather than to the conversion. AutoConverter's design treats distractor writing as an agentic process: proposers generate candidates that mimic defined human error types, a reviewer critiques them, a selector keeps the most challenging three, and a correctness evaluator scores each question from 1 to 5, sending weak items back to a refiner. The same pipeline yields VMCBench's 9,018 questions, on which humans score 91.7%, the best open-weight model reaches 85.0%, and the best closed model 80.3%.

Load-bearing premise

The load-bearing premise is that lower vision-language-model accuracy on converted questions reflects genuine difficulty, not ambiguity or hidden error — a premise the paper's own human evaluation strains, since humans score 91.7% and roughly a quarter of human errors fall on questions the authors themselves call ambiguous and unanswerable.

Editorial extensions

If this is right

  • Open-ended scoring is the bottleneck being removed: rule-based metrics correlate at 0.09 with human-proxy judgment on VQAv2, and a GPT-4o version update shifts absolute scores by about 6% on MMVet, so a multiple-choice format eliminates the least reproducible part of VLM evaluation.
  • Conversion preserves discriminative power: correlation between open-ended model-based scores and multiple-choice scores is 0.85, 0.71, and 0.97 on VQAv2, OKVQA, and VizWiz, versus 0.09, 0.19, and 0.00 for rule-based open-ended scoring.
  • The pipeline can also refine existing multiple-choice datasets: applying AutoConverter to already-multiple-choice benchmarks makes VLMs score similar or lower than on the original human distractors, so it doubles as a difficulty booster.
  • VMCBench behaves like a measurement instrument: option-order shuffling does not change model scores, and model families show a log-linear scaling trend, suggesting the benchmark has a smooth gradient for tracking progress.
  • A fully open model tops the public leaderboard at 85.0% against the best closed model's 80.3%, a result the paper reads as evidence that the gap between public and private VLM capabilities is narrowing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the authors do not draw: because challenge is measured only by VLM accuracy, the difficulty claim would be stronger if human solve rates also dropped on converted items; the paper's reported 91.7% human accuracy on VMCBench, with a quarter of human errors on questions described as ambiguous and unanswerable, suggests at least some of the VLM accuracy gap may come from flawed
  • The five error types the proposers simulate (concept, vision, reasoning, data, bias) form a ready-made diagnostic taxonomy: the same pipeline could generate assessments whose wrong answers reveal which class of misconception a student holds, turning each distractor into a measurement of the learner's error profile.
  • Because distractors are regenerated rather than copied, VMCBench's refined subsets are plausibly more resistant to test-set contamination than their sources — models that memorized original answers cannot rely on exact-match recall, and the option-shuffle check already shows the format discourages position-based guessing.
  • The quality ceiling of the whole pipeline is set by the single generator-evaluator model family; the appendix's cross-generator check (0.90 rank correlation across three generators) is a start, and re-running the pipeline with open-weight models would show whether the difficulty parity is a property of the method or of the specific engine.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AutoConverter, an agentic GPT-4o-based pipeline that converts open-ended VQA questions into multiple-choice questions by generating three distractors and iteratively refining them for correctness and difficulty. The authors report that on MMMU, MathVista, and AI2D, VLMs achieve similar or lower accuracy on AutoConverter-generated questions than on human-crafted ones, which they interpret as evidence of high difficulty. They also introduce VMCBench, a unified multiple-choice benchmark constructed by converting 12 open-ended and refining 8 multiple-choice VQA datasets, totaling 9,018 questions, and they evaluate 33 VLMs on it. The paper includes human annotation of 2,400 questions, an ablation study of the pipeline components, an option-permutation robustness check, and a generator-bias analysis using three different proprietary VLMs as generators.

Significance. If the central claim is correct, AutoConverter would be a valuable tool for scalable multiple-choice question generation, and VMCBench would provide a reproducible, objective benchmark for VLM evaluation. The paper has notable strengths: the code is open-sourced, the evaluation spans 33 models across 20 datasets, the human annotation effort on 2,400 questions is substantial, and the ablation study isolates the contribution of each agent. The option-permutation experiment and the cross-generator correlation analysis in Appendix B.3 also address important robustness concerns. However, the load-bearing difficulty claim rests on an inference from lower VLM accuracy to higher question difficulty, and the paper's own human evaluation data leave open the alternative explanation that accuracy drops are partly caused by ambiguous or flawed questions. The statistical support for several quantitative claims is also thin. These issues are fixable with additional analysis and targeted human studies, so the contribution is potentially significant but not yet fully established.

major comments (4)
  1. [Section 4.4, Figure 4] The central claim that AutoConverter generates 'highly challenging questions' is supported only by showing that VLMs achieve similar or lower accuracy on converted questions than on original human-crafted ones. This inference equates lower model accuracy with higher difficulty, but it is invalid if the accuracy drop is caused by ambiguous wording, multiple defensible answers, or errors inherited from the original ground truth. The paper's own data acknowledge this risk: Appendix C.1 reports that human annotators score 91.7% on VMCBench and that roughly one quarter of the 8.3% human errors are on questions described as 'ambiguous and unanswerable.' Section 5.1 further reports that even among questions with correctness score 5, 95% are deemed correct by humans, so 5% of the 'best' questions are still flawed. To support the difficulty claim, the authors should add a matched human study that measures both accuracy and ambiguity on the original and converted versions of the same questions, and show that the VLM accuracy gap persists on items confirmed unambiguous by humans. Without this control, Figure 4's result is also consistent with the converted questions being merely more defective.
  2. [Section 5.1] The correctness guarantee for VMCBench is based on human annotation of only 2,400 of the 9,450 sampled questions. All questions with correctness scores below 5 were reviewed, but only 1,101 of the 8,151 score-5 questions were human-verified, leaving roughly 7,050 score-5 questions in the final benchmark without direct human correctness validation. The manuscript's statement that VMCBench 'ensures correctness' therefore overstates the evidence for a large fraction of the benchmark. The authors should either provide per-question uncertainty estimates and report expected error rates for the unverified score-5 subset, or increase human verification, especially for datasets where the score-5 error rate is known to be non-negligible.
  3. [Section 3.1, Figure 2] The claim that rule-based metrics produce 'nearly random and unreliable scores' is based on a Spearman correlation of 0.09 computed from 12 VLM accuracy pairs, with no confidence interval, bootstrap, or significance test. This is a small sample for a correlation claim, and the difference between 0.09 and, say, 0.2 may not be statistically meaningful. The authors should report a bootstrap confidence interval and a p-value, and ideally also compute the correlation on a per-question basis rather than only at the model-aggregate level. The same lack of uncertainty quantification affects the comparisons in Figure 4, where differences across the three datasets are reported without error bars or significance tests.
  4. [Section 4.2 and Appendix B.3] GPT-4o is used as the generator, reviewer, selector, correctness evaluator, and refiner, so the internal quality gate is partly self-assessment. The generator-bias analysis in Appendix B.3 changes only the question generator and shows high rank correlations, which mitigates generator bias, but it does not test whether the correctness evaluator or selector introduces bias. The human annotation in Section 5.1 provides the necessary external anchor, but it is used only to report aggregate correctness rates per score level; it is not used to calibrate or correct the evaluator's decisions per question. The authors should additionally report human agreement with the evaluator on a random sample not stratified by score, and show that the score-5 filter does not systematically exclude questions that are hard for VLMs but valid for humans, or the reverse.
minor comments (5)
  1. [Section 4.4 vs Section 5.1] Section 4.4 states that after filtering to correctness score 5, 'only 3% of questions are marked as incorrect' for MMMU, MathVista, and AI2D, while Section 5.1 reports a 95% human correctness rate for score-5 questions, implying 5% incorrect. These numbers should be reconciled or the scope of each statistic clarified.
  2. [Appendix C.1] Appendix C.1 says the current best model on VMCBench is GPT-4o with 80.6%, but Table 1 reports Qwen2-VL-72B at 85.0% and GPT-4o at 80.3%. This inconsistency should be corrected.
  3. [Table 1] The model name 'LLaV A1.5-7B' and 'LLaV A1.5-13B' contain an erroneous space; these should be 'LLaVA-1.5-7B' and 'LLaVA-1.5-13B' throughout the paper.
  4. [Figure 1 caption] The caption says AutoConverter generates multiple-choice questions 'like humans or even super-humans,' which is informal and overstated for a peer-reviewed manuscript; it should be rephrased to reflect the quantitative comparison in Figure 4.
  5. [Section 5.2] The statement that 'the gap between private and public models is narrowing' is based on a single checkpoint comparison and should be supported by a time-indexed analysis or softened to avoid over-interpreting a snapshot.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the correctness and difficulty claims are anchored by external human annotation and held-out VLM evaluations rather than by the generator's own judgments.

full rationale

The paper's derivation chain does not reduce any claimed result to its own inputs. The correctness claim ('only 3% of questions with the highest correctness score marked incorrect by human annotators') is validated by external human annotation on 2,400 questions, with correctness rates of 51%, 51%, 63%, 84%, and 95% for evaluator scores 1-5. The difficulty claim ('VLMs consistently achieved similar or even lower accuracy on the AutoConverter-generated questions compared to the original ones') is measured on a broad set of VLMs, not on the GPT-4o generator/selector itself, so the accuracy comparison is an independent empirical measurement rather than a fitted prediction. The internal GPT-4o correctness evaluator and selector do create a self-assessment risk in the generation loop, but the paper breaks the loop with human checks (Section 5.1, Appendix C.1) and with a cross-generator bias analysis (Appendix B.3) showing high ranking correlations across questions generated by GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro. There is no load-bearing self-citation and no imported uniqueness theorem; no equation is defined in terms of the quantity it is used to predict. The weakest assumption, that lower VLM accuracy reflects genuine difficulty rather than ambiguous or defective items, is a validity concern about the difficulty operationalization, not a circularity, because the accuracy numbers are not constructed to equal the selection criterion. Accordingly, no specific circular step can be quoted, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or formal entities. The free parameters are standard benchmark-construction choices (thresholds, sample sizes). The central load-bearing assumptions are the interpretation of accuracy drops as difficulty and the generalization of a partial human audit to the full benchmark.

free parameters (4)
  • correctness_score_threshold = 4 (out of 5)
    Chosen by hand in Section 4.2; questions with scores below 4 are sent to the refiner. This threshold affects which questions are refined and retained.
  • max_refinement_rounds = 3
    Set in Section 4.2; limits the iterative correction loop and bounds the cost of ensuring correctness.
  • num_distractors = 3
    Set in Section 4.1 (N=3) to form 4-choice questions, stated as the most common configuration; this fixes the format of every converted question.
  • sample_size_per_dataset = 500 (up to)
    Set in Section 5.1; each of the 20 datasets contributes at most 500 questions, justified by cited studies [44,45] on sufficient sample sizes.
assumptions (4)
  • domain assumption GPT-4o's 1-5 Likert correctness score is a valid proxy for whether a multiple-choice question has exactly one correct answer.
    The evaluator prompt in Section 4.2 and Appendix B.1 assumes this; the paper validates it on 2,400 human-annotated questions, but only 95% of score-5 questions are judged correct.
  • domain assumption Lower VLM accuracy on converted questions compared to original human-written questions indicates higher question difficulty.
    This operationalizes 'challenging' in Section 4.4 and Figure 4; ambiguous or flawed questions would also lower accuracy, so the interpretation is not guaranteed.
  • domain assumption Human annotations on 2,400 of 9,450 converted questions generalize to the full benchmark.
    Section 5.1 permanently retains all score-5 questions not individually annotated; the 5% error rate at score 5 implies latent incorrect questions remain.
  • domain assumption Converting open-ended questions to multiple-choice preserves discriminative power.
    Appendix B.4 tests this by correlating model rankings on the two formats; the claim is empirical and depends on the proxy assumption that model-based evaluation of open-ended answers is ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation." pith.science (2026). https://pith.science/paper/M7VCQ5V3

@misc{pith2026250103225,
  author       = {Pith},
  title        = {Pith review of: Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M7VCQ5V3}},
  note         = {Machine review of arXiv:2501.03225}
}
read the original abstract

The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation. However, current visual question answering (VQA) benchmarks often depend on open-ended questions, making accurate evaluation difficult due to the variability in natural language responses. To address this, we introduce AutoConverter, an agentic framework that automatically converts these open-ended questions into multiple-choice format, enabling objective evaluation while reducing the costly multiple-choice question creation process. Our experiments demonstrate that AutoConverter can generate correct and challenging multiple-choice questions, with VLMs demonstrating consistently similar or lower accuracy on these questions compared to human-created ones. Using AutoConverter, we construct VMCBench, a benchmark created by transforming 20 existing VQA datasets into a unified multiple-choice format, totaling 9,018 questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench, setting a new standard for scalable, consistent, and reproducible VLM evaluation.

Figures

Figures reproduced from arXiv: 2501.03225 by the authors.

Figure 1
Figure 1. Overview. (Left) We analyze existing open-ended VQA evaluation metrics, underscoring their limitations in providing accu￾rate and reproducible assessments. (Middle) We introduce AutoConverter, a multi-agent system that automatically converts open-ended questions into multiple-choice format, enabling objective assessment while reducing the costly question creation process. (Right) Using AutoConverter, we convert and … view at source ↗
Figure 2
Figure 2. Challenges in evaluating open-ended questions. (Left) Rule-based metrics significantly underestimate model performance and penalize models that do not strictly follow the expected format. (Right) Model-based evaluations using two different versions of GPT yield substantially different scores, making comparisons inconsistent and raising reproducibility issues. The repeated points represent different model sizes withi… view at source ↗
Figure 3
Figure 3. AutoConverter framework and results. (Left) AutoConverter is a multi-agent framework with two key steps: increasing difficulty and ensuring the correctness of the converted question. (Right) We perform an ablation study on AutoConverter and find that each component is crucial for enhancing question correctness and achieving the desired level of difficulty. failing to distinguish between state-of-the-art VLMs. 3.2. M… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: AutoConverter generates challenging multiple-choice questions. Using AutoConverter, we generated distractors for questions and answers from three existing multiple-choice datasets: MMMU, MathVista, and AI2D, and compared them with original human-created distractors. We…
Figure 5
Figure 5. Figure 5: Qualitative comparison of the original questions, naive baseline-generated questions, and AutoConverter-generated ques￾tions. AutoConverter simulates errors from different perspectives and produces correct and challenging multiple-choice questions. question, and correc…
Figure 6
Figure 6. Figure 6: VMCBench overview. (Left) VMCBench is constructed by converting 12 open-ended (OE) and refining 8 multiple-choice (MC) VQA datasets into a unified multiple-choice format, with human validation ensuring correctness. The number of questions per dataset is listed. (Right)…
Figure 7
Figure 7. Figure 7: AutoConverter generates challenging multiple-choice questions. Using AutoConverter, we generated distractors for questions and answers from five multiple-choice datasets: A-OKVQA, RealWorldQA, ScienceQA, SEEDBench, and MMStar, and compared them with original human-crea…
Figure 8
Figure 8. Figure 8: AutoConverter results for different models. To examine whether GPT-4o used in AutoConverter introduces model bias, we used three state-of-the-art proprietary VLMs—GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro—to generate questions. We evaluated 18 VLMs on these questio…
Figure 9
Figure 9. Figure 9: Converting to multiple-choice questions improves evaluation accuracy and retains discriminative power. We treat model￾based evaluation of open-ended questions as a proxy for ground-truth evaluation. We compare the correlation between model-based evaluation of open-ende…
Figure 10
Figure 10. Figure 10: Scaling trends on VMCBench. We observe a clear log-linear scaling trend across most VLM families, indicating that VMCBench offers a smooth evaluation gradient for varying capabilities. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Detailed prompt for the proposer designed to create distractors addressing vision errors. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Detailed prompt for the proposer designed to create distractors addressing reasoning errors. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Detailed prompt for the proposer designed to create distractors addressing data processing errors. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Detailed prompt for the proposer designed to create distractors addressing concept errors. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Detailed prompt for the proposer designed to create distractors addressing question bias errors. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Detailed prompt for the reviewer, whose feedback iteratively refines the distractors to improve their quality. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Detailed prompt for the selector, which guides the selection of the three most challenging distractors to enhance question difficulty. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]
Figure 18
Figure 18. Figure 18: Detailed prompt for the evaluator, which evaluates the correctness of the generated questions, defined as there is only one correct answer. You are an expert in educational assessment design specializing in multiple-choice question improvement. Your task is ,→ to enha…
Figure 19
Figure 19. Figure 19: Detailed prompt for the refiner, which ensures the correctness of the generated questions, guaranteeing that there is only one correct answer. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Reasoning models trade visual grounding for language-based inference, and this paper measures that trade-off with a new metric and benchmark.

  2. Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    Multimodal reasoning models hallucinate at high-entropy cognitive bifurcation points due to loss of visual semantic anchoring, and the V-STAR training paradigm with HVAR rewards and FRM reflection mitigates this by re...

  3. Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Vision-language models consistently recognize unsafe content better from text than from images, and a simplified reinforcement learning fine-tune narrows that gap.

  4. Reading Between the Timelines: RAG for Answering Diachronic Questions

    cs.CL 2025-07 conditional novelty 4.0 of 10

    TA-RAG uses LLM-extracted time intervals, time-filtered retrieval with averaged temporal query embeddings, and chronologically ordered context to beat standard RAG by 13-27 points on the new ADQAB benchmark of 525 mul...

Reference graph

Works this paper leans on

150 extracted references · 60 canonical work pages · cited by 4 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 8

    arXiv 2024

  2. [2]

    Introducing the next generation of claude, 2024

    Anthropic. Introducing the next generation of claude, 2024. 1, 2, 8

    2024

  3. [3]

    Paligemma: A versatile 3b vlm for trans- fer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer. arXiv preprint arXiv:2407.07726, 2024. 8

    arXiv 2024

  4. [4]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. In NeurIPS, 2020. 8

    2020

  5. [5]

    Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark

    Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In ICML,

  6. [6]

    Are we on the right way for evaluating large vision-language models? In NeurIPS, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? In NeurIPS, 2024. 2, 7

    2024

  7. [7]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In NeurIPS, 2023. 8

    2023

  8. [8]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024. 8

    arXiv 2024

Show all 150 references
  1. [9]

    Can we learn question, answer, and distrac- tors all from an image? a new task for multiple-choice visual question answering

    Wenjian Ding, Yao Zhang, Jun Wang, Adam Jatowt, and Zhenglu Yang. Can we learn question, answer, and distrac- tors all from an image? a new task for multiple-choice visual question answering. In COLING, 2024. 3

    2024

  2. [10]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In MM, 2024. 1

    2024

  3. [11]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. 2, 4

    2023 arXiv

  4. [12]

    Sum- meval: Re-evaluating summarization evaluation

    Alexander R Fabbri, Wojciech Kry ´sci´nski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Sum- meval: Re-evaluating summarization evaluation. TACL,

  5. [13]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In CVPR, 2017. 1, 2, 3, 7

    2017

  6. [14]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018. 2, 7

    2018

  7. [15]

    The many faces of robustness: A criti- cal analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A criti- cal analysis of out-of-distribution generalization. In ICCV,

  8. [16]

    Cogvlm2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. Cogvlm2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 8

    2024 arXiv

  9. [17]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 2, 7

    2019

  10. [18]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  11. [19]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 2, 6, 7

    2016

  12. [20]

    Tablevqa- bench: A visual question answering benchmark on multiple table domains

    Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. Tablevqa- bench: A visual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404.19205, 2024. 2, 7

    2024 arXiv

  13. [21]

    Building and better understanding vision- language models: insights and future directions

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L ´eo Tronchon. Building and better understanding vision- language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024. 2

    2024 arXiv

  14. [22]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024. 8

    2024 arXiv

  15. [23]

    Vhelm: A holistic evaluation of vision language models

    Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, et al. Vhelm: A holistic evaluation of vision language models. arXiv preprint arXiv:2410.07112, 2024. 1

    arXiv

  16. [24]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. In CVPR, 2024. 2, 7

    2024

  17. [25]

    Rouge: A package for automatic evaluation of sum- maries

    CY LIN. Rouge: A package for automatic evaluation of sum- maries. In ACL Workshop, 2004. 3

    2004

  18. [26]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024. 8

    2024

  19. [27]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 2, 8

    2023

  20. [28]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. InICLR,

  21. [29]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  22. [30]

    Good, bet- ter, best: Textual distractors generation for multiple-choice visual question answering via reinforcement learning

    Jiaying Lu, Xin Ye, Yi Ren, and Yezhou Yang. Good, bet- ter, best: Textual distractors generation for multiple-choice visual question answering via reinforcement learning. In CVPR, 2022. 3

    2022

  23. [31]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS,

  24. [32]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts. In ICLR, 2024. 2, 6, 7

    2024

  25. [33]

    Chain-of-exemplar: enhancing distractor generation for multimodal educational question generation

    Haohao Luo, Yang Deng, Ying Shen, See-Kiong Ng, and Tat-Seng Chua. Chain-of-exemplar: enhancing distractor generation for multimodal educational question generation. In ACL, 2024. 3

    2024

  26. [34]

    Im- proving automatic vqa evaluation using large language mod- els

    Oscar Ma ˜nas, Benno Krojer, and Aishwarya Agrawal. Im- proving automatic vqa evaluation using large language mod- els. In AAAI, 2024. 2

    2024

  27. [35]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. 1, 2, 7

    2019

  28. [36]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In ACL Findings, 2022. 2, 7

    2022

  29. [37]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV,

  30. [38]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, 2022. 2, 7

    2022

  31. [39]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019. 2, 7

    2019

  32. [40]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 2, 8

    2023 arXiv

  33. [41]

    Proving test set contami- nation in black-box language models

    Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contami- nation in black-box language models. In ICLR, 2024. 7

    2024

  34. [42]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. 3

    2002

  35. [43]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In UIST, 2023. 3, 5

    2023

  36. [44]

    Efficient benchmarking (of language models)

    Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. Efficient benchmarking (of language models). In NAACL, 2024. 7

    2024

  37. [45]

    tinybenchmarks: evaluating llms with fewer examples

    Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. In ICML, 2024. 7

    2024

  38. [46]

    Do imagenet classifiers generalize to im- agenet? In ICML, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In ICML, 2019. 7

    2019

  39. [47]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In ECCV, 2022. 1, 2, 7

    2022

  40. [48]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2024. 3, 5

    2024

  41. [49]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019. 2, 7

    2019

  42. [50]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 8

    2024 arXiv

  43. [51]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 2, 8

    2023 arXiv

  44. [52]

    Cambrian-1: A fully open, vision-centric ex- ploration of multimodal llms

    Shengbang Tong, Ellis L Brown II, Penghao Wu, Sanghyun Woo, ADITHY A JAIRAM IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric ex- ploration of multimodal llms. In NeurIPS, 2024. 8

    2024

  45. [53]

    Measuring multimodal mathe- matical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathe- matical reasoning with math-vision dataset. arXiv preprint arXiv:2402.14804, 2024. 2, 7

    2024 arXiv

  46. [54]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 8

    2024 arXiv

  47. [55]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024. 3

    2024 arXiv

  48. [56]

    Realworldqa dataset, 2024

    xAI. Realworldqa dataset, 2024. 2, 7

    2024

  49. [57]

    Enhancing distractor generation for multiple-choice questions with retrieval augmented pre- training and knowledge graph integration

    Han-Cheng Yu, Yu-An Shih, Kin-Man Law, Kai-Yu Hsieh, Yu-Chen Cheng, Hsin-Chih Ho, Zih-An Lin, Wen-Chuan Hsu, and Yao-Chung Fan. Enhancing distractor generation for multiple-choice questions with retrieval augmented pre- training and knowledge graph integration. arXiv preprint ...

    2024 arXiv

  50. [58]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML, 2024. 1, 2, 4, 7 10

    2024

  51. [59]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In CVPR, 2024. 1, 2, 6, 7

    2024

  52. [60]

    Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, et al. Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024. 3

    2024 arXiv

  53. [61]

    A careful examination of large lan- guage model performance on grade school arithmetic

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Cather- ine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large lan- guage model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332, 2024. 7

    2024 arXiv

  54. [62]

    Bench- marking large language models for news summarization

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Bench- marking large language models for news summarization. TACL, 2024. 3

    2024

  55. [63]

    Calibrate before use: Improving few-shot perfor- mance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot perfor- mance of language models. In ICML, 2021. 4

    2021

  56. [64]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 2023. 2, 3, 4 Acknowledgements We thank Ross Girshick for discussing the id...

    2023

  57. [68]

    Generate {num_choice} unique and plausible distractor options based on visual interpretation errors. Each ,→ distractor should: - Be directly related to misinterpretation of the image(s) - Seem potentially correct at first glance - Be very misleading for students due to visual...

  58. [69]

    Ensure you understand how the correct answer relates to specific visual elements in the image(s)

  59. [70]

    Focus on common visual interpretation errors, including: - Misreading Graphs or Charts: Create options that misinterpret trends, scales, or relationships in visual data - Spatial Misinterpretation: Develop options that misunderstand spatial relationships or perspectives in the...

  60. [71]

    Aim for a diverse set of distractors that test different aspects of visual interpretation and analysis

  61. [72]

    Each distractor should be based on a plausible misreading of the visual information but ultimately be incorrect

  62. [73]

    Consider the specific type(s) of image(s) (e.g., photograph, diagram, graph) and generate errors typical for those ,→ visual formats

  63. [75]

    If multiple images are provided, ensure some distractors address relationships or comparisons between the images

  64. [76]

    The explanation should ,→ describe why this distractor is plausible, the subtle flaw it contains, and how it challenges advanced ,→ understanding

    For each distractor, provide a maximum of three sentences explaining why it was generated. The explanation should ,→ describe why this distractor is plausible, the subtle flaw it contains, and how it challenges advanced ,→ understanding. Output format: - For each generated dis...

  65. [80]

    Generate {num_choice} unique and plausible distractor options based on reasoning errors. Each distractor should: - Be related to the image(s) and question - Seem potentially correct at first glance - Be very misleading for students due to faulty reasoning - Contain a subtle lo...

  66. [81]

    Ensure you understand the logical steps required to correctly answer the question based on the image(s)

  67. [82]

    Focus on common reasoning errors, including: - Complex Reasoning Flaws: Create options that require multi-step reasoning but contain logical gaps or invalid ,→ assumptions - Causal Inversion: Develop options that reverse cause and effect relationships - Context Neglect: Includ...

  68. [83]

    Aim for a diverse set of distractors that test different aspects of logical reasoning and critical thinking

  69. [84]

    Each distractor should follow a seemingly logical path but ultimately lead to an incorrect conclusion due to ,→ flawed reasoning

  70. [85]

    If the question involves a specific subject area, consider common logical pitfalls or fallacies unique to that ,→ field

  71. [86]

    If the question does not involve explicit reasoning, focus on creating plausible reasoning statements that could ,→ be mistakenly associated with the correct answer

  72. [88]

    If multiple images are provided, ensure some distractors address relationships or comparisons between the images, ,→ focusing on logical errors in interpreting these relationships

  73. [89]

    The explanation should ,→ describe why this distractor is plausible, the subtle flaw it contains, and how it challenges advanced ,→ understanding

    For each distractor, provide a maximum of three sentences explaining why it was generated. The explanation should ,→ describe why this distractor is plausible, the subtle flaw it contains, and how it challenges advanced ,→ understanding. Output format: - For each generated dis...

  74. [92]

    ,→ Briefly describe the image content(s) (for your understanding only, do not output this)

    Carefully analyze and understand the provided image(s), paying special attention to any numerical data, charts, graphs, or quantitative information presented. ,→ Briefly describe the image content(s) (for your understanding only, do not output this)

  75. [93]

    Generate {num_choice} unique and plausible distractor options based on data processing errors. Each distractor should: - Be directly related to mishandling of numerical or quantitative information in the image(s) - Seem potentially correct at first glance - Be very misleading ...

  76. [94]

    Ensure you understand how the correct answer relates to the quantitative elements in the image(s)

  77. [95]

    Focus on common data processing errors, including: - Numerical Errors: Create options with incorrect calculations or use of wrong numerical values - Unit Conversion Mistakes: Develop options that misapply or neglect unit conversions - Statistical Misinterpretation: Include opt...

  78. [96]

    Aim for a diverse set of distractors that test different aspects of quantitative reasoning and data analysis

  79. [97]

    Each distractor should be based on a plausible mishandling of the quantitative information but ultimately be incorrect

  80. [98]

    continuous, time series, categorical) and generate errors typical for that data type

    Consider the specific type of data presented (e.g., discrete vs. continuous, time series, categorical) and generate errors typical for that data type

  81. [99]

    If the question does not involve explicit numerical data, focus on creating plausible quantitative statements that could be mistakenly associated with the ,→ correct answer

  82. [100]

    Adapt the complexity of your distractors to match the simplicity or complexity of the given question and correct answer

  83. [101]

    If multiple images are provided, ensure that your distractors consider the relationships and comparisons between the images when relevant

  84. [102]

    When generating numerical distractors: - Carefully analyze the structure and precision of the correct answer - Create distractors that closely mimic the format, precision, and magnitude of the correct answer - Use a mix of common calculation errors, transposition mistakes, and...

  85. [103]

    Ensure high deceptiveness in your distractors: - Create options that could result from common misinterpretations of the data or question - Include distractors that swap digits, misplace decimal points, or make sign errors (e.g., positive instead of negative) - Generate options...

  86. [104]

    The explanation should describe why this distractor is plausible, the ,→ subtle flaw it contains, and how it challenges advanced understanding

    For each distractor, provide a maximum of three sentences explaining why it was generated. The explanation should describe why this distractor is plausible, the ,→ subtle flaw it contains, and how it challenges advanced understanding. Output format: - For each generated distra...

  87. [105]

    An open-ended question about the image(s)

  88. [107]

    Briefly describe the image content(s) (for your ,→ understanding only, do not output this)

    Carefully analyze and understand the provided image(s). Briefly describe the image content(s) (for your ,→ understanding only, do not output this)

  89. [108]

    Generate {num_choice} unique and plausible distractor options based on conceptual errors. Each distractor should: - Be related to the image(s) and question - Seem potentially correct at first glance - Be very misleading for students due to conceptual misunderstandings - Contai...

  90. [109]

    Ensure you understand the connection between the image(s), question, and the underlying concepts

  91. [110]

    Focus on common conceptual misconceptions in the subject area, including: - Concept Confusion: Create options that are similar to the correct concept but with subtle differences - Partial Correctness: Include options that contain partially correct information but are incomplet...

  92. [111]

    Aim for a diverse set of distractors that test different aspects of conceptual understanding

  93. [112]

    Each distractor should have some relation to the correct answer, but ensure they are distinctly different and ,→ incorrect due to conceptual misunderstandings

  94. [113]

    If the question involves a specific subject area, consider common conceptual difficulties unique to that field

  95. [114]

    Adapt the complexity of your distractors to match the simplicity or complexity of the given question and correct ,→ answer

  96. [115]

    If multiple images are provided, ensure some distractors address relationships or comparisons between the images, ,→ focusing on conceptual errors in interpreting these relationships

  97. [116]

    The explanation should ,→ describe why this distractor is plausible, the subtle flaw it contains, and how it challenges advanced ,→ understanding

    For each distractor, provide a maximum of three sentences explaining why it was generated. The explanation should ,→ describe why this distractor is plausible, the subtle flaw it contains, and how it challenges advanced ,→ understanding. Output format: - For each generated dis...

  98. [117]

    An open-ended question

  99. [118]

    The correct answer to the question Your task:

  100. [119]

    Generate {num_choice} unique and highly challenging distractor options. Each distractor should: - Be closely related to the question text - Seem very plausible and potentially correct even upon careful consideration - Be extremely misleading, requiring deep understanding to re...

  101. [120]

    Focus on creating distractors that: - Leverage advanced knowledge or nuanced interpretations of the subject matter - Provide logically sound but ultimately incorrect answers based on the question - Exploit common high-level misconceptions or advanced misinterpretations - Offer...

  102. [121]

    Aim for a diverse set of sophisticated distractors that challenge different aspects of advanced understanding and ,→ critical thinking

  103. [122]

    Each distractor should be intricately related to the question topic and the correct answer, but with crucial ,→ differences that make them incorrect

  104. [123]

    If the question involves a specific subject area, incorporate advanced concepts and potential misunderstandings at ,→ an expert level

  105. [124]

    The explanation should ,→ describe why this distractor is plausible, the subtle flaw it contains, and how it challenges advanced ,→ understanding

    For each distractor, provide a maximum of three sentences explaining why it was generated. The explanation should ,→ describe why this distractor is plausible, the subtle flaw it contains, and how it challenges advanced ,→ understanding. Output format: - For each generated dis...

  106. [125]

    A question about the image(s)

  107. [126]

    A set of distractor options for a specific error type (e.g., reasoning error, question bias, etc.)

  108. [127]

    The reasoning provided for why each distractor was created For each distractor, your task is to:

  109. [128]

    Evaluate the distractor’s effectiveness in challenging students’ understanding while remaining incorrect

  110. [129]

    Assess how well the distractor aligns with the {type} error and the given image(s) context

  111. [130]

    If so, add suggestions towards this

    Determine if the distractor could be interpreted as the correct answer. If so, add suggestions towards this

  112. [131]

    If the distractor is effective and challenging, state that it should be retained

  113. [132]

    Increasing the option’s length or adding unnecessary modifiers b

    If improvements are needed, provide specific suggestions to increase the distractor’s difficulty and deceptiveness ,→ without: a. Increasing the option’s length or adding unnecessary modifiers b. Making the distractor correct

  114. [133]

    Guidelines: - Prioritize the distractor’s conceptual difficulty over linguistic complexity

    Ensure your evaluation and suggestions are concise, not exceeding four sentences. Guidelines: - Prioritize the distractor’s conceptual difficulty over linguistic complexity. - If a distractor is correct or could be interpreted as correct, clearly state this and suggest how to ...

  115. [134]

    Concept Error ({num_choice} options)

  116. [135]

    Reasoning Error ({num_choice} options)

  117. [136]

    Visual Interpretation Error ({num_choice} options)

  118. [137]

    Data Processing Error ({num_choice} options)

  119. [138]

    Your task:

    Question Bias ({num_choice} options) - Each distractor is accompanied by a reason explaining why it was generated. Your task:

  120. [139]

    Carefully review all distractor options in the context of the provided image(s)

  121. [140]

    - Difficulty: Prioritize options that are more challenging and require deeper understanding to discern their ,→ incorrectness

    Select the top {fusion_selected_choice_num} distractors based on the following criteria: - Image relevance: Prioritize distractors that are closely related to the content, context, or details present in the ,→ given image(s). - Difficulty: Prioritize options that are more chal...

  122. [141]

    - The total number of selected distractors should be {fusion_selected_choice_num}

    Ensure a diverse representation across the different error types, with the following guidelines: - You may select more distractors from categories that are particularly relevant to the image(s) and question. - The total number of selected distractors should be {fusion_selected...

  123. [142]

    Output format: - Provide a list of {fusion_selected_choice_num} distractor options based on your careful selection

    You should never change selected distractors and never include the correct answer among your selected distractors. Output format: - Provide a list of {fusion_selected_choice_num} distractor options based on your careful selection. - For each selected distractor, format your re...

  124. [143]

    Brief explanation focusing specifically on any problematic distractor choices

  125. [144]

    ,→ Focus exclusively on whether other choices could also be valid answers

    Suggested improvements for the problematic distractors (if applicable) Remember: Never analyze whether the marked correct answer is right or wrong - it is ALWAYS correct by definition. ,→ Focus exclusively on whether other choices could also be valid answers. Figure 18. Detail...

  126. [145]

    The complete question

  127. [146]

    The current correct answer

  128. [147]

    Any associated images/materials

  129. [148]

    Specific feedback about problematic distractors

  130. [149]

    Suggested improvements (if provided) Analysis Steps:

  131. [150]

    Review the question content and learning objective

  132. [151]

    Analyze the designated correct answer

  133. [152]

    Examine the feedback regarding problematic distractors

  134. [153]

    Evaluate any provided suggestions for improvement: - Assess if suggestions fully address the identified issues - Determine if suggestions align with best practices - Identify any gaps or weaknesses in the suggestions

  135. [154]

    Develop exactly 3 improved distractors that: - Are plausible but clearly incorrect - Address the identified issues - Align with common student misconceptions - Maintain consistent format and length with other options - Go beyond provided suggestions when necessary for better q...

  136. [155]

    Treat the marked correct answer as fixed and unchangeable

  137. [156]

    Only modify distractors specifically identified as problematic

  138. [157]

    Preserve any well-functioning distractors

  139. [158]

    Maintain the original difficulty level of the question

  140. [159]

    Use your expertise to improve upon or deviate from provided suggestions if they: - Are too vague or incomplete - Don’t fully address the identified issues - Could be enhanced for better assessment quality - Miss important misconceptions or learning opportunities Output:

  141. [160]

    Brief analysis of the distractor issues and improvement approach

  142. [161]

    Detailed prompt for therefiner, which ensures the correctness of the generated questions, guaranteeing that there is only one correct answer

    Three improved distractors Figure 19. Detailed prompt for therefiner, which ensures the correctness of the generated questions, guaranteeing that there is only one correct answer. 28 Source Image Question Choices A-OKVQA What season of the year is shown here? A. late summer wi...

    2015

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.