{"id":"97d576af-7cac-4585-a3d3-9e0f908ed3f0","arxiv_id":"2501.03225","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AutoConverter converts open-ended VQA questions into multiple-choice format via multi-agent GPT-4o, and VMCBench applies it to 20 datasets to evaluate 33 vision-language models.","lead":"AutoConverter is an AI system that turns open-ended visual questions into multiple-choice format, and the authors use it to build VMCBench, a 9,018-question benchmark for vision-language models. The work aims to make VLM evaluation more objective and reproducible, but its claim that generated questions are as hard as human-written ones is only partially supported.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lower VLM accuracy on converted questions is read as \"challenging,\" but the paper never rules out that the drop comes from ambiguous or flawed distractors; a matched human difficulty/ambiguity study is needed.","rationale":"The reader identified the same load-bearing assumption: lower VLM accuracy is treated as evidence of genuine difficulty, while the paper's own human evaluation shows that some questions are ambiguous or unanswerable. I agree with that assessment and with the CONDITIONAL verdict. The concern is not that the pipeline is useless; the paper provides real supporting evidence, including open-source code, 2,400 question-level human correctness annotations, an ablation showing each agent contributes, and an option-permutation robustness check on VMCBench. However, these checks do not directly validate the specific claim that converted questions are challenging rather than flawed. The 3% human-flagged error rate among correctness-score-5 questions and the roughly 2% ambiguous/unanswerable questions on VMCBench show that the generation process is good but not clean, and Figure 4 does not separate difficulty from defect. No confidence intervals are reported for the accuracy comparisons, so part of the claimed difference may also be statistical noise. A matched human accuracy and ambiguity study would settle the question and should be a condition of acceptance. Because the reader already recommended exactly this kind of validation, my stress-test does not move the verdict; it strengthens the reason for keeping it conditional.","tokens_in":30396,"tokens_out":3649,"duration_ms":37562,"concrete_test":"Take a stratified random sample of, for example, 200 questions per dataset from the MMMU, MathVista, and AI2D comparisons in Figure 4, and prepare both the original and AutoConverter versions of each item. Have at least three expert annotators per item complete two tasks without knowing which version is converted: (1) select the correct answer, and (2) flag any question that is ambiguous, has more than one defensible answer, or is unanswerable from the image. Compare human accuracy and ambiguity-flag rates between original and converted versions, with confidence intervals. If human accuracy on converted items is equal or lower and the ambiguity rate is not higher, the difficulty interpretation holds; if converted items show more ambiguity or multiple-answer flags, the accuracy drop is at least partly an artifact and the claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.4, Figure 4, equates lower VLM accuracy on AutoConverter-generated questions with higher difficulty. This inference is only valid if the accuracy gap is not caused by defective items: ambiguous wording, multiple defensible answers, or errors inherited from the original ground truth. The paper's own correctness evidence leaves this open. After filtering to correctness score 5, humans still mark 3% of converted questions as incorrect, and 52% of those errors trace to wrong original answers. Appendix C.1 reports that roughly one quarter of the 8.3% human errors on VMCBench are \"ambiguous and unanswerable\" questions. Neither number is connected to the original-vs-converted comparisons in Figure 4, and no human difficulty measurement is made on the same questions. Thus the headline \"challenging\" claim is supported by a proxy that could also be produced by flawed items. The missing control is a matched human study that measures both accuracy and ambiguity on original and converted versions of the same questions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoConverter, an agentic GPT-4o-based pipeline that converts open-ended VQA questions into multiple-choice questions by generating three distractors and iteratively refining them for correctness and difficulty. The authors report that on MMMU, MathVista, and AI2D, VLMs achieve similar or lower accuracy on AutoConverter-generated questions than on human-crafted ones, which they interpret as evidence of high difficulty. They also introduce VMCBench, a unified multiple-choice benchmark constructed by converting 12 open-ended and refining 8 multiple-choice VQA datasets, totaling 9,018 questions, and they evaluate 33 VLMs on it. The paper includes human annotation of 2,400 questions, an ablation study of the pipeline components, an option-permutation robustness check, and a generator-bias analysis using three different proprietary VLMs as generators.","tokens_in":30703,"tokens_out":4094,"duration_ms":40312,"significance":"If the central claim is correct, AutoConverter would be a valuable tool for scalable multiple-choice question generation, and VMCBench would provide a reproducible, objective benchmark for VLM evaluation. The paper has notable strengths: the code is open-sourced, the evaluation spans 33 models across 20 datasets, the human annotation effort on 2,400 questions is substantial, and the ablation study isolates the contribution of each agent. The option-permutation experiment and the cross-generator correlation analysis in Appendix B.3 also address important robustness concerns. However, the load-bearing difficulty claim rests on an inference from lower VLM accuracy to higher question difficulty, and the paper's own human evaluation data leave open the alternative explanation that accuracy drops are partly caused by ambiguous or flawed questions. The statistical support for several quantitative claims is also thin. These issues are fixable with additional analysis and targeted human studies, so the contribution is potentially significant but not yet fully established.","major_comments":[{"comment":"The central claim that AutoConverter generates 'highly challenging questions' is supported only by showing that VLMs achieve similar or lower accuracy on converted questions than on original human-crafted ones. This inference equates lower model accuracy with higher difficulty, but it is invalid if the accuracy drop is caused by ambiguous wording, multiple defensible answers, or errors inherited from the original ground truth. The paper's own data acknowledge this risk: Appendix C.1 reports that human annotators score 91.7% on VMCBench and that roughly one quarter of the 8.3% human errors are on questions described as 'ambiguous and unanswerable.' Section 5.1 further reports that even among questions with correctness score 5, 95% are deemed correct by humans, so 5% of the 'best' questions are still flawed. To support the difficulty claim, the authors should add a matched human study that measures both accuracy and ambiguity on the original and converted versions of the same questions, and show that the VLM accuracy gap persists on items confirmed unambiguous by humans. Without this control, Figure 4's result is also consistent with the converted questions being merely more defective.","section":"Section 4.4, Figure 4"},{"comment":"The correctness guarantee for VMCBench is based on human annotation of only 2,400 of the 9,450 sampled questions. All questions with correctness scores below 5 were reviewed, but only 1,101 of the 8,151 score-5 questions were human-verified, leaving roughly 7,050 score-5 questions in the final benchmark without direct human correctness validation. The manuscript's statement that VMCBench 'ensures correctness' therefore overstates the evidence for a large fraction of the benchmark. The authors should either provide per-question uncertainty estimates and report expected error rates for the unverified score-5 subset, or increase human verification, especially for datasets where the score-5 error rate is known to be non-negligible.","section":"Section 5.1"},{"comment":"The claim that rule-based metrics produce 'nearly random and unreliable scores' is based on a Spearman correlation of 0.09 computed from 12 VLM accuracy pairs, with no confidence interval, bootstrap, or significance test. This is a small sample for a correlation claim, and the difference between 0.09 and, say, 0.2 may not be statistically meaningful. The authors should report a bootstrap confidence interval and a p-value, and ideally also compute the correlation on a per-question basis rather than only at the model-aggregate level. The same lack of uncertainty quantification affects the comparisons in Figure 4, where differences across the three datasets are reported without error bars or significance tests.","section":"Section 3.1, Figure 2"},{"comment":"GPT-4o is used as the generator, reviewer, selector, correctness evaluator, and refiner, so the internal quality gate is partly self-assessment. The generator-bias analysis in Appendix B.3 changes only the question generator and shows high rank correlations, which mitigates generator bias, but it does not test whether the correctness evaluator or selector introduces bias. The human annotation in Section 5.1 provides the necessary external anchor, but it is used only to report aggregate correctness rates per score level; it is not used to calibrate or correct the evaluator's decisions per question. The authors should additionally report human agreement with the evaluator on a random sample not stratified by score, and show that the score-5 filter does not systematically exclude questions that are hard for VLMs but valid for humans, or the reverse.","section":"Section 4.2 and Appendix B.3"}],"minor_comments":[{"comment":"Section 4.4 states that after filtering to correctness score 5, 'only 3% of questions are marked as incorrect' for MMMU, MathVista, and AI2D, while Section 5.1 reports a 95% human correctness rate for score-5 questions, implying 5% incorrect. These numbers should be reconciled or the scope of each statistic clarified.","section":"Section 4.4 vs Section 5.1"},{"comment":"Appendix C.1 says the current best model on VMCBench is GPT-4o with 80.6%, but Table 1 reports Qwen2-VL-72B at 85.0% and GPT-4o at 80.3%. This inconsistency should be corrected.","section":"Appendix C.1"},{"comment":"The model name 'LLaV A1.5-7B' and 'LLaV A1.5-13B' contain an erroneous space; these should be 'LLaVA-1.5-7B' and 'LLaVA-1.5-13B' throughout the paper.","section":"Table 1"},{"comment":"The caption says AutoConverter generates multiple-choice questions 'like humans or even super-humans,' which is informal and overstated for a peer-reviewed manuscript; it should be rephrased to reflect the quantitative comparison in Figure 4.","section":"Figure 1 caption"},{"comment":"The statement that 'the gap between private and public models is narrowing' is based on a single checkpoint comparison and should be supported by a time-indexed analysis or softened to avoid over-interpreting a snapshot.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the venue and the contribution is potentially useful. The main risk is that the difficulty claim is not yet separated from the ambiguity-flaw explanation; the authors' own human data in Appendix C.1 make this risk concrete. The recommended major revision should require the matched human difficulty/ambiguity study and the statistical uncertainty analysis. I would not reject, because the core methodology and benchmark are reproducible and the human annotation effort is substantial, but the current wording overstates what the evidence shows."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: VMCBench is a real contribution and the AutoConverter pipeline is solid, reproducible engineering. But the claim that the generated questions are 'challenging' in the proper sense is not fully supported—the paper shows VLMs do worse, not that the questions are harder rather than sloppier. That's the main thing to know.\n\nWhat's genuinely new: a 9,018-question unified multiple-choice benchmark across 20 VQA datasets, with 33 models scored and a public leaderboard/code. The multi-agent generation loop—five error-type proposers, a reviewer, a selector, a correctness evaluator and refiner—is more than a prompt hack; the ablations show each component contributes. The human annotation on 2,400 questions is real evidence: the model's correctness score tracks human judgment well (from 51% correct at score 1 to 95% at score 5), and the error attribution is honest about how many errors come from the original datasets. The three-generator rank-correlation check (0.90 Spearman) is a good answer to the 'it's all GPT-4o' worry.\n\nNow the soft spots. The difficulty claim in Section 4.4 uses lower VLM accuracy on converted questions as evidence of higher difficulty. That only works if the drop isn't caused by ambiguous or broken items. The paper's own human data suggest some of that is happening: 3% of the highest-correctness questions are still wrong, and a quarter of human errors on VMCBench are 'ambiguous and unanswerable.' Nobody measured difficulty directly on the same questions for original vs converted versions, so the headline 'super-human' difficulty is over-claimed. The 0.09 correlation in Section 3 is also on 12 models with no error bars; it's suggestive, not definitive. The GPT-4o-evaluates-GPT-4o loop is a real concern, though the human calibration and the three-generator check soften it.\n\nWho should read this: anyone working on VLM evaluation who wants a cheap, objective, standardized alternative to open-ended scoring with API judges. It deserves a serious referee. I'd ask for confidence intervals, a matched human difficulty/ambiguity study on original vs converted questions, and a documented error audit of the final 9,018 questions. With those, VMCBench would be a solid benchmark; without them, the 'challenging' tag should be treated as provisional.","headline":"Useful benchmark, solid engineering, but the 'challenging' claim needs a matched human-difficulty control before it earns the headline.","tokens_in":31164,"tokens_out":3497,"would_cite":true,"duration_ms":30969,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoConverter, a multi-agent GPT-4o pipeline, converts open-ended visual questions into multiple-choice items with machine-written distractors at least as challenging as human-crafted ones, yielding the 9,018-question VMCBench benchmark.","keywords":["vision language models","VQA evaluation","multiple-choice question generation","distractor generation","agentic framework","VMCBench","benchmark construction","GPT-4o"],"falsifier":"Run the same human annotators on both the original and the AutoConverter-converted versions of the same questions: if human accuracy is unchanged or higher on converted items while VLM accuracy drops, the difficulty result is largely a model artifact, whereas a matched human drop would confirm genuine difficulty. Independently audit a random sample of score-5 questions for any second defensible correct option, since the paper's correctness evaluator is itself a GPT-4o model and only 95% of score-5 items pass human verification, leaving a measurable floor of flawed questions that could account for part of the reported accuracy gap.","tokens_in":30176,"feed_emoji":"🤖","tokens_out":17674,"duration_ms":140583,"temperature":0.7,"pith_summary":"Open-ended visual question answering is hard to score: the paper shows that word-overlap rules barely agree with human judgment, and that LLM-based graders shift scores when the grading model is updated. Its proposed fix is AutoConverter, a multi-agent system built on GPT-4o that rewrites open-ended questions as four-option multiple-choice items with automatically generated plausible wrong answers, called distractors. The central claim is that these machine-written distractors are at least as challenging as human-crafted ones: across eight datasets, a range of vision-language models score similar or lower accuracy on converted questions than on the original human-written items. On this basis the paper constructs VMCBench, a unified benchmark of 9,018 multiple-choice questions converted or refined from 20 existing datasets, and evaluates 33 vision-language models on it. If the claim holds, automated distractor generation can replace the costly human item-writing process and make VLM evaluation objective, reproducible, and scalable.","feed_headline":"Machine-written test options match or beat human-written ones","feed_subtitle":"Converting open-ended visual questions into four-choice items makes model evaluation objective and repeatable.","key_machinery":"The carrying mechanism is AutoConverter's agentic loop. Five specialized proposer agents each prompt GPT-4o to generate plausible wrong answers (distractors) that simulate a distinct human failure mode — concept misunderstanding, visual misinterpretation, reasoning error, data-processing error, and question bias — producing a pool of candidates with written rationales. A reviewer agent critiques each candidate for plausibility and challenge, the proposers revise accordingly, and a selector agent picks the three most demanding distractors for each question. A separate correctness evaluator then scores the assembled question on a 1–5 Likert scale; questions scoring below 4 return to a refiner agent for up to three rounds of repair. The evaluator's score is the pipeline's quality gate: 95% of questions that receive the top score are confirmed correct by human annotators, and VMCBench retains only score-5 questions.","core_discovery":"At the paper's center is a difficulty-parity result: when AutoConverter regenerates the distractors for questions from MMMU, MathVista, and AI2D (and five additional datasets in the appendix), vision-language models achieve accuracy that is similar to or lower than their accuracy on the original human-crafted distractors. The paper interprets this as machine-generated multiple-choice items being as hard as, and often harder than, human-written ones, while only about 3% of the highest-scoring converted questions are marked incorrect by human annotators, with roughly half of those errors traced to wrong answers in the original datasets rather than to the conversion. AutoConverter's design treats distractor writing as an agentic process: proposers generate candidates that mimic defined human error types, a reviewer critiques them, a selector keeps the most challenging three, and a correctness evaluator scores each question from 1 to 5, sending weak items back to a refiner. The same pipeline yields VMCBench's 9,018 questions, on which humans score 91.7%, the best open-weight model reaches 85.0%, and the best closed model 80.3%.","pith_inferences":["A testable consequence the authors do not draw: because challenge is measured only by VLM accuracy, the difficulty claim would be stronger if human solve rates also dropped on converted items; the paper's reported 91.7% human accuracy on VMCBench, with a quarter of human errors on questions described as ambiguous and unanswerable, suggests at least some of the VLM accuracy gap may come from flawed","The five error types the proposers simulate (concept, vision, reasoning, data, bias) form a ready-made diagnostic taxonomy: the same pipeline could generate assessments whose wrong answers reveal which class of misconception a student holds, turning each distractor into a measurement of the learner's error profile.","Because distractors are regenerated rather than copied, VMCBench's refined subsets are plausibly more resistant to test-set contamination than their sources — models that memorized original answers cannot rely on exact-match recall, and the option-shuffle check already shows the format discourages position-based guessing.","The quality ceiling of the whole pipeline is set by the single generator-evaluator model family; the appendix's cross-generator check (0.90 rank correlation across three generators) is a start, and re-running the pipeline with open-weight models would show whether the difficulty parity is a property of the method or of the specific engine."],"forward_implications":["Open-ended scoring is the bottleneck being removed: rule-based metrics correlate at 0.09 with human-proxy judgment on VQAv2, and a GPT-4o version update shifts absolute scores by about 6% on MMVet, so a multiple-choice format eliminates the least reproducible part of VLM evaluation.","Conversion preserves discriminative power: correlation between open-ended model-based scores and multiple-choice scores is 0.85, 0.71, and 0.97 on VQAv2, OKVQA, and VizWiz, versus 0.09, 0.19, and 0.00 for rule-based open-ended scoring.","The pipeline can also refine existing multiple-choice datasets: applying AutoConverter to already-multiple-choice benchmarks makes VLMs score similar or lower than on the original human distractors, so it doubles as a difficulty booster.","VMCBench behaves like a measurement instrument: option-order shuffling does not change model scores, and model families show a log-linear scaling trend, suggesting the benchmark has a smooth gradient for tracking progress.","A fully open model tops the public leaderboard at 85.0% against the best closed model's 80.3%, a result the paper reads as evidence that the gap between public and private VLM capabilities is narrowing."],"supporting_citations":[{"why":"Supplies one of the three datasets (MMMU) on which AutoConverter-converted distractors are shown to match or exceed the difficulty of human-crafted ones.","marker":"[59]"},{"why":"Supplies MathVista, the second dataset in the same head-to-head difficulty comparison.","marker":"[32]"},{"why":"Supplies AI2D, the third dataset in the head-to-head difficulty comparison.","marker":"[19]"},{"why":"Provides VQAv2, the dataset behind the rule-based evaluation experiment that yields the 0.09 correlation motivating the conversion approach.","marker":"[13]"},{"why":"Provides MMVet, the dataset behind the model-based evaluation experiment showing an approximate 6% absolute score shift between GPT-4o versions.","marker":"[58]"},{"why":"Justifies the four-option design (three distractors) as a guard against option-selection bias in language models.","marker":"[63]"},{"why":"Supplies the self-reflection technique that the reviewer and refiner agents use for iterative distractor improvement.","marker":"[48]"},{"why":"Supplies the role-playing multi-agent pattern on which the proposer, reviewer, and selector loop is built.","marker":"[43]"}],"fun_headline_variants":["AI-generated test options match human difficulty for VLMs","AutoConverter: machine-written MCQs rival human ones","Machine-made multiple choice as challenging as human","Auto-generated distractors stump VLMs like human ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that lower vision-language-model accuracy on converted questions reflects genuine difficulty, not ambiguity or hidden error — a premise the paper's own human evaluation strains, since humans score 91.7% and roughly a quarter of human errors fall on questions the authors themselves call ambiguous and unanswerable.","fun_headline_variants_meta":{"raw":{"variants":["AI-generated test options match human difficulty for VLMs","AutoConverter: machine-written MCQs rival human ones","Machine-made multiple choice as challenging as human","Auto-generated distractors stump VLMs like human ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1385,"prompt_tokens":930,"completion_tokens":455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":546,"tokens_out":455,"duration_ms":5050,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:19.813610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same human annotators on both the original and the AutoConverter-converted versions of the same questions: if human accuracy is unchanged or higher on converted items while VLM accuracy drops, the difficulty result is largely a model artifact, whereas a matched human drop would confirm genuine difficulty. Independently audit a random sample of score-5 questions for any second defensible correct option, since the paper's correctness evaluator is itself a GPT-4o model and only 95% of score-5 items pass human verification, leaving a measurable floor of flawed questions that could account for part of the reported accuracy gap.","supporting_citations":[],"review_version":1}