{"id":"7198e136-955f-4cbe-9d1b-265497c85496","arxiv_id":"2501.13949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On 6,990 ophthalmology MCQs, OpenAI o1 beat five other LLMs on answer accuracy yet lagged GPT-4o and GPT-4 on text-similarity metrics used to gauge reasoning.","lead":"Researchers put OpenAI's o1 and five other large language models through 6,990 ophthalmology exam questions. O1 answered the most correctly, but its written explanations scored lower than GPT-4o and GPT-4 on similarity-based reasoning metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that o1's reasoning lags is unsupported because the five text-generation metrics are unvalidated proxies for reasoning quality and their known length sensitivity penalizes o1's longer outputs; the paper's own qualitative review points the other way.","rationale":"My review identifies the same load-bearing assumption the reader flagged: the validity of the five text-generation metrics as proxies for reasoning. The accuracy result (o1: 0.88, macro-F1 0.70) is a well-defined descriptive outcome and appears credible; no concern arises there. The reasoning ranking, however, rests on metrics the paper itself concedes are non-medical and length-sensitive, and the evidence within the paper (BARTScore/METEOR superiority, qualitative review showing more detailed o1 reasoning) is inconsistent with a simple 'o1 reasons worse' narrative. The paper's own Discussion and Limitations sections acknowledge this, which is a point in its favor, but the abstract and conclusion still assert the reasoning lag without a validity check. The proposed concrete test—human expert ratings correlated with metric scores and adjusted for output length—would directly settle whether the metric-based ordering reflects reasoning quality or a length artifact. Because the descriptive accuracy result stands and the reasoning claim could survive with a validation study or appropriate reframing, the reader's CONDITIONAL verdict is appropriate; I do not find grounds to move it.","tokens_in":17809,"tokens_out":2529,"duration_ms":26026,"concrete_test":"Select a random sample of about 200 items from the 6,990. Have at least two ophthalmologists, blinded to model identity, independently rate the reasoning quality of o1 and GPT-4o outputs (e.g., correctness of clinical steps, use of relevant facts, coherence, absence of critical errors). Compute the correlation of each text-generation metric with the human ratings, and fit a regression of each metric on output length, model, and their interaction. If human-rated reasoning does not place o1 third, or if controlling for output length removes the metric gap between o1 and GPT-4o/GPT-4, then the reasoning-lag claim fails and the paper should be revised to present the metric results as hypothesis-generating rather than as a measure of reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central interpretive claim—that o1 ranks third in reasoning behind GPT-4o and GPT-4—depends entirely on the assumption that the five text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore, METEOR) validly measure reasoning quality in ophthalmology. The paper provides no validation of this assumption. These metrics compare model outputs to MedMCQA ground-truth explanations using lexical, embedding, or generation-based similarity; they do not directly evaluate logical validity, clinical soundness, or stepwise inference. The authors themselves concede in the Discussion that the metrics are general, not medical-specific, and are sensitive to length mismatches, and that o1 produces longer outputs due to chain-of-thought. The aggregate 'weighted normalised score' is actually an unweighted average of min-max normalized scores across six models, so the third-place ranking is driven by ROUGE-L, BERTScore, and AlignScore—exactly the metrics most likely to penalize length. Meanwhile, o1 ranks first on BARTScore and METEOR, and the human qualitative review of 40 items finds o1's responses more structured and detailed, with several examples where o1's reasoning is superior (Supplementary Tables 8a, 8c). The sensitivity analyses (correct-answer-only and ≥100-word subsets) do not control for output length, so they do not resolve the confound. Therefore, the conclusion that o1's reasoning enhancements 'may not extend equivalently to ophthalmology' is not supported by the metric evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript evaluates OpenAI's o1 against five other large language models (GPT-4o, GPT-4, GPT-3.5, Llama-3-8b, Gemini 1.5 Pro) on 6,990 ophthalmology multiple-choice questions drawn from the MedMCQA dataset. The authors measure performance using accuracy and macro-F1, and reasoning using five text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore, METEOR) plus a normalized aggregate score. They report that o1 achieves the highest accuracy (0.88 ± 0.33) and macro-F1 (0.70), but ranks third behind GPT-4o and GPT-4 on the aggregate reasoning score. Subgroup analyses examine performance by ophthalmic subtopic and ground-truth explanation length, and a qualitative human review of 40 items is also included. The paper concludes that o1's reasoning enhancements may not extend equivalently to ophthalmology, underscoring the need for domain-specific LLMs.","tokens_in":18063,"tokens_out":5092,"duration_ms":46768,"significance":"The accuracy benchmark is a useful contribution: it is, to the authors' claim, the largest ophthalmology MCQ evaluation to date, uses a public dataset, covers six models, and applies standardized prompting. The finding that o1 outperforms other models in answer accuracy is straightforward and clearly reported. If the reasoning conclusion were well supported, the paper would be important for guiding decisions about whether domain-specific medical LLMs are needed. However, the reasoning conclusion rests on text-generation metrics that are not validated as measures of reasoning quality, and the paper itself acknowledges their limitations. The statistical analysis of those metrics also uses an inappropriate test. The qualitative review is anecdotal and does not resolve the inconsistency. The paper's significance is therefore mainly in the accuracy benchmarking and in raising the question of how to evaluate reasoning, not in establishing that o1's reasoning lags in ophthalmology.","major_comments":[{"comment":"The five text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore, METEOR) are similarity measures between model outputs and the MedMCQA reference explanations, not validated measures of reasoning quality in ophthalmology. The Discussion concedes that these metrics are general, not medical-specific, and are sensitive to length mismatches, and that o1 tends to produce longer outputs due to chain-of-thought reasoning. The sensitivity analyses in Table 3 and Supplementary Tables 4 and 5 stratify by ground-truth explanation length or by correct-answer status, but they do not control for model output length, so they do not address the length confound. Consequently, the central claim that o1's reasoning lags behind GPT-4o and GPT-4 is not supported; the metrics support at most a claim about lexical and semantic similarity to the reference explanations, not about reasoning quality.","section":"Methods: Evaluation Metrics; Discussion"},{"comment":"For the text-generation metrics, all models are evaluated on the same 6,990 items, so the observations are paired by question. The paper states that the two-tailed Wilcoxon rank-sum test was used; the rank-sum test (Mann-Whitney U) assumes independent samples and is therefore inappropriate for paired data. The correct test would be the Wilcoxon signed-rank test or a paired permutation test. This error affects the validity of every reported p-value for the text-generation metrics in Table 2, Table 3, and the supplementary tables, and the authors should re-run these analyses with an appropriate paired test.","section":"Methods: Statistical Analysis"},{"comment":"The 'weighted normalised aggregate score' is described as weighted, but it is computed as an unweighted mean of min-max normalized scores across the five metrics, with the normalization performed across the six models in this study. The term 'weighted' is misleading. More importantly, because the min-max normalization is relative to the specific set of models evaluated, the aggregate score and the resulting third-place ranking for o1 are not absolute properties of the model; they would change if the model set changed. The equal weighting is also an arbitrary choice, and the ranking is heavily influenced by ROUGE-L, BERTScore, and AlignScore, which are the metrics most susceptible to length differences. The authors should either remove the claim of weighting, justify the equal weighting, or report a sensitivity analysis over alternative aggregation schemes.","section":"Methods: Evaluation Metrics; Table 2 footnote"},{"comment":"The qualitative review is based on only 40 randomly selected items, with no structured rating instrument, no inter-rater reliability statistic, and no quantitative summary of the evaluators' judgments. As presented, it consists of selected examples and is anecdotal rather than a systematic evaluation. The Discussion's characterization of o1's responses as 'more detailed and methodical' is in tension with the text-metric ranking that places o1 third, and this inconsistency is not resolved. The qualitative review should either be formalized (e.g., with a scoring rubric and agreement measures) or explicitly labeled as illustrative, and it should not be used to support the conclusion that o1's reasoning is superior or inferior without further evidence.","section":"Methods: Subset with Human Expert Qualitative Review; Results: Human Expert Review"}],"minor_comments":[{"comment":"The capitalization of the model name 'o1' is inconsistent (for example, 'O1' appears in the abstract's Meaning section and elsewhere); please standardize to 'o1' throughout.","section":"Abstract and Throughout"},{"comment":"The definition of accuracy contains a typo: 'proportion of correct predictions made out of the total predictions mad' should read '... total predictions made.'","section":"Methods: Accuracy definition"},{"comment":"The sentence on greedy decoding says 'temperature set to 0'; please clarify whether this was temperature = 0 or temperature = 0.0 and note whether the OpenAI o1 API allows exact control of this parameter, since o1's internal reasoning settings may differ from standard chat models.","section":"Methods: Prompt Engineering and hyperparameters"},{"comment":"Reference 29 (Xie et al.) is cited for o1's superior performance on ROUGE-1, but the current study reports ROUGE-L; please clarify which metric the cited study used and avoid ambiguity.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The accuracy benchmark is solid and likely useful for the community, but the reasoning claim is the main contribution and it is not currently supported by the evidence. I would encourage the editors to ask the authors to reframe the paper around what the text-generation metrics actually measure (similarity to reference explanations), to correct the statistical test, and to temper the conclusions accordingly. The authors' prior work on an ophthalmology-specific LLM is not a circularity issue, but it is worth keeping in mind when evaluating the prominence given to the domain-specific-refinement conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The accuracy half of this paper is solid and useful. This is the largest o1 ophthalmology evaluation I know of: 6,990 MedMCQA questions, six models, consistent prompting, greedy decoding, standard statistics with multiple-comparison corrections. The finding that o1 reaches 0.88 accuracy and 0.70 macro-F1, ahead of GPT-4o and Llama-3, is new and likely reproducible from the description even without released code or outputs. The subtopic and explanation-length analyses are a nice extra, and the paper is easy to read.\n\nThe reasoning half is much weaker, and the stress-test note has it right. The paper calls a weighted aggregate of ROUGE-L, BERTScore, BARTScore, AlignScore, and METEOR a measure of reasoning, but these are lexical/semantic similarity scores against MedMCQA reference explanations. They do not assess logical validity, clinical soundness, or stepwise inference. The authors themselves concede the metrics are general and length-sensitive, and o1's longer chain-of-thought outputs predictably lower its similarity scores. On the two metrics less sensitive to exact phrasing—BARTScore and METEOR—o1 ranks first or joint-first. The qualitative review of 40 items points the other way, with o1's responses described as more structured and detailed. The sensitivity analyses on correct-answer-only and longer-explanation subsets do not control for output length, so they do not break the confound. The aggregate score is also not weighted in any real sense; it is an unweighted average of min-max normalized scores, which is a minor but annoying mislabel.\n\nThat said, the paper is honest about these limitations in the Discussion, and the accuracy result stands independently. The problem is the framing: the conclusion that o1's reasoning enhancements \"may not extend to ophthalmology\" is not supported by the metric evidence as presented. It is a plausible hypothesis, but the data here mostly show that verbose outputs match reference explanations poorly on string-overlap metrics.\n\nWho benefits? Anyone doing model selection for medical QA, or building domain-specific ophthalmology LLMs, will want the accuracy numbers and subtopic rankings. The reasoning section needs a rewrite—either a validation of the text-gen metrics against human judgment, or a lighter interpretation that limits the claim to \"similarity to reference reasoning\" rather than \"reasoning ability.\"\n\nMy recommendation: this deserves peer review, not desk rejection. The accuracy benchmark is worth publishing, but I would insist the reasoning claim be reframed or supported with a proper human evaluation before acceptance.","headline":"Solid accuracy benchmark for o1 in ophthalmology, but the reasoning claim rests on unvalidated text-similarity metrics and should be reframed.","tokens_in":18682,"tokens_out":1803,"would_cite":true,"duration_ms":19314,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenAI o1 answers ophthalmology MCQs more accurately than five rival models, yet its explanations rank behind GPT-4o and GPT-4 when scored by text-generation metrics.","keywords":["OpenAI o1","large language models","ophthalmology","medical question answering","MedMCQA","reasoning evaluation","text-generation metrics","GPT-4o"],"falsifier":"Rescore the same 6,990 explanations with output length normalised, or have ophthalmologist raters judge reasoning quality blind to model identity on a larger sample than 40 items. If o1 then ties or beats GPT-4o and GPT-4, the claim that its reasoning lags in ophthalmology is not general; if it still ranks third, the claim is confirmed.","tokens_in":17589,"feed_emoji":"👁️","tokens_out":7188,"duration_ms":65714,"temperature":0.7,"pith_summary":"This paper asks whether OpenAI o1's general-purpose reasoning enhancements carry over to a specialised medical field, ophthalmology. Using 6,990 multiple-choice questions from the MedMCQA benchmark, it compares o1 with GPT-4o, GPT-4, GPT-3.5, Llama-3-8b, and Gemini 1.5 Pro under a standardised prompt and greedy decoding. o1 achieves the highest answer accuracy (0.88 ± 0.33) and macro-F1 (0.70), significantly above every comparator. But when reasoning is scored by five text-generation metrics that compare each model's explanation with the dataset's reference explanations, o1 ranks third behind GPT-4o and GPT-4 on the weighted aggregate. The authors take this accuracy–reasoning gap as evidence that o1's enhancements may not transfer to ophthalmology and that domain-specific refinement is still needed.","feed_headline":"o1 wins 88% of eye MCQs but ranks 3rd in reasoning","feed_subtitle":"o1 beats all rivals on 6,990 eye MCQs, but GPT-4o and GPT-4 score higher on reasoning metrics.","key_machinery":"The argument runs through the 6,990-item ophthalmology subset of MedMCQA and a five-metric text-generation battery. Each model receives the same zero-shot prompt ('assistant specializing in ophthalmology'), with temperature set to 0, and its explanation is scored against the MedMCQA ground-truth explanation by ROUGE-L, BERTScore, BARTScore, AlignScore, and METEOR; per-metric scores are normalised to 0–1 and averaged into a weighted aggregate. The accuracy results come from the same dataset's answer labels, so one benchmark carries both halves of the dissociation. Sensitivity analyses subset the questions by reference-explanation length, by five largest ophthalmic topic groups, by items all models answered correctly, and by a 100-item set with complete explanations, with a 40-item human qualitative review as a check on what the automatic metrics miss.","core_discovery":"The central finding is a dissociation between answering correctly and explaining like the reference. o1 answers 88% of the 6,990 MCQs correctly, the best of the six models, and its macro-F1 of 0.70 also leads. On the weighted normalised aggregate of ROUGE-L, BERTScore, BARTScore, AlignScore, and METEOR, however, o1's score (0.72) is below GPT-4o (0.83) and GPT-4 (0.74). The deficit is driven by ROUGE-L, BERTScore, and AlignScore, where o1 lags; o1 leads on BARTScore and ties with GPT-4o on METEOR. Subgroup analyses show o1 first in Lens and Glaucoma, second to GPT-4o in three other topic groups, and stronger on items whose reference explanations exceed 100 words.","pith_inferences":["A length-controlled rescoring of the same 6,990 outputs, truncating or summarising o1's explanations to the reference length, would show whether the o1 deficit on ROUGE-L, BERTScore, and AlignScore is mostly a verbosity artifact; this is a direct test the paper did not run.","The 'reasoning' ranking is really a ranking of similarity to MedMCQA's reference explanation style, so a clinician reading the same outputs could legitimately rank o1's step-by-step answers above the aggregate score without contradicting the reported numbers.","The same evaluation protocol could be applied to MedMCQA's other specialty subsets to test whether the accuracy-reasoning dissociation is specific to ophthalmology or a general o1 pattern.","The qualitative examples of confident, structured errors, such as confusing two dry-eye tests, suggest that even a top-accuracy model needs independent verification before its explanations are used in clinical education or triage."],"forward_implications":["If o1's accuracy lead is what matters for exam-style knowledge retrieval, o1 is currently the strongest of the six evaluated models on this ophthalmology benchmark.","If the reasoning scores are trusted, o1's chain-of-thought design does not automatically beat earlier GPT models on ophthalmology explanations, so general reasoning claims should not be extrapolated to specialty medicine.","Domain-specific performance varies by subtopic: o1 is best in Lens and Glaucoma but trails GPT-4o in Corneal and external diseases, Vitreous and retina, and Oculoplastic and orbital diseases.","Because o1's relative reasoning scores improve when reference explanations are longer, evaluations on short-reference items may understate o1's reasoning quality.","Benchmark accuracy on MCQs does not establish clinical utility; the authors frame the result as a first step toward structured, domain-specific validation datasets."],"supporting_citations":[{"why":"Supplies the 6,990 ophthalmology MCQs and their ground-truth explanations, the benchmark that produces both the accuracy and reasoning rankings.","marker":"32"},{"why":"States the design claim that o1 'thinks before it responds,' which is the hypothesis being tested for ophthalmology.","marker":"26,27"},{"why":"Reports o1's superior ROUGE-1 in general medical evaluation, the contrast that makes this study's third-place reasoning rank notable.","marker":"29"},{"why":"Defines the five text-generation metrics whose weighted normalized aggregate yields the reasoning ranking.","marker":"36–40"},{"why":"Supports the paper's own caveat that text-generation metrics alone may not fully capture reasoning, motivating the human qualitative review.","marker":"25,41"}],"fun_headline_variants":["o1 wins 88% eye MCQs but ranks 3rd in reasoning","Top eye MCQs score, third in reasoning for o1","o1 nails 6,990 eye questions, lags in reasoning","Eye exams: o1 tops accuracy, trails GPT-4o in reasoning","o1's 88% accuracy vs third-place reasoning gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that o1's reasoning lags GPT-4o and GPT-4 rests on the assumption that the five explanation-similarity metrics are valid proxies for reasoning quality; if those metrics mostly punish o1's longer, differently styled outputs, the reasoning ranking loses its force even though the accuracy result stands.","fun_headline_variants_meta":{"raw":{"variants":["o1 wins 88% eye MCQs but ranks 3rd in reasoning","Top eye MCQs score, third in reasoning for o1","o1 nails 6,990 eye questions, lags in reasoning","Eye exams: o1 tops accuracy, trails GPT-4o in reasoning","o1's 88% accuracy vs third-place reasoning gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1198,"prompt_tokens":909,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":194}},"tokens_in":525,"tokens_out":289,"duration_ms":3363,"temperature":1.0,"reasoning_tokens":194,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:30:25.877158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rescore the same 6,990 explanations with output length normalised, or have ophthalmologist raters judge reasoning quality blind to model identity on a larger sample than 40 items. If o1 then ties or beats GPT-4o and GPT-4, the claim that its reasoning lags in ophthalmology is not general; if it still ranks third, the claim is confirmed.","supporting_citations":[{"cited_title":"Medmcqa: A large-scale multi-subject multi- choice dataset for medical domain question answering","cited_arxiv_id":null,"evidence_quote":"Supplies the 6,990 ophthalmology MCQs and their ground-truth explanations, the benchmark that produces both the accuracy and reasoning rankings."},{"cited_title":"A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor? arXiv preprint arXiv:240915277","cited_arxiv_id":null,"evidence_quote":"Reports o1's superior ROUGE-1 in general medical evaluation, the contrast that makes this study's third-place reasoning rank notable."}],"review_version":1}