{"id":"b0e00495-53ca-44d8-abfd-cc1dc9b6c956","arxiv_id":"2509.09254","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"MMOral is a large new dental X-ray instruction dataset and benchmark, but the proposed model's 24.73% improvement is from fine-tuning and then testing on the same data pool.","lead":"A new dataset and benchmark for panoramic dental X-ray analysis brings 20,563 images and 1.3 million instruction pairs, but the evaluation suite is drawn from the same pool used to train the proposed model. A reader may care because the resource aims to measure and improve AI for a common dental imaging modality, but the headline improvement is compromised by train/test overlap.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MMOral-Bench is drawn from the MMOral training pool, so the reported 24.73% OralGPT gain may reflect memorization, not generalization.","rationale":"The reader's weakest assumption correctly identifies the decisive confound: MMOral-Bench is sampled from MMOral, and OralGPT is fine-tuned on MMOral, with no disjoint test split described in the main text or supplementary. §4.1 states that benchmark cases are chosen from MMOral; §5.3 states that SFT is performed on MMOral instruction data; no exclusion of benchmark images is mentioned. The improvement numbers match the table exactly: baseline 21.46, full OralGPT 46.19, difference 24.73. This is the paper's central claim. A model trained on 1.3 million instruction instances, including the 100 benchmark images and their derived QA pairs, can score highly by memorization after one epoch of SFT, so the result does not demonstrate generalization. The zero-shot evaluation of 64 LVLMs could survive independently, but that is not the paper's headline claim. Appendix F's admission that ground-truth labels from external datasets were not independently verified is a real secondary limitation, but the primary issue is the missing held-out split. The concrete test—retraining with benchmark images excluded—would settle the matter. If the improvement persists, the central claim is credible; if it collapses, the current verdict stands. Since my analysis aligns with the reader's, no verdict adjustment is needed.","tokens_in":37979,"tokens_out":4044,"duration_ms":43830,"concrete_test":"Re-run the §5.3 SFT protocol after removing from the training set all MMOral instances whose source image is one of the 100 MMOral-Bench images (matched by SHA-256 hash or canonical filename), including MMOral-Report, MMOral-VQA, and MMOral-Chat instances derived from those images. If OralGPT's MMOral-Bench average drops from 46.19 toward the 21.46 baseline, the reported 24.73% gain is explained by train/benchmark overlap. As a supplementary check, compute the maximum ROUGE-L or n-gram overlap between benchmark QA pairs and training QA pairs to quantify direct question leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that one epoch of SFT on MMOral yields a 24.73% average improvement (21.46→46.19, Table 3/9)—requires MMOral-Bench to be an independent test set. The paper never describes a train/test split. §4.1 says MMOral-Bench cases are 'manually chosen and checked from the MMOral', and its 100 images are selected from Hoang Viet Do [19], one of the two image sources used to build the 20,563-image MMOral (§2.1). §5.3 fine-tunes Qwen2.5-VL-7B on MMOral-Report, MMOral-VQA, and MMOral-Chat for one epoch, with no exclusion of benchmark images or benchmark-derived QA pairs. Because MMOral-VQA and MMOral-Chat are generated from the same grounding captions and medical reports that underlie the benchmark QA pairs, the fine-tuned model can memorize the exact images and near-identical questions. The zero-shot ranking of 64 LVLMs is not affected by this leakage, but the headline SFT result and the conclusion that MMOral improves dental LVLMs are not established. Appendix F's concession that automated labels lack independent third-party validation is secondary but compounds the concern: even a held-out split would still inherit potentially noisy ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MMOral, a large multimodal instruction dataset for panoramic X-ray analysis (20,563 images, 1.3M instruction instances across attribute extraction, report generation, VQA, and chat), together with MMOral-Bench, an evaluation suite of 100 images and 1,100 closed/open QA pairs spanning five dental diagnostic dimensions. The authors evaluate 64 LVLMs zero-shot, reporting that even GPT-4o reaches only 41.45% average accuracy, and propose OralGPT, a Qwen2.5-VL-7B model fine-tuned for one epoch on MMOral instruction data, claiming a 24.73% average improvement on MMOral-Bench. The main contributions are the dataset, the benchmark, the large-scale model evaluation, and the SFT result.","tokens_in":38285,"tokens_out":4203,"duration_ms":50406,"significance":"If MMOral-Bench were an independent, expertly validated test set, this would be a valuable contribution to dental AI: the instruction dataset is large, the evaluation covers 64 models with standard tools (VLMEvalKit), the report-generation pipeline includes human evaluation, and the LLM-as-judge for open-ended answers is accompanied by repeatability and dentist-agreement checks. The zero-shot leaderboard, in particular, is a useful resource irrespective of the SFT claims. However, the central SFT claim is not supported because the benchmark is drawn from the same image pool and instruction data used for training, and the anatomical ground truth is generated by automated specialist models without independent validation. These two issues undermine the paper's headline conclusion about the value of MMOral for improving dental LVLMs.","major_comments":[{"comment":"The headline result—one epoch of SFT on MMOral improves Qwen2.5-VL-7B by 24.73% (21.46→46.19, Table 9)—is not supported because MMOral-Bench is not disjoint from the MMOral training data. §4.1 states that all MMOral-Bench cases are 'manually chosen and checked from the MMOral,' and its 100 images are selected from Hoang Viet Do [19], one of the two image sources of MMOral (§2.1). The SFT in §5.3 trains on MMOral-Report, MMOral-VQA, and MMOral-Chat, which are generated from the same grounding captions and medical reports that underlie the benchmark QA pairs (§2.4). The model can therefore memorize the benchmark images and near-identical questions; the reported improvement measures train-set fit, not generalization to new panoramic X-rays. The zero-shot ranking of 64 LVLMs is unaffected by this overlap, but the paper's central claim about MMOral's effectiveness is not established. The auth","section":"§4.1, §5.3, Table 9"},{"comment":"The benchmark ground truth is not independently validated. Appendix F concedes that the accuracy of the ground-truth labels from the constituent public datasets 'has not been validated by independent third-party organizations,' and the visual specialist models are trained on those labels. MMOral-Bench QA pairs are then built from the resulting grounding captions and LLM-generated reports (§4.1), so a model's score partly reflects agreement with potentially noisy automated annotations. Human evaluation in §3.2(3) covers report quality, not the anatomical labels or benchmark answers; §4.1 does not specify that the 'manually chosen and checked' step involved dental experts or measured inter-rater agreement. The claim that GPT-4o's 41.45% represents a clinically meaningful performance ceiling requires an expert-reviewed sample of MMOral-Bench QA pairs with reported agreement statistics.","section":"Appendix F, §2.1–§2.4, §4.1"},{"comment":"The paper's Finding 1 states that MMOral-Bench 'poses significant challenges to ALL LVLMs' based on the 41.45% GPT-4o score. Given the ground-truth derivation issues above, the absolute difficulty of the benchmark is unclear: low scores could indicate either genuine model weakness or annotation noise/ambiguity in the benchmark itself. The authors should quantify label quality, e.g., by having two or more dentists independently answer a random sample of MMOral-Bench questions and reporting human performance and agreement. Without this, the interpretation of all absolute scores on MMOral-Bench is ambiguous.","section":"§5.2, Finding 1"}],"minor_comments":[{"comment":"The term 'Oral-Bench' appears once in the introduction and in §5.2 ('on our Oral-Bench'); elsewhere it is 'MMOral-Bench.' Please standardize.","section":"§1"},{"comment":"The main text refers to 'Table 9' for the SFT results on Qwen2.5-VL-7B, but the table in the main text is numbered Table 3; Table 9 appears in the supplementary. Please fix the cross-reference.","section":"§5.3"},{"comment":"The text calls the OCR model 'OpenOCR' and cites reference [21], but the reference is titled 'Svtrv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition.' Please clarify whether OpenOCR is the deployed model and cite the appropriate source.","section":"§2.1"},{"comment":"Figure 3(c) shows the diagnostic distribution of MMOral-Bench, but the figure caption is terse. Please list the exact counts or percentages for the five categories in the caption or text.","section":"§3.1 / Figure 3"},{"comment":"The equations are standard, but the definition of N_c in Eq. (2) should be explicit about how samples assigned to multiple categories are handled in per-category scores.","section":"§4.2, Eq. (1)–(2)"}],"recommendation":"reject","confidential_remarks":"The dataset and zero-shot leaderboard could be useful, but the central SFT claim is circular because MMOral-Bench is constructed from the training pool, and the ground truth inherits unvalidated automated labels. These are load-bearing issues for the paper's main contribution as stated. If the authors can provide a genuinely disjoint held-out set and independent verification of benchmark answers, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper builds the largest panoramic dental X-ray instruction dataset to date: 20,563 images and 1.3 million instruction instances across attribute extraction, report generation, VQA, and dialogue. The curation pipeline is elaborate—ten specialist models, spatial relationship matching, two-stage LLM report generation, and dentist evaluation—and the zero-shot evaluation of 64 LVLMs is a useful resource, showing that even GPT-4o only reaches 41.45% and that medical LVLMs have no clear edge. That part is worth taking seriously.\n\nThe problem is the central SFT claim. MMOral-Bench is not an independent test set. Section 4.1 says its cases are \"manually chosen and checked from the MMOral,\" and the 100 images are selected from Hoang Viet Do, one of the two sources used to build MMOral. The fine-tuning in Section 5.3 trains on MMOral-Report, MMOral-VQA, and MMOral-Chat with no exclusion of benchmark images or their derived QA pairs. So the reported jump from 21.46% to 46.19%—the 24.73% improvement that anchors the abstract—may largely reflect memorization, not generalization. The paper never describes a train/test split, and this is a load-bearing omission, not a minor wrinkle.\n\nThe secondary weakness is ground-truth reliability. Appendix F concedes the automated anatomical labels have not been validated by an independent third party. The paper does report dentist agreement on open-ended evaluation and report quality, which is partial credit, but the underlying labels feeding both training and benchmark remain unverified. That means even a proper held-out split would inherit potentially noisy ground truth.\n\nThe zero-shot evaluation of existing LVLMs is not affected by the leakage, and the dataset itself could still be useful for training even if the benchmark is compromised. But as written, the main empirical conclusion is not established. A revision that constructs a genuinely held-out benchmark—ideally with independent expert annotation of the test images—and re-runs the SFT comparison would address the core objection.\n\nMy take: this deserves a serious referee, and I would engage with it in review, but I would reject it in its current form. The resource has value; the evaluation protocol needs to be redone before the benchmark and the OralGPT claim can be trusted. The authors seem to have the tools and clinical connections to fix this, so I'd encourage a careful resubmission rather than shelving it.","headline":"The dataset is a real contribution, but the headline 24.73% SFT gain is not credible because MMOral-Bench is drawn from the same image pool the model was trained on.","tokens_in":38788,"tokens_out":1806,"would_cite":false,"duration_ms":23100,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MMOral, a large instruction dataset and benchmark for panoramic dental X-rays, and claims that a single epoch of fine-tuning lifts a 7-billion-parameter vision-language model's benchmark score by 24.73 percentage point","keywords":["panoramic X-ray","dental AI","vision-language models","instruction tuning","benchmark","visual question answering","medical report generation","OralGPT"],"falsifier":"Pick 100 panoramic X-rays from a second imaging center that were never part of MMOral, have two dentists label the same five diagnostic dimensions, and run OralGPT zero-shot; if its score falls near the 21–25% baseline instead of the 46% reported, the large gain came from memorizing the benchmark pool.","tokens_in":37857,"feed_emoji":"🦷","tokens_out":7673,"duration_ms":82611,"temperature":0.7,"pith_summary":"MMOral is a new dataset and evaluation suite built for panoramic dental X-rays, a modality with dense anatomy and subtle disease cues that general medical benchmarks skip. The paper claims that a single epoch of supervised fine-tuning on MMOral's instruction data lifts a 7-billion-parameter vision-language model from 21.5% to 46.2% average accuracy on its own benchmark. It also reports that 64 current vision-language models, including the strongest proprietary ones, score at most 41.45%, so the field is far from reliable automated interpretation. If these numbers hold, MMOral provides both the reading curriculum and the measuring stick that dental AI has been missing.","feed_headline":"One training pass lifts dental X-ray AI by 25 points","feed_subtitle":"A 20k-image dental instruction set lifts a 7B model from 21% to 46% while top models stay under 42%.","key_machinery":"The pipeline starts with ten visual specialist models detecting 49 categories of anatomical structures in panoramic X-rays. A post-processing chain deduplicates overlapping detections with threshold filtering and non-maximum suppression, links findings to FDI tooth numbers through spatial relationship matching, and inserts clinical rules such as recommending extraction when a wisdom tooth lacks an opposing counterpart. From this structured output, template-based grounding captions and two-stage LLM-generated medical reports are built, and then GPT-style prompting produces closed-ended and open-ended question-answer pairs plus patient-radiologist dialogues. All instruction data is organized a","core_discovery":"The central claim is that panoramic X-ray interpretation can be taught to a general vision-language model with one training epoch using automatically generated instruction data, and that the same data can expose how far current models are from clinical competence. MMOral contains 20,563 annotated images, 1.3 million instruction instances, 904k bounding boxes, and report, VQA, and chat subsets. MMOral-Bench offers 1,100 questions over 100 images across five diagnostic dimensions: condition of teeth, pathological findings, historical treatments, jawbone observations, and clinical summary and recommendations. On this benchmark the fine-tuned OralGPT improves from 21.46% to 46.19% average score,","pith_inferences":["Because the benchmark is selected from the same MMOral pool used for fine-tuning, the 24.73% gain likely measures how well the model fits that pool; an external set from another scanner would be needed to know whether the skill generalizes.","The paper's supplementary section concedes that the underlying anatomical labels have not been independently validated; if those labels are noisy, both the instruction data and the benchmark inherit that noise.","The visual-specialist plus LLM pipeline could be reused for other dense-anatomy imaging such as periapical X-rays or CBCT, but the bottleneck is the quality of the specialist labels.","If the five-dimension diagnostic taxonomy proves robust, it could become a standard reporting structure for dental AI, letting models output per-tooth findings plus summary recommendations in a form clinicians can audit."],"forward_implications":["On MMOral-Bench, the strongest proprietary model reaches only 41.45% average accuracy, so current vision-language models are not yet reliable readers of panoramic X-rays.","A single epoch of supervised fine-tuning on MMOral instruction data lifts a 7-billion-parameter model from 21.46% to 46.19%, meaning large gains come from domain instruction data rather than architecture changes.","Open-ended questions are disproportionately hard: a large share of open-source models score below 25%, so models can recognize options but cannot generate accurate descriptions.","Models perform best on coarse jaw-level anatomy and worst on fine-grained tooth-level questions, locating the specific weakness in current multimodal encoders.","Medical-specific vision-language models do not outperform general-purpose ones on this benchmark, suggesting dental skill is not inherited from generic medical training."],"fun_headline_variants":["One epoch on dental X-ray data lifts 7B model from 21% to 46%","20k dental scans, 1.3M prompts: single-pass fine-tune beats GPT-4o","Dental AI benchmark: GPT-4o scores 41%, fine-tuned 7B hits 46%","Single training epoch turns 7B VLM into top dental X-ray AI","Multimodal dental dataset boosts 7B model 25 points in one pass"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that MMOral-Bench measures generalization: Section 4.1 draws every benchmark case from the same MMOral pool used for fine-tuning in Section 5.3, and the supplementary notes the underlying labels have not been independently validated.","fun_headline_variants_meta":{"raw":{"variants":["One epoch on dental X-ray data lifts 7B model from 21% to 46%","20k dental scans, 1.3M prompts: single-pass fine-tune beats GPT-4o","Dental AI benchmark: GPT-4o scores 41%, fine-tuned 7B hits 46%","Single training epoch turns 7B VLM into top dental X-ray AI","Multimodal dental dataset boosts 7B model 25 points in one pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1211,"prompt_tokens":873,"completion_tokens":338,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":216}},"tokens_in":617,"tokens_out":338,"duration_ms":4414,"temperature":1.0,"reasoning_tokens":216,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:23:42.348373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick 100 panoramic X-rays from a second imaging center that were never part of MMOral, have two dentists label the same five diagnostic dimensions, and run OralGPT zero-shot; if its score falls near the 21–25% baseline instead of the 46% reported, the large gain came from memorizing the benchmark pool.","supporting_citations":[],"review_version":1}