{"id":"833aa3dc-dbf0-48ab-99e5-6c8b22b40676","arxiv_id":"2508.20416","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new bilingual dental QA benchmark and corpus reveals large performance gaps in LLMs for dentistry, and shows that domain adaptation with the corpus improves accuracy.","lead":"This paper introduces DentalBench, a bilingual English-Chinese benchmark of 36,597 dental questions and a 337-million-token dental corpus, and tests 14 AI language models on it. It shows that current models struggle with open-ended dental reasoning but that fine-tuning and retrieval improve their scores, which matters for building trustworthy AI assistants in dentistry.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-answer correctness is never validated; a check on the unseen test split would settle whether the benchmark and all adaptation deltas are trustworthy.","rationale":"The reader's weakest assumption is broadly correct: the benchmark's reliability rests on unvalidated question quality in the unseen 99.2% of DentalQA. My stress-test sharpens this: even the 300 manually reviewed samples do not check gold-answer correctness, because the manual review only validates GPT-4o's category labels. Thus the load-bearing risk is not just extrapolation from 300 to 36,597; it is that the stated validation protocol cannot detect the most damaging error type (wrong answer keys) even within the reviewed sample. This is the single most load-bearing concern because every reported accuracy, BERTScore, model ranking, and adaptation delta in Table 1 inherits any gold-label error. I do not see an internal mathematical inconsistency; the construction pipeline is described in detail, the question counts sum correctly, and the adaptation results are directionally plausible. The lack of released dataset/code and the absence of error bars are additional limitations, but they are secondary to the unresolved answer-correctness question. Because the reader already returned CONDITIONAL and this concern reinforces that verdict rather than moving it, I recommend UNCHANGED: the paper should be accepted only if the proposed validation passes or the claims are scaled back accordingly.","tokens_in":83560,"tokens_out":4764,"duration_ms":62926,"concrete_test":"Draw a stratified random sample of 400 held-out DentalQA test questions (~1% of the test set, covering all seven task-language cells in proportion). Have two independent dental professionals, blinded to model outputs, label each item as (a) valid dental question, and (b) gold answer correct / incorrect / ambiguous. Compute the gold-answer error rate with a Wilson confidence interval and inter-annotator agreement. Then recompute the Table 1 rows and the SFT/RAG deltas after excluding invalid items. If per-task scores shift by more than 2–3 points or any model ranking changes, the central claims need revision; if the error rate is below ~2% and rankings are stable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 reports 'Human Validation' as manual review of 300 samples, but the protocol only checks agreement with GPT-4o's classification into oral / non-oral / insufficient (50 samples per language-category cell). It does not verify that the retained gold answers are correct, that MCQ/MAQ answer keys match the questions, or that OEQ/DEF reference answers are faithful. Since English OEQs are 'preserved in their original form without modification' and MCQ/MAQ answers are inherited from upstream exam and QA datasets, any source-level label errors propagate directly into every number in Table 1. The paper provides no estimate of the gold-answer error rate on the 99.2% of items not manually reviewed, and no error bars or significance tests. If even a few percent of answers are wrong, the 14-model rankings and the claimed SFT/RAG gains (e.g., MCQ-ZH +11.43, MAQ-ZH +9.88) could be materially biased. This is not an allegation of fraud; it is an unverified necessary condition for the benchmark's central claim to measure dental understanding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DentalBench, a bilingual (English-Chinese) resource for dentistry-domain LLM evaluation and adaptation. It has two components: DentalQA, 36,597 questions across four task formats (MCQ, MAQ, OEQ, DEF) and 16 dental subfields, and DentalCorpus, a 337.35-million-token corpus for supervised fine-tuning and retrieval-augmented generation. The authors evaluate 14 proprietary, open-source, and medical LLMs in zero-shot settings on a held-out test split, and report that domain adaptation of Qwen-2.5-3B via SFT, RAG, and SFT+RAG improves performance, especially on Chinese MCQ/MAQ and open-ended tasks. The paper also includes a limitations section acknowledging cross-lingual asymmetry and the absence of English MAQ items.","tokens_in":83761,"tokens_out":3676,"duration_ms":48749,"significance":"If the benchmark and corpus are valid, this is a useful and reusable resource: dentistry is under-served by current medical QA benchmarks, and the bilingual design plus 16-subfield coverage is a genuine gap. The paper provides a relatively transparent construction pipeline, including filtering prompts and keyword lists, and evaluates a broad model suite. The main contribution is empirical and resource-based rather than methodological. The strongest advertised claims are the benchmark's quality and the conclusion that domain adaptation 'significantly improves' performance; both currently rest on validation and statistical evidence that is thinner than the claims require. The reported experiments are plausible, but the manuscript needs strengthening in gold-answer validation, contamination analysis, and significance testing before the central claims can be accepted as established.","major_comments":[{"comment":"The reported manual review of 300 samples validates only the GPT-4o topical classification (oral / non-oral / insufficient), not the correctness of the retained gold answers, the consistency of MCQ/MAQ answer keys, or the faithfulness of OEQ/DEF reference answers. Because English items are inherited from upstream datasets and Chinese items from exams/textbooks, any source-level label errors propagate directly into every number in Table 1 and into the SFT/RAG deltas. The paper provides no estimate of the gold-answer error rate on the 99.2% of items not manually reviewed, and no expert check on a sample of the held-out test split. A validation study that flags incorrect or ambiguous answers and reports corrected performance would be a necessary condition for the benchmark's central claim to measure dental understanding.","section":"Section 2.1, Human Validation"},{"comment":"The main results are reported as point estimates without error bars, confidence intervals, or significance tests, and the domain-adaptation experiments use a single SFT seed. Several of the headline gains are small enough to be within run-to-run noise, especially on English OEQ/DEF (e.g., SFT+RAG vs. SFT: 37.68 vs. 37.74 BERTScore for EN-OEQ and 29.65 vs. 30.79 for EN-DEF, i.e., negative deltas; RAG alone on EN-DEF is +0.42). The claim that domain adaptation 'significantly improves' performance needs bootstrapped confidence intervals or multiple-seed runs with significance tests, at least for the central MCQ/MAQ gains and the open-ended task comparisons.","section":"Section 3.1 / Table 1 / Section 3.2"},{"comment":"The RAG experiments retrieve from DentalCorpus, which is built from PubMed full-text articles and MedRAG PubMed snippets, while English DentalQA sources include PubMedQA and MedQuAD-derived content. There is no contamination or overlap analysis between the test set and the retrieval corpus. If the corpus contains passages that are near-duplicates of test questions or reference answers, the RAG gains (e.g., MCQ-ZH +11.43, MAQ-ZH +9.88) could partly reflect direct answer lookup rather than improved dental understanding. The authors should quantify overlap between test items and corpus chunks, and ideally run an ablation where retrieval excludes chunks that contain answer-derived or question-overlapping text, or use a temporally/structurally held-out retrieval index.","section":"Section 3.2 RAG / Section 2.2 DentalCorpus"},{"comment":"OEQ and DEF performance is measured only by BERTScore F1, with absolute scores mostly in the 15-35 range. The paper provides no evidence that BERTScore differences of 1-3 points correspond to meaningful quality differences in dentistry answers, nor any human or expert correlation study for the metric on this domain. Since the adaptation claims for open-ended tasks rest on these numbers, a small expert rating sample or an additional metric (e.g., LLM-as-judge with a rubric, or token-level F1 against reference keywords) is needed to make the improvements interpretable.","section":"Section 3.1, Evaluation Metrics"}],"minor_comments":[{"comment":"Typo: 'Precistion' should be 'Precision'.","section":"Table 1 caption"},{"comment":"The text says 'seven public medical QA datasets' but the enumeration lists eight sources (MMLU, MedQA, MedMCQA, MedQuAD, PubMedQA, iCliniq, Medical Meadow Flashcards, Medical Meadow Wikidoc). Please correct the count or split the list.","section":"Section 2.1, Data Sources"},{"comment":"The sentence 'Fig. 5 shows the prompt formats...' duplicates the text in Appendix A.3 and appears to be a copy-paste error; the intended cross-reference is missing.","section":"Appendix C.2"},{"comment":"In the provided PDF, the supplementary performance figures appear as encoded strings rather than readable plots, so the per-subfield and per-model results they are meant to convey are not accessible. These should be regenerated as standard figures.","section":"Appendix figures 8-17"},{"comment":"The limitations section acknowledges that MAQ is Chinese-only and that the bilingual resources are asymmetric. This is useful, but the abstract and introduction's 'bilingual benchmark' claim should explicitly qualify that the four task types are not all available in both languages.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"This is a resource/benchmark paper whose value depends on dataset quality and on the reliability of the reported gains. The central design is sound and the resource is likely to be useful, but the manuscript currently lacks gold-answer validation on the test split, error bars/significance tests, and contamination analysis for the RAG setup. These are fixable within the paper's scope and should be required before acceptance. I would also encourage the editor to request that the authors commit to releasing the benchmark and corpus with a datasheet, since reproducibility of resource papers depends on availability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DentalBench is a real new resource—the first bilingual dental QA benchmark covering 4 task types and 16 subfields, plus a 337M-token domain corpus used for SFT and RAG experiments across 14 LLMs. The construction is careful: keyword filtering, TF-IDF-derived term lists, UMLS definitions, manual spot checks on corpus relevance (99%/96%). The authors also openly flag the EN/ZH source asymmetry and MAQ being Chinese-only. That is honest and useful.\n\nThe soft spot is the one the stress-test note pins down, and it lands. The human validation of 300 samples only measures agreement with GPT-4o's classification into oral/non-oral/insufficient. It never checks whether the retained gold answers are correct, whether MCQ/MAQ answer keys match questions, or whether OEQ/DEF reference answers are faithful. Since most items inherit answers from upstream exam and QA datasets, any label errors propagate directly into every number in Table 1. There is no estimate of answer error on the 99.2% not manually reviewed. That is a load-bearing gap: if a few percent of answers are wrong, the 14-model rankings and the SFT/RAG deltas (e.g., +11.43 MCQ-ZH) are suspect.\n\nSecondary issues in proportion: no error bars or significance tests on the headline table; single-seed SFT; no contamination analysis; and the 'substantially improves' claim is overstated for English OEQ/DEF, where gains are small and inconsistent. Also, the dataset and code are not released, which currently blocks both reuse and external verification.\n\nNone of this means the central idea fails. The benchmark fills a real gap, and the flaws are fixable: release the data, run a gold-answer correctness check on a random sample of the test split, add basic statistics, and cool the domain-adaptation wording. The reader's conditional verdict is about right. I'd send this to peer review—it deserves a serious referee—but I would expect major revision before acceptance. For my own work, I wouldn't cite the numbers until the answer validation and release are in place.","headline":"DentalBench is a genuinely new bilingual dental benchmark and corpus, but the reported numbers rest on unverified gold answers and thin statistics.","tokens_in":84254,"tokens_out":2473,"would_cite":false,"duration_ms":30071,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DentalQA presents a bilingual test for how well LLMs understand dentistry, plus a corpus for improving them.","keywords":["dental LLM benchmark","bilingual QA","dentistry natural language processing","retrieval-augmented generation","supervised fine-tuning","domain adaptation","dental corpus","medical LLM evaluation"],"falsifier":"Have several dental specialists independently verify answer correctness and dental relevance on a random sample of roughly 1,000 questions drawn from the 99.2% of DentalQA that was not manually reviewed; if label-error or relevance-error rates substantially exceed the 4-8% range seen in the 300-sample check, the reported performance and adaptation numbers cannot be trusted as measures of dental understanding.","tokens_in":83446,"feed_emoji":"🦷","tokens_out":8710,"duration_ms":90250,"temperature":0.7,"pith_summary":"DentalBench is an attempt to give dentistry its own evaluation and adaptation resources for large language models. It introduces DentalQA, a 36,597-question English-Chinese benchmark in four task formats across 16 dental subfields, and DentalCorpus, a 337.35-million-token dental corpus for fine-tuning and retrieval-augmented generation. Evaluating 14 models, the paper finds that even strong general models struggle most with open-ended clinical questions and term definitions, while medical-specific models lag on multiple-choice dental facts. On a 3B model, supervised fine-tuning, retrieval support, and their combination each improve scores, with combined gains largest on multiple-choice tasks. If the benchmark is reliable, it provides a reusable way to measure and steer dental LLM capabilities in both languages.","feed_headline":"DentalQA: 36,597-question bilingual test exposes LLM gaps","feed_subtitle":"English-Chinese dental QA plus a 337M-token corpus: fine-tuning and retrieval lift small-model scores.","key_machinery":"The object carrying the argument is DentalQA, a bilingual question bank with 36,597 items in four formats: single-answer multiple choice (MCQ), multi-answer multiple choice (MAQ, Chinese only), open-ended clinical questions (OEQ), and definition questions (DEF). It spans 16 dental subfields and is scored by accuracy for MCQs, precision/recall/F1 for MAQs, and BERTScore for OEQ/DEF. A companion resource, DentalCorpus, supplies 337.35M tokens of filtered textbooks, PubMed articles, and medical datasets; it is used both to fine-tune a model and to retrieve supporting passages during inference (RAG).","core_discovery":"On the paper's own terms, the central discovery is that currently available large language models, including those trained for medicine, are not reliably dental-literate: multiple-choice factual accuracy is moderate, open-ended clinical reasoning and terminology scores are far lower, and which model leads depends on language (DeepSeek-R1 leads Chinese, GPT-4o leads English). The discovery is made concrete by showing that in-domain adaptation of one 3B model (Qwen-2.5-3B) with fine-tuning or retrieval moves scores upward, and the two are additive on Chinese multiple-choice questions. The benchmark is the load-bearing instrument: without DentalQA, these comparisons and adaptation gains would n","pith_inferences":["Beyond the paper, the same benchmark could be extended with an English MAQ format; whether the reported cross-lingual gaps persist would separate language coverage from true dental knowledge.","A reasonable next stress test would be to run the SFT+RAG recipe on larger open models; the 3B gains may shrink or grow with scale, and that is not measured here.","Because DentalQA reuses public exam and QA datasets filtered by keyword lists, some items may resemble pretraining text; scoring on fresh, unpublished dental licensing questions would reveal contamination.","A practical clinical extrapolation: retrieval over DentalCorpus raised open-ended Chinese scores the most, suggesting RAG-style support could help with dental documentation or patient education even when full fine-tuning is impractical."],"forward_implications":["Any future dental LLM can be scored on the same 16 subfields and four task formats, making model comparisons consistent across languages.","Per-task reporting matters: a single aggregate score would hide that open-ended questions and definitions are the weak points.","In-domain text helps in two different ways—fine-tuning boosts factual multiple-choice accuracy, retrieval boosts open-ended generation—so they should be combined rather than treated as alternatives.","Current medical-specific LLMs are not automatically strong on dentistry, so medical adaptation alone is not enough for specialized dental applications.","The Chinese and English halves rank models differently, so a dental safety claim based on one language may not carry to the other."],"supporting_citations":[{"why":"Supplies MMLU items that are filtered into the English MCQ portion of DentalQA.","marker":"Hendrycks et al., 2021"},{"why":"Supplies MedQA medical exam questions used for English MCQ construction.","marker":"Jin et al., 2020"},{"why":"Supplies MedMCQA questions filtered into English MCQs.","marker":"Pal et al., 2022"},{"why":"Supplies MedQuAD QA pairs used as English OEQ sources.","marker":"Ben Abacha and Demner-Fushman, 2019"},{"why":"Supplies PubMedQA items used in the English question pool.","marker":"Jin et al., 2019"},{"why":"Provides UMLS term definitions used to generate English DEF questions.","marker":"U.S. National Library of Medicine, 2025b"},{"why":"Supplies MMedC, filtered by dental keywords into DentalCorpus.","marker":"Qiu et al., 2024"},{"why":"Supplies MedRAG PubMed snippets filtered into DentalCorpus.","marker":"Zhao et al., 2025"},{"why":"Defines BERTScore, the metric used to grade open-ended and definition answers.","marker":"Zhang et al., 2019"},{"why":"Defines the Qwen-2.5-3B base model used in the domain adaptation experiments.","marker":"Qwen et al., 2025"}],"fun_headline_variants":["LLMs flunk dental exam: bilingual benchmark finds big gaps","DentalBench: 36,597 bilingual questions expose weak dental AI","Bilingual dental quiz shows LLM limits, fine-tuning lifts scores","DentalQA: 36,597 questions, two languages, one big gap","3B model gains from dental fine-tuning, benchmark reveals"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole benchmark depends on the 36,597 questions being correctly labeled and truly dental; only 300 (0.8%) were manually reviewed, so a materially higher error rate in the unseen questions would make the measured gaps and adaptation gains unreliable.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flunk dental exam: bilingual benchmark finds big gaps","DentalBench: 36,597 bilingual questions expose weak dental AI","Bilingual dental quiz shows LLM limits, fine-tuning lifts scores","DentalQA: 36,597 questions, two languages, one big gap","3B model gains from dental fine-tuning, benchmark reveals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1095,"prompt_tokens":759,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":503,"tokens_out":336,"duration_ms":4403,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:05:00.128949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several dental specialists independently verify answer correctness and dental relevance on a random sample of roughly 1,000 questions drawn from the 99.2% of DentalQA that was not manually reviewed; if label-error or relevance-error rates substantially exceed the 4-8% range seen in the 300-sample check, the reported performance and adaptation numbers cannot be trusted as measures of dental understanding.","supporting_citations":[],"review_version":1}