{"id":"2943c258-1349-4249-9f67-c6db5ac99e61","arxiv_id":"2607.27378","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PanDent links tooth-level dental annotations to template-aligned reports and shows that current MLLMs fail fine-grained dental diagnosis while fine-tuning on it improves localization and accuracy.","lead":"PanDent is a new benchmark of 9,524 dental panoramic X-rays with expert tooth-level labels and matching text reports, used to test whether AI models can describe dental findings accurately. Current multimodal models write fluent reports but mistarget or miss many tooth-level findings; fine-tuning on the dataset improves diagnostic accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set composition: the 500-case benchmark appears to be nearly the entire in-house batch, and Table 4 shows large prevalence shifts; if so, benchmark and fine-tuning conclusions are confounded by domain shift.","rationale":"In good faith, PanDent is a useful resource: the template-driven report construction, expert validation, public data/code, and fine-grained tooth-level annotations are real contributions. The central claim, however, is that the benchmark reliably measures tooth-level structure-language consistency and that fine-tuning on PanDent substantially improves it. That claim depends on the held-out test set being a meaningful sample of the target distribution. The arithmetic of the release — 505 in-house cases and a 500-case test set — combined with the prevalence shifts in Table 4 makes it very likely that the test set is almost entirely in-house. This is not a theoretical worry: if true, every zero-shot model is being evaluated on a distribution essentially absent from the training split, and the reported fine-tuning gains could reflect adaptation to one clinic's acquisition and annotation style plus template format, rather than durable clinical improvement. The paper's 'disjoint from the training split' guarantee does not address source composition. I see this as the single most load-bearing concern because it directly undermines the benchmark's external validity and the paper's central fine-tuning claim. The reader's weakest_assumption identified the same issue, and I agree. My proposed check — re-running on a source-stratified or public-only held-out split — would settle whether the concern lands. If the gains persist, the conclusion is substantially strengthened; if not, the benchmark needs re-splitting or re-reporting. This confirms the reader's CONDITIONAL verdict rather than changing it.","tokens_in":23421,"tokens_out":5462,"duration_ms":56210,"concrete_test":"Request or derive per-case source metadata for all 9,524 cases and verify whether the 500 test cases coincide with the in-house batch. Then re-run the core comparisons (e.g., GPT-5.4, Qwen3-VL-4B zero-shot, and Qwen3-VL-4B/Qwen3.5-4B fine-tuned) on a new held-out set sampled randomly from public-source cases, or stratified by source, using the same attribute prompts and exact-match protocol. If the fine-tuned-minus-zero-shot deltas and model rankings change materially, the reported conclusions are confounded by domain shift; if they persist on the public-source split, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that PanDent evaluates and improves tooth-level clinical consistency across the target distribution. The weakest load-bearing assumption is that the held-out 500 cases are a representative sample of the same distribution as the training resource. The paper reports 9,019 public + 505 in-house cases, split into 9,024 train / 500 test; this strongly suggests the test set is virtually the entire in-house batch. Table 4 shows large prevalence shifts consistent with source/annotation differences: missing teeth 88.4% test vs 68.7% train; maxillary GCBL 54.2% vs 30.8%; crowns 32.8% vs 17.7%; mandibular angular bone loss 8.0% vs 1.9%; and rare findings (TMJ, maxillary sinus, maxillary structural abnormality) at 0% in test. Disjointness from the training split does not control for source composition; if the test set is in-house, zero-shot failures may reflect OOD acquisition/annotation style, and the fine-tuning gains (e.g., Table 2 average 0.401 to 0.638 vs OralGPT) may reflect adaptation to a single clinic distribution and template format rather than generalizable clinical improvement. The paper provides no source-stratified analysis or cross-source held-out evaluation. This concern is about benchmark validity, not about the dataset's usefulness as a resource.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PanDent, a dental panoramic radiography (OPG) benchmark of 9,524 cases with expert-validated tooth-level structured annotations, from which free-text radiology reports are generated via a clinician-designed template bank. The authors evaluate a wide range of general-domain and medical-domain MLLMs on report-level NLG metrics and attribute-level exact-match accuracy, report that current models produce fluent but clinically inconsistent reports, and show that fine-tuning Qwen-based models on PanDent substantially improves both linguistic and clinical accuracy. The central claim is that PanDent provides a rigorous benchmark and training resource for tooth-level structure-language consistency.","tokens_in":23824,"tokens_out":4134,"duration_ms":41935,"significance":"If the evaluation is valid, PanDent would be a valuable contribution: the dataset is large, expert-validated, publicly released with code, and the attribute-level exact-match protocol is a principled way to test fine-grained clinical grounding. The deterministic template-based report generation and explicit mapping between structured findings and text are also useful design elements. However, the empirical conclusions rest on several untested assumptions about test-set representativeness, the normalization pipeline, and the clinical meaningfulness of the report-level metrics. These issues are load-bearing because the paper's headline claims — that MLLMs fail at tooth-level consistency and that fine-tuning on PanDent fixes this — are only as strong as the benchmark's validity.","major_comments":[{"comment":"The test-set composition is almost certainly confounded by source. The dataset contains 9,019 public and 505 in-house cases, and the split is 9,024 train / 500 test, so the test set is effectively the entire in-house batch. Table 4 confirms large prevalence shifts between train and test (e.g., missing teeth 68.7%→88.4%, maxillary GCBL 30.8%→54.2%, crowns 17.7%→32.8%, mandibular angular bone loss 1.9%→8.0%, with rare findings at 0% in test). Thus zero-shot failures may reflect out-of-distribution acquisition/annotation style, and fine-tuning gains may reflect adaptation to a single clinic distribution and template format rather than generalizable clinical improvement. 'Disjoint from the training split' is insufficient. Please provide source-stratified results (public-only test, in-house-only test) or re-split so the test set is representative of the combined training distribution.","section":"Sec. 3, Table 4"},{"comment":"Several exact-match accuracies are implausibly low relative to trivial baselines. For binary regional attributes, a majority-class predictor on the reported test prevalences would achieve roughly 0.49–0.54 accuracy (e.g., maxillary GCBL prevalence 54.2%, mandibular 49.2%), yet GPT-5.4 scores 0.418/0.419 and MAIRA-2 scores near 0.10. Many tooth-index attributes also show near-zero accuracy for some models. These numbers suggest the LLM-based normalizer π_j may be systematically mis-mapping model outputs, or that the metric is computed incorrectly. The paper does not report the majority-class baseline, per-attribute positive/negative counts, or a manual analysis of normalization errors. Without this, the conclusion that MLLMs 'fail' at clinical consistency is not trustworthy.","section":"Table 2, Eq. (A.8)"},{"comment":"The 'structure-language consistency' evaluation is partially circular. Ground-truth free-text reports are generated from the same structured annotations using a template bank, and the normalized-report NLG metrics rewrite model outputs into the same template-aligned format before comparing against these template references. This largely measures template mimicry, not clinical understanding. The attribute-level exact-match protocol is anchored to expert-validated labels and is the valid clinical measure, but the paper's language-level 'clinical consistency' claims rely on the more circular NLG comparison. Please either de-emphasize the NLG-based consistency conclusions, or validate them against independent expert-written reports.","section":"Sec. 3.3, Sec. 4.2"},{"comment":"The highlighted attribute set used for the headline clinical accuracy does not include several important tooth-level findings, such as missing teeth, partially erupted/impacted teeth, furcation radiolucency, and angular bone loss, even though these are annotated and appear in the dataset. Rare findings have 0% prevalence in the test split, so they cannot be evaluated at all. Consequently, the claim of evaluating 'comprehensive tooth-level structure-language consistency' is overstated. Please either expand the evaluated attribute set or qualify the claim to reflect the attributes actually measured.","section":"Sec. 4.1, Table 2"}],"minor_comments":[{"comment":"There are typos and informal draft remnants: 'involvment' (Sec. 3.1), 'teet #16' (Table 3), and Appendix A contains the placeholder sentence 'The final version can directly replace the current draft with the exact implementation details...' This is not appropriate for a submission-ready manuscript.","section":"Throughout"},{"comment":"The manual audit of only 200 normalized reports is small relative to the 500-case test set, and the normalization prompt explicitly suppresses negative findings not stated in the input. Please report inter-auditor agreement and the rate of normalization-induced errors.","section":"Sec. 4.2"},{"comment":"No confidence intervals or significance tests are provided for the fine-tuning gains. With N=500, differences of a few points may be within noise; please report bootstrap intervals or at least per-attribute variances.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like a draft in places (e.g., the placeholder in Appendix A). More importantly, the test-set composition issue is severe enough that the current results cannot be interpreted as benchmarking generalizable dental MLLM ability; the authors should be asked to re-run with source-stratified or re-split evaluation. The dataset itself, however, has clear value, and the attribute-level evaluation protocol is a good idea if the normalization pipeline is validated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the resource is real, the benchmark is currently overclaimed. PanDent is a large OPG dataset — 9,524 images — with expert-validated, FDI-indexed tooth-level annotations across 21 findings, plus template-generated free-text reports. That combination is genuinely new for dental radiology, and the data/code are public. The paper also evaluates a broad set of proprietary, open, and medical MLLMs, and the attribute-level exact-match protocol is anchored to the expert labels, which is the right way to get past BLEU-style fluency scoring.\n\nWhere it gets soft: first, the test split. The corpus is 9,019 public plus 505 in-house cases, and the split is 9,024 train / 500 test. The test set is almost certainly the in-house batch. Table 4 makes the problem concrete: missing teeth 88.4% test vs 68.7% train, maxillary generalized crestal bone loss 54.2% vs 30.8%, crowns 32.8% vs 17.7%, and several rare findings at 0% in test. That is distribution shift, not a random held-out sample of the same population. The zero-shot failures and the fine-tuning gains could then reflect adaptation to one clinic's acquisition and annotation style rather than general clinical improvement. There is no source-stratified analysis. This is the load-bearing flaw for the 'rigorous benchmark' claim.\n\nSecond, the numbers in Table 2 do not behave. GPT-5.4's binary regional accuracies are around 0.42 on attributes whose test prevalence is roughly 50%; that is below the always-predict-majority baseline. That points to a broken normalization/exact-match pipeline, not to the model suddenly failing a binary question. Appendix A still says the final version will replace the current draft with exact implementation details, which is not acceptable for a benchmark paper. There are also no error bars anywhere, and some relative-improvement claims in the text do not line up with the table averages.\n\nThird, the report-level NLG part is partially circular: the references are generated from the same structured labels used for evaluation, and model outputs are rewritten by an LLM into the same template style. The fine-tuned models train on the same template style, so large normalized BLEU/ROUGE jumps are partly format mimicry. The attribute-level exact-match results, because they are anchored to the expert-validated labels, are the trustworthy part.\n\nVerdict: the dataset deserves serious peer review, and the authors should be pushed to fix the split and the evaluation pipeline before the benchmark claims are taken at face value. If they add a cross-source held-out analysis and repair the normalizer, this becomes a useful benchmark. I would cite it as a dataset resource, but not yet as a clean benchmark.","headline":"A genuinely useful dental OPG dataset with tooth-level structured annotations, but the benchmark test split and normalization pipeline need major repair before the headline claims hold.","tokens_in":24243,"tokens_out":4507,"would_cite":true,"duration_ms":46110,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current multimodal models can write fluent dental reports yet fail to tie findings to the correct teeth, and fine-tuning on the new PanDent benchmark substantially improves that structure-language consistency.","keywords":["dental panoramic radiography","orthopantomogram","multimodal large language models","structure-language consistency","tooth-level diagnosis","radiology report generation","clinical benchmark","fine-grained localization"],"falsifier":"Use the released code to reproduce the exact-match numbers on the 500-case split, then run the same zero-shot versus fine-tuned models on a fresh OPG cohort whose missing-tooth, bone-loss, and crown prevalences are matched to the training split; if the fine-tuning advantage collapses or reverses, the claimed clinical gain is a distribution artifact rather than genuine tooth-level grounding.","tokens_in":23339,"feed_emoji":"🦷","tokens_out":5506,"duration_ms":58196,"temperature":0.7,"pith_summary":"The paper seeks to establish that multimodal large language models—models that read dental panoramic X-rays and produce free-text reports—can sound fluent while making clinically consequential mistakes: they miss teeth that are absent, attribute findings to the wrong tooth numbers, and overlook restorations or bone loss. To make this failure measurable, it introduces PanDent, a set of 9,524 panoramic radiographs with expert-validated tooth-level annotations and template-generated reports that tie each sentence to specific findings. On a held-out 500-case set, the paper shows that fine-tuning general-purpose models on PanDent roughly doubles their average exact-match clinical accuracy, which still leaves tooth-index grounding as the weakest skill. A sympathetic reader would care because structure-language consistency is the missing evaluation axis for medical image-text models: fluency scores do not tell whether a model actually sees the disease at the correct anatomical location. The paper itself concedes that rare findings are long-tailed and unevenly covered.","feed_headline":"Fluent dental AI reports name the wrong teeth","feed_subtitle":"A 9,524-image benchmark with expert tooth-level labels shows fine-tuning lifts average clinical accuracy from 27% to 64% on held-out cases.","key_machinery":"The load-bearing object is the PanDent annotation-report pair, built from 21 normalized clinical findings, tooth-index-aware sets under FDI numbering, and binary regional findings. A clinician-defined template bank converts these structured findings into free-text reports with deterministic structure-language alignment; an LLM-assisted verifier and expert audit check consistency. Evaluation proceeds by prompting a model separately for each highlighted attribute and exact-matching its normalized answer against expert labels, which isolates visual localization and diagnostic correctness from language style.","core_discovery":"The central claim is that current MLLMs fail to produce clinically consistent tooth-level descriptions and that PanDent's structured annotations provide supervision that substantially fixes this. The authors argue that the gap is hidden by conventional report-level metrics: BLEU/ROUGE scores make models look adequate, while exact-match comparisons at the level of individual FDI tooth numbers reveal large errors. Their experiments on proprietary, open-source, and medical-domain models show the same pattern—fluency without grounding—and their fine-tuned models achieve the largest gains on the benchmark's clinical accuracy track. The paper also claims that this evaluation protocol, combining re","pith_inferences":["The template-generated reports may make fine-tuning gains look larger than they would on free-form clinical dictations; a clinician-reading study on unstructured reports would test whether the improvement is about radiological content or about mimicking template style.","The large prevalence shifts between train and test (for example, missing teeth in 88.4% of test cases versus 68.7% of training cases) suggest that part of the measured gain could be a domain-shift artifact; evaluating on prevalence-matched cohorts would separate benchmark-specific overfitting from genuine tooth-level understanding.","A natural next step would be to add location-aware supervision, such as detection heads or grounding tokens at the tooth level, since even fine-tuned models still fail at exact FDI indexing."],"forward_implications":["If PanDent is right, fine-tuning general-domain multimodal models on it yields substantial gains in both report-level similarity and exact-match clinical accuracy.","Fluency metrics alone are not a reliable proxy for clinical quality; attribute-level exact match is the limiting evaluation.","Medical-domain models trained on other modalities (for example, chest X-ray) do not transfer well to dental panoramic imaging, performing worse than general-purpose models.","Tooth-index grounding remains the hardest failure mode even after fine-tuning, so progress will require domain-specific visual grounding methods.","The dual-track evaluation (linguistic coherence plus structural factuality) can be replicated for other anatomy where findings must be localized to specific structures."],"fun_headline_variants":["From 27% to 64%: dental AI gets better with expert tooth labels","Dental AI reports often name wrong teeth; PanDent fixes that with 9,524 scans","Tooth-level benchmark reveals: MLLMs are fluent but not clinically consistent","Fluency without grounding: PanDent tracks dental MLLMs to the tooth"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 500-case held-out test set fairly represents the same clinical distribution as the 9,024 training images; the reported prevalence shifts (missing teeth 88.4% vs 68.7%; maxillary bone loss 54.2% vs 30.8%) raise the possibility that fine-tuning gains come from domain shift rather than tooth-level understanding.","fun_headline_variants_meta":{"raw":{"variants":["From 27% to 64%: dental AI gets better with expert tooth labels","Dental AI reports often name wrong teeth; PanDent fixes that with 9,524 scans","Tooth-level benchmark reveals: MLLMs are fluent but not clinically consistent","Fluency without grounding: PanDent tracks dental MLLMs to the tooth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001129,"raw_usage":{"total_tokens":4560,"prompt_tokens":802,"completion_tokens":3758,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":3667}},"tokens_in":546,"tokens_out":3758,"duration_ms":27511,"temperature":1.0,"reasoning_tokens":3667,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T08:26:00.070330+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the released code to reproduce the exact-match numbers on the 500-case split, then run the same zero-shot versus fine-tuned models on a fresh OPG cohort whose missing-tooth, bone-loss, and crown prevalences are matched to the training split; if the fine-tuning advantage collapses or reverses, the claimed clinical gain is a distribution artifact rather than genuine tooth-level grounding.","supporting_citations":[],"review_version":1}