{"id":"59091aa3-408f-482d-9df3-99741381d679","arxiv_id":"2411.15666","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"An ontology-guided constrained decoding method produces specialty-specific clinical summaries and lowers hallucination scores on MIMIC-III relative to greedy and beam search baselines.","lead":"This paper describes a method that uses a medical ontology to guide language models when summarizing electronic health records, producing summaries tailored to specialties like nursing, radiology, or cardiology. It also constrains the model's step-by-step generation to reduce hallucinated or ungrounded statements, and evaluates the approach on the MIMIC-III intensive care dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hallucination-reduction evidence is circular: MedCAT steers decoding and scores the outcome; independent annotation check needed.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern I also find: the MedCAT annotator is used both to guide decoding and to compute the hallucination metrics, creating a circularity that could invalidate the central empirical claim. I agree that this warrants a conditional verdict rather than strong acceptance. I do not see a need to change the reader's verdict because the concern is testable and the paper has other independent support (reproducible code, clear methodology, plausible mechanism); the right response is to require the proposed check before accepting the hallucination-reduction claim. I considered also raising the lack of ablations separating the ontology scores (H, P) from the ROUGE similarity score (S), and the domain-score evaluator's potential circularity with the DCF pruning, but the MedCAT circularity is the most decisive because it affects both the mechanism and the measurement. The concrete test I propose would settle the concern directly: if an independent annotator reproduces the gains, the claim survives; if not, the reported improvements are likely evaluator artifacts. This is a good-faith objection: the authors themselves flag the annotator dependence and absence of human evaluation in §6.4, so the test is aligned with their stated limitations and would strengthen the paper regardless of outcome.","tokens_in":13789,"tokens_out":2394,"duration_ms":22701,"concrete_test":"Re-run the hallucination and groundedness evaluations (Table 2 and the HS/AHS columns of Table 3) on a random sample of 100 admissions from Ω, replacing MedCAT with an independent SNOMED-CT annotator (e.g., QuickUMLS or cTAKES), or on a clinician-verified gold standard of concepts for a subset. If the constrained-generation advantage over greedy search collapses or becomes statistically insignificant (bootstrap 95% CI includes zero), the hallucination-reduction claim is an artifact of annotator circularity rather than a property of the decoding procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim, that ontology-guided decoding reduces hallucinations, rests on the reliability and neutrality of the MedCAT annotator. The same annotator is used twice: in §3.3 MedCAT tags each beam to compute the hierarchy and property scores that steer decoding, and in §5.3 MedCAT tags the source notes, reference summaries, and generated summaries to compute HS and AHS (and the groundedness scores in Table 2). If MedCAT is noisy or biased toward the phrasings that the constrained decoder is rewarded to produce, then the reported improvements (e.g., groundedness 0.90 vs 0.83, HS 37.95 vs 45.70 for Phi-3) could reflect annotator alignment rather than true faithfulness. The authors explicitly concede in §6.4 that the approach 'heavily relies on a good annotator' and that the automated NLI/annotation evaluation 'may not be always trustworthy.' A further confounder is the similarity score S (ROUGE-2 to the note, Sbf=10), which alone could improve groundedness; without ablations, the ontology-specific contribution is unidentified. A single independent re-annotation study would settle whether the measured hallucination reduction is real or an artifact of annotator circularity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an ontology-guided constrained decoding method for generating domain-specific clinical summaries from electronic health records. The approach uses SNOMED-CT and the MedCAT annotator to tag clinical notes, builds a Domain-Class-Frequency (DCF) dictionary from MIMIC-III CATEGORY labels to prune extracted structured representations to a target domain, and steers beam search with hierarchy, property, and lexical-similarity scores. Experiments on MIMIC-III report large improvements in a learned domain score (e.g., Phi-3 from 0.41 to 0.86), modest improvements in NLI-based groundedness and relevance, and reductions in hallucination scores for the BHC summarization task. The authors acknowledge limitations including computational overhead, hyperparameter sensitivity, reliance on a good annotator, and the absence of human assessment.","tokens_in":13971,"tokens_out":4094,"duration_ms":35376,"significance":"If the reported results are trustworthy, the paper contributes a novel use of ontologies for constrained decoding that is model-agnostic and produces interpretable, structured summaries. The ability to adapt summaries to different medical specialties without prompt engineering is practically valuable. The paper is also transparent about several limitations. However, the current evidence is weakened by evaluation circularity: the domain score metric is trained on the same MIMIC-III CATEGORY labels that define the DCF pruning signal, and the hallucination metrics use the same MedCAT annotator that steers decoding. The lack of independent validation (manual or with a different annotator) makes the central claims about domain adaptation and hallucination reduction not yet firmly established.","major_comments":[{"comment":"The domain score evaluation is partly circular. The DCF dictionary is constructed by annotating MIMIC-III notes with MedCAT and using the CATEGORY column to group notes by domain, while the evaluator is a BERT classifier trained on the same CATEGORY labels. The pruning step directly uses the DCF classes, so the method is rewarded for producing summaries that match the evaluator's training signal. The large improvement in Table 1 (e.g., 0.41 to 0.86 for Phi-3) may therefore be inflated. The authors should either use a human evaluation, an evaluator trained on independent labels, or an ablation that removes the overlap between the DCF construction and the evaluator training data to show that the domain score reflects genuine domain adaptation.","section":"§3.1 and §5.1, Table 1"},{"comment":"The same MedCAT annotator is used both to compute the hierarchy and property scores that steer decoding (Section 3.3) and to compute the hallucination scores HS and AHS (Section 5.3) and the groundedness evaluation (Section 5.2). If MedCAT is noisy, biased toward certain phrasings, or imperfect in the target domains, the reported hallucination reduction (e.g., HS from 45.70 to 37.95 for Phi-3 in Table 3) could reflect annotator alignment rather than true clinical faithfulness. The authors' own concession in Section 6.4 that automated evaluation 'may not be always trustworthy' applies directly to these metrics. An independent annotation study, ideally with clinician involvement or at least a second, different concept annotator, is needed to validate the hallucination-reduction claim.","section":"§3.3 and §5.3 (also §5.2)"},{"comment":"The contribution of the ontology-based scores is not isolated. The similarity score S (ROUGE-2 to the clinical note, with Sbf = 10) alone could improve groundedness by favoring beams that lexically resemble the note. Without an ablation that varies the presence of H, P, and S, it is unclear whether the observed groundedness improvements in Table 2 are due to the ontology constraints or simply to the lexical-similarity term. The authors should report results with S only, H+P only, and the full combination.","section":"§3.3, Table 2"},{"comment":"The groundedness evaluation relies on a DEBERTA model fine-tuned on MNLI, which is not trained on medical data. The authors note this in Section 6.4, but the claim in Section 6.2 that 'summaries generated using our custom ontology-guided decoding process are more likely to be factually consistent' is not supported by a medically validated NLI model. Additionally, no confidence intervals or significance tests are reported; with 1000 samples, the difference between 0.83 and 0.90 for Phi-3 groundedness may or may not be statistically reliable. Please report error bars or a significance test.","section":"§5.2, Table 2"},{"comment":"The hallucination scores HS and AHS depend on the sets of concepts extracted by MedCAT from the notes and the generated summary. Beyond the circularity concern, this metric can be sensitive to annotator precision and recall in both the source and the summary. The adjusted score AHS uses the reference summary to allow inferred concepts, but it still relies on the same annotator. The authors should provide evidence that MedCAT's annotations are reliable in this setting, or supplement the automated scores with human evaluation, before claiming that the method reduces hallucinations.","section":"§5.3, Eqs. (2)-(3), Table 3"}],"minor_comments":[{"comment":"There are typos: 'is is not surprising' in Section 6.3 and 'highly sensible to hyperparameters' in Section 6.4 should be 'highly sensitive'.","section":"§6.3 and §6.4"},{"comment":"The notation is confusing: N is used for the number of samples in Eq. (1) and for the set of concepts in the notes in Eqs. (2)-(3). Please use distinct symbols to avoid ambiguity.","section":"§5.3, Eqs. (2)-(3)"},{"comment":"The figure captions are incomplete and do not explain all the abbreviations and arrows (e.g., what 'CSR' stands for in Figure 3 is defined, but the captions for Figures 1 and 2 could be more explicit).","section":"Figures 1 and 2"},{"comment":"Some references contain malformed publisher fields, e.g., references [2] and [16] include '[object Object]' in the publisher entry. These should be cleaned up.","section":"References"},{"comment":"The claim that this is the 'first approach that employs ontologies in conjunction with LLMs to constrain generation' is strong. Given existing work on knowledge-graph-constrained decoding (e.g., reference [6]), it would be safer to phrase this as 'to the best of our knowledge' and to clarify the distinction between ontologies and knowledge graphs.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the idea of using ontology-guided decoding to produce domain-adapted summaries is creative. However, the evaluation design has a central circularity issue: the same MIMIC-III CATEGORY labels and the same MedCAT annotator are used both for steering the method and for scoring the results. This is a load-bearing problem for the two main claims (domain adaptation and hallucination reduction). The authors already acknowledge many limitations, but the manuscript would need a substantial revision with independent evaluation (e.g., human assessment, a second annotator, or at least ablations that break the circularity) before the claims can be accepted. I recommend major revision rather than rejection because the core method is plausible and the paper is transparent about its weaknesses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely new: using SNOMED-CT structure (hierarchy, restriction properties) to score beams during decoding, then pruning with a domain-class-frequency dictionary built from MIMIC-III categories. That combination is not in the prior constrained-generation work they cite, which mostly uses grammars, FSMs, or knowledge graphs. The method is also clearly described and the code is public. Second, the evidence for the headline claim, that this reduces hallucinations, is weaker than the abstract suggests. The same MedCAT annotator both steers decoding (§3.3) and computes the hallucination scores HS/AHS (§5.3), and the NLI model used for groundedness was not trained on medical text. The domain score is similarly close to circular: the evaluator is trained on the same CATEGORY labels that define the DCF pruning signal, so the 0.86 vs 0.41 jump partly reflects the model being told which domain to match. The authors themselves concede most of this in Section 6.4, which is to their credit, but the concession doesn't fix the inference. No error bars, significance tests, or ablations are reported, so the similarity-score component alone could be driving some of the groundedness gain. That said, the method is plausible and the components are sensible. The extraction-plus-pruning pipeline is modular, and the domain-adaptation idea is neat even if the evaluation is self-referential. The improvements are modest (3–7% groundedness, 0.2–0.4 domain score) and the BHC summarization results are below fine-tuned baselines, which they honestly report. The central contribution is a recipe for ontology-constrained decoding, not a demonstrated clinical win. The paper deserves a serious referee because the method is new and the limitations are stated rather than hidden. But it needs major revision: independent annotation or gold-standard concepts for hallucination metrics, human expert evaluation, ablations on Hbf/Pbf/Sbf and the pruning hyperparameters, and confidence intervals. Without those, the empirical claims should stay conditional. Who gets value? Researchers working on constrained decoding for structured generation, or clinical summarization with ontologies. I'd bring it to reading group and would probably cite it if I worked on decoding-time constraints, but I'd not rely on its numbers. Send it to peer review, but expect heavy revision. If a revision adds a clean independent evaluation, this could be a solid contribution.","headline":"Novel ontology-guided beam scoring for clinical summarization; promising pipeline, but the headline evaluation is partly circular and needs a serious referee before strong claims.","tokens_in":14572,"tokens_out":1492,"would_cite":true,"duration_ms":16070,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ontology-guided decoding makes clinical summaries more domain-specific and less hallucinated.","keywords":["clinical summarization","ontology-guided decoding","hallucination reduction","SNOMED-CT","constrained generation","domain adaptation","electronic health records","beam search"],"falsifier":"Re-run the generation on the same test set with MedCAT replaced by an independent clinical concept annotator; if the domain-score and hallucination-score gaps between constrained and greedy decoding shrink to near zero, the reported improvements are an artifact of the scorer and evaluator sharing the same annotator.","tokens_in":13496,"feed_emoji":"🩺","tokens_out":5498,"duration_ms":49176,"temperature":0.7,"pith_summary":"This paper sets out to show that a medical ontology can be used at decoding time to make LLM-generated clinical summaries more relevant to a chosen specialty and less likely to invent information. It proposes generating structured concept-level extractions from electronic health records with ontology-guided constrained decoding, then pruning those extractions to a domain and verbalizing them. On MIMIC-III the method raises the domain score for Phi-3 from 0.41 with greedy search to 0.86, and raises groundedness from 0.83 to 0.90, with similar but smaller gains for Zephyr. The hope is that clinicians can receive tailored, queryable summaries without fine-tuning a model per specialty.","feed_headline":"Ontology-steered decoding cuts clinical summary hallucinations","feed_subtitle":"Constraining beam search to SNOMED-CT concepts doubles Phi-3's domain score and lifts groundedness from 0.83 to 0.90.","key_machinery":"The engine is an ontology-guided beam search. After each generation window of ten tokens, the newly generated text is tagged with the same MedCAT annotator used on the source notes, and each beam receives a score with three components: a hierarchy score rewarding concepts that are descendants of the prompt's base ontology class, a property score rewarding mentions of classes linked to the base class by restriction properties, and a similarity score measuring ROUGE-2 overlap with the original note. The final beam score is LogSoftmax(H + P + S). A Domain-Class-Frequency dictionary, built by annotating domain texts and normalizing across domains, then drives the pruning step that selects which extracted concepts appear in the final summary.","core_discovery":"The paper's central claim is that constraining an LLM's generation to align with ontology concepts and relationships reduces the likelihood of generating information that contradicts established domain knowledge, while also making the output more relevant to a target clinical domain. The evidence comes from MIMIC-III: the ontology-constrained method more than doubles the domain score for Phi-3 compared with greedy search (0.86 versus 0.41), improves groundedness from 0.83 to 0.90, and lowers the hallucination score on the discharge-summary task. The same pipeline yields both a structured, queryable representation of each clinical note and an unstructured summary, and the pruning step lets one extraction be adapted to multiple specialties without regenerating.","pith_inferences":["Inference: The measured gains likely depend on how consistently MedCAT tags both the source notes and the generated beams; if the annotator is biased toward the model's preferred phrasing, the reported hallucination reduction could be inflated.","Inference: Because the approach only needs token probabilities and an ontology with an annotator, it could transfer to other structured knowledge domains, provided a reliable concept annotator exists.","Inference: Decoupling extraction from adaptation suggests a practical deployment pattern in which each admission is summarized once and then filtered per specialty, reducing redundant computation at the cost of a heavy one-time extraction.","Inference: The main barrier to clinical use is computational overhead from many inference passes and beam search; a testable extension would reduce the number of generation windows or cache beam scores across notes."],"forward_implications":["Domain-specific summaries can be produced without explicitly prompting the model for a specialty; the pruning step performs the adaptation using only the ontology and the extracted concepts.","Hallucination scores on the Brief Hospital Course task drop for Phi-3, indicating that structured extraction plus ontology-based beam scoring is a viable faithfulness lever for smaller models.","The same extraction pass can feed multiple domains, since adaptation happens after extraction by pruning with the domain dictionary.","The method works with any LLM that exposes token probabilities, because it operates entirely at decoding time rather than requiring fine-tuning.","Treating the Brief Hospital Course section as a domain shows that the notion of a domain can extend beyond note categories to arbitrary target summaries."],"supporting_citations":[{"why":"Supplies the MIMIC-III clinical notes and Brief Hospital Course reference summaries used in all experiments.","marker":"[14]"},{"why":"The MedCAT annotator maps both source notes and generated beams to SNOMED-CT concepts, feeding the decoding scores and the hallucination metrics.","marker":"[15]"},{"why":"SNOMED-CT provides the class hierarchy and restriction properties used to define domains and score beams.","marker":"[25]"},{"why":"Diverse beam search is the base algorithm that the ontology-guided constrained decoding extends.","marker":"[31]"},{"why":"Defines the discharge-summary hospital-course summarization task and provides the dual Transformer baseline.","marker":"[22]"},{"why":"Provides the SPEER baseline and the hallucination-score style used for the BHC comparison.","marker":"[2]"},{"why":"The DeBERTa NLI model computes groundedness and relevance scores in the hallucination evaluation.","marker":"[13]"},{"why":"The element-aware summarization prompting adapted by the paper with ontology-guided prompts.","marker":"[32]"},{"why":"The RAG-style prompt augmentation used to inject ontology restriction properties into the prompt.","marker":"[16]"}],"fun_headline_variants":["Ontology steering curbs hallucinations in clinical notes","Domain-adapted clinical summaries with fewer hallucinations","Ontology-constrained generation grounds clinical summaries","Ontology guidance doubles clinical domain score in notes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline and its evaluation assume that MedCAT reliably maps clinical text to SNOMED-CT concepts in both the source notes and the generated beams.","fun_headline_variants_meta":{"raw":{"variants":["Ontology steering curbs hallucinations in clinical notes","Domain-adapted clinical summaries with fewer hallucinations","Ontology-constrained generation grounds clinical summaries","Ontology guidance doubles clinical domain score in notes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1423,"prompt_tokens":823,"completion_tokens":600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":543}},"tokens_in":439,"tokens_out":600,"duration_ms":5747,"temperature":1.0,"reasoning_tokens":543,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:02:47.617775+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the generation on the same test set with MedCAT replaced by an independent clinical concept annotator; if the domain-score and hallucination-score gaps between constrained and greedy decoding shrink to near zero, the reported improvements are an artifact of the scorer and evaluator sharing the same annotator.","supporting_citations":[{"cited_title":"Scientific Data3(1), 160035 (May 2016).https://doi.org/10.1038/ sdata.2016.35, https://www.nature.com/articles/sdata201635","cited_arxiv_id":null,"evidence_quote":"Supplies the MIMIC-III clinical notes and Brief Hospital Course reference summaries used in all experiments."},{"cited_title":"Proc AMIA Symp pp","cited_arxiv_id":null,"evidence_quote":"SNOMED-CT provides the class hierarchy and restriction properties used to define domains and score beams."},{"cited_title":"In: International Conference on Learning Representations (2021), https:// openreview.net/forum?id=XPZIaotutsD","cited_arxiv_id":null,"evidence_quote":"The DeBERTa NLI model computes groundedness and relevance scores in the hallucination evaluation."},{"cited_title":"Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method","cited_arxiv_id":"2305.13412","evidence_quote":"The element-aware summarization prompting adapted by the paper with ontology-guided prompts."}],"review_version":1}