REVIEW 3 major objections 6 minor 29 references
Closed-loop validation-repair lifts clinical LLM schema compliance from roughly 89% to 99% across three open-source models.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Closed-loop validation-repair lifts schema compliance of three 7–9B clinical LLMs from ~89% to 99% on 320 synthetic scenarios, with failures mostly format-level.
T0 review reviewed 2026-07-31 challenge →
load-bearing objection Solid paired multi-model evidence that format noncompliance is cross-vendor and mostly fixable by a cheap validate-repair loop—useful ops result, oversold as deployment readiness. the 3 major comments →
Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
Across 960 model–scenario pairs, schema noncompliance is consistent across three vendors (baseline 85.9–91.6%), 96% of validator-detected failures are representation-level format or enumeration violations, and a closed-loop validation-repair procedure raises overall compliance to 99.0% (98.4–99.4% per model) with statistically significant paired gains of 7.8 to 12.5 percentage points and most errors resolved by the second iteration.
What carries the argument
Closed-loop validation-repair: a deterministic field-level validator (presence, type, enum, regex format, range, safety flags) acts as a compliance sensor; detected errors become a targeted repair prompt that drives the same model to regenerate until compliance or a small iteration cap (K_max=3).
Load-bearing premise
That success on a study-defined format schema over 320 guideline-style vignettes—not live EHR notes, full terminology checks, or clinical correctness—is a fair stand-in for readiness to plug into real clinical systems.
What would settle it
Run the same paired baseline versus validation-repair protocol on real de-identified EHR note extracts with institutional FHIR/ICD/CPT validators that also check code existence and profile rules; if cross-vendor baseline gaps shrink sharply or repair fails to reach high-90s compliance, the deployment claim does not hold.
If this is right
- Healthcare AI stacks can add validation-repair as vendor-independent middleware instead of picking models mainly for format compliance.
- Most residual risk is representation mismatch (BID vs BD, bare CPT digits vs prefixed codes), so IT standards docs matter as much as more clinical text in training.
- Smaller local models (around 7B) can match or beat larger peers on post-repair compliance, supporting privacy-preserving on-prem deployment.
- Bounded repair (about 1.1–1.2 model calls per case on average) is cheap enough to sit in the critical path before billing or EHR write-back.
- Unresolved ~1% cases need an explicit human-review fallback rather than silent acceptance.
Where Pith is reading between the lines
- If format gaps are corpus-wide, fine-tuning on HL7/ICD/CPT style guides plus repair traces may cut first-pass failures more than scaling clinical QA data alone.
- The same sensor-controller split could transfer to other regulated structured outputs (claims, prior auth, lab order sets) where checkers already exist but generators do not.
- Pairing this schema loop with separate clinical-correctness and hallucination checks would be the natural next stack layer the paper deliberately leaves open.
- Institutions might version schemas as living contracts and measure model drift by Valid@K curves rather than only by diagnostic accuracy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates schema compliance of three locally deployed open-weight LLMs (Qwen2.5 7B, Llama 3.1 8B, Gemma2 9B) on 320 guideline-grounded clinical vignettes spanning ten specialties (960 model–scenario pairs), under a paired design: a strong few-shot baseline with explicit schema specification versus a closed-loop validation-repair loop (deterministic field-level validator, targeted repair prompts, K_max=3). It reports (i) cross-vendor baseline noncompliance of 8.4–14.1%, (ii) a taxonomy in which 96% of validator-detected failures are ENUM/FORMAT violations, and (iii) post-repair compliance of 99.0% overall with zero regressions, exact McNemar p < 1e-7, bootstrap CIs, and convergence within two iterations at ~1.1–1.2 LLM calls per scenario. The authors are explicit that the validator checks format/enum/range only — not terminology existence, full FHIR profile conformance, or clinical correctness — and scope their conclusions accordingly in §VI.D.
Significance. If the results hold, the paper provides a clean, well-instrumented demonstration that small open-weight models systematically violate simple healthcare interoperability constraints even under strong prompting, and that a cheap deterministic feedback loop closes most of the gap. Strengths worth naming: the paired design with temperature-0 determinism; exact McNemar tests with scenario-level bootstrap CIs; zero regressions across all three models; internally consistent counts across Tables I–IV (I verified the discordant pairs, percentages, and per-iteration convergence counts); a transparent residual-failure analysis (the mg/kg/hour case); and unusually candid scoping of the validator's coverage in §III.A and §VI.D. The cross-vendor consistency finding, while limited to three 7–9B models, is a useful empirical datapoint for the field, and the work is a reasonable complement to constrained-decoding and FHIR-mapping literature.
major comments (3)
- [§II.C, §VI.B] The paper positions itself against constrained/grammar-guided decoding (citing [8], [9], [29]) but never evaluates it. Every observed failure class in Table II — INVALID ENUM, INVALID FORMAT, TYPE MISMATCH, OUT OF RANGE — is expressible as a JSON-schema/grammar constraint, so constrained decoding would plausibly achieve ~100% compliance at exactly one call, versus 1.11–1.20 calls and 98.4–99.4% here. The recommendation in §VI.A/§VII that validation-repair be integrated 'as a standard middleware layer' therefore rests on an unmade comparison against the obvious deterministic alternative. Either an experimental baseline (e.g., grammar-constrained decoding on the same three models) or an explicit argument for what repair buys that constrained decoding cannot (e.g., repair of semantically motivated violations, applicability to black-box models without logit access) is needed to support the s
- [§V.B, §VI.C] The finding that '96% of failures are representation-level' is partly bounded by instrument coverage: Algorithm 2 checks presence, type, enum, regex, and range only, and §III.A states terminology-level existence is not verified. A well-formed but nonexistent ICD-10 code, or a wrong-but-in-range dosage, passes silently. The paper partially acknowledges this ('96 percent of validator-detected failures', §V.B), but §VI.C then infers shared properties of medical training corpora ('teaching models that BD and BID are interchangeable') — a causal claim about training data that the instrument cannot see and three models cannot establish. The taxonomy should be framed consistently as characterizing detectable failure modes, and §VI.C's training-corpora inference softened to a hypothesis, or supported with a terminology-existence probe on the compliant outputs.
- [§III.A–B, §V.C] Two schema design choices undermine the 'safeguard' framing. (1) Safety flags (allergy checking, drug-interaction screening) are enforced as S.enum[f] = {true}: a model that outputs false is 'repaired' to true without any underlying check being performed, so compliance on these fields certifies the flag value, not the safeguard. (2) The unit enum {mg, ml, g, mcg, units} excludes clinically standard composite units; the residual mg/kg/hour case (§V.C) shows repair pressure toward content-altering conversions, and no clinical-correctness check guards the repair step ('preserving clinical accuracy' is a prompt instruction, not a verified property). For the deployment-readiness claim, the paper needs at minimum a content-preservation audit showing repairs change only representation (BD→BID, 'CPT 74140'→74140) and never clinical content, plus a redesign or honest treatment of the safety-flag
minor comments (6)
- [§I, §V.C] The 98% 'deployment-oriented' threshold (RQ3, §V.C) is introduced without motivation; either justify it from downstream requirements or drop the threshold framing.
- [§IV] No statement on release of the 320-scenario benchmark, prompts, validator code, or raw outputs. Given the temperature-0 determinism claim and that the benchmark is author-constructed without expert review (disclosed in §IV.A), artifact availability is important for independent verification; please add an explicit availability statement.
- [§V.D, Fig. 3] Figure 3 reports baseline per-specialty rates but the post-repair per-specialty results are 'not shown'; since the ≥95% claim across all 30 model–specialty cells is load-bearing for §V.D, include the post-repair matrix (at least in an appendix).
- [§V.C, Table III] Zero regressions (Table III) is a net-scenario result; per-field churn is not reported. It would strengthen §V.C to state whether repair ever introduced a new violation in a previously compliant field that was itself later corrected.
- [§III, §VI.A] The cybernetic framing (sensor/controller layers, Fig. 4, Wiener citation) adds rhetorical weight without technical content beyond the loop in Algorithm 1; consider trimming.
- [§V.C, Table IV] K_max=3 is used but 'no additional scenarios converge at the third iteration' (Fig. 2); the cap discussion is fine, but Table IV would be clearer with an explicit K=3 column of zeros.
Circularity Check
No circular derivation: empirical before/after compliance against an external deterministic schema, not a self-forced prediction.
full rationale
This is a multi-model systems evaluation, not a first-principles derivation. Baseline compliance (85.9–91.6%), the error taxonomy (Table II), Valid@K convergence (Table IV / Fig. 2), and McNemar-tested gains (+7.8 to +12.5 pp to 99.0% overall) are measured outcomes of running three LLMs on 320 scenarios under paired conditions. The study-defined validator (Algorithm 2) is an external scoring instrument; repair uses its field-level error set as feedback, which is ordinary closed-loop design, not reduction of a claimed prediction to its fitted inputs. Residual failures after K_max=3 (10 pairs) show success is not forced by construction. The observation that 96% of detected failures are INVALID ENUM/FORMAT is an empirical count over the four observed categories the validator can emit (including TYPE MISMATCH and OUT OF RANGE, which were rare), not a definitional identity. Scope limits of the validator (format/enum only; no terminology existence or clinical correctness) affect external validity of deployment claims, not circularity of the derivation chain. The sole author self-citation [21] is peripheral (RAG hallucination) and not load-bearing. No fitted constants renamed as predictions, no uniqueness-by-self-citation, no smuggled ansatz. Score 0; steps empty.
Axiom & Free-Parameter Ledger
free parameters (4)
- K_max repair iterations =
3
- Study schema enumerations and patterns =
study-defined enums/regexes (not full FHIR/terminology)
- Benchmark size and specialty mix =
320 scenarios; e.g. Cardiology 48 (15%)
- Strong-baseline prompt design =
2 format examples; T=0
axioms (5)
- domain assumption Deterministic field-level checks on code format, enums, types, ranges, and required safety flags are a meaningful interoperability gate even without terminology-server code existence checks or full FHIR profile validation.
- domain assumption Guideline-grounded structured case vignettes sufficiently stress schema compliance behavior for conclusions about downstream clinical system integration readiness.
- domain assumption Schema compliance can be evaluated independently of clinical correctness; representation-level fixes preserve usable clinical content.
- standard math McNemar paired tests on binary compliance before/after repair are the appropriate significance test for the primary improvement claim.
- ad hoc to paper Cross-vendor similarity of baseline noncompliance implies shared medical-corpus gaps rather than architecture-specific defects.
invented entities (2)
-
Study-defined FHIR-oriented interoperability schema (combined ICD-10/CPT/frequency/unit/safety-flag constraints)
no independent evidence
-
Closed-loop validation-repair middleware framed as cybernetic sensor-controller layers
no independent evidence
Cite this review
Pith. "Pith review of Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs." pith.science (2026). https://pith.science/paper/D2B2PN53
@misc{pith2026260724371,
author = {Pith},
title = {Pith review of: Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/D2B2PN53}},
note = {Machine review of arXiv:2607.24371}
}
read the original abstract
Healthcare interoperability requires AI systems to produce structured outputs conforming to standardized schemas including ICD-10 for diagnostic coding, CPT for procedure billing, and HL7 FHIR for data exchange. While large language models demonstrate clinical reasoning capabilities, their integration into electronic health record systems faces a critical barrier: schema noncompliance. We evaluate three open-source models, Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B, via local deployment across 320 clinical scenarios spanning ten medical specialties, yielding 960 model-scenario pairs assessed under paired baseline and validation-repair conditions. First, schema noncompliance is consistent across the three model families, with baseline compliance rates ranging from 85.9 to 91.6 percent despite varying architectures and training data, suggesting shared gaps in medical training corpora rather than model-specific limitations. Second, 96 percent of validator-detected failures are representation-level format violations such as alternative medical abbreviations and code prefixes, indicating models follow clinical writing conventions but lack awareness of healthcare IT standards. Third, the validation-repair framework achieves 99.0 percent overall compliance, ranging from 98.4 to 99.4 percent across models, with most errors resolving within one or two iterations. Exact McNemar p-values below 0.001 and absolute improvements of 7.8 to 12.5 percentage points across model sizes confirm statistical significance. These results support closed-loop validation-repair as an effective system-level safeguard for healthcare interoperability, improving schema-level readiness for downstream clinical system integration.
Figures
Reference graph
Works this paper leans on
-
[1]
Large language models encode clinical knowledge,
K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, et al., “Large language models encode clinical knowledge,”Nature, vol. 620, pp. 172–180, 2023
2023
-
[2]
Towards expert-level medical question answering with large language models,
K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, et al., “Towards expert-level medical question answering with large language models,”arXiv preprint arXiv:2305.09617, 2023
Pith/arXiv arXiv 2023
-
[3]
Capabilities of GPT-4 on medical challenge problems,
H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of GPT-4 on medical challenge problems,”arXiv preprint arXiv:2303.13375, 2023
Pith/arXiv arXiv 2023
-
[4]
HL7 FHIR Release 4,
HL7 International, “HL7 FHIR Release 4,” Health Level Seven Inter- national, 2019
2019
-
[5]
International Statistical Classification of Diseases and Related Health Problems, 10th Revision,
World Health Organization, “International Statistical Classification of Diseases and Related Health Problems, 10th Revision,” 2016
2016
-
[6]
Current Procedural Terminology,
American Medical Association, “Current Procedural Terminology,” Chicago, IL: American Medical Association, 2023
2023
-
[7]
No small change for the health information economy,
K. D. Mandl and I. S. Kohane, “No small change for the health information economy,”New England Journal of Medicine, vol. 360, no. 13, pp. 1278–1281, Mar. 2009
2009
-
[8]
Lexically constrained decoding for sequence generation using grid beam search,
C. Hokamp and Q. Liu, “Lexically constrained decoding for sequence generation using grid beam search,” inProc. 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 1535–1546
2017
-
[9]
PICARD: Parsing in- crementally for constrained auto-regressive decoding from language models,
T. Scholak, N. Schucher, and D. Bahdanau, “PICARD: Parsing in- crementally for constrained auto-regressive decoding from language models,” inProc. 2021 Conference on Empirical Methods in Natural Language Processing, 2021
2021
-
[10]
Reflexion: Language agents with verbal reinforcement learning,
N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” in Proc. NeurIPS, 2023
2023
-
[11]
Self-refine: Iterative refinement with self-feedback,
A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, et al., “Self-refine: Iterative refinement with self-feedback,” inProc. NeurIPS, 2023
2023
-
[12]
Constitutional AI: Harmlessness from AI feedback,
Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, et al., “Constitutional AI: Harmlessness from AI feedback,”arXiv preprint arXiv:2212.08073, 2022
Pith/arXiv arXiv 2022
-
[13]
Loscalzo, A
J. Loscalzo, A. S. Fauci, D. L. Kasper, S. Hauser, D. Longo, and J. L. Jameson,Harrison’s Principles of Internal Medicine, 21st ed. McGraw-Hill Education, 2022
2022
-
[14]
Feather, D
A. Feather, D. Randall, and M. Waterhouse, eds.,Kumar and Clark’s Clinical Medicine, 10th ed. Elsevier, 2020
2020
-
[15]
National Ambulatory Medical Care Survey: 2019 National Summary Tables,
L. Santo and K. Kang, “National Ambulatory Medical Care Survey: 2019 National Summary Tables,” National Center for Health Statistics, 2023
2019
-
[16]
ClinicalBERT: Modeling clinical notes and predicting hospital readmission,
K. Huang, J. Altosaar, and R. Ranganath, “ClinicalBERT: Modeling clinical notes and predicting hospital readmission,”arXiv preprint arXiv:1904.05342, 2019
Pith/arXiv arXiv 1904
-
[17]
Wright, D
A. Wright, D. F. Sittig, J. S. Ash, J. Feblowitz, S. Meltzer, C. McMullen, K. Guappone, J. Carpenter, J. Richardson, L. Simonaitis, R. S. Evans, W. P. Nichol, and B. Middleton, “Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record sys...
2011
-
[18]
E. S. Berner, ed.,Clinical Decision Support Systems: Theory and Practice, 3rd ed. Cham, Switzerland: Springer, 2016
2016
-
[19]
Elec- tronic health records, medical research, and the Tower of Babel,
R. D. Kush, E. Helton, F. W. Rockhold, and C. D. Hardison, “Elec- tronic health records, medical research, and the Tower of Babel,”New England Journal of Medicine, vol. 358, no. 16, pp. 1738–1740, Apr. 2008
2008
-
[20]
P. T. O’Gara, F. G. Kushner, D. D. Ascheim, D. E. Casey, M. K. Chung, J. A. de Lemos, et al., “2013 ACCF/AHA guideline for the management of ST-elevation myocardial infarction: A report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines,”Circulation, vol. 127, no. 4, 2013
2013
-
[21]
J. Shen, “Evidence graph consistency in retrieval-augmented gener- ation: A model-dependent analysis of hallucination detection,”arXiv preprint arXiv:2606.06748, 2026
Pith/arXiv arXiv 2026
-
[22]
FHIR- GPT enhances health interoperability with large language models,
Y . Li, H. Wang, H. Z. Yerebakan, Y . Shinagawa, and Y . Luo, “FHIR- GPT enhances health interoperability with large language models,” NEJM AI, vol. 1, no. 8, Art. no. AIcs2300301, Jul. 2024
2024
-
[23]
Assessing the potential of an LLM-powered system for enhancing FHIR resource validation,
P. Tabari, A. Piscitelli, G. Costagliola, and M. de Rosa, “Assessing the potential of an LLM-powered system for enhancing FHIR resource validation,”Studies in Health Technology and Informatics, vol. 327, pp. 803–807, 2025
2025
-
[24]
Qwen Team, “Qwen2.5 technical report,”arXiv preprint arXiv:2412.15115, 2024
Pith/arXiv arXiv 2024
-
[25]
A. Grattafiori, et al., “The Llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024
Pith/arXiv arXiv 2024
-
[26]
Gemma 2: Improving open language models at a practical size,
Gemma Team, “Gemma 2: Improving open language models at a practical size,”arXiv preprint arXiv:2408.00118, 2024
Pith/arXiv arXiv 2024
-
[27]
Wiener,Cybernetics: Or Control and Communication in the Animal and the Machine, 2nd ed
N. Wiener,Cybernetics: Or Control and Communication in the Animal and the Machine, 2nd ed. Cambridge, MA: MIT Press, 1961
1961
-
[28]
Resource $validate operation,
HL7 International, “Resource $validate operation,” HL7 FHIR Release 4 (v4.0.1), 2019
2019
-
[29]
Grammar- constrained decoding for structured NLP tasks without finetuning,
S. Geng, M. Josifoski, M. Peyrard, and R. West, “Grammar- constrained decoding for structured NLP tasks without finetuning,” inProc. 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023, pp. 10932–10952
2023
This paper was first reviewed by grok-4.5 on July 31, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.