{"id":"de42a21b-3d04-445a-bb90-fcfd5985d987","arxiv_id":"2507.03067","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A retrieval-augmented LLM pipeline maps MIMIC-IV tables to HL7 FHIR resources and attributes, with 100% resource identification in a curated baseline but 68% attribute accuracy under real-world-like conditions.","lead":"This paper builds a pipeline that uses large language models plus text embeddings and clustering to map hospital database tables into the HL7 FHIR interoperability standard. It reports near-perfect resource identification but only moderate attribute-level mapping accuracy on MIMIC-IV, showing that semi-automated standardization is feasible with expert review.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval and mapping accuracies may be inflated because the same 45-resource corpus and gold-standard mappings used for evaluation also drive configuration selection; the 64 excluded baseline attributes are never scored and the abstract's 94% refers only to resource retrieval.","rationale":"The reader's verdict (CONDITIONAL) and the weakest-assumption analysis align closely with my stress-test. I agree that the manuscript's central claim is a feasible semi-automated mapping pipeline, supported by concrete experiments and transparent error analysis, and rejection would be inappropriate. However, the most load-bearing technical weakness is the lack of a clean separation between configuration selection and evaluation: the 94% figure came after switching to biomedical embeddings, the clustering configuration was chosen by quality metrics on the same data, and the 100% baseline resource-identification was achieved after iterating on embedding approaches. This is a correctness-risk issue rather than a question of fraud or internal inconsistency; the paper's own methods section records the iterations (word-frequency embeddings 'did not produce satisfactory results', then hybrid retrieval, then biomedical embeddings for real-world). The second concern, that 64 of 183 baseline attributes were excluded before scoring and the abstract reports 94% (retrieval) instead of 67.7-69.9% (attribute mapping), is also real and material to how a reader interprets the headline. The reader identified the gold-standard correctness as the weakest assumption; my analysis agrees but adds that even with correct gold labels, the reported numbers are in-sample selection-best-case estimates because the same evaluation data were used to choose embeddings, clustering, and prompts. The proposed concrete test (pre-registration or cross-validation, plus scoring all 183 attributes, plus correcting the abstract) is the minimal check that would distinguish an optimistic but valid feasibility study from a systematically inflated one. I do not see a need to change the verdict: CONDITIONAL remains the honest assessment, with the additional requirement that the authors provide held-out or cross-validated numbers and publish the excluded attribute list.","tokens_in":10552,"tokens_out":1923,"duration_ms":20663,"concrete_test":"Re-run the real-world retrieval and mapping pipeline under a locked protocol: (a) pre-register the clustering algorithm, number of clusters, and embedding model before seeing the 68 column gold mappings (or use leave-one-table-out cross-validation); (b) score the 94% resource-retrieval figure on the held-out split; (c) also report per-attribute accuracy on all 183 baseline attributes, including the 64 excluded ones, with exclusion criteria documented in a supplement; (d) recompute the abstract headline using the attribute-level accuracy (67.7-69.9%) rather than the 94% retrieval figure. If the held-out retrieval accuracy drops materially below 94%, the reported headline overstates pipeline performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility claim rests on two accuracy numbers: 100% baseline resource identification and 94% real-world resource retrieval (with 67.7-69.9% attribute mapping). Both numbers are configuration-dependent and the evaluation protocol does not separate configuration selection from evaluation. In the baseline, the hybrid retrieval system (TFIDF, BM25, USE, Word2Vec with RRF) was chosen because word-frequency embeddings 'did not produce satisfactory results', and the result reported is the accuracy of the final, selected configuration on the same 17 tables used for development. In the real-world scenario, the authors explicitly tried general embeddings (87.9%) then biomedical embeddings and report 94% after switching embedding models; the clustering algorithm, number of clusters, and top-5 resource selection were all tuned using clustering-quality metrics and presumably the same 68-attribute gold mapping. With no held-out split, cross-validation, or fixed configuration pre-registered before seeing the gold labels, the 94% and 100% figures are selection-best-case estimates, not unbiased estimates of pipeline accuracy. A second, distinct concern is that 64 of 183 baseline attributes (35%) were excluded after 'exhaustive review' as redundant or lacking feasible FHIR representation; if exclusion is subjective, the 67-74% attribute accuracy applies only to an author-filtered subset, and the paper does not report what those 64 attributes were or how exclusion criteria map to FHIR profiling decisions. This is not a fatal flaw because the paper is transparent about the need for expert review, but it means the claim 'semi-automated mapping is feasible' is supported by optimistic in-sample estimates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a semi-automated pipeline that combines text embeddings, clustering, retrieval-augmented generation, and large language models (GPT-4o and Llama 3.2 405B) to map structured clinical data from MIMIC-IV to HL7 FHIR resources and attributes. In a baseline scenario with 17 tables and 119 attributes (after excluding 64 of 183), the authors report 100% resource-level identification and GPT-4o attribute-level accuracy of 67.02–73.88% (95% CI), while Llama 3.2 reaches 43.79–52.98%. In a simulated real-world scenario with a single 68-attribute table, resource retrieval reaches 94% and attribute-level mapping is 67.7–69.9% for GPT-4o and 54.5–57.8% for Llama 3.2, with the best results obtained at specific temperature settings. The authors conclude that the approach demonstrates feasibility of context-aware, LLM-driven semi-automated FHIR interoperability workflows.","tokens_in":10804,"tokens_out":4228,"duration_ms":45315,"significance":"If the reported accuracy numbers were unbiased, the paper would make a useful contribution to clinical data interoperability by showing that LLMs combined with retrieval and clustering can reduce the manual effort required to map legacy tabular schemas to FHIR. The work is timely, the pipeline is concretely described, and the authors provide prompts and results on GitHub, which supports reproducibility. However, the evaluation protocol has load-bearing methodological weaknesses: configuration selection and evaluation share the same gold standard, the gold standard itself lacks inter-annotator agreement assessment, and the abstract's headline 94% conflates resource retrieval with the much lower attribute-level mapping accuracy. The central feasibility claim is plausible but currently rests on accuracy figures that may be inflated and are reported in a way that overstates the actual attribute mapping performance.","major_comments":[{"comment":"The reported 94% resource-retrieval accuracy is a selection-best-case estimate rather than an unbiased evaluation. The paper states that initial embeddings achieved 87.9% accuracy and that switching to biomedical embeddings raised accuracy to 94%, and that clustering algorithms and configurations were selected using Silhouette, Calinski-Harabasz, and Davies-Bouldin metrics. Since the same 68-attribute gold-standard mapping appears to drive both configuration selection and the final accuracy report, with no held-out split, cross-validation, or pre-registered configuration, the 94% figure likely overstates the expected pipeline accuracy. Please separate configuration selection from evaluation, e.g., by using a development subset for tuning and a disjoint test subset for reporting, or by reporting accuracy across all tried configurations rather than only the best.","section":"Real World Scenario – Context Building; Results"},{"comment":"The abstract's statement that 'under real-world conditions, accuracy dipped slightly to 94%' refers to resource-level identification (retrieval), not attribute-level mapping. The attribute-level accuracy in the real-world scenario is considerably lower: 67.7–69.9% for GPT-4o at the best temperature setting. Because the paper's stated objective is mapping table attributes to FHIR elements, the headline number is misleading. The abstract and the Results section should explicitly distinguish resource-level retrieval accuracy from attribute-level mapping accuracy and report both numbers, with the attribute-level figure given equal prominence in the abstract.","section":"Abstract; Results"},{"comment":"The baseline evaluation excludes 64 of 183 attributes (35%) as 'redundant' or 'lacking a feasible FHIR representation' after an 'exhaustive review,' but the paper does not list the excluded attributes or specify operational exclusion criteria. If the exclusion is subjective, the reported 67–74% attribute-level accuracy applies only to an author-filtered subset and cannot be interpreted as accuracy over the original table schemas. Please provide the list of excluded attributes, the criteria used, and ideally a sensitivity analysis that treats excluded attributes as failed mappings in a conservative scoring.","section":"Methods – Baseline Scenario, Data Processing"},{"comment":"The gold-standard mappings used to compute all accuracy figures are (for the baseline) manual author mappings and (for the real-world scenario) mappings 'made previously by MIMIC developers.' No inter-annotator agreement, adjudication procedure, or independent validation is reported, so the correctness and unambiguity of the reference labels are unestablished. Ambiguous or erroneous gold-standard mappings would directly mismeasure every reported accuracy, and the risk is compounded in the real-world scenario where the same gold standard appears to drive configuration selection and final evaluation. Please make the gold-standard mapping file publicly available, describe how mappings were derived, and report inter-annotator agreement or provide a convincing argument that the mappings are unambiguous.","section":"Methods – Evaluation and Gold Standard"}],"minor_comments":[{"comment":"The abstract states that 'resource identification achieved a perfect F1-score,' but the paper reports only accuracy (proportion correct) for resource-level identification and does not compute an F1-score anywhere. Please correct the terminology to 'perfect accuracy.'","section":"Abstract"},{"comment":"Tables 3 and 4 report only 95% confidence intervals; the text reports means (e.g., 68.8% for GPT-4o at t=0.5). Including the mean and the CI method (e.g., exact binomial, Wilson) in the tables would improve clarity.","section":"Results, Tables 3 and 4"},{"comment":"The paper reports N=4 for the baseline prompt experiments and N=10 for the real-world temperature experiments but never defines what N represents (number of runs, number of attributes, or number of prompt variants). Please clarify.","section":"Methods, LLM Interaction"},{"comment":"There are several typos and stylistic inconsistencies: 'GPT-4o shown' should be 'GPT-4o showed,' 'differents temperature configurations' should be 'different temperature configurations,' and 'LLaMA' is used inconsistently alongside 'Llama.' Please copyedit.","section":"Throughout"},{"comment":"The figures are referenced but not included in the manuscript text. They are important for understanding the pipeline; please ensure they are legible and accompanied by captions that clearly label steps 1.1–1.3 and 2.1–2.3.","section":"Figures 1 and 2"},{"comment":"Several reference entries are incomplete or malformed (e.g., references 2 and 3 appear as 'CT S; 2007' and 'LOINC; 1994' with no titles or authors). Please format all references consistently according to the journal style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant topic for JAMIA and the proposed pipeline is potentially useful. However, the evaluation protocol's failure to separate configuration selection from evaluation, the lack of gold-standard validation, and the abstract's conflation of resource retrieval (94%) with attribute mapping (~68%) are load-bearing issues that must be fixed before publication. I believe the authors can address these concerns within the scope of a major revision by adding a held-out evaluation, reporting attribute-level accuracy prominently, documenting the gold-standard creation, and disclosing the excluded attributes. I did not find evidence of circularity in the sense of the LLM being evaluated against its own labels; the concern is optimizer's bias on the evaluation set, which is a correctable methodological issue rather than a fatal flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful applied paper. It shows that a RAG-plus-clustering pipeline can map structured MIMIC-IV columns to FHIR resources and attributes with attribute-level accuracy around 68% for GPT-4o. The 100% and 94% figures are resource-retrieval numbers, not overall mapping accuracy, and the abstract blurs that distinction.\n\nWhat's new: prior LLM-to-FHIR work mostly processed free text or a single resource. Here you get 45 candidate FHIR resources, two scenarios, two LLMs, multiple prompt strategies, and temperature ablations. The GitHub repo with prompts and full outputs is a real plus. The authors are also honest about hallucinations, granularity mismatches, and the need for expert validation. That's the correct framing for a semi-automated tool.\n\nSoft spots, in order. The evaluation protocol doesn't separate configuration selection from evaluation. The hybrid retrieval was chosen after word-frequency embeddings underperformed, and the real-world retrieval jumped from 87.9% to 94% after switching to biomedical embeddings. No held-out split, no cross-validation. So the retrieval accuracies are selection-best-case, not unbiased estimates. This is a real concern but not fatal, since attribute mapping is the actual bottleneck and it doesn't swing as much. Second, 64 of 183 baseline attributes were excluded after 'exhaustive review' and are never scored. The paper doesn't list them or state the exclusion criteria. That could inflate the attribute accuracy. Third, the baseline gold standard is the authors' own manual mapping without inter-annotator agreement; the real-world gold standard came from MIMIC developers, which is better but still needs independent adjudication. Fourth, the abstract should say 'resource retrieval accuracy' when it reports 94%, and should headline the attribute-level ~68% as the mapping accuracy.\n\nNone of these are rejection-level. The central feasibility claim holds up: with retrieval context, LLMs can propose sensible FHIR mappings that narrow the manual ETL work, and the paper says plainly that experts still need to review. I'd send this to peer review. The authors should be asked to fix the abstract, publish the excluded attribute list, and add a held-out or cross-validated evaluation of retrieval.\n\nI'd cite it as a baseline for structured-to-FHIR mapping, and I'd bring it to reading group as a discussion piece, mainly because the evaluation-protocol issues are instructive for applied LLM papers.","headline":"Useful applied baseline for LLM-to-FHIR mapping, but the retrieval accuracies are selection-best-case and the abstract overstates them.","tokens_in":11415,"tokens_out":4128,"would_cite":true,"duration_ms":41354,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Context-aware LLMs can map structured clinical tables into HL7 FHIR with high resource-level accuracy, making semi-automated interoperability workflows feasible.","keywords":["HL7 FHIR","Large Language Models","Retrieval Augmented Generation","Semantic Clustering","Data Interoperability","MIMIC-IV","Clinical Data Standardization","Prompt Engineering"],"falsifier":"Have independent clinical informaticists annotate the same 68-attribute subset and the 119 baseline attributes to FHIR, and compare their consensus against the gold standard used here; if agreement between annotators or with the gold standard is much lower than the reported accuracy, the central accuracy numbers are not measuring what they claim.","tokens_in":10310,"feed_emoji":"🏥","tokens_out":7858,"duration_ms":68155,"temperature":0.7,"pith_summary":"This paper asks whether large language models can take over the labor-intensive task of converting structured clinical data into HL7 FHIR, the interoperability standard used to exchange healthcare records. It argues yes, for a semi-automated workflow, and backs that claim with experiments on the MIMIC-IV intensive-care database. The central result is that a pipeline combining retrieval-augmented generation, semantic clustering, and structured prompting identifies the correct FHIR resource for real-world attribute groups 94% of the time, and that GPT-4o maps roughly 68% of individual attributes to the correct FHIR element. That would matter because FHIR mapping currently requires scarce domain experts and is too slow to scale across the many local data models in healthcare.","feed_headline":"LLMs map clinical data to FHIR standards semiautomatically","feed_subtitle":"On MIMIC-IV, GPT-4o identifies the right FHIR resource 94% of the time and maps roughly 68% of attributes correctly.","key_machinery":"The load-bearing mechanism is a three-stage pipeline. First, input tables are preprocessed into JSON with column descriptions and representative values. Second, context building uses hybrid retrieval, combining TF-IDF, BM25, Universal Sentence Encoder, and Word2Vec embeddings via Reciprocal Rank Fusion, and in the real-world scenario specialized clinical embeddings, to rank candidate FHIR resources for each table or attribute cluster. Third, LLM interaction prompts GPT-4o or Llama 3.2 to map each attribute to FHIR elements, with structured outputs and JSON schemas enforcing a consistent response format. Semantic clustering groups related attributes when table-level context is absent.","core_discovery":"The paper's claim is that context-aware LLM-driven transformation of structured clinical data into HL7 FHIR is feasible for semi-automated interoperability workflows. In the curated baseline, resource-level identification reaches 100% accuracy; in a simulated real-world scenario with unsorted attributes and no table-level context, retrieval identifies the correct FHIR resource 94% of the time using biomedical embeddings, and GPT-4o maps 67.7% to 69.9% of attributes correctly depending on temperature, notably outperforming Llama 3.2 405b. The authors also find that including FHIR resource JSON schemas in prompts narrows confidence intervals and reduces mapping ambiguity, while missing source descriptions and occasional hallucinations remain limitations requiring manual validation.","pith_inferences":["The roughly two-thirds attribute accuracy may be enough for expert-in-the-loop production use, but not for unattended automation; a reasonable next benchmark is measuring how much expert time this actually saves versus manual mapping.","The same retrieval-plus-prompting pipeline should transfer to other standards such as HL7 CDA or OMOP CDM, since it only needs a corpus of target schema descriptions; testing that would be a cheap extension.","The absence of inter-annotator agreement suggests the true mapping accuracy could be higher or lower than reported; a proper uncertainty estimate would require adjudicated gold standards.","A testable extension is to replace the generic LLM with a fine-tuned open-weight model on FHIR-specific corpora, which the authors list as future work and which would clarify whether the gap with GPT-4o is model size or training data."],"forward_implications":["Resource-level identification in curated tables can reach 100%, so table-to-resource assignment can be fully automated under well-documented schemas.","In real-world settings with only column descriptions, retrieval still finds the right FHIR resource 94% of the time, meaning the main bottleneck moves to attribute-level mapping.","Attribute-level accuracy around 68% with GPT-4o implies a human-in-the-loop workflow is needed, but one that removes most of the manual mapping effort.","Adding FHIR JSON schemas to prompts narrows confidence intervals and reduces ambiguity, so machine-readable context is a concrete lever for reliability.","GPT-4o consistently outperforms Llama 3.2 405b and is less sensitive to temperature, suggesting closed frontier models currently give the most stable mappings."],"supporting_citations":[{"why":"Supplies the source clinical tables from MIMIC-IV used in both scenarios.","marker":"[9]"},{"why":"Provides the pre-existing FHIR mapping used as gold standard in the real-world scenario.","marker":"[14]"},{"why":"Defines the set of 45 candidate FHIR resources and their descriptions used for retrieval.","marker":"[10]"},{"why":"Reciprocal Rank Fusion algorithm that aggregates embedding rankings to select the FHIR resource.","marker":"[31]"},{"why":"Structured Outputs functionality that enforces the JSON response format for LLM mappings.","marker":"[32]"},{"why":"PubMedBERT embeddings used as a specialized clinical embedding source in real-world retrieval.","marker":"[42]"},{"why":"MedEmbed embeddings used as a specialized clinical embedding source in real-world retrieval.","marker":"[43]"},{"why":"ClinicalBERT embeddings that contributed to the 94% resource-retrieval accuracy in the real-world scenario.","marker":"[44]"},{"why":"BioBERT embeddings that contributed to the specialized retrieval corpus and higher resource-retrieval accuracy.","marker":"[45]"}],"fun_headline_variants":["LLMs hit 94% accuracy in FHIR resource mapping","GPT-4o maps 68% of clinical attributes to FHIR","LLM-driven FHIR conversion: 94% right resource, 68% right fields","Semi-automated FHIR mapping: LLMs get resources right 94%","LLMs transform clinical data to FHIR at 94% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy numbers assume that each MIMIC-IV attribute has exactly one correct FHIR target, but the paper does not measure inter-annotator agreement, so a mapping the authors call an error could be a legitimate alternative.","fun_headline_variants_meta":{"raw":{"variants":["LLMs hit 94% accuracy in FHIR resource mapping","GPT-4o maps 68% of clinical attributes to FHIR","LLM-driven FHIR conversion: 94% right resource, 68% right fields","Semi-automated FHIR mapping: LLMs get resources right 94%","LLMs transform clinical data to FHIR at 94% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00079,"raw_usage":{"total_tokens":3494,"prompt_tokens":966,"completion_tokens":2528,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2428}},"tokens_in":582,"tokens_out":2528,"duration_ms":20813,"temperature":1.0,"reasoning_tokens":2428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:20:12.348284+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent clinical informaticists annotate the same 68-attribute subset and the 119 baseline attributes to FHIR, and compare their consensus against the gold standard used here; if agreement between annotators or with the gold standard is much lower than the reported accuracy, the central accuracy numbers are not measuring what they claim.","supporting_citations":[{"cited_title":"Physionet, editor","cited_arxiv_id":null,"evidence_quote":"Supplies the source clinical tables from MIMIC-IV used in both scenarios."},{"cited_title":"MIMIC-IV on FHIR","cited_arxiv_id":null,"evidence_quote":"Provides the pre-existing FHIR mapping used as gold standard in the real-world scenario."},{"cited_title":"FHIR resources; 2011","cited_arxiv_id":null,"evidence_quote":"Defines the set of 45 candidate FHIR resources and their descriptions used for retrieval."},{"cited_title":"Reciprocal rank fusion outperforms condorcet and individual rank learning methods","cited_arxiv_id":null,"evidence_quote":"Reciprocal Rank Fusion algorithm that aggregates embedding rankings to select the FHIR resource."},{"cited_title":"Structured Outputs; 2024","cited_arxiv_id":null,"evidence_quote":"Structured Outputs functionality that enforces the JSON response format for LLM mappings."},{"cited_title":"Embeddings for Medical Literature; 2023","cited_arxiv_id":null,"evidence_quote":"PubMedBERT embeddings used as a specialized clinical embedding source in real-world retrieval."},{"cited_title":"MedEmbed: Fine-Tuned Embedding Models for Medical / Clinical IR","cited_arxiv_id":null,"evidence_quote":"MedEmbed embeddings used as a specialized clinical embedding source in real-world retrieval."},{"cited_title":"Clinicalbert: Model- ing clinical notes and predicting hospital readmission","cited_arxiv_id":null,"evidence_quote":"ClinicalBERT embeddings that contributed to the 94% resource-retrieval accuracy in the real-world scenario."},{"cited_title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","cited_arxiv_id":null,"evidence_quote":"BioBERT embeddings that contributed to the specialized retrieval corpus and higher resource-retrieval accuracy."}],"review_version":1}