{"id":"4a4f66ef-b1ac-428a-a0d6-0a5c2b85d64b","arxiv_id":"2501.13687","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuned small LLMs are claimed to outperform GPT-4 on synthetic FHIR question answering, but the benchmark is weakened by GPT-4-generated ground truth and likely train/test leakage.","lead":"This paper tests whether small, privately hosted language models can answer questions about electronic health records by first picking out relevant FHIR data and then generating answers. The authors report that fine-tuned 8B and 12B models beat GPT-4 on their synthetic benchmarks, but the evaluation is built on GPT-4-generated labels and tiny test sets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The resource-level random splits in §4.1.1 and §4.1.2 leak query text into training: each query appears in 10 rows, so a 95-5 or 98-2 ungrouped split places the same query in both train and test, letting fine-tuned models memorize query–resource associations and inflate the reported gains over…","rationale":"The reader identified two intertwined assumptions: GPT-4-generated labels as ground truth and the split not leaking query identities. I agree with the reader's overall REJECT verdict, but I single out the split leakage as the most load-bearing concern because it is concrete, falsifiable, and sufficient by itself to invalidate the headline comparison. The query-leakage mechanism is explicit in the data-preparation description: each query yields 10 resource-level examples with the same query text. A random example-level split, with no mention of grouping, will place rows from the same query into both train and test. This allows the model to memorize query-resource associations during training and apply them to the leaked test rows, inflating accuracy and F1. The reported margins over GPT-4 are small in Task 1 (≈0.5 F1 points) and large in Task 2 (≈42% relative METEOR), but both could be affected by memorization. The GPT-4-generated labels are a concern for external validity, but they do not by themselves break the internal comparison; the split does. The concrete test—re-running with a grouped split and reporting confidence intervals—would settle whether the claim survives. If the margins persist on a leakage-free split, the paper would be a solid pilot; if they vanish, the current evaluation is not trustworthy. Given the absence of any such grouped split in the manuscript, the correct verdict remains REJECT with moderate confidence, as the reader stated.","tokens_in":12422,"tokens_out":4820,"duration_ms":42951,"concrete_test":"Re-split the Task-1 and Task-2 datasets by query (or patient) so that no query appears in both training and test, then retrain the fine-tuned models from scratch and re-compute F1 and METEOR against GPT-4 with 95% confidence intervals. If the fine-tuned models no longer beat GPT-4, the reported superiority is an artifact of leakage. As a quick check, count how many test-set query strings appear in the training split; if the count is nonzero, leakage is confirmed.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that fine-tuned 8B–12B models outperform GPT-4 on FHIR QA—rests on evaluations whose test splits are not grouped by query. In §4.1.1, for each of 50 patients, 10 FHIR resources are selected and GPT-4 generates one query referencing one or more of them, labeling each resource relevant/irrelevant. This produces 10 examples per query, each with identical query text. The 95-5 split is described only as 'a 95-5 training-test split for the dataset' (no grouping by query or patient). Similarly, §4.1.2 creates 5,000 Task-2 examples (one per query) and applies a 98-2 split. If the split is at the example level, a query that appears in training also appears in test with a subset of its resources. The model can then memorize which resource strings co-occur with a given query in training and use that to predict relevance for the held-out rows of the same query. Because the labels themselves are GPT-4-generated and deterministic given the query generation, matching them on seen queries is a memorization task, not a test of generalization to new patient questions. This directly compromises both the Task-1 F1 comparison (95.52% vs. 95.00%) and the Task-2 METEOR comparison (0.5333 vs. 0.3752). The test sets are also small (≈250 and 100 examples respectively) with no confidence intervals, so even a correct split would need statistical verification of the margins. The GPT-4-as-ground-truth issue is secondary: the labels are synthetic but internally consistent; the leakage is the load-bearing flaw because it invalidates the evaluation as a measure of out-of-query generalization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage semantic QA pipeline over synthetic FHIR patient records: Task 1 classifies whether a FHIR resource is relevant to a natural-language query, and Task 2 generates a patient-friendly answer from the relevant resources. The authors generate a synthetic dataset of 5,000 query–resource examples using GPT-4 to create queries, relevance labels, and reference answers, then fine-tune open-weight models (Llama 3.1 8B and Mistral NeMo 12B) with QLoRA. They report that the fine-tuned models outperform GPT-4 on both tasks, with Llama 3.1 Base reaching 98.82% accuracy and 95.52% F1 on Task 1 versus GPT-4's 95.00% F1, and Mistral NeMo Base reaching 0.5333 METEOR on Task 2 versus GPT-4's 0.3752. The paper also studies sequential fine-tuning, training-set size effects, and LLM-as-a-judge self-preference bias.","tokens_in":12855,"tokens_out":5330,"duration_ms":45239,"significance":"If the reported results were reliable, the paper would make a meaningful practical contribution: it demonstrates that small, privately hostable models can potentially match or beat a large API-based model on a structured medical QA task, with public model and dataset release. The exploration of sequential fine-tuning and narcissistic evaluation is also a useful addition to the literature. However, the central empirical claim rests on an evaluation design with a likely train/test leakage in Task 1, a circular gold-standard construction, and very small test sets with no uncertainty quantification. These issues are load-bearing for the headline conclusion, so the current manuscript does not provide trustworthy evidence for its main claim.","major_comments":[{"comment":"The 95-5 split is described only as 'a 95-5 training-test split for the dataset' with no grouping by query or patient. Since each query generates 10 examples (one per FHIR resource), a random example-level split will place rows from the same query in both training and test, allowing the model to memorize query–resource associations during training and apply them to held-out rows of the same query. This directly inflates the reported Task 1 accuracy (98.82%) and F1 (95.52%). The same concern applies to Task 2: the 10 queries per patient share the same pool of resources, so an ungrouped split can leak patient-level information. The evaluation must be grouped by query (or patient), and results should be reported on never-seen queries only.","section":"Section 4.1.1, Dataset Creation"},{"comment":"The reference answers used for METEOR scoring are generated by GPT-4, and the fine-tuned models are trained on GPT-4-generated answers. The reported 'outperform GPT-4' result on Task 2 therefore measures how closely the models reproduce GPT-4's own text, not agreement with an independent ground truth. An external human or clinical gold standard is needed to support the claim that the fine-tuned models answer queries correctly. Additionally, the test set contains only 100 examples, so the 0.5333 vs. 0.3752 METEOR difference is reported without any confidence interval or significance test, making it impossible to assess whether the margin is real.","section":"Section 4.1.2, Task 2 Data Preparation"},{"comment":"The headline Task 1 result is a 0.55% F1 margin (95.52% vs. 95.00%) on roughly 250 test examples. No confidence intervals, bootstrap estimates, or significance tests are provided. At this sample size, the margin is within sampling noise; the authors should report uncertainty estimates and a paired significance test (e.g., McNemar's test for the binary labels) before claiming that the fine-tuned model outperforms GPT-4. The same applies to other comparisons in Sections 5.2.1 and 5.2.3.","section":"Section 5.1.1, Experiment 1"},{"comment":"Even apart from the split issue, the gold labels are circular: GPT-4 generates the query, decides which resources are 'relevant' or 'irrelevant,' and later writes the reference answers. The benchmark is internally consistent, but it does not establish that these labels reflect real medical relevance or that the reference answers are clinically correct. The paper's conclusion that fine-tuned models 'outperform GPT-4' should be reframed as 'better approximate GPT-4's judgments' unless human-validated labels and references are introduced.","section":"Sections 4.1.1 and 4.1.2, Dataset Creation"}],"minor_comments":[{"comment":"The phrase '42% on Meteor Task in Task2' is imprecise; the reported numbers indicate a relative METEOR improvement of about 42%, not a percentage-point gain. Please state the metric and the comparison explicitly.","section":"Abstract"},{"comment":"The sentence 'LLaMA Base fine-tuned with 4900 examples showed a 4.39% higher METEOR score compared to Mistral NeMo Base fine-tuned with 500 examples' confounds model choice with dataset size; the dataset-size conclusion should compare the same model trained on 500 vs. 4,900 examples.","section":"Section 5.2.2, Experiment 2"},{"comment":"Only LoRA rank 16 and 5 epochs are reported for all fine-tuning runs, with no sensitivity analysis or rationale for these hyperparameter choices.","section":"Section 4.2, Fine-tuning"},{"comment":"The text contains duplicated figure numbers (e.g., 'Figure 3 3', 'Figure 4 4'), which should be corrected.","section":"Figures 3–9"},{"comment":"The claim that GPT-4 has 1760B parameters is based on an unverified public estimate; please cite a source or remove the specific number.","section":"Section 5.1.1"},{"comment":"The judge model 'Claude Sonnet' is named without a version; please specify the exact model version used for reproducibility.","section":"Section 5.2.4, Experiment 4"},{"comment":"The limitations section mentions only the use of synthetic patient data; the reliance on GPT-4 for labels and reference answers, the lack of statistical significance testing, and the small test-set sizes are equally important limitations and should be acknowledged.","section":"Section 6.2, Limitations"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful pilot study for private FHIR QA, and the authors ship their models, data, and prompts. But the headline result — fine-tuned 8–12B models beating GPT-4 — rests on two evaluations that don't both hold up. The Task 1 split leaks query text; the Task 2 comparison is against GPT-4-written references on 100 examples.\n\nWhat's new: the two-stage retrieval-then-answer decomposition, the dataset-size study, the sequential fine-tuning experiments, and the self-preference evaluation for Llama-3.1 and Mistral-NeMo. The closest prior work (SUPaHOT) used three tasks, so this is a modest but distinct variant. The catastrophic forgetting result — F1 drops from 95.5 to 31.6 after fine-tuning on Task 2 — is a clean, useful observation.\n\nWhat's done well: the pipeline is transparent and reproducible. Synthea data, QLoRA, and METEOR are standard choices. Prompts are in the appendix. Releasing the artifacts is real credit.\n\nSoft spots, in order of severity.\n\nFirst, the Task 1 split. Each query produces 10 rows (one per resource), all with identical query text. The paper describes a 95-5 split without grouping by query. Under a random row-level split, the same query almost certainly appears in both train and test. The model can memorize which resource strings co-occur with a query and label held-out rows of seen queries. That makes the reported 95.52 F1 vs GPT-4's 95.00 an artifact of memorization, not generalization to new questions. This is load-bearing for Task 1.\n\nSecond, Task 2. I don't agree with the stress-test note that the Task 2 split leaks the same way: each Task 2 example is one query with its relevant resources, so a row-level split does not repeat a query in both train and test. The problem there is different: the test set is 100 examples, with no confidence intervals, and the reference answers are GPT-4's own outputs. The fine-tuned models were trained to imitate GPT-4's style, so a higher METEOR score partly reflects distribution matching, not better medical answers. A human-validated reference set, or at least a non-GPT-4-generated one, is needed before claiming an edge.\n\nThird, minor: in Experiment 2, the sentence comparing Llama-4900 to Mistral-500 conflates model and dataset size.\n\nThe GPT-4-as-ground-truth issue is present in both tasks, but for Task 1 it's secondary to the split leakage.\n\nWho this is for: practitioners building private FHIR QA systems and researchers studying fine-tuning behavior on medical tasks. A serious referee should see it, because the flaws are fixable and the artifacts are useful. But the abstract's performance claims should be rewritten as pilot results until the evaluation is redone with query-grouped splits, larger tests, and independent references.","headline":"A useful, reproducible pilot on private fine-tuned FHIR QA, but the head-to-head with GPT-4 is undermined by a query-leaking Task 1 split and a 100-example GPT-4-written Task 2 test.","tokens_in":13380,"tokens_out":4708,"would_cite":false,"duration_ms":38979,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned 8B model beats GPT-4 on medical-record QA","keywords":["FHIR","question answering","fine-tuning","LLM","privacy","synthetic data","METEOR","narcissistic evaluation"],"falsifier":"Re-run both tasks on a test set whose queries and resources are entirely new, with human clinicians writing the gold labels and reference answers; if the fine-tuned models no longer beat GPT-4 on F1 and METEOR, the reported advantage does not generalize.","tokens_in":12210,"feed_emoji":"🏥","tokens_out":7470,"duration_ms":54505,"temperature":0.7,"pith_summary":"This paper tries to establish that a privacy-preserving, two-stage question-answering system over FHIR medical records can outperform GPT-4 while using models about 250 times smaller. The first stage decides which FHIR resources are relevant to a patient's natural-language query; the second stage generates a plain-language answer from those resources. The authors fine-tune LLaMA 3.1 8B and Mistral NeMo 12B with QLoRA on synthetic data produced by Synthea and GPT-4, and report that the small fine-tuned models beat GPT-4 by 0.55% in F1 on the relevance task and by 42% on METEOR for answer generation. If that claim holds, it matters because medical-record question answering could be run on private servers or edge devices without sacrificing accuracy.","feed_headline":"Fine-tuned 8B model beats GPT-4 on medical-record QA","feed_subtitle":"Privately hostable and 250x smaller, it still beats GPT-4 on both stages of a FHIR QA pipeline.","key_machinery":"The central machinery is a two-stage decomposition of semantic QA over EHRs: Task 1 is binary classification $F(q,r) = I\\{0,1\\}$ deciding whether a FHIR resource $r$ is relevant to query $q$, and Task 2 generates a patient-friendly answer conditioned on the query and the relevant resources. The adaptation mechanism is QLoRA fine-tuning (rank 16, 5 epochs) applied to 8-12B base models, which keeps the models small enough for private hosting. The supervision pipeline uses Synthea to generate synthetic FHIR records and GPT-4 to write queries, relevance labels, and reference answers; METEOR is the evaluation metric for answer generation. The comparison set includes GPT-4, GPT-4o, and Meditron-7B.","core_discovery":"The paper's central claim is that fine-tuned, openly available models match or beat frontier models on a two-stage FHIR question-answering pipeline when each stage is optimized separately. On Task 1, the fine-tuned LLaMA 3.1 Base 8B reaches 98.82% accuracy and 95.52% F1, compared with GPT-4's 98.43% accuracy and 95% F1. On Task 2, the fine-tuned Mistral NeMo Base 12B reaches a METEOR score of 0.5333, versus 0.3752 for GPT-4. The paper also claims that increasing training data from 500 to 4,900 examples improves Task 2 METEOR by about 4-5 points, that sequential fine-tuning on one task sharply degrades performance on the other, and that LLM-as-a-judge evaluation shows a clear self-preference bias only when the judging model knows it is evaluating its own output.","pith_inferences":["If the results hold on independently annotated data, the practical takeaway is that parameter-efficient fine-tuning of mid-size open models can substitute for frontier API calls in structured EHR QA, and that private deployment need not cost accuracy.","The 0.55% F1 margin on Task 1 is small enough that a different split or annotation source could flip the ordering; the METEOR margin is larger, but METEOR computed against GPT-4-written references may reward models that imitate GPT-4's phrasing.","Because the same GPT-4 that generated the labels and references is the benchmark being outperformed, the fairest next test would be an external human-annotated evaluation set with stricter query-level separation.","The architecture-dependent sequential fine-tuning results (Mistral loses, LLaMA gains) point to multi-task learning or continual pre-training as more reliable routes to a single model handling both tasks."],"forward_implications":["A privately hostable 8-12B model can serve as the core of a FHIR question-answering system without sending patient data to a cloud API, at accuracy comparable to or better than GPT-4.","Fine-tuning on task-specific synthetic data is what unlocks the gains: pretrained Llama 3.1 Base and Mistral NeMo Base score only around 54% accuracy on Task 1 before fine-tuning.","More fine-tuning data helps: models trained on 4,900 examples beat those trained on 500 by roughly 4-5 points of METEOR on Task 2.","Sequential fine-tuning is fragile: moving from Task 1 to Task 2 drops Task 1 F1 from 95.52 to 31.62, and extending the prompt with classification instructions only recovers it to 56.62.","LLM-as-a-judge evaluation shows only minor self-preference in blind comparisons, but a clear 13-point win-rate drop for GPT-4 once it knows it is judging its own response."],"supporting_citations":[{"why":"QLoRA; supplies the parameter-efficient fine-tuning method used to adapt the small models.","marker":"[15]"},{"why":"LLaMA 3.1; base model family that is fine-tuned for both tasks.","marker":"[11]"},{"why":"Mistral open-source models; base model family, including Mistral NeMo, used for fine-tuning.","marker":"[12]"},{"why":"Meditron-7B; medical-domain baseline that the fine-tuned models are compared against.","marker":"[20]"},{"why":"LLM on FHIR; prior GPT-4-based app whose querying setup this work extends to privately fine-tuned models.","marker":"[18]"},{"why":"SUPaHOT; prior approach replacing GPT-4 with fine-tuned models on FHIR data, used as a reference point for the two-stage design.","marker":"[19]"},{"why":"Synthea; generates the synthetic FHIR patient records that seed the dataset.","marker":"[23]"},{"why":"METEOR; the automatic answer-quality metric that defines Task 2 performance.","marker":"[24]"},{"why":"LLMs as narcissistic evaluators; prior finding that LLMs inflate scores for their own outputs, which Experiment 4 probes.","marker":"[25]"}],"fun_headline_variants":["Private fine-tuned LLMs beat GPT-4 on medical record QA","8B fine-tuned model outperforms GPT-4 in FHIR question answering","Small fine-tuned LLMs top GPT-4 on private medical QA","Edge-deployable LLMs excel at medical record Q&A","Fine-tuned LLMs, 250x smaller, beat GPT-4 on EHR queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes both that GPT-4's relevance labels and reference answers are correct ground truth and that the random split of 5,000 resource-level examples never puts resources from the same query into both training and test.","fun_headline_variants_meta":{"raw":{"variants":["Private fine-tuned LLMs beat GPT-4 on medical record QA","8B fine-tuned model outperforms GPT-4 in FHIR question answering","Small fine-tuned LLMs top GPT-4 on private medical QA","Edge-deployable LLMs excel at medical record Q&A","Fine-tuned LLMs, 250x smaller, beat GPT-4 on EHR queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1394,"prompt_tokens":1013,"completion_tokens":381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":629,"tokens_out":381,"duration_ms":2976,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:41:09.825842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both tasks on a test set whose queries and resources are entirely new, with human clinicians writing the gold labels and reference answers; if the fine-tuned models no longer beat GPT-4 on F1 and METEOR, the reported advantage does not generalize.","supporting_citations":[{"cited_title":"QLoRA: Efficient finetuning of quantized LLMs","cited_arxiv_id":null,"evidence_quote":"QLoRA; supplies the parameter-efficient fine-tuning method used to adapt the small models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaMA 3.1; base model family that is fine-tuned for both tasks."},{"cited_title":"Meditron-7b: Scaling medical pretraining for large language models, 2023","cited_arxiv_id":null,"evidence_quote":"Meditron-7B; medical-domain baseline that the fine-tuned models are compared against."},{"cited_title":"LLM on FHIR – demystifying health records","cited_arxiv_id":null,"evidence_quote":"LLM on FHIR; prior GPT-4-based app whose querying setup this work extends to privately fine-tuned models."},{"cited_title":"SUPaHOT: Universally scalable and private method to demystify FHIR health records","cited_arxiv_id":null,"evidence_quote":"SUPaHOT; prior approach replacing GPT-4 with fine-tuned models on FHIR data, used as a reference point for the two-stage design."},{"cited_title":"Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record","cited_arxiv_id":null,"evidence_quote":"Synthea; generates the synthetic FHIR patient records that seed the dataset."},{"cited_title":"LLMs as narcissistic evaluators: When ego inflates evaluation scores","cited_arxiv_id":null,"evidence_quote":"LLMs as narcissistic evaluators; prior finding that LLMs inflate scores for their own outputs, which Experiment 4 probes."}],"review_version":1}