{"id":"79c3bd48-45e0-4a70-b308-409471dd19d0","arxiv_id":"2501.05479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Fine-tuned Phi-3 language models matched or beat GPT-4o on surgical billing code generation while running locally on four GPUs.","lead":"Researchers fine-tuned two small language models on 192,585 surgical reports and their assigned billing codes, and found the fine-tuned models matched or beat GPT-4o at generating ICD-10 and CPT codes. The work suggests that cost-effective, privacy-preserving AI coding assistants can be built with modest hardware and open-source tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4o baseline prompt and decoding settings are undisclosed, so the headline comparison is unauditable; this is the load-bearing weak point.","rationale":"The reader's weakest assumption and my load-bearing concern are the same: the GPT-4o baseline is not fairly or reproducibly specified, and the main claim depends on that comparison. Additional issues in the manuscript, such as the abstract reporting ICD-10 recall as 72% while Table 2 reports 0.65, the 'Table X' placeholder, and omitted training hyperparameters, are real but lower-stakes; they affect precision and reproducibility without changing the qualitative ordering of models. The GPT-4o concern is structural because it questions whether the headline comparison is an artifact of an uncontrolled baseline. I do not see a reason to move the reader's verdict: CONDITIONAL remains appropriate, with conditions being full disclosure of the GPT-4o setup and correction of the numerical inconsistency. The paper's honest limitations section, large real dataset, and detailed local-model methodology are credit-worthy and support the plausibility of the finding, but auditability of the SOTA comparator is necessary before the comparative claim can be accepted.","tokens_in":11679,"tokens_out":4548,"duration_ms":45946,"concrete_test":"Request the exact GPT-4o configuration from the authors (prompt, model version, temperature, max tokens, decoding settings) and rerun GPT-4o on the same held-out test set under at least three conditions: (1) the fine-tuned Phi-3 prompt adapted to chat syntax with explicit format instructions, (2) the base/RAG prompt with the output format instruction, and (3) a minimal prompt, each with greedy decoding (temperature 0) and with default sampling. If any reasonable GPT-4o configuration reaches or exceeds the fine-tuned Medium's ICD-10, CPT, and modifier recall/precision, the paper's central comparison is not robust; if GPT-4o remains below under all tested configurations, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is explicitly comparative: 'Both fine-tuned models performed better or as well as GPT-4o.' The evidence for this claim depends on a GPT-4o baseline that is not specified. Methods (pp. 4-5) states only that 'GPT-4o was used to establish a comparative benchmark representing large state of the art (SOTA) models.' No prompt template, system message, output-format instruction, model version, temperature, decoding strategy, or repetition penalty is given for GPT-4o, while the Phi-3 configurations are described in full, including a task-specific system prompt, placeholder tokens, greedy decoding with a repetition penalty of 1.1, and explicit formatting requirements. The local fine-tuned models were trained on exactly the prompt and output format used at inference, giving them a structural advantage in producing parseable, correctly formatted claims. If GPT-4o was queried with a bare prompt, a different prompt, default sampling, or no format specification, its lower recall/precision and higher fabrication rate could reflect the benchmark configuration rather than a genuine capability gap. This is not an internal inconsistency but a missing control, and it is load-bearing because the headline conclusion is a comparison and because no reader can audit or replicate the GPT-4o baseline without these details.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates four locally deployed Phi-3-based configurations (base Mini, Mini with RAG, fine-tuned Mini, fine-tuned Medium) against GPT-4o on the task of generating ICD-10-CM, CPT, and modifier codes from operative reports. The authors fine-tune Phi-3 Mini and Phi-3 Medium with QLoRA on institutional surgical billing data, run inference with greedy decoding and repetition penalty, and measure exact-code recall/precision, fabrication rates, and output-format fidelity. The headline claim is that the fine-tuned models perform as well as or better than GPT-4o while requiring only four 24GB consumer GPUs for training. The study uses a large real-world dataset (192,585 encounters), a held-out split in principle, bootstrap confidence intervals, and validity checks against reference code lists. The abstract reports that the fine-tuned Phi-3 Medium achieves ICD-10 recall/precision of 72%/72%, CPT recall/precision of 77%/79%, and modifier recall/precision of 63%/64%.","tokens_in":11905,"tokens_out":4866,"duration_ms":44739,"significance":"If the comparative results are sound, this is a practically valuable demonstration: small open models fine-tuned on institution-specific data can approach or match a much larger commercial model on a real clinical documentation task while keeping data on-premises and using modest hardware. The strengths of the paper are its scale of real patient data, the explicit local-development and privacy-preserving setup, the use of bootstrap confidence intervals, and the fabrication-rate validity checks. The main weakness is that the GPT-4o baseline is not specified, which makes the central comparative claim unauditable as reported. There is also an ambiguity about which split (validation vs. test) underlies the main results, and the abstract's ICD-10 numbers do not match Table 2. These issues are fixable but must be resolved before the conclusions can be accepted.","major_comments":[{"comment":"The GPT-4o baseline is not specified. The Methods state only that 'GPT-4o was used to establish a comparative benchmark representing large state of the art (SOTA) models,' with no prompt template, system message, output-format instructions, model version or date, temperature, sampling strategy, repetition penalty, or maximum token count, while the Phi-3 configurations are described in full. Because the headline conclusion is explicitly comparative ('Both fine-tuned models performed better or as well as GPT-4o'), the comparison is not auditable without this information. Please provide the exact GPT-4o prompt (ideally the same system prompt and output-format instructions used for the other configurations), the specific model version, all decoding settings, and any post-processing applied to its output, or explicitly weaken the comparative claim to what can be supported without that information.","section":"Methods, 'Language Model Configurations' and 'Inference'"},{"comment":"It is unclear whether the main results in Table 2 and Figure 3 are computed on the held-out test set or on the validation set. The paper says the data were split 60/20/20 into training, validation, and test, and the inference section says the evaluation test set was used, but the bootstrap paragraph in Evaluation says 'The validation set is sampled with replacement (N = 39052)' for the confidence intervals shown in Figure 3. If the metrics are computed on the validation set, the held-out claim is compromised; if they are computed on the test set, the bootstrap text is incorrect. Additionally, N = 39,052 does not match 20% of 192,585 (which is 38,517). Please clarify which split produced the numbers in Table 2 and Figure 3 and correct the sample size.","section":"Evaluation, 'Data Extraction and Processing' and Figure 3"},{"comment":"The abstract reports ICD-10 recall and precision for the fine-tuned Phi-3 Medium as 72% and 72%, but Table 2 lists ICD-10 recall as 0.65 and precision as 0.72 for the same configuration. The CPT and modifier numbers in the abstract match Table 2, suggesting a typo, but this is a headline result and must be corrected or reconciled.","section":"Abstract and Table 2"}],"minor_comments":[{"comment":"The phrase 'provided in Table X' should be replaced with the actual table number (Table 2).","section":"Results, 'Code Validity Analysis'"},{"comment":"The precision and recall formulas contain unreadable special characters and should be typeset using standard mathematical notation.","section":"Evaluation, equations"},{"comment":"The F1 formula is garbled; please replace it with a properly typeset expression and state explicitly that F1 is computed from the mean precision and mean recall rather than averaged per-case F1.","section":"Evaluation, F1 equation"},{"comment":"The sentence 'augment and existing LLM' should read 'augment an existing LLM.'","section":"Background"},{"comment":"The model name is inconsistently capitalized as 'PHI-3' and 'Phi-3'; please use 'Phi-3' consistently.","section":"Throughout"},{"comment":"The size of the test set is not reported; please report the number of encounters in the training, validation, and test splits.","section":"Results, 'Data Extraction and Processing'"},{"comment":"The 'ROUGE L Sum' metric is not standard in the cited reference; please define it or cite a source that defines it.","section":"Evaluation, 'Format Consistency Analysis'"}],"recommendation":"major_revision","confidential_remarks":"The GPT-4o baseline specification is the main risk to the paper's central claim. If the authors can provide the exact prompt and decoding settings, and if the evaluation-set ambiguity is resolved, the paper would be a useful practical contribution. The abstract/Table 2 inconsistency and the N=39,052 issue should be caught during revision; they are not deeply problematic but need correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful benchmark paper with one load-bearing transparency gap. The authors fine-tune Phi-3 Mini and Medium with QLoRA on 192,585 real operative report-to-billing-code pairs from a single health system, then compare against base Phi-3, a RAG variant, and GPT-4o. The dataset is large and real, the evaluation uses a proper held-out split, bootstrap confidence intervals, and reference-list validity checks, and the fine-tuning details (rank 64, alpha 16, dropout 0.1, targeted modules, 4x A5000 GPUs) are given in enough detail to be reproducible. That is genuine empirical work, and the honest discussion—admitting the models are not ready to replace coders and that ICD-10 performance suffers without the H&P note—deserves credit.\n\nThe soft spots are real but fixable. The most serious is that the GPT-4o baseline is effectively unspecified. The Methods say only that GPT-4o was used as a SOTA benchmark; there is no prompt template, model version, temperature, or decoding settings, while the Phi-3 configurations get full prompts and greedy decoding with repetition penalty 1.1. Because the fine-tuned models were trained on the exact prompt and output format used at inference, and the RAG/base models got extra formatting instructions, GPT-4o may have been at a structural disadvantage if it received a bare prompt or default sampling. That makes the headline claim—that fine-tuned models match or beat GPT-4o—not fully auditable. This is a missing control, not an internal contradiction, and it can be fixed by reporting the GPT-4o setup.\n\nThere are a few smaller issues: the abstract reports ICD-10 recall of 72% for the Medium model while Table 2 shows 65% (the abstract's numbers match precision, not recall); there is a \"Table X\" placeholder in the Code Validity Analysis; and no code or trained adapters are released. None of these undercut the core finding, but they add friction for a reader trying to verify or build on the work. I also note that the RAG variant's top-2 similar-note retrieval is a thin baseline, but the authors are straightforward about that being a design choice for simplicity.\n\nWho is this for? People working on practical LLM deployment in hospital revenue-cycle management, and anyone benchmarking small open models against commercial APIs on structured clinical tasks. It deserves a serious referee: the empirical design is mostly sound, the dataset is valuable, and the main flaw is a missing methods detail, not a wrong conclusion. I'd recommend engaging with it after the GPT-4o baseline is disclosed and the internal inconsistencies are cleaned up.","headline":"A useful, mostly sound empirical benchmark of fine-tuned Phi-3 models for surgical billing coding, but the undisclosed GPT-4o baseline makes the headline comparison unauditable.","tokens_in":12577,"tokens_out":1410,"would_cite":true,"duration_ms":14243,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A locally fine-tuned small language model can generate surgical billing codes from operative reports as accurately as GPT-4o, while training on four consumer-grade 24GB GPUs.","keywords":["medical billing and coding","ICD-10-CM","CPT codes","modifier codes","QLoRA fine-tuning","Phi-3 small language models","surgical operative reports","healthcare generative AI"],"falsifier":"Re-run the same test set with GPT-4o using the exact structured prompt, special-token segmentation, greedy decoding, and output-format instruction used for the fine-tuned Phi-3 models, and compare exact-match recall and precision on ICD-10, CPT, and modifier codes. If GPT-4o then matches or exceeds the fine-tuned Phi-3 Medium, the paper's central comparison would not survive.","tokens_in":11460,"feed_emoji":"🧾","tokens_out":10306,"duration_ms":91847,"temperature":0.7,"pith_summary":"This paper reports a practical recipe for building a generative-AI medical billing coder from a small open-weight language model instead of a large commercial API. The authors fine-tuned Phi-3 Mini and Phi-3 Medium on operative reports paired with the billing claims actually submitted for payment, and compared them against the base model, a retrieval-augmented version, and GPT-4o. Their central claim is that both fine-tuned models match or beat GPT-4o on exact ICD-10, CPT, and modifier code generation, with the fine-tuned Phi-3 Medium performing best while fabricating only about 1% of ICD-10 codes and 0.6% of CPT codes. The significance is practical: if true, a health system could train and run a competitive coder entirely on local, protected infrastructure with four 24GB GPUs, avoiding both the cost and the privacy exposure of sending patient notes to a hosted model.","feed_headline":"Small fine-tuned model matches GPT-4o on surgical coding","feed_subtitle":"Locally trained Phi-3 hit 1% fabricated codes and beat GPT-4o on real operative reports, using four 24GB GPUs.","key_machinery":"The load-bearing mechanism is QLoRA (quantized low-rank adaptation), a parameter-efficient fine-tuning method that trains small adapter matrices on a 4-bit quantized base model, applied to the Phi-3 family of small open-weight language models. The authors fine-tuned Phi-3 Mini and Phi-3 Medium on paired operative reports and billing claims, using the model's unused special tokens to segment the prompt into system, user, and assistant components so the model learns the claim format. At inference they used greedy decoding with a repetition penalty and a 512-token limit, and they merged the trained adapters back into the models. This mechanism carries the argument because it is what converts a general chat model into a structured billing-claim generator while keeping training feasible on four 24GB GPUs.","core_discovery":"The paper claims that a small open-weight language model, fine-tuned with parameter-efficient supervised learning on institution-specific operative reports and their associated billing claims, can generate ICD-10, CPT, and modifier codes from an operative report as accurately as or more accurately than GPT-4o. In the authors' benchmark, the fine-tuned Phi-3 Medium performed best, with ICD-10 recall and precision of 72% and 72%, CPT recall and precision of 77% and 79%, and modifier recall and precision of 63% and 64%; the fine-tuned Phi-3 Mini also performed better than or as well as GPT-4o. The Medium model fabricated only 1% of ICD-10 codes and 0.6% of CPT codes, which the authors contrast with roughly half of base-model ICD-10 codes being invalid. The authors' conclusion is that task-specific fine-tuning on local data, with simple open-source tooling and four 24GB GPUs, can close the gap with much larger general-purpose models on a real clinical documentation task.","pith_inferences":["If the result generalizes, the same local fine-tuning recipe could be applied to other structured clinical-extraction tasks, such as pulling diagnoses from pathology or radiology reports, where privacy and exact vocabulary matter.","The reported comparison would be strengthened by a head-to-head rerun in which GPT-4o receives the exact same structured prompt and decoding settings as the Phi-3 models; the paper does not disclose GPT-4o's prompt, so part of the measured gap could be prompt-format rather than model capability.","Adding the History and Physical text, which the paper identifies as the source coders use for diagnosis codes, would likely improve ICD-10 recall and precision beyond the reported levels.","Because the evaluation counts only exact code-set matches as correct, the reported recall and precision may understate the clinical usefulness of outputs that are semantically appropriate but differ from the submitted bill; a family-level or coder-reviewed metric would test that."],"forward_implications":["A health system with four 24GB GPUs can build a private, on-site billing-code generator without sending protected health information to an external API.","Fine-tuned small models reduce code fabrication to roughly 1%, which is what makes a coder-assist tool practical; the base and RAG configurations fabricated codes far more often.","Fine-tuning on institutional data outperforms retrieval-augmented prompting for exact code-set generation, suggesting that for highly structured outputs in-domain weights matter more than in-context examples.","The paper's own conclusion bounds the claim: these models are not coder replacements and did not show reduced staffing or faster claim submission, but they could serve as a lower-cost alternative to commercial computer-assisted coding software.","Because both fine-tuned models matched or exceeded GPT-4o, model scale is not the decisive factor for this structured coding task; task-specific adaptation is."],"supporting_citations":[{"why":"Supplies the Phi-3 Mini and Phi-3 Medium base models that all local configurations build on.","marker":"[24]"},{"why":"Provides the QLoRA method and the hyperparameter settings (rank 64, alpha 16, dropout 0.1) used for fine-tuning.","marker":"[29]"},{"why":"Documents that large language models are poor medical coders, the problem the paper targets and the baseline expectation for GPT-4o.","marker":"[11]"},{"why":"Shows that training a health-system-scale foundational model requires extensive resources, motivating the fine-tuning alternative.","marker":"[13]"},{"why":"Supports the use of identifiable patient records by showing that data type affects model performance.","marker":"[23]"},{"why":"Supplies evidence that greedy decoding is a robust deterministic choice for comparable language models.","marker":"[34]"},{"why":"Defines the ROUGE-L metric used to score output format consistency.","marker":"[37]"},{"why":"Defines the METEOR metric used to score output structure.","marker":"[38]"}],"fun_headline_variants":["Fine-tuned Phi-3 medium beats GPT-4o on surgical coding","Local fine-tuned model rivals GPT-4o with 1% code fabrication","Small model fine-tuned on local data matches GPT-4o in billing codes","Cheap fine-tuned Phi-3 outperforms GPT-4o at surgical code generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o baseline was given a fair and comparably optimized prompt and decoding setup; the paper reports the Phi-3 prompts in full but does not specify GPT-4o's prompt, model version, temperature, or generation settings.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned Phi-3 medium beats GPT-4o on surgical coding","Local fine-tuned model rivals GPT-4o with 1% code fabrication","Small model fine-tuned on local data matches GPT-4o in billing codes","Cheap fine-tuned Phi-3 outperforms GPT-4o at surgical code generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2862,"prompt_tokens":1118,"completion_tokens":1744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":734,"completion_tokens_details":{"reasoning_tokens":1656}},"tokens_in":734,"tokens_out":1744,"duration_ms":11661,"temperature":1.0,"reasoning_tokens":1656,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:43:59.211938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same test set with GPT-4o using the exact structured prompt, special-token segmentation, greedy decoding, and output-format instruction used for the fine-tuned Phi-3 models, and compare exact-match recall and precision on ICD-10, CPT, and modifier codes. If GPT-4o then matches or exceeds the fine-tuned Phi-3 Medium, the paper's central comparison would not survive.","supporting_citations":[{"cited_title":"METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments [Internet]","cited_arxiv_id":null,"evidence_quote":"Defines the METEOR metric used to score output structure."},{"cited_title":"Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying","cited_arxiv_id":null,"evidence_quote":"Documents that large language models are poor medical coders, the problem the paper targets and the baseline expectation for GPT-4o."},{"cited_title":"A large language model for electronic health records","cited_arxiv_id":null,"evidence_quote":"Shows that training a health-system-scale foundational model requires extensive resources, motivating the fine-tuning alternative."},{"cited_title":"Towards Medical Billing Automation: NLP for Outpatient Clinician Note Classification [Internet]","cited_arxiv_id":null,"evidence_quote":"Supports the use of identifiable patient records by showing that data type affects model performance."},{"cited_title":"ROUGE: A Package for Automatic Evaluation of Summaries [Internet]","cited_arxiv_id":null,"evidence_quote":"Defines the ROUGE-L metric used to score output format consistency."}],"review_version":1}