{"id":"9250e915-7c64-4f97-9254-71fcac2949ce","arxiv_id":"2504.16448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EMRModel fine-tunes Qwen2.5-7B-Instruct with LoRA and code-style prompts to extract structured medical records from consultations, reaching 88.1 percent weighted F1.","lead":"This paper reports that fine-tuning a general language model with LoRA and code-style prompts converts medical consultation dialogues into structured electronic medical records with 88.1 percent F1 on a private test set. The result is a practical reminder that prompt format and lightweight fine-tuning can matter as much as, or more than, using a code-specialized model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed benefit of code-style prompts over NL prompts is a 0.4-point F1 gap (88.1 vs 87.7) with no significance testing; on 536 test dialogues this is within plausible sampling noise, so the paper's central mechanism is not yet established.","rationale":"The reader's weakest assumption concerns annotation quality and dataset representativeness. Those are legitimate, but they affect the absolute F1 more than the relative comparison between prompt styles, because annotation noise would likely influence both the NL-prompt and code-prompt conditions similarly. The more load-bearing problem is the absence of any significance testing for the very small differences that support the paper's main methodological contribution. The paper's novelty is not merely that a fine-tuned model reaches 88.1% F1, but that code-style prompts plus LoRA on a natural-language model outperform natural-language prompts and coder-model combinations. That claim rests on differences of 0.4 and 0.4 points, with no measure of uncertainty. Given 536 test dialogues, random variation could easily account for such gaps unless the per-sample variance is unusually small, which the paper does not demonstrate. A paired bootstrap or equivalent test would settle this directly. Until such evidence is provided, the empirical support for the prompt-design mechanism is weak, though the overall conditional acceptance remains appropriate given the possibility that the authors can supply the missing analysis and artifacts.","tokens_in":10842,"tokens_out":3973,"duration_ms":36698,"concrete_test":"Ask the authors to perform a paired bootstrap over the 536 test dialogues: resample the dialogue indices with replacement (e.g., 10,000 replicates), recompute the field-level weighted F1 for NL+Code and NL+NL on each resample, and report the 95% bootstrap confidence interval for the mean difference. If the interval includes zero, the code-prompt advantage is not significant and the central mechanistic claim should be downgraded. A secondary check is to independently re-derive the per-field F1 under an explicitly stated exact-match definition from the released outputs to confirm the 0.4-point gap is not an artifact of fuzzy character matching.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanistic claim is that code-style prompts improve extraction performance over natural-language prompts (Section 4.2). The evidence is a weighted F1 increase from 87.7% to 88.1% for the NL model and from 85.0% to 85.4% for the coder model—both 0.4 points. No confidence intervals, error bars on the mean, or hypothesis tests are reported; the reported standard deviation is the spread of per-sample F1, not the standard error of the aggregate estimate. With N=536 test dialogues, a per-sample F1 standard deviation of even 10 points implies a standard error around 0.43 points, so the 0.4-point gap is on the order of one SE. The paper also does not specify the matching granularity (character-level, token-level, or exact match) used to compute per-field precision and recall, and the weights are proportional to ground-truth character count, so long fields dominate the metric. Because the entire contribution of the prompt-design component rests on this small, untested difference, the headline result that code-style prompts enhance performance is not statistically supported as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes EMRModel, which fine-tunes Qwen2.5-7B-Instruct with LoRA using code-style prompts to convert Chinese medical consultation dialogues into structured EMRs. The authors construct a private dataset of 8,665 dialogue-EMR pairs from 46,910 raw consultations via NLP-assisted manual annotation, and evaluate on 536 held-out dialogues using a field-weighted F1 score. They compare four LoRA-fine-tuned configurations (NL vs Coder base model × NL vs Code prompt) and four zero-shot baselines, reporting that NL+Code achieves 88.1% F1, outperforming NL+NL (87.7%) and Coder+Code (85.4%), and that code-style prompts improve both base models. A field-level analysis and qualitative deployment descriptions are also included.","tokens_in":11100,"tokens_out":5569,"duration_ms":50579,"significance":"If the reported 88.1% F1 and the ranking of strategies are reliable, the paper offers a practical recipe for clinical NLP: a general-purpose instruct model adapted with LoRA and code-style prompts outperforms both natural-language-prompt fine-tuning and a code-specialized model, at low training cost. The work also introduces a rare private resource (a multi-department Chinese consultation-dialogue corpus) and a fine-grained field-level evaluation methodology. Strengths include the held-out test set, the systematic comparison of eight configurations, and the explicit sensitivity analysis over base models. However, the core mechanistic claim--that code-style prompting is beneficial under fine-tuning--is supported only by a 0.4-point F1 difference with no uncertainty quantification, and the evaluation depends entirely on a private, self-annotated benchmark. The contribution is therefore promising but not yet established at the level claimed.","major_comments":[{"comment":"The central claim that code-style prompts improve extraction performance over natural-language prompts rests on weighted-F1 differences of 0.4 points (88.1% vs 87.7% for the NL model; 85.4% vs 85.0% for the Coder model) on a single test set of 536 dialogues. No confidence intervals, bootstrap estimates, or significance tests are reported; the standard deviation shown in Figure 5 is the per-sample spread of F1, which is not the standard error of the aggregate mean. With N=536, a per-sample SD of roughly 10 points already implies a standard error of about 0.43 points, making the 0.4-point gap comparable to one SE. The conclusion that code-style prompts 'significantly' improve performance (Section 4.2) and the abstract's improvement claims should be supported by paired tests or bootstrap CIs, or else be reworded as a non-significant trend.","section":"4.2, Figure 5"},{"comment":"The dataset contribution is asserted as high-quality, but two aspects are not substantiated. First, the reduction from 46,910 raw records to 8,665 retained records is unexplained; if the excluded records are noisier or more complex, the reported F1 may not reflect deployment conditions. Second, the gold annotations are created by NLP pre-extraction followed by manual review, which is the same method family as the model being evaluated; without inter-annotator agreement statistics or an audit of a random test subset, the absolute 88.1% cannot be distinguished from annotation-pipeline bias. The authors should report annotation agreement and a characterization of the filtering step, and ideally release the test set or a public sample for external benchmarking.","section":"3.2.1, Eq. (2)"},{"comment":"The evaluation metric is under-specified: the manuscript does not state how predicted field values are matched to ground truth (exact character match, token-level, or fuzzy), and the weights are ground-truth character counts, so the few long unstructured fields (e.g., 'treatment recommendations') dominate the aggregate. This matters because the reported advantage of code-style prompts is largest for exactly those fields (Section 4.3.3, Figure 8), and the per-field F1 differences are also reported without uncertainty. The authors should define the matching rule, report per-field sample sizes and variances, and show whether the 0.4-point overall gap survives when short structured fields are removed or when alternative weighting schemes (e.g., uniform across fields) are used.","section":"4.1.2, Eq. (8)"}],"minor_comments":[{"comment":"The phrase 'improving by 49.5% over standard pre-trained models' is ambiguous; the number appears to be an absolute percentage-point difference between 88.1% and the zero-shot coder model's 38.6%, not a relative improvement. Please specify whether the gain is absolute or relative.","section":"Abstract"},{"comment":"The notation |D'| in the objective function (7) is not introduced; also, the dimensions in Eq. (6) (A ∈ R^{r×k}, B ∈ R^{d×r}) are inconsistent with W ∈ R^{d×d} unless k=d. Please clarify the dimension notation.","section":"3.3.2, Eqs. (6)-(7)"},{"comment":"The 'Scenario Verification' section describes two deployments but provides no quantitative evaluation; it should be labeled a qualitative case demonstration rather than a verification.","section":"4.4"},{"comment":"Several typos appear: 'EMRModeL' in Section 4.4, 'maritaland' in Section 4.3.3, and 'the i physician-patient dialogue text' in Eq. (4). These should be corrected.","section":"4.3.3, Figure 8"},{"comment":"The dataset is a core contribution, but no data availability statement is provided. The authors should state whether, and under what privacy conditions, the dataset or a sample can be shared to allow external verification.","section":"3.2"},{"comment":"Reference [3] concerns image processing tool condition monitoring and appears off-topic; consider replacing it with a relevant medical-NLP reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper would be substantially strengthened by releasing at least a small public dev/test sample and by adding statistical uncertainty quantification for the headline comparison. If the authors cannot release data or provide significance tests, the abstract and conclusions should be scaled back to avoid overstating the prompt-design benefit. The work fits a clinical NLP application venue, but as a scientific claim the core mechanism is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know before you read this one. First, the dataset is real and valuable: 8,665 physician-patient dialogues from multiple Chinese hospitals, cleaned and annotated into structured EMR fields. That part deserves attention. Second, the paper's central mechanistic claim—that code-style prompts help after LoRA fine-tuning—is not supported by the evidence as reported. The entire gain is 0.4 points on weighted F1 (88.1 vs 87.7), and with 536 test dialogues that is roughly one standard error.\n\nWhat the paper does well: it runs eight strategies in a systematic sweep, comparing general NL models against coder models, with and without code prompts, with and without LoRA. The finding that LoRA fine-tuning lifts both model families from roughly 40-50% F1 to over 85% is robust and large by any measure. The observation that a general-purpose NL model fine-tuned with LoRA beats a coder model on this extraction task, once fine-tuning is in play, is a useful empirical point that goes beyond the cited ICL-only work (C-ICL, CodeIE, CodeKGC). The field-level breakdown is also a reasonable diagnostic: structured fields are near ceiling, and the hard field ('treatment recommendations') is where the models genuinely struggle.\n\nThe soft spots are real but mostly in one cluster. The code-prompt-vs-NL-prompt comparison is the advertised novelty, and it falters on statistics. No confidence intervals, no significance tests, and the reported standard deviation is the per-sample F1 spread, not the standard error of the aggregate. On top of that, the matching granularity for precision/recall is never specified, and the field weights are character-count based, so long narrative fields dominate the metric. The dataset is private, no code is released, hyperparameters are not given, and the retained 8,665 records out of 46,910 may carry selection bias from the cleaning pipeline. Those are all fixable in revision.\n\nThe paper is not circular. The evaluation is on a held-out test set, and the F1 is not a fitted parameter. But the benchmark is self-constructed, and external validation is absent.\n\nWho should read this: anyone working on medical NLP or EMR generation, especially in Chinese clinical settings. It deserves a serious referee: the data collection effort is substantial, the experiments are internally consistent, and a revision requiring significance testing and artifact release could make it solid.\n\nMy recommendation: send it to peer review, but with the expectation of a substantial revision. The prompt-design claim needs to be either supported by error bars or softened; the authors should release at least the benchmark and code.","headline":"Solid dataset and a credible LoRA fine-tuning story, but the paper's signature claim about code-style prompts is a 0.4-point F1 gap that is statistically indistinguishable from noise.","tokens_in":11656,"tokens_out":2715,"would_cite":false,"duration_ms":24647,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a general-purpose instruction-tuned language model fine-tuned with LoRA on code-style prompts converts medical consultation dialogues into structured records at 88.1 percent weighted F1, outperforming…","keywords":["electronic medical record generation","medical consultation dialogue","information extraction","Low-Rank Adaptation","code-style prompting","structured output","clinical NLP","benchmark dataset"],"falsifier":"Annotate a random sample of the 536 test dialogues with a second, independent annotation team and measure character-level agreement; if agreement on free-text fields such as treatment recommendations is low, the 0.4-point gap between code prompts and natural-language prompts is within annotation noise. Alternatively, re-run the same LoRA-plus-code-prompt recipe on an independently collected public medical dialogue set; if the code-prompt advantage disappears, the reported gain is dataset-specific.","tokens_in":10655,"feed_emoji":"🩺","tokens_out":8798,"duration_ms":73333,"temperature":0.7,"pith_summary":"This paper tries to establish that a general-purpose instruction-tuned language model, adapted by Low-Rank Adaptation (LoRA) and prompted with code-style templates, is the best of eight compared strategies for turning doctor-patient consultation dialogues into structured electronic medical records. It reports a weighted F1 of 88.1 percent on a private 536-dialogue test set, well above both its zero-shot baselines and the other fine-tuning configurations. The result matters because medical records are still largely created by hand or by rigid rules, and the paper shows a lightweight, parameter-efficient fine-tuning recipe plus structured prompts can close much of the gap. The paper also contributes a curated dataset of 8,665 dialogues and a field-weighted evaluation metric for this extraction task.","feed_headline":"Code-style prompts hit 88.1% F1 in medical record extraction","feed_subtitle":"LoRA fine-tuning plus code prompts beats coder models and plain prompts on medical dialogue-to-EMR conversion","key_machinery":"The load-bearing mechanism is the code-style prompt template combined with LoRA fine-tuning. A prompt-encoder embeds each dialogue into a code snippet that turns EMR generation into a constrained code-completion problem; LoRA updates only low-rank matrices A and B while freezing the base weights, keeping adaptation cheap and preserving general language ability; a prompt-decoder parses the code-shaped output back into structured fields. Evaluation uses a weighted F1 in which each field is weighted by its character count in the ground truth, so longer, harder fields such as treatment recommendations carry more weight.","core_discovery":"The paper's central claim is that structured electronic medical records can be generated from consultation dialogues by combining a general-purpose natural-language model's semantic understanding with a code-completion-style output format. Its proposed method, EMRModel, applies Low-Rank Adaptation (LoRA) to an instruction-tuned 7-billion-parameter language model, training only small low-rank weight updates on dialogue-to-EMR examples that are formatted as code templates. On the private 536-dialogue test set this configuration achieves a weighted F1 of 88.1 percent. The same base model with natural-language prompts reaches 87.7 percent, a code-specialized base model with code prompts reaches 85.4 percent, and non-fine-tuned baselines range from 40.1 to 81.9 percent. The authors take this ordering as evidence that code-style prompts provide format control while the natural-language base provides the semantic grasp, and that this pairing matters more than using a code-specialized model.","pith_inferences":["My inference: the reported gap between code prompts and natural-language prompts is only 0.4 F1 points overall, so the practical advantage may be concentrated in a few free-text fields; on already structured fields like age and gender, code prompts likely add little.","My inference: because the test set comes from the same annotation pipeline that produced training data, the absolute 88.1 percent figure may not transfer to other note templates or hospitals; the relative ordering of the eight strategies is the stronger claim.","My inference: the character-count weighting of fields means that a field with longer gold text dominates the score; a clinically weighted metric might rank the strategies differently.","My inference: the code-prompt effect could be isolated further by ablating the prompt syntax (function signature versus JSON schema versus class definition) to identify which structural feature drives the gain."],"forward_implications":["LoRA fine-tuning alone lifts extraction F1 from below 50 percent for a general-purpose base to above 80 percent, making task-specific EMR generation feasible at low parameter cost.","Code-style prompts improve accuracy most on semi-structured and flexibly phrased fields such as treatment recommendations, where output-format constraints matter.","A natural-language base model outperforms a code-specialized base model after fine-tuning, so coding ability in the base is not the decisive factor for this extraction task.","The field-weighted F1 metric and the de-identified dataset of 8,665 dialogues provide a reusable benchmark for medical consultation extraction research.","The two deployed tools, a messaging-platform assistant and a hospital information-system module, show the approach can fit into existing clinical workflows."],"supporting_citations":[{"why":"Defines the LoRA low-rank adaptation method that EMRModel uses for parameter-efficient fine-tuning.","marker":"[11]"},{"why":"Shows that code-style prompts improve in-context learning for information extraction, motivating the prompt design.","marker":"[13]"},{"why":"Provides the LoRA formulation of freezing base weights and training low-rank matrices.","marker":"[20]"},{"why":"Demonstrates that code generation models with code-style prompts are stronger few-shot information extractors.","marker":"[23]"},{"why":"Shows code-style prompts improve generative knowledge graph construction, another structured-output task.","marker":"[25]"},{"why":"Reviews clinical information extraction applications and motivates the need for structured records from unstructured clinical text.","marker":"[21]"}],"fun_headline_variants":["Code prompts with LoRA hit 88.1% F1 for medical record extraction","Semantic base model beats code-specialized using code-style prompts","LoRA plus code prompts: 88.1% F1 on dialogue-to-EMR conversion","Generic LLM + code format outperforms code-specific models on EMRs","Why natural-language base wins with code prompts for medical records"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gold-standard EMR labels created by NLP pre-extraction plus manual review are accurate and representative enough that the reported 88.1 percent F1 and the ranking of the eight strategies reflect real extraction quality.","fun_headline_variants_meta":{"raw":{"variants":["Code prompts with LoRA hit 88.1% F1 for medical record extraction","Semantic base model beats code-specialized using code-style prompts","LoRA plus code prompts: 88.1% F1 on dialogue-to-EMR conversion","Generic LLM + code format outperforms code-specific models on EMRs","Why natural-language base wins with code prompts for medical records"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2885,"prompt_tokens":944,"completion_tokens":1941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1849}},"tokens_in":560,"tokens_out":1941,"duration_ms":13857,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:02:19.907051+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a random sample of the 536 test dialogues with a second, independent annotation team and measure character-level agreement; if agreement on free-text fields such as treatment recommendations is low, the 0.4-point gap between code prompts and natural-language prompts is within annotation noise. Alternatively, re-run the same LoRA-plus-code-prompt recipe on an independently collected public medical dialogue set; if the code-prompt advantage disappears, the reported gain is dataset-specific.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the LoRA low-rank adaptation method that EMRModel uses for parameter-efficient fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reviews clinical information extraction applications and motivates the need for structured records from unstructured clinical text."}],"review_version":1}