{"id":"419f0c93-8b92-4f1f-8056-224e3ad2034e","arxiv_id":"2501.09384","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Prompt formatting, feature selection, and in-context example choice significantly affect LLM performance on EHR table extraction and retrieval, with query-based demonstrations helping extraction.","lead":"This paper tests two open-source language models on extracting and retrieving patient information from electronic health record tables. It finds that prompt formatting and feature selection can shift performance by up to roughly 27%, and that retrieval is harder than extraction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains are chosen from test-set maxima without a validation split, so the 26.79% and 5.95% magnitudes are likely inflated by selection bias; a held-out configuration-selection rerun is needed.","rationale":"Reader's weakest_assumption is the same one I would flag. The paper has real assets: two new datasets, code release, multiple baselines, and a plausible qualitative story consistent with prior tabular-LLM work. But the central claim is quantitative ('up to 26.79%', '+5.95%'), and the route to those numbers is test-set selection. This matters for the contribution: practitioners following the proposed guidelines need an unbiased estimate of the gain, not the maximum over many prompt configurations. A second under-specified point is that sgen is defined as an LLM-generated description 'in function of a given question' without naming the generator; if the generator is the evaluated model, the serialization comparison is partly circular. I keep that secondary because the text is ambiguous, while the validation leak is directly documented. Given the reproducible artifacts, I would not reject the paper; the appropriate outcome is the reader's conditional verdict pending a validation-based rerun.","tokens_in":13200,"tokens_out":8693,"duration_ms":87518,"concrete_test":"Use the released dev splits (MIMICask dev=96, MIMICsearch dev=368) to select, per model, the single best (Fp, phi, instruction, sigma, k) by dev Bscore/MAP; then evaluate only those configurations on the held-out test splits and report bootstrap 95% CIs per metric. If the chosen-configuration advantage over the rnd/rndavg baseline falls below 26.79% (and 5.95% for ICL) or includes zero, the abstract's magnitudes are selection artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims are not supported by a valid model-selection protocol. Section 4.1 says 'we chose the best settings from Table 2' after inspecting test performance, and Sections 4.2-4.3 keep choosing 'best-performing settings' from Tables 3-4 before reporting final comparisons. Because the best configuration is selected from the same test evaluations that are then used to report improvements, the 26.79% and 5.95% figures are maxima over 12-16 unpaired configurations rather than estimates of an expected effect; no confidence intervals, seeds, or multiple-comparison correction are provided. The Delta metric itself is an average of per-metric relative gains, so the 26.79% headline is dominated by an 80% relative ROUGE-1 jump from a 12.69 baseline while BERTScore rises only about 9%. The qualitative direction (serialization and feature selection matter; retrieval is hard) may survive, but the quantitative magnitude at the center of the abstract does not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents an empirical study of how prompt design affects the ability of two open 7B LLMs (Llama2 and Meditron) to perform patient data extraction and retrieval from tabular electronic health records. The authors introduce two new datasets derived from MIMICSQL, MIMICask and MIMICsearch, and evaluate combinations of feature selection (all, all aggregated, random, random aggregated), serialization (template, X-separate, self-generated), instruction type (guided vs non-guided), and in-context demonstration selection (patient-based, query-based, random) using metrics ROUGE-1 and BERTScore for extraction and MAP and Recall@100 for retrieval. The paper reports that feature selection and serialization improve performance by up to 26.79%, that query-based demonstrations improve extraction by 5.95%, that guided instructions have minimal impact, and that retrieval is more difficult than extraction. It concludes with practical guidelines for prompting LLMs on EHR data.","tokens_in":13344,"tokens_out":10714,"duration_ms":125805,"significance":"If the quantitative findings were supported by a sound evaluation protocol, this would be a useful contribution to the growing literature on LLMs and tabular EHR data. The paper provides two new benchmark datasets, releases code, compares against several baselines including fine-tuned models and TREQS, and addresses a practical question that is relevant to health-search applications. However, the headline improvement magnitudes are currently threatened by a configuration-selection protocol that uses the test set, and by an aggregate metric that is dominated by a single low-baseline component. The qualitative direction of the findings is plausible and the datasets and code are valuable, but the central quantitative claims need to be re-established with a valid selection procedure and with uncertainty quantification.","major_comments":[{"comment":"The optimal settings are selected from the test evaluations themselves: Section 4.1 states 'we chose the best settings from Table 2', and Sections 4.2 and 4.3 then fix 'best-performing settings' from the same tables before reporting final comparisons. Consequently, the headline gains of 26.79% and 5.95% are maxima over many unpaired configurations rather than unbiased estimates of an expected effect; they are likely inflated by selection bias. Please either hold out a validation split for configuration selection and report test results only for the final configuration, or use a selection-aware evaluation with appropriate correction for multiple comparisons.","section":"Section 4.1, Table 2"},{"comment":"All results appear to be single runs with no variance estimates or significance tests. Several comparisons are extremely small, e.g., Meditron retrieval MAP 10.34 vs 10.30 in Table 2 and BERTScore differences near 0.1 in Table 4, so summary claims such as '5 out of 8 metrics' and '9 out of 16 metrics' may reflect noise. Please report multiple seeds or bootstrap confidence intervals, and account for the number of configurations compared when claiming that one setting is optimal.","section":"Tables 2-4"},{"comment":"The Delta aggregate in Table 2 is an average of per-metric relative changes. For the Llama 'all/txt' row versus the 'rnd/txt' baseline, the relative gains are approximately +8.9% BERTScore, +80.0% ROUGE-1, +6.7% MAP, and +11.7% Recall@100, which average to the reported 26.79%. The headline is therefore dominated by a single low-baseline metric. Please report per-metric deltas alongside the aggregate and justify the aggregation, because the current abstract implies a more uniform across-the-board improvement than the data show.","section":"Table 2 and abstract"},{"comment":"The random feature-selection strategy keeps only 60% of the features (Section 3.2, footnote 3), so the 'all' versus 'rnd' and 'allavg' versus 'rndavg' comparisons vary both the feature subset and the number of features. The guideline 'Context is improved when using all available EHR features' is therefore not a clean feature-selection result. A comparison with matched feature counts, or with a systematic feature-selection method, is needed to attribute the gain to selection rather than to the sheer quantity of information.","section":"Section 3.2, Table 2"},{"comment":"The claim that retrieval is more difficult than extraction is not directly established by the presented comparisons, because the two tasks are evaluated with different metrics (Bscore/R-1 vs MAP/R@100) and the cross-task argument relies on relative improvements over task-specific baselines. Please reframe this as a claim about LLM performance relative to strong task-specific baselines, or provide a common evaluation protocol that makes task difficulty commensurable.","section":"Section 4.1, Section 6"}],"minor_comments":[{"comment":"Table 1 appears misaligned: the column labeled '# k/n' shows values 34 and 557, which do not equal the ratios k/n for MIMICask (5414/100) and MIMICsearch (19970/4000). Please correct the table or the column label.","section":"Table 1"},{"comment":"The sentence 'explored LLMs perform better than all baselines, with the best score for Lma* with Bscore=62.44, except BART_ft' is self-contradictory; Table 5 shows that Lma*_ft reaches the best Bscore (84.79). Please reword to distinguish zero-shot LLMs, fine-tuned LLMs, and fine-tuned pre-trained baselines.","section":"Section 5, Table 5"},{"comment":"The statement that in retrieval 'studied LLMs only outperform MonoBERT' is imprecise: Med* beats MonoBERT on MAP but not on Recall@100, and Lma* does not beat it on either metric. Please report the per-metric comparisons accurately.","section":"Section 5, Table 5"},{"comment":"The text says 'We used the MIMICsearch small dataset for our study', but the experimental setup does not explicitly state that the reported retrieval results use the 250-query test set rather than the 1101-query set. Please state this clearly in the setup.","section":"Section 3.3, Table 1"},{"comment":"The formal definition of the demonstration retrieval function sigma says it provides examples for both extraction and retrieval, but the patient-based function sigma_p is only applicable to extraction. Please clarify the domain of each retrieval function.","section":"Section 3.2, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The datasets and code are real assets, and the qualitative conclusions (serialization and feature choices matter, retrieval remains hard for these models) are likely to survive a re-analysis. However, the current quantitative claims rest on test-set selection and an aggregate metric that masks large per-metric variation. A revision that performs configuration selection on a validation split, reports confidence intervals or multiple seeds, and decomposes the Delta metric would allow the central claims to be assessed fairly. The cross-task 'retrieval is harder' claim should also be reframed to avoid comparing non-commensurable metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about 2501.09384. First, it ships two new EHR benchmark datasets (MIMICask, MIMICsearch) and code, which is real value. Second, the headline numbers—26.79% and 5.95%—are not trustworthy as reported, because the best configurations were chosen from the same test set used for the final comparison. That is not a fatal flaw, but it means the magnitudes are likely inflated.\n\nWhat is new: the paper applies known serialization and ICL techniques to EHR extraction and retrieval, and shows retrieval is harder than extraction for both Llama2 and Meditron. That qualitative result is plausible and consistent with prior tabular-LLM work. The finding that query-based demonstrations help extraction more than patient-based ones is also useful. The datasets fill a small gap and the baselines are reasonable.\n\nWhere it gets soft. Section 4.1 says \"we chose the best settings from Table 2\" after examining the table; Sections 4.2 and 4.3 do the same. Selecting on the test set makes the reported improvements maxima over 12–16 configurations, not expected effects. There are no confidence intervals or significance tests. The Delta metric is an unweighted average of relative gains per metric, so the 26.79% is dominated by an 80% relative ROUGE-1 jump from a 12.69 baseline; BERTScore improves only about 9% in the same row. That should be stated clearly in any revision. Also, the study covers only two 7B Llama-based models, so the guidelines are narrower than the title suggests.\n\nIs it still worth refereeing? Yes. The datasets and code are reproducible, the qualitative direction is credible, and the methodological issue is fixable: rerun on a held-out validation split or use a proper selection protocol, report per-metric changes with confidence intervals, and avoid averaging relative improvements across metrics. I would bring it to the reading group for a discussion of selection bias in empirical LLM papers, and I'd cite the datasets if I worked in this area. Send it to a serious referee, but the referee should ask for the validation rerun before acceptance.","headline":"Useful new EHR benchmark datasets and credible qualitative findings, but the headline effect sizes are inflated by test-set selection and by averaging relative gains across metrics.","tokens_in":13902,"tokens_out":3190,"would_cite":true,"duration_ms":28463,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt design changes LLM performance on tabular EHR extraction and retrieval by up to 26.79%.","keywords":["large language models","electronic health records","tabular data","information extraction","information retrieval","in-context learning","prompt engineering","MIMICSQL"],"falsifier":"Run the same MIMICask and MIMICsearch experiments but select the prompt configuration on a held-out validation split and evaluate only on untouched test patients; if the best-configuration advantage over the naive baseline shrinks below the reported 26.79% and 5.95% or disappears, the claim of large prompt-design gains is not supported.","tokens_in":13019,"feed_emoji":"🏥","tokens_out":8580,"duration_ms":74869,"temperature":0.7,"pith_summary":"This paper asks whether the way an EHR table is turned into text determines how well large language models can extract a single patient's answer from it and retrieve which patients match a query. Using the MIMICSQL patients, it compares Llama2 and Meditron across serialization formats, feature-selection strategies, guided versus non-guided instructions, and in-context demonstrations. It claims that combining all features (or their averaged values) with an LLM self-generated table description improves task performance by up to 26.79% over naive random feature choices, and that query-matched demonstrations improve extraction by 5.95%. It also claims retrieval is consistently harder than extraction for both models. The practical payoff is a set of prompt-design guidelines for building LLM-based health search tools.","feed_headline":"Prompt design changes LLM EHR performance by up to 26.79%","feed_subtitle":"Which features you include and how you write the table matters more than instructions; retrieval lags extraction.","key_machinery":"The load-bearing object is the prompt template `\\textless Instruction [Demonstration] Context \\textgreater` with `Context = $\\phi(T_i) + q$`, where `$\\phi$` is a serialization function and `$T_i$` is the patient's EHR table restricted to a feature subset. The paper compares three serializations (plain text template `txt`, HTML-tag-separated `xsep`, and LLM self-generated `sgen`) crossed with four feature selections (all, all averaged, random, random averaged). The second mechanism is the demonstration retriever `$\\sigma$`, which chooses in-context examples by patient similarity or query similarity; it is the component that makes ICL help extraction. Serialization plus feature selection carries most of the observed performance spread; instruction type carries little.","core_discovery":"The paper's central claim is that a prompt's context construction—specifically the combination of which medical features are kept and how the table is serialized—is the dominant factor in LLM performance on tabular EHR tasks, more than instruction phrasing. With the best settings, Llama2 and Meditron improve by 26.79% and 21.53% respectively over naive random feature selection; the best context uses all features (or averaged longitudinal values) and often an LLM self-generated description. In-context learning helps extraction when demonstrations are selected by query similarity (+5.95% BERTScore), but does not help retrieval, which performs best zero-shot. The authors read this as evidence that LLMs can extract fine-grained patient data from well-serialized tables but still struggle to match coarse patient-level queries to the right tables.","pith_inferences":["A replication that selects prompt configurations on a held-out validation split rather than the test set would clarify how much of the reported 26.79% and 5.95% gains are genuine versus selection-dependent.","The serialization-and-feature-selection interaction may transfer to other structured clinical tables, such as pharmacy or imaging metadata, but the paper only studies MIMIC-derived tables, so that transfer is untested.","The extraction-versus-retrieval gap suggests EHR retrieval may need task-specific decoding or summary-based candidate generation rather than pointwise relevance judgments; the paper does not explore that alternative."],"forward_implications":["Prompt context, not instruction wording, is the main lever: feature selection plus serialization should be tuned before trying to improve instructions.","Using all available features or their averaged values beats random feature subsets for both models, so defaulting to all features is a low-cost improvement.","Query-similar demonstrations should be the default ICL choice for extraction; patient-similar or random demonstrations offer no consistent benefit and can hurt.","Retrieval tasks should be treated separately: zero-shot prompting outperforms ICL for retrieval, and LLM pointwise rankers underperform BM25 and MonoT5 on the studied data.","MIMICask and MIMICsearch provide reusable benchmarks for future EHR extraction and retrieval evaluation."],"supporting_citations":[{"why":"Supplies the MIMICSQL question-SQL pairs and patient tables on which MIMICask and MIMICsearch are built.","marker":"[27]"},{"why":"Provides the MIMIC-III EHR database underlying the study's patient tables.","marker":"[10]"},{"why":"Defines table-serialization sensitivity findings and the self-generated table description approach (sgen) used as a competing context format.","marker":"[24]"},{"why":"Provides the X-separate serialization strategy used as a competing context format.","marker":"[25]"},{"why":"Motivates the template- and separator-based serialization choices tested in the prompt designs.","marker":"[21]"},{"why":"Shows LLM classification with tabular serialization and serves as a baseline for extraction-style tasks.","marker":"[8]"},{"why":"Supports the idea that retrieving high-quality demonstrations improves in-context learning, the basis of the query-based demonstration retriever.","marker":"[19]"},{"why":"Supports selective example selection for in-context learning, motivating the comparison of demonstration strategies.","marker":"[34]"},{"why":"Supplies the zero-shot LLM pointwise re-ranker setup used for the retrieval task.","marker":"[36]"}],"fun_headline_variants":["Context beats instructions in LLM EHR extraction tasks","Feature selection boosts LLM patient data extraction by 26.79%","LLM EHR: How you write the table matters most","Prompt context dominates LLM accuracy on health records"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gains assume that picking the best settings from Table 2 on the test set is a valid way to measure those settings' benefit; without a separate validation split, the reported improvements may be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Context beats instructions in LLM EHR extraction tasks","Feature selection boosts LLM patient data extraction by 26.79%","LLM EHR: How you write the table matters most","Prompt context dominates LLM accuracy on health records"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1299,"prompt_tokens":878,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":494,"tokens_out":421,"duration_ms":5014,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:04:36.650244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same MIMICask and MIMICsearch experiments but select the prompt configuration on a held-out validation split and evaluate only on untouched test patients; if the best-configuration advantage over the naive baseline shrinks below the reported 26.79% and 5.95% or disappears, the claim of large prompt-design gains is not supported.","supporting_citations":[{"cited_title":"Sci Data3 (2016)","cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-III EHR database underlying the study's patient tables."},{"cited_title":"In: Findings of the Association for Computational Lin- guistics: EMNLP 2024 (2024) 16 Author","cited_arxiv_id":null,"evidence_quote":"Provides the X-separate serialization strategy used as a competing context format."},{"cited_title":"In: Table Representation Learning Workshop at NeurIPS 2023 (December 2023)","cited_arxiv_id":null,"evidence_quote":"Motivates the template- and separator-based serialization choices tested in the prompt designs."},{"cited_title":"In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V","cited_arxiv_id":null,"evidence_quote":"Supports the idea that retrieving high-quality demonstrations improves in-context learning, the basis of the query-based demonstration retriever."},{"cited_title":"In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval","cited_arxiv_id":null,"evidence_quote":"Supplies the zero-shot LLM pointwise re-ranker setup used for the retrieval task."}],"review_version":1}