{"id":"7de539b3-c664-402d-b59d-a076f875569d","arxiv_id":"2504.17360","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"PatientDx merges a math-specialized LLM with a medical or instruct LLM via SLerp and reports mortality-prediction gains on MIMIC-IV, but the merging weight is tuned on the test set, undermining the claimed improvement.","lead":"PatientDx merges existing large language models, such as a math-specialized model and a medical or instruction-tuned model, to predict patient mortality without fine-tuning on patient data. The authors report AUROC gains on MIMIC-IV, but the merging weight was selected on the same test set used for evaluation, so the headline results need confirmation on a truly held-out set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported AUROC gain is not evidence of generalization because lambda was selected on the same test partition used for evaluation; the paper's own Limitations section admits this.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: the test partition is used for hyperparameter selection, so the reported AUROC improvements are in-sample and cannot support the generalization claim. I agree with this assessment and find it confirmed by the manuscript's own text: footnote 2 in Section 4.1 and the Limitations section both acknowledge that selection was performed on the test partition. This is not a matter of disagreeing with current consensus; it is an internally admitted evaluation leak. The central claim—that merging without patient-data adaptation yields a model that outperforms baselines—collapses if the only way to get that performance is to tune λ on the test labels. The privacy claim is also affected: the framework does use patient data for model selection, even though it does not train the LLM, so the abstract's \"without requiring fine-tuning nor adaptation on patient data\" is too strong. The paper does have positive aspects: a clear framing, a public model release, and an ablation study; these are real contributions but they do not rescue the main quantitative result. The proposed concrete test would settle the concern by measuring whether the headline gain survives a clean validation split. Since the reader already reached REJECT and my analysis supports that verdict, no change to the reader's verdict is needed.","tokens_in":15103,"tokens_out":3566,"duration_ms":37128,"concrete_test":"Re-run the PatientDx pipeline with a proper held-out test fold: on the remaining data, create a validation split and select λ for each merge by maximizing AUROC on that validation split only (e.g., grid over λ ∈ {0.1, 0.2, ..., 0.9}). Evaluate the chosen model once on the untouched test fold and compare PatientDx 8B's AUROC with Meditron 7B and the input models, reporting bootstrap confidence intervals. If the gap shrinks to within chance, or if the validation-selected λ differs from λ*=0.4, the Table 2 headline is a selection artifact rather than evidence of generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that PatientDx 8B improves over Meditron 7B and the input models—depends on the test partition being a clean held-out set. Section 4.1 states that hyperparameter selection was performed with a 2-fold partition, and footnote 2 clarifies: \"Only in test partition given the computational cost.\" The Limitations section then confirms that \"hyper-parameter selection could be performed on the full training set (without k-fold on test as described in §4.1) but at significantly higher computational cost.\" Thus the λ* values in Table 2 (e.g., λ*=0.4 for PatientDx 8B) were chosen by maximizing AUROC on the same Mortality and Mortality-hard test instances used to report the headline numbers. The reported 0.6338 AUROC is therefore an in-sample selection maximum over the λ grid, not an unbiased estimate of performance on unseen patients. The comparison against zero-shot, untuned baselines is unfair: the merged models receive test-set adaptation while the baselines do not. This also weakens the privacy claim. The abstract says PatientDx works \"without requiring fine-tuning nor adaptation on patient data,\" yet the framework tunes hyperparameters on patient examples and selects the best configuration on test outcomes. While no gradient training occurs, the model is still chosen using private patient data, so \"no adaptation on patient data\" is inaccurate. The DLT perplexity metrics in Table 4 do not detect this selection-based use of test data, so they do not establish that PatientDx is privacy-preserving in the claimed sense.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PatientDx, a framework that builds task-specialized LLMs for healthcare prediction by merging publicly available pre-trained models using SLerp or linear interpolation. The proposed configurations combine an instruction-tuned or biomedical model with a math-specialized model (Mathstral or DART-math), and are evaluated on MIMIC-IV mortality and mortality-hard prediction tasks. The paper reports AUROC/AUPRC gains over zero-shot baselines, an ablation of merging components, perplexity-based DLT leakage diagnostics, a qualitative medical question-answering example, and a query-expansion retrieval experiment on ReCDS-PPR. The central claim is that model merging avoids fine-tuning or adaptation on patient data while improving predictive performance and reducing leakage risk.","tokens_in":15423,"tokens_out":7910,"duration_ms":73267,"significance":"If the central claim were supported, the paper would make a useful contribution: model merging would be a lightweight, privacy-preserving alternative to fine-tuning for numerical electronic health record tasks, and the choice of a math-specialized pivot is a concrete, transferable design insight. The authors are also to be credited for releasing the best model, for using a standard merging toolkit, and for including an ablation and a leakage diagnostic that go beyond task accuracy. However, the experimental protocol as reported does not support the generalization claim: the hyperparameters in Table 2 are selected on the same test partition used for the reported scores, so the headline improvement is an in-sample selection artifact. The leakage analysis, while a useful diagnostic, does not measure the influence of the selection procedure on the released model. The core idea remains worth pursuing, but the evidence presented here needs a clean held-out evaluation before the claims can be accepted.","major_comments":[{"comment":"The protocol described in §4.1 and in the Limitations section means that the λ* values in Table 2 are chosen by maximizing AUROC on the exact test partition that is then used to report the headline results. The footnote 'Only in test partition given the computational cost' and the Limitations sentence 'hyper-parameter selection could be performed on the full training set (without k-fold on test as described in §4.1)' make this explicit. Consequently, the PatientDx 8B AUROC of 0.6338 is the value at the best λ on the test fold, not an unbiased estimate of performance on unseen patients, and the comparison against zero-shot, untuned baselines is not a fair evaluation of generalization. This directly undermines the answer to RQ1 and the abstract's claim that the method works without adaptation on patient data. A clean re-evaluation with λ chosen on a training fold and reported on a truly held-out fold, or a nested cross-validation, is required before the improvement claim can be supported.","section":"§4.1, footnote 2; §5 Limitations"},{"comment":"The statement that PatientDx works 'without requiring fine-tuning nor adaptation on patient data' is not supported by the reported procedure. Section 3.2 says that λ is optimized using a performance metric on examples, and Section 4.1 places that optimization on the test partition. Thus the final model is selected using patient inputs and patient outcomes, even though no gradient updates are performed. The word 'adaptation' in the abstract should be restricted to 'without gradient-based training'; otherwise the privacy motivation is overstated. The DLT metrics in Table 4 assess the released model's perplexity, but they cannot detect the fact that the model's hyperparameters were chosen using the test labels, so they do not establish that the selection process preserved privacy.","section":"§3.2 and Abstract"},{"comment":"No uncertainty quantification or significance test is provided for any of the AUROC/AUPRC comparisons. With 6,155 patients and a 10.22% positive rate, the difference between PatientDx 8B (0.6338) and the strongest baseline Meditron 7B (0.5890) is 0.0448 AUROC, and the difference from the input models is about 0.13; neither difference is accompanied by a confidence interval or a test such as DeLong or bootstrap. Several baselines sit at exactly 0.50, which suggests low signal, and the ablation percentages in Table 3 are likewise reported without any indication of variance. Without this, the central 'outperforms' claim is not statistically grounded.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"The sentence 'PatientDx 8B improves by 0.07 absolute points, the strongest baseline' is inconsistent with the table: 0.6338−0.5890=0.0448. The abstract's 'up to 7%' should also be labeled as relative rather than absolute improvement, or corrected.","section":"§4.2, Table 2"},{"comment":"The Hugging Face URL in the abstract points to 'mistral_merged_0_4', but the best model in Table 2 is PatientDx 8B, a Llama-based merge at λ*=0.4; please make sure the released artifact matches the reported configuration and the URL is described correctly.","section":"Abstract / §4.2"},{"comment":"Figure 1 is referenced in §3.1 to support Observation 2, but the figure itself is not included in the text; either include it or describe the data behind the curves in prose.","section":"Figure 1"},{"comment":"The caption says 'rows represent better performance' for the Δ metrics, which is ambiguous because Δ1 and Δ2 move in opposite directions; the caption should state explicitly that larger Δ1 and smaller Δ2 are better.","section":"§4.3, Table 4"},{"comment":"The statement that 'statistical tests show no significance between the two' should name the test used and report the p-value, or be removed; otherwise the claim is not verifiable.","section":"§4.4, Table 6"},{"comment":"The ablation percentages do not state whether the drops are relative or absolute; Table 3 gives e.g. '↓21.1%' for PatientDx 8B w/o Math, but the parenthetical percentages should be defined.","section":"§4.2, Table 3"}],"recommendation":"major_revision","confidential_remarks":"I believe the core idea is promising, but the current protocol is not acceptable as evidence for the headline claim. The authors themselves acknowledge the computational shortcut in the Limitations section. If the authors can re-run the selection on a proper training fold and report the held-out numbers, this could become a solid contribution; if the held-out numbers collapse, the claims need to be downgraded to in-sample or exploratory. Please also verify that the released Hugging Face model matches the reported PatientDx 8B configuration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper to know about: PatientDx merges instruct and math-specialized LLMs (SLerp) to build mortality predictors on MIMIC-IV without fine-tuning on patient data. The framing—model merging as a privacy-preserving alternative to fine-tuning—is genuinely new and worth taking seriously. The authors release the best model and include ablations, DLT perplexity measurements, and a small case study. That is real work.\n\nThe problem is that the headline AUROC numbers are not evidence of generalization. The merging weight λ is chosen on the test partition: footnote 2 says hyperparameter selection was done “only in test partition given the computational cost,” and the Limitations section confirms that selecting on the full training set would require higher cost. So the 0.6338 AUROC for PatientDx 8B is an in-sample maximum over the λ grid on the same patients used to report performance. The comparison against zero-shot baselines is therefore unfair: the merged models get test-set adaptation, the baselines do not. No error bars or significance tests are reported either.\n\nThis also undercuts the privacy claim. The abstract says “without requiring fine-tuning nor adaptation on patient data,” but the framework does use patient examples to pick λ and then selects the best configuration on test outcomes. No gradients are involved, but the model is still chosen using private patient data, and the DLT perplexity metrics in Table 4 cannot detect that selection-based use.\n\nWhat survives? The observation that a math-specialized pivot model helps when merging for numeric EHR tasks is plausible, and the recipe is cheap and easy to reproduce. With a proper train/validation/test split and significance testing, this could be a useful practical result.\n\nAs it stands, the central quantitative claim does not hold. I would not reject the idea—I would send it back for a major revision with a clean validation procedure. If the authors redo the λ selection on training data and the gains persist, you have a decent paper. Right now, the numbers are selection artifacts.\n\nSerious referee? Yes—the question matters and the flaw is fixable. But it shouldn't be accepted in this form.\n\nBest,\n[You]","headline":"Test-set lambda tuning makes the headline AUROC gains in-sample, but the privacy-preserving-merging framing is new and salvageable.","tokens_in":15992,"tokens_out":3335,"would_cite":false,"duration_ms":28823,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PatientDx shows that merging two off-the-shelf language models—one math-specialized, one instruction-tuned—beats fine-tuned medical LLMs on mortality prediction without ever training on patient records.","keywords":["model merging","privacy-preserving LLM","MIMIC-IV mortality","SLerp","healthcare prediction","data leakage","language model merging","numerical reasoning"],"falsifier":"Re-run the PatientDx pipeline with lambda chosen on a dedicated validation split (or nested cross-validation) and evaluate on a test fold never used for selection; if PatientDx 8B's AUROC no longer exceeds Meditron 7B's 0.5890, the headline improvement is a selection artifact.","tokens_in":14933,"feed_emoji":"🩺","tokens_out":7064,"duration_ms":59184,"temperature":0.7,"pith_summary":"The paper tries to establish that a hospital can build a strong predictive language model without ever fine-tuning on patient data: merge two publicly available base models—one instruction-tuned, one math-specialized—using spherical linear interpolation, and tune only a single scalar weight λ against a performance metric. On the MIMIC-IV in-hospital mortality task, the resulting 8B model reaches 0.6338 AUROC, beating the strongest prior biomedical model (Meditron 7B at 0.5890) and both of its own inputs (around 0.50). The paper also claims the merged model is substantially less prone to patient-data leakage than LoRA-fine-tuned alternatives, and that it transfers to numerical medical questions. The reason a reader would care is that it points to a privacy-preserving alternative to fine-tuning: performance gains without storing or exposing patient records in the training loop.","feed_headline":"Merged LLMs beat fine-tuned rivals without touching patient data","feed_subtitle":"PatientDx 8B scores 0.634 AUROC on MIMIC-IV mortality while leaking less than fine-tuned models.","key_machinery":"The load-bearing mechanism is spherical linear interpolation (SLerp), an angular interpolation between the parameter vectors of two same-architecture LLMs: θ* = [sin((1-λ)Ω)/sin(Ω)]θ1 + [sin(λΩ)/sin(Ω)]θ2, where Ω is the angle between the two weight vectors. PatientDx tunes the single scalar λ (e.g., λ=0.4 for PatientDx 8B) on the task metric without updating any weights. The second essential ingredient is the choice of a math-specialized model (DART-math) as the pivot, which supplies numerical reasoning competence that the instruct/biomedical model lacks, letting the merged model parse numeric EHR features without having seen patient data.","core_discovery":"The central discovery is that model merging, not fine-tuning, can be the effective adaptation mechanism for privacy-sensitive clinical prediction. PatientDx 8B combines Llama-3.1-8B-Instruct with DART-math-8B via SLerp at λ=0.4, and the merged parameters alone produce better mortality prediction than either input model, the biomedical specialist Meditron 7B, and fine-tuned baselines, while the DLT metrics show no evidence of memorizing the training set. The authors take this as empirical support for their postulate: with a math-capable pivot model, there exists a merge setting that outperforms the input models on private data and does so without adapting the weights to any patient record. Ablations show the math pivot is the most critical ingredient: removing DART-math costs 21.1% in AUROC for PatientDx 8B, more than removing SLerp (9.0%).","pith_inferences":["Editorial inference: the same recipe should transfer to other numeric-heavy clinical prediction tasks (e.g., length-of-stay, readmission, dosing), because the mechanism being exploited—arithmetic competence injected by a math-pivot model—is task-agnostic; the paper only tests mortality.","Editorial inference: the privacy claim is comparative, not absolute; PatientDx inherits whatever memorized content its input models already contain, and the DLT metrics measure ease of detecting training data, not immunity to membership-inference attacks, so a stronger privacy guarantee would need a formal leakage definition such as differential privacy.","Editorial inference: a cheaper way to test the hypothesis would be to vary the pivotal math model across families (e.g., different math-tuned 7–8B checkpoints) and see whether the AUROC gain tracks the pivot's arithmetic benchmark score; the paper does not report this correlation.","Editorial inference: the RRF fusion improvement in the retrieval task was not statistically significant, suggesting the transfer benefit may be limited to tasks requiring numerical computation rather than general retrieval."],"forward_implications":["Merging can replace fine-tuning for at least some healthcare predictive tasks: PatientDx 8B reaches 0.6338 AUROC on MIMIC-IV mortality without parameter updates on patient data.","The math pivot is load-bearing: removing DART-math from the merge drops PatientDx 8B by 21.1% in AUROC, and replacing SLerp with linear interpolation drops it by 9.0%.","Merged models are less prone to patient-data leakage than fine-tuned models under the DLT perplexity metrics, with Δ1 near 3.85–3.89 for PatientDx versus negative values for fine-tuned models.","PatientDx 8B transfers to numerical medical questions, producing a reasoned dose calculation (166.6 mg/8h) where a biomedical baseline drifts off-task.","The framework needs only a performance metric to tune λ, not private labels for training, keeping the interaction between model and patient data one-way."],"supporting_citations":[{"why":"Establishes weight-averaging (Model Soup) as the baseline merging strategy that PatientDx generalizes to SLerp.","marker":"Wortsman et al., 2022"},{"why":"Provides the SLerp angular interpolation formula used to combine the two input models.","marker":"Jang et al., 2024"},{"why":"Supplies DART-math, the math-specialized pivotal model whose inclusion drives the AUROC gain.","marker":"Tong et al., 2024"},{"why":"Provides the MIMIC-IV patient dataset and mortality tasks used in all evaluation and leakage analyses.","marker":"Johnson et al., 2023"},{"why":"Defines the DLT metrics (Δ1, Δ2) used to claim the merged model leaks less than fine-tuned models.","marker":"Wei et al., 2023"},{"why":"Supplies the EHR-to-text serialization and prompt that convert patient tables into LLM input.","marker":"Lovon-Melgarejo et al., 2024"},{"why":"Meditron 7B is the strongest biomedical baseline that PatientDx 8B must outperform.","marker":"Chen et al., 2023"},{"why":"Med42 8B is the fine-tuned clinical baseline used both as an accuracy competitor and a leakage comparison.","marker":"Christophe et al., 2024"},{"why":"MergeKit is the implementation used to perform the merges reported in the experiments.","marker":"Goddard et al., 2024"},{"why":"BioMistral serves as a biomedical baseline and as a prior example of SLerp use in the biomedical domain.","marker":"Labrak et al., 2024"}],"fun_headline_variants":["Model merging outperforms fine-tuning for private health prediction","PatientDx: Merge LLMs, skip fine-tuning, keep patient data private","Math-pivot LLM merge beats fine-tuning on MIMIC-IV without data leak","No fine-tuning needed: Merged LLMs predict mortality with less leakage","LLM merging: A privacy-safe alternative to fine-tuning for clinical tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper selects the lambda that maximizes performance using the same test partition on which it reports final AUROC, so the claimed gains assume that partition is a genuinely held-out set; if it is not clean, the improvement could be a tuning artifact rather than a property of merging.","fun_headline_variants_meta":{"raw":{"variants":["Model merging outperforms fine-tuning for private health prediction","PatientDx: Merge LLMs, skip fine-tuning, keep patient data private","Math-pivot LLM merge beats fine-tuning on MIMIC-IV without data leak","No fine-tuning needed: Merged LLMs predict mortality with less leakage","LLM merging: A privacy-safe alternative to fine-tuning for clinical tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1835,"prompt_tokens":989,"completion_tokens":846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":749}},"tokens_in":605,"tokens_out":846,"duration_ms":6944,"temperature":1.0,"reasoning_tokens":749,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:41:24.166905+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the PatientDx pipeline with lambda chosen on a dedicated validation split (or nested cross-validation) and evaluate on a test fold never used for selection; if PatientDx 8B's AUROC no longer exceeds Meditron 7B's 0.5890, the headline improvement is a selection artifact.","supporting_citations":[{"cited_title":"Moreno, and Lynda Tamine","cited_arxiv_id":null,"evidence_quote":"Supplies the EHR-to-text serialization and prompt that convert patient tables into LLM input."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BioMistral serves as a biomedical baseline and as a prior example of SLerp use in the biomedical domain."}],"review_version":1}