{"id":"636afadf-3e2a-4b9d-8f67-560e6cc26001","arxiv_id":"2607.22984","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A pilot of metamorphic testing on heart-disease classifiers found high rates of clinically implausible predictions despite good AUROC, but the headline fault-detection result is an artifact of the plausibility filter, not a measured violation-rate increase.","lead":"A framework paper proposes metamorphic testing—checking that related patient inputs produce clinically expected changes in model output—as a complement to AUROC for clinical ML models. Its pilot found high violation rates, but the headline injected-fault result compares a violation rate to \"unevaluable\", not to zero violations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated pilot MRs—cholesterol↑ non-decreasing risk, sex invariance—may be clinically wrong, so Table 3 violation rates do not establish model faults.","rationale":"The framework proposal and five-layer validation strategy are coherent, and the MIMIC MR catalog is a reasonable starting point. However, the pilot evidence is not sufficient to establish the central claim. The most load-bearing assumption is that the pilot MRs are valid clinical oracles; without that, none of the violation rates in Table 3 can be read as evidence of model behavioral faults. MR-H3 and MR-H5 are especially suspect because they encode a universal monotone cholesterol effect and sex invariance, respectively, both of which conflict with established cardiology knowledge. The paper's own §7 states L2/L3 validation has not been applied, yet it presents the violation rates as clinically meaningful. The injected-fault result is also flawed (VR_F=0 represents unevaluable cases, not a zero violation rate), but that concern is secondary: even a correctly reported fault-detection signal would be uninterpretable if the MR itself is invalid. A concrete L2/L3 validation of the pilot MRs would settle the concern. The reader's REJECT verdict is appropriate; no adjustment is needed.","tokens_in":6736,"tokens_out":5510,"duration_ms":53345,"concrete_test":"Run the paper's L3 empirical direction check on the pilot MRs using a held-out cohort (e.g., the full UCI dataset or an external dataset such as MIMIC-III matched features): for MR-H3, stratify by age and sex, compute Spearman correlation between total cholesterol and disease/mortality; if any stratum shows a significant negative association, the monotone direction of MR-H3 fails. For MR-H5, compare model outputs for male vs female patients with identical other features (propensity-matched or synthetic pairs); if outputs differ significantly—as clinical risk does—the invariance MR is invalid. Also run L2: have at least three cardiologists independently judge each MR; reject any MR with <80% agreement. If MR-H3 or MR-H5 fail these checks, recalculate Table 3 excluding them; if remaining violation rates are small or null, the central evidence collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical pillar is that models with AUROC 0.849–0.900 show MT violation rates of 27–87% (Table 3), interpreted as clinically inconsistent behavior. This interpretation requires each pilot MR to be a correct clinical oracle. MR-H3 (cholesterol +30 mg/dl should not lower risk) is not a universally valid monotone relation: total cholesterol's association with coronary disease is age- and LDL/HDL-dependent, and inverse associations appear in older or high-risk subgroups. MR-H5 (sex swap should not change prediction) demands invariance of a feature that is a legitimate, guideline-endorsed risk predictor (e.g., ACC/AHA Pooled Cohort Equations include sex); treating sex as irrelevant is a normative fairness claim, not a clinical sensibility constraint. The paper's own §7 admits L2/L3 validation has not been applied to pilot MRs. Therefore the high violation rates on H3/H5—and the overall 27–87% range—may simply reflect invalid test oracles, not model faults. The injected-fault result is also misreported (VR_F=0 is 'unevaluable', not zero violations), but that result's interpretation still depends on MR-H2's validity; the unvalidated MRs are the more foundational weakness. If they are wrong, the pilot provides no evidence that MT is a valuable complement, and the conclusion that 'models with competitive AUROC can behave inconsistently with clinical expectations' is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ClinMT, a framework for applying metamorphic testing (MT) to clinical machine-learning models on structured EHR data. It contributes a catalog of 12 candidate metamorphic relations (MRs) for MIMIC-III/IV mortality, decompensation, and sepsis-onset tasks, each grounded in a clinical guideline, together with a five-layer validation strategy (L1–L5), of which L1, L4, and L5 are applied and L2/L3 are planned. A pilot study on the UCI Heart Disease dataset with three classifiers reports MR violation rates of 27–87% despite AUROC 0.849–0.900, and an injected sign-negation fault in resting blood pressure is claimed to increase the MR violation rate by 31–67 percentage points while leaving AUROC essentially unchanged. The paper concludes that MT is a useful complement to conventional performance metrics.","tokens_in":7110,"tokens_out":6344,"duration_ms":62241,"significance":"If the empirical claims held, the paper would make a useful contribution: MT for structured EHR prediction is underexplored, and tying MRs to clinical guidelines with a formal validation protocol is a sensible response to the oracle problem. The paper is also candid about the pilot's small size and about L2/L3 validation being planned rather than executed. However, the two central empirical pillars—that unvalidated pilot MRs reveal clinical inconsistency in high-AUROC models, and that MT detects a fault that AUROC misses—are not established as reported. The injected-fault result conflates a non-evaluation with a zero violation rate, and several pilot MRs are not clinically uncontroversial. These issues are addressable in revision, but the current claims overstate the evidence.","major_comments":[{"comment":"The abstract's claim of a '31–67 percentage-point shift in MT violation rate' is not supported by the experiment. Table 4 reports VR_F = 0, but the footnote states that the L4 plausibility filter excluded all test cases (n_tested = 0) because the fault produces negative blood pressures; the text itself calls the MR 'unevaluable' rather than violated. The difference |ΔVR| is therefore a comparison between a violation rate and a non-evaluation. This is load-bearing because the injected-fault experiment is the paper's main evidence that MT detects a fault AUROC misses. The abstract and conclusion must be corrected, or the experiment redesigned so that the faulty model's inputs remain physiologically plausible for a nonzero number of cases.","section":"Abstract and §5.2, Table 4"},{"comment":"The interpretation of the 27–87% violation rates as 'clinically inconsistent behavior' assumes each pilot MR is a valid clinical oracle. MR-H3 (cholesterol↑+30 mg/dl should not lower predicted risk) and MR-H5 (sex swap should be invariant) are not established monotone/invariance constraints: total cholesterol has age- and subgroup-dependent associations with coronary disease, and sex is a legitimate predictor in widely used risk scores. The paper's own §7 states that L2/L3 validation has not been applied to the pilot MRs. Consequently, the high violation rates on H3/H5—and the overall range in Table 3—may reflect invalid test oracles rather than model faults. This undermines the central claim that competitive AUROC can coexist with behavior that is 'inconsistent with clinical expectations.'","section":"§5.2, Table 3 and §7"}],"minor_comments":[{"comment":"MR-D1 is listed as 'SpO2↓(−5%)' with type MI, but the MI definition in §3.1 is stated for an increase in the feature. The direction is confusing; either define an MD variant or restate the perturbation as 'SpO2 increase should not raise risk' to match the type column.","section":"Table 1"},{"comment":"The sensitivity ratio S is undefined for LR and the text says the release script substitutes a 'large finite placeholder.' This is not a scientific result and should be omitted or reported as undefined without reference to script behavior.","section":"§5.2, Table 4"},{"comment":"Reference [7] contains an inserted note: 'Unverified: could not confirm this citation independently; please check against the SANER 2020 proceedings before submission.' Such an unverified citation should be confirmed or removed before publication.","section":"References"},{"comment":"The line 'Accepted at the AIware 2026 arXiv Track' in the manuscript header is nonstandard for a journal submission and should be removed or moved to a footnote.","section":"Header"},{"comment":"The mapping of pilot MR-H2 ('Resting BP↑ +15 mmHg') to catalog MR-D4 ('MAP↓ <65 mmHg') is not transparent because the feature and direction differ. Clarify how the pilot perturbation corresponds to the catalog relation.","section":"§5.1"},{"comment":"Minor wording: 'AUROC = 0.849-0.900' should be 'AUROC 0.849–0.900'; also, the pilot's 61-patient test set yields large binomial confidence intervals, and reporting them would help readers gauge the stability of the violation rates.","section":"Abstract and §5.2"}],"recommendation":"major_revision","confidential_remarks":"The framework proposal and MR catalog are worth pursuing, and the authors are transparent about the pilot's exploratory nature. However, the headline empirical claims in the abstract and conclusion are not supported by the data as presented: the injected-fault result is based on non-evaluation, and the unvalidated MRs—particularly H3 and H5—leave the violation-rate interpretation open to the alternative that the oracles are invalid. These are fixable with a revised framing and/or a redesigned experiment, but they are central enough to require a major revision rather than minor polish. I would also ask the authors to confirm the unverified reference [7] and to provide code/data for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the ClinMT framework and the 12-MR catalog are worth reading. The five-layer validation strategy is a sensible addition to the MT-for-healthcare literature, and grounding MRs in clinical guidelines is the right instinct. The paper is also honest about its limitations in Section 7, which makes the abstract's overreach more annoying.\n\nThe problem is the empirical pillars. The injected-fault experiment reports a 31–67 percentage-point increase in violation rate, but Table 4 and the footnote show VR_F=0 because the L4 plausibility filter excluded all 61 test cases—the faulty model receives negative blood pressures, so the MR cannot be evaluated. That is not a violation rate; it is a non-evaluation. So the claim that MT catches a fault AUROC misses is unsupported. The paper does include the footnote, but the abstract and conclusions don't carry the caveat.\n\nThe second issue is the pilot MRs themselves. MR-H3 (cholesterol increase should not lower risk) and MR-H5 (sex swap should not change prediction) are not obviously valid clinical oracles. Cholesterol is not a clean monotone risk factor across all subgroups, and sex is a legitimate predictor in standard risk scores. Treating invariance over sex as a clinical sensibility constraint is a normative fairness claim, not a clinical one. The paper's own L2/L3 validation steps, which would catch this, are planned but not applied. So the 27–87% violation rates may simply reflect invalid test oracles, not model faults. The stress-test note is correct on this.\n\nWhat is genuinely new: the 12-MR catalog for ICU tasks (mortality, decompensation, sepsis) tied to MIMIC-style features, and the five-layer validation protocol. That is a solid extension of prior MT work on imaging and NLP to structured EHR time-series. The paper's related work is reasonable and the self-assessment in Section 7 is honest.\n\nI would send this to peer review, but with the expectation of major revision. The framework deserves referee time, and the catalog is a useful resource for the community. The pilot, as reported, should not be the basis for the abstract's conclusions. A revised version that either fixes the injected-fault protocol or reports it honestly as 'unevaluable,' and that validates or drops the questionable MRs, could be a genuinely useful paper. If the journal's bar is 'results must support claims,' this is a reject; if the bar includes 'is this a promising framework with fixable flaws,' it's a revise-and-resubmit. I'd lean toward sending it out.","headline":"A useful framework proposal with a pilot that overstates its evidence: the injected-fault result is a non-event and the pilot MRs are unvalidated.","tokens_in":7532,"tokens_out":2683,"would_cite":false,"duration_ms":25251,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Metamorphic testing catches clinical-AI faults AUROC misses","keywords":["metamorphic testing","clinical machine learning","AUROC","behavioral oracle","ICU prediction","MIMIC","model validation","patient safety"],"falsifier":"A decisive test: have clinicians validate the five pilot MRs (L2) and empirically check their direction on a held-out cohort (L3). If violation rates collapse to near zero after invalid MRs are removed, the paper's central evidence disappears. In parallel, repeat the injected-fault experiment with a fault that preserves physiological plausibility (e.g., negate a z-scored feature instead of raw mmHg); if MT violation rates do not rise while AUROC remains flat, the claimed sensitivity advantage is an artifact.","tokens_in":6677,"feed_emoji":"🩺","tokens_out":8364,"duration_ms":67379,"temperature":0.7,"pith_summary":"This paper argues that the standard metric for clinical prediction models, AUROC, measures only whether patients are ranked in the right order—not whether individual predictions respect medical knowledge. To expose such behavioral faults, it proposes metamorphic testing (MT), which checks whether a model's output moves in a clinically expected direction when a patient record is perturbed, without needing ground-truth labels. The paper designs a catalog of 12 candidate metamorphic relations for ICU prediction tasks and a five-layer validation protocol, then runs a pilot on the UCI Heart Disease dataset. In the pilot, three models with AUROC between 0.849 and 0.900 violated the five pilot relations in 27–87% of applicable test cases, and an injected sign-negation fault left AUROC essentially unchanged while causing the relevant relation to become unevaluable because all perturbed inputs fell outside physiological bounds—which the paper reads as a sensitivity signal. The broader point is that clinical models should be audited for behavioral consistency, not just ranking accuracy.","feed_headline":"Metamorphic testing catches clinical-AI faults AUROC misses","feed_subtitle":"In a heart-disease pilot, models with AUROC 0.85-0.90 broke medically expected input-output rules in 27-87% of tests.","key_machinery":"The load-bearing mechanism is the metamorphic relation (MR)—a rule that fixes how a model's output should change when its input is perturbed in a clinically meaningful way (for example, increasing age should not lower predicted mortality risk). The paper's ClinMT pipeline applies these relations by perturbing each patient record, scoring both versions, and checking the output relation, with a five-layer validation protocol (L1–L5) designed to ensure MRs are clinically sound before deployment. The catalog of 12 candidate MRs for MIMIC-III/IV mortality, decompensation, and sepsis tasks, plus the five pilot MRs on the UCI Heart Disease data, are what carry the argument: they translate clinical","core_discovery":"The central claim is that metamorphic testing can serve as a label-free behavioral oracle for clinical machine learning models, revealing inconsistencies with established medical knowledge that high AUROC scores do not reflect. On the paper's own terms, the pilot demonstrates this: all three classifiers (logistic regression, random forest, MLP) achieved AUROC between 0.849 and 0.900, yet violated the pilot metamorphic relations in a large fraction of test cases—most strikingly, 77–87% of applicable cases on the relation that an increase in ST depression should not lower predicted risk. The injected-fault experiment, in which the resting blood pressure feature's sign was negated, produced alm","pith_inferences":["The 31–67 percentage-point sensitivity claim in the abstract is not a fair comparison: the faulty model's violation rate is reported as 0% because all test cases were filtered out, not because the model passed the relation. A fair test would inject a fault that keeps perturbed inputs within physiological bounds (e.g., sign-negate a standardized feature rather than raw mmHg) and then compare violat","The pilot MRs themselves—particularly cholesterol-increase-should-not-lower-risk and sex-invariance—may not be unqualified medical truths; cholesterol is not a universal monotone risk factor and sex is a real predictor. If L2/L3 validation rejects these MRs, the headline 27–87% violation rates would shrink, possibly dramatically.","The framework's L4 filter could be used the other way: rather than merely excluding impossible inputs, an audit could deliberately sample perturbations near physiological boundaries, where encoding errors like sign negation are most likely to surface, potentially making MT even more sensitive.","If MT proves out, one could imagine a continuous monitoring system where every patient record is checkable: feed the same record with a small, clinically justified perturbation and verify the prediction moves in the expected direction in real time."],"forward_implications":["If MT catches faults that AUROC misses, clinical model validation could add a label-free behavioral test alongside AUROC, flagging models that contradict guidelines even when their ranking performance is excellent.","A model that violates basic clinical constraints—'older means less risk' or 'sex changes the prediction'—should probably not reach the clinic, so MT could serve as a pre-deployment gate.","The 12-MR catalog gives ICU benchmark builders a concrete checklist to test mortality, decompensation, and sepsis models for behavioral consistency on MIMIC-III/IV.","Because fairness constraints (e.g., race invariance) are just invariance MRs, the same MT harness could unify clinical-sensibility and demographic-fairness audits in one run.","The high violation rates on a simple, 13-feature dataset imply that real ICU models, with many more features and complex nonlinearities, are likely to show even broader behavioral inconsistency unless explicitly tested."],"fun_headline_variants":["High AUROC masks clinical AI rule violations","Clinical AI fails medical logic despite high AUROC","Metamorphic testing exposes AI medical rule breaks","AUROC blind to clinical AI logic errors"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The pilot's metamorphic relations are assumed to be clinically valid on this population—especially that increasing cholesterol never lowers heart-disease risk and that swapping sex never changes risk; if any of these relations is not a faithful medical regularity, the reported violation rates blame the model for what is actually a faulty oracle.","fun_headline_variants_meta":{"raw":{"variants":["High AUROC masks clinical AI rule violations","Clinical AI fails medical logic despite high AUROC","Metamorphic testing exposes AI medical rule breaks","AUROC blind to clinical AI logic errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1135,"prompt_tokens":792,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":298}},"tokens_in":536,"tokens_out":343,"duration_ms":3875,"temperature":1.0,"reasoning_tokens":298,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T03:56:57.565882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test: have clinicians validate the five pilot MRs (L2) and empirically check their direction on a held-out cohort (L3). If violation rates collapse to near zero after invalid MRs are removed, the paper's central evidence disappears. In parallel, repeat the injected-fault experiment with a fault that preserves physiological plausibility (e.g., negate a z-scored feature instead of raw mmHg); if MT violation rates do not rise while AUROC remains flat, the claimed sensitivity advantage is an artifact.","supporting_citations":[],"review_version":1}