{"id":"172e529e-1746-45c7-a865-cf4a9eb3c0d7","arxiv_id":"2608.03862","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CRS-Triage, which jointly models modality reliability, cross-modal consistency, and a learned confidence score, improves triage accuracy and reduces under-triage on MIMIC-IV-ED compared with evidential and fusion baselines.","lead":"A machine-learning model for emergency triage estimates how reliable the structured and text parts of each patient's record are, and defers cases it is unsure about. On the MIMIC-IV-ED dataset it reports better acuity prediction and a lower under-triage rate than fusion baselines, especially when records are incomplete or conflicting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 7's reliability-as-completeness score is untested under informative missingness; if missingness signals high acuity, CRS-Triage may downweight exactly the structured evidence it needs most.","rationale":"The reader identified the same weakest assumption: Eq. 7 equates reliability with completeness. I agree this is the soft spot, but I would add two nuances. First, the missingness mask is already an input to the structured encoder, so the model is not strictly incapable of learning informative-missingness signals; the danger is that the fusion weight and confidence score are explicitly built around a monotone completeness score, and no experiment shows the encoder-level signal can overcome that bias. Second, the decisive gap is empirical: Table 5 only perturbs data by random MCAR masking and random token deletion, which cannot distinguish 'incomplete but reliable' from 'missing because the patient is severely ill.' Thus the central safety claim is conditional on missingness being ignorable, which is not argued and is likely false in emergency settings. No internal inconsistency was found in the equations; the concern is that the key operationalization of reliability has not been validated under the missingness mechanism most relevant to the application. This supports the reader's CONDITIONAL verdict rather than upgrading or rejecting it.","tokens_in":13708,"tokens_out":6476,"duration_ms":72399,"concrete_test":"First, test for MNAR in MIMIC-IV-ED: fit a regression of high-acuity ESI (or an objective outcome such as admission or critical intervention) on missingness indicators of triage vitals, controlling for observed vitals. If missingness predicts higher acuity, construct a test-time MNAR perturbation that preferentially masks vitals for high-acuity encounters, preserving the empirically observed missingness-acuity odds ratios, and re-run the Table 5 protocol at 10%, 30%, and 50% masking. Compare CRS-Triage against base evidential fusion on Macro-F1, under-triage rate, and expected triage penalty at 80% coverage. If CRS-Triage degrades substantially more under MNAR than under random masking, or if under-triage among accepted cases increases, Eq. 7's completeness-based reliability score is the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central safety claim depends on Eq. 7, where structured-data reliability rho^s_i is a weighted fraction of observed features with nonnegative learned weights. This operationalizes reliability as completeness. If missingness is informative in the opposite direction (e.g., high-acuity patients have fewer triage vitals because they are immediately resuscitated), then low rho^s_i marks high acuity, not low reliability. Since rho^s_i enters the fusion weight q^s_i (Eq. 10) and the confidence score s_i (Eq. 12), the model can suppress the most predictive structured evidence for the very patients it is designed to protect, and can defer or under-triage them. The missingness mask is also fed to the encoder (Eq. 4), so the model is not completely blind to informative missingness, but no experiment verifies that this compensation is sufficient. The robustness experiments in Table 5 use random MCAR masking and random token deletion, which cannot reveal this failure mode. Without an MNAR evaluation or a direct validation that rho^s_i tracks actual per-modality error, the claim that CRS-Triage remains reliable under incomplete clinical evidence is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CRS-Triage proposes a multimodal emergency triage model for incomplete and conflicting EHR data. Structured data and clinical text are encoded separately into evidential (Dirichlet) predictions; per-modality reliability scores are computed from feature completeness (Eq. 7) and text length/certainty (Eq. 8), then combined with cross-modal disagreement (Eq. 9) to weight modality evidence before fusion (Eqs. 10-11). A learned confidence score (Eq. 12) is trained with a selective loss (Eq. 13) and an asymmetric triage penalty (Eq. 2) to favor over-triage over under-triage, and defer low-confidence cases. On MIMIC-IV-ED, the model reports Macro-F1 0.742, QWK 0.846, under-triage rate 9.6% (vs. 13.1% for base evidential fusion), improved ECE, better risk-coverage trade-offs at 80% and 90% coverage, and favorable robustness under random masking, text degradation, and cross-modal conflict.","tokens_in":14049,"tokens_out":4878,"duration_ms":56154,"significance":"If the empirical claims hold, the paper's contribution is a useful end-to-end selective triage framework that jointly models modality reliability, cross-modal consistency, and asymmetric error costs. The evaluation has genuine strengths: a held-out patient-level split, results over five seeds, ablation studies, and perturbation tests that are not circular. The comparison to SelectiveNet and evidential baselines is appropriate. However, the significance is conditional on the reliability definition: structured reliability is operationalized as completeness (Eq. 7), and no direct evidence shows that the learned reliability scores track per-modality error. The informative-missingness failure mode is especially relevant for emergency triage and is not tested. As a result, the broad claim that CRS-Triage 'remains reliable under incomplete clinical evidence' is not yet fully established.","major_comments":[{"comment":"The structured-data reliability rho^s_i is a weighted fraction of observed features, i.e., reliability is equated with completeness. The robustness experiments mask observed entries randomly (MCAR), so they cannot detect a failure mode in which missingness is informative in the opposite direction. If high-acuity patients have fewer recorded vitals because they are taken to resuscitation immediately, Eq. 7 assigns low rho^s to exactly the patients the model should protect. Since rho^s enters the fusion weight q^s_i (Eq. 10) and the confidence score s_i (Eq. 12), the model may downweight or defer the most informative structured evidence for those patients. The missingness mask is provided to the encoder (Eq. 4), which is a partial mitigation, but no experiment verifies that this compensation is sufficient. Please add an MNAR-style evaluation or otherwise demonstrate that rho^s tracks actua","section":"Eq. (7) and Table 5"},{"comment":"The reliability scores are never directly validated. No table or figure reports, for example, the correlation or calibration between rho^s_i / rho^t_i and the actual per-modality classification error, either on clean or perturbed inputs. The paper's central claim is that fusion is 'reliability-aware,' but the only support is indirect through ablations. A direct analysis showing that low reliability scores correspond to higher per-modality error rates would substantially strengthen the paper and would address the concern that Eq. 7 is simply a completeness measure in disguise.","section":"Eqs. (7)-(12), Table 1"},{"comment":"The training objective contains many hyperparameters that are never specified: lambda_pen, lambda_sel, lambda_dis, lambda_KL, lambda_aux, gamma, beta, kappa, epsilon, and the exact values of lambda_under/lambda_over beyond the ratio of 3 shown in Table 4. Without these values the multi-term loss is not reproducible, and the claimed advantage of joint training cannot be independently checked. Please provide the full hyperparameter configuration, including any scheduling or annealing (e.g., for lambda_KL).","section":"Eq. (17) and Experimental Setup"}],"minor_comments":[{"comment":"No significance tests are reported over the five seeds. Many key differences appear separated by more than the reported standard deviations, but some metrics overlap (e.g., balanced accuracy for CRS-Triage is 0.734 +/- 0.009 vs. TMC's 0.738 +/- 0.009). Please add paired significance tests or confidence intervals.","section":"Tables 1-3"},{"comment":"The confidence threshold tau is described as predefined, but the experiments at fixed coverage appear to use an empirical score quantile. Please state explicitly how tau is chosen in practice and whether the reported results use a fixed threshold or coverage-based selection.","section":"Selective Prediction, Table 2"},{"comment":"The text states that deferred encounters have 'substantially higher error rates and expected triage penalties,' but the deferred-case metrics are not shown in a table. Please include these numbers.","section":"Selective Prediction, Section 4.2"},{"comment":"Fix formatting issues such as 'to0.742from' in Section 4.1 and missing spaces around numbers/operators in equations and text. Also, the sentence beginning '418,100 adult encounters' should be rephrased for clarity.","section":"Overall text"},{"comment":"If the KL regularization coefficient lambda_KL is annealed during training, as is common in evidential learning, please state the schedule. If it is fixed, that should also be stated explicitly.","section":"Eq. (14)"}],"recommendation":"major_revision","confidential_remarks":"The referee report focuses on the gap between the paper's broad reliability claim and the specific operationalization of reliability in Eq. 7. The experiments are honest and the ablation/perturbation design is a strength. The missing hyperparameters are a reproducibility issue that is fixable. I see no circularity concern: the reported results come from a trained model evaluated on held-out data. The paper would be a solid candidate after the MNAR/direct-reliability validation is added and the experimental details are completed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers what it promises: a selective-triage model that fuses structured EHR data and text by explicitly weighting each modality's reliability, cross-modal consistency, and an asymmetric under-triage penalty, then decides when to defer. The empirical work is honest and above the median for this area: held-out test split, patient-level separation, five seeds, ablations, and perturbation experiments (masking, token removal, conflict). The gains over the baselines are consistent and the risk–coverage results at 80% and 90% coverage look real. Credit where due: this is not a toy, and the safety framing is appropriate.\n\nThe actual novelty is modest but real. Building blocks are recognizable—Dirichlet evidential learning, SelectiveNet-style coverage loss, JS divergence from Xu et al. 2024a—but the combination of per-modality reliability scores, learned fusion weights, and a confidence score trained jointly with an under-triage penalty is not something I've seen in triage. The ablation study gives some support for each piece.\n\nSoft spots, in order of importance. First, the structured-data reliability ρ^s in Eq. 7 is a weighted fraction of observed features. That defines reliability as completeness. The stress-test note is correct: if missingness is informative in the opposite direction—sicker patients less likely to have complete vitals because they get whisked to resuscitation—then this score downweights the most useful evidence for the very patients the model should protect. The missingness mask is fed to the encoder, so the model is not blind, but nothing verifies that the compensation works. This is the main weakness and it deserves a direct MNAR experiment or at least a discussion. Second, no significance tests over the five seeds; the stds overlap in places. Third, hyperparameters for the multi-term loss are not reported; reproducibility suffers. Fourth, the closest evidential-conflict baseline (Xu et al. 2024a) is cited but not run. Minor: the reliability estimates are never directly validated against per-modality error.\n\nDespite these, the central argument holds up as far as the paper claims. The results are not circular, and the reported performance is plausible for MIMIC-IV-ED. I'd send it to a serious referee, with the expectation that the missingness-direction issue be addressed before publication. For anyone building triage models under incomplete EHRs, this is worth reading and likely worth citing once code and hyperparameters are available.","headline":"Solid applied clinical ML paper with a real gap in the reliability definition: it treats missingness as unreliability, and never tests the opposite direction.","tokens_in":14540,"tokens_out":1040,"would_cite":true,"duration_ms":13623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Emergency triage gets safer when the model scores the reliability of each data source and defers low-confidence cases: under-triage drops from 13.1% to 9.6% on MIMIC-IV-ED.","keywords":["emergency triage","selective prediction","evidential learning","multimodal fusion","incomplete EHR data","under-triage penalty","confidence score","MIMIC-IV-ED"],"falsifier":"Run CRS-Triage on a version of the held-out MIMIC-IV-ED set where missingness is informative: mask 30–50% of triage vitals preferentially from high-acuity (ESI 1–2) encounters while leaving low-acuity records intact. If reliability-as-completeness is sound, accepted high-acuity cases keep near the random-missingness under-triage rate; if deferral concentrates on the sickest and their penalty rises sharply, the scheme distrusts its most acute evidence. A simpler correlational check: if ESI 1–2 encounters have fewer observed structured features than ESI 4–5, the model is systematically discounti","tokens_in":13615,"feed_emoji":"🚑","tokens_out":32123,"duration_ms":245810,"temperature":0.7,"pith_summary":"Emergency triage must judge a patient's acuity from records that arrive incomplete, poorly typed, or mutually contradictory. This paper claims that a triage model can be made both more accurate and safer by estimating, for every encounter, how reliable the structured vitals and the clinical text each are, whether the two modalities agree, and how confident the resulting prediction is. That confidence score decides whether the model issues an acuity label or defers the case for human assessment, and under-triage errors are penalized more heavily than over-triage. On MIMIC-IV-ED, the method lifts the average per-class F1 to 0.742, cuts under-triage from 13.1% to 9.6%, and improves the risk–coverage trade-off: at 80% coverage the expected triage penalty falls from 0.267 to 0.208, the lowest of any selector tested. If right, ML triage can say 'I don't know' when the evidence is poor instead of guessing, and can push residual errors toward the less harmful over-triage direction.","feed_headline":"Under-triage drops to 9.6% when AI scores data reliability","feed_subtitle":"The model weighs how complete and mutually consistent the evidence is, then decides when to predict and when to pause.","key_machinery":"Structured reliability ρ^s (Eq. 7) is the weighted fraction of observed features; text reliability ρ^t (Eq. 8) combines text length, prediction certainty, and evidential certainty (Dirichlet-based). Disagreement δ (Eq. 9) is the normalized Jensen–Shannon divergence between the modalities' probability vectors; it drives independent, non-normalized fusion weights (Eq. 10), so unreliable or conflicting evidence is downweighted and both modalities can contribute little at once. The same signals, with fused uncertainty and margin, feed the learned confidence score s_i (Eq. 12) that decides predict-vs-defer. The selective loss (Eq. 13) minimizes accepted-case triage penalty under a coverage constr","core_discovery":"The claim: confidence and prediction must be learned together, not tuned afterward. CRS-Triage estimates, per encounter, the completeness of structured vitals, the reliability of the clinical text, and cross-modal disagreement (normalized Jensen–Shannon divergence). These signals weight each modality's evidence in evidential fusion (a Dirichlet-based mechanism yielding probabilities plus an uncertainty) and feed a learned confidence score that decides predict-vs-defer. Training minimizes expected triage penalty — under-triage weighted above over-triage — over accepted cases under a coverage constraint. On MIMIC-IV-ED this yields Macro-F1 0.742, under-triage down from 13.1% to 9.6%, and bette","pith_inferences":["The structured-reliability score treats 'reliable' as 'complete': a missing value always lowers it. If informative missingness runs the opposite way — the sickest patients have fewer vitals recorded because resuscitation starts immediately — the model would systematically distrust its most informative records; perturbing the test set by removing vitals preferentially from high-acuity encounters wo","The coverage constraint is an average over training, and the confidence threshold is applied only at inference; letting the threshold respond to clinician workload or queue congestion is a natural extension the paper does not explore.","Text length enters the text-reliability score, so a dense one-line note like 'STEMI' is scored less reliable than a padded note; weighting informativeness rather than length, and testing conflicts that co-occur with missingness (rather than in isolation), are testable refinements.","The deferral decision is binary; a graded escalation — predict with a conservative acuity label rather than defer entirely when confidence is mid-range — is a natural follow-on that would soften the accuracy–coverage trade-off."],"forward_implications":["The CRS confidence score improves selective deferral at fixed coverage: at 80% coverage the expected triage penalty drops from 0.267 (evidential certainty) to 0.208, and under-triage among accepted cases from 5.9% to 4.7%.","The under-triage penalty ratio is a policy dial: ratio 3 gives 9.6% under-triage and 15.1% over-triage; ratio 5 gives 8.0% and 18.5%, so a hospital can choose its position on the over-resourcing curve.","Reliability weighting degrades gracefully under test-time perturbation: at 50% additional structured missingness, Macro-F1 stays at 0.681 and triage penalty at 0.520, versus 0.612 and 0.690 for base evidential fusion, and the margin over baselines is largest under cross-modal conflict.","Deferred encounters carry substantially higher error and triage penalty than accepted ones, so the predict-or-defer decision concentrates human review on the cases the model is least suited to judge."],"supporting_citations":[{"why":"Supplies the Dirichlet-based evidential learning machinery: class probabilities and an uncertainty estimate derived from nonnegative evidence vectors, used in the modality-specific and fused prediction heads.","marker":"(Sensoy, Kaplan, and Kandemir 2018)"},{"why":"Provides the MIMIC-IV-ED dataset (418,100 adult encounters with five-level acuity labels) on which all experiments and comparisons are run.","marker":"(Johnson et al. 2023)"},{"why":"Defines selective prediction and the SelectiveNet baseline, establishing the predict-or-defer framing and risk–coverage evaluation that CRS-Triage builds on.","marker":"(Geifman and El-Yaniv 2019)"},{"why":"Trusted multi-view classification with dynamic evidential fusion; the TMC baseline that CRS-Triage is compared against for evidential fusion with disagreement.","marker":"(Han et al. 2023)"},{"why":"DrFuse, the clinical multimodal fusion baseline for incomplete and inconsistent modalities that CRS-Triage must beat in the main comparison.","marker":"(Yao et al. 2024)"},{"why":"Provides BioClinicalBERT, the pretrained clinical text encoder used to encode chief complaints for the text modality.","marker":"(Alsentzer et al. 2019)"},{"why":"Reliable conflictive multi-view learning; source of the softplus evidence transform and the cross-modal disagreement formulation used in CRS-Triage.","marker":"(Xu et al. 2024a)"},{"why":"Documents the clinical asymmetry between under- and over-triage that motivates the larger under-triage penalty.","marker":"(Huabbangyang et al. 2023)"},{"why":"Argues that EHR missingness is informative and reflects the clinical measurement process, motivating per-encounter reliability estimation.","marker":"(Groenwold 2020)"}],"fun_headline_variants":["AI learns to defer when clinical data is shaky","Triage model cuts under-triage to 9.6% by weighing data reliability","Confidence-aware triage balances predict and defer for emergency care","Fewer missed high-acuity patients when AI scores data completeness","CRS-Triage: selective triage that weighs evidence and defers when unsure"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that 'reliable structured data' equals 'complete structured data': Eq. 7 defines reliability as the weighted fraction of observed features, so a missing value always lowers it. If informative missingness runs the other way — the sickest patients have fewer vitals recorded at triage because care starts immediately — the model systematically downweights its most informative evidence exactly for the patients it exists to protect.","fun_headline_variants_meta":{"raw":{"variants":["AI learns to defer when clinical data is shaky","Triage model cuts under-triage to 9.6% by weighing data reliability","Confidence-aware triage balances predict and defer for emergency care","Fewer missed high-acuity patients when AI scores data completeness","CRS-Triage: selective triage that weighs evidence and defers when unsure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1226,"prompt_tokens":777,"completion_tokens":449,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":357}},"tokens_in":521,"tokens_out":449,"duration_ms":4550,"temperature":1.0,"reasoning_tokens":357,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:52:05.362489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CRS-Triage on a version of the held-out MIMIC-IV-ED set where missingness is informative: mask 30–50% of triage vitals preferentially from high-acuity (ESI 1–2) encounters while leaving low-acuity records intact. If reliability-as-completeness is sound, accepted high-acuity cases keep near the random-missingness under-triage rate; if deferral concentrates on the sickest and their penalty rises sharply, the scheme distrusts its most acute evidence. A simpler correlational check: if ESI 1–2 encounters have fewer observed structured features than ESI 4–5, the model is systematically discounti","supporting_citations":[],"review_version":1}