{"id":"5b6306c5-d0f3-4e7f-835b-849fcab41205","arxiv_id":"2507.01437","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A standard Transformer with sigmoid multi-label classification is reported to reach 77.8% accuracy on MIMIC-IV disease prediction, but the evaluation is not reproducible and the baselines are not comparable.","lead":"This paper applies a standard Transformer attention model to predict multiple diseases from MIMIC-IV clinical notes, reporting 77.8% accuracy. The reported advantage over prior baselines is not credible because the baselines come from unrelated tasks and no implementation details are provided.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 cannot support the central superiority claim: each cited baseline [22]–[26] addresses a different task or dataset, so the reported 77.8% versus 68.4–74.6% is not a controlled comparison.","rationale":"The reader's weakest assumption is precisely the one that breaks the paper: the comparability of the five baselines in Table 1. I independently checked the cited references, and each describes a different predictive task, dataset, or system: an ICU time-series LSTM, an IoT breast cancer CNN, a blockchain-storage BiLSTM, an RFID record-access system, and a variant-pathogenicity model. None supports the implied claim that the listed numbers are results of the same MIMIC-IV multi-label disease-prediction task. Without a shared task, preprocessing, split, and metric definition, the table cannot be read as a controlled comparison, so the central claim 'consistently outperforms representative existing approaches' is unsupported. The paper also lacks code, model checkpoints, seeds, error bars, and a precise training/evaluation protocol, which leaves no way to independently verify even the proposed model's 77.8% accuracy. I agree with the reader's REJECT verdict: the main empirical assertion fails at its load-bearing point, not merely in presentation. My concern does not require attributing intent to the authors; the issue is that the evidence as presented is not commensurable across rows. The concrete test I propose would settle the matter by forcing all methods onto one shared protocol and checking reproducibility. If the test passes, the rejection could be revisited; if it fails, the central claim remains unsupported.","tokens_in":6699,"tokens_out":2538,"duration_ms":36235,"concrete_test":"Retrieve the full texts of references [22]–[26] and check whether any one reports MIMIC-IV clinical notes with multi-label disease prediction. Then rerun each baseline architecture on exactly the same preprocessed MIMIC-IV text subset, label set, train/test split, and random seeds as the proposed model, computing accuracy, precision, and recall with confidence intervals. If the baseline numbers in Table 1 cannot be reproduced under this shared protocol, or differ by more than one percentage point, then the claimed consistent superiority is not established.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim, stated in the abstract and Section IV-B, is that the proposed method consistently outperforms representative existing approaches. The only evidence is Table 1, but no experimental protocol is given: no data split, label set, sample size, seeds, hyperparameters, or error bars. More fundamentally, the cited baseline papers do not implement the same task. [22] is an LSTM for irregularly collected sequential ICU time series, not clinical free-text multi-label diagnosis; [23] is a CNN for telehealth breast cancer detection via IoT; [24] is a BiLSTM for blockchain-based secure cloud storage in IoT healthcare; [25] is an RFID-based medical record access system from 2011, not a prediction model; and [26] is DITTO, a genomics model for transcript-specific variant pathogenicity. None of these references reports MIMIC-IV clinical-note multi-label disease prediction. Thus the load-bearing assumption that all rows of Table 1 come from the same benchmark task and split is unsupported and contradicted by the citations themselves. Even if the numbers were transcribed accurately, Table 1 would still not establish consistent superiority without a shared evaluation protocol. The central empirical claim therefore fails at its base: there is no demonstrable comparison to outperform.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a Transformer-based multi-label classifier with multi-head self-attention and a sigmoid output layer for predicting ICD diagnosis codes from MIMIC-IV clinical notes. The authors report 77.8% accuracy, 75.9% precision, and 73.2% recall, claim consistent superiority over five baselines (LSTM, CNN, BiLSTM, RF-MediSys, DITTO), and present sensitivity analyses for learning rate, sample size, and input noise. The manuscript does not include the actual content of the referenced figures, code, data splits, or a detailed experimental protocol.","tokens_in":6991,"tokens_out":3726,"duration_ms":42043,"significance":"If the empirical claims were reproducible, the paper would offer a modest confirmation that Transformer-based models outperform older sequence models on MIMIC-IV multi-label diagnosis prediction. However, the contribution is primarily empirical, and the evidence as presented does not support it. The authors provide no code, no data-split details, and no shared evaluation protocol, and Table 1 compares against baselines from unrelated tasks and datasets. The conceptual framing (attention, sigmoid multi-label classification, binary cross-entropy) follows standard practice and is not a new methodological contribution.","major_comments":[{"comment":"The central claim of consistent superiority is unsupported because the cited baselines are not evaluated on the same task or dataset. Reference [22] is a time-series LSTM for irregularly collected ICU data, [23] is a CNN for IoT-based breast cancer detection, [24] is a BiLSTM for blockchain-secured cloud storage in IoT healthcare, [25] is an RFID-based medical record access system from 2011, and [26] is a genomics model for transcript-specific variant pathogenicity prediction. None of these works performs MIMIC-IV clinical-note multi-label disease prediction, so the reported differences (77.8% versus 68.4-74.6%) do not constitute a controlled comparison and cannot establish superiority over any method.","section":"Section IV-B, Table 1"},{"comment":"No experimental protocol is reported: the number of training, validation, and test samples is not given, the label set size is not specified, the exact data split is not described, the preprocessing thresholds for 'appropriate length and complete labels' are not defined, and the number of runs, random seeds, and error bars are absent. The only hyperparameter discussed is the learning rate. These omissions make the numbers in Table 1 and the sensitivity analyses impossible to verify or reproduce.","section":"Section IV-A and Section IV-B"},{"comment":"The sensitivity and robustness claims (learning-rate effects, sample-size trends, and noise tolerance) rest on figures that are not present in the manuscript. Since Figures 2, 3, and 4 are referenced but their data and axes are not shown, the claims that accuracy peaks at a learning rate of 1e-4, stabilizes above 80% of the training data, and drops below 0.67 at 20% noise cannot be checked.","section":"Section IV-B, Figures 2-4"},{"comment":"The paper claims unified modeling for information extraction and multi-label disease prediction, but no information-extraction task, entity labels, or extraction metrics are defined or evaluated anywhere in the experimental section. All reported experiments are multi-label classification metrics, so the information-extraction component of the contribution is unsubstantiated.","section":"Abstract and Section III"}],"minor_comments":[{"comment":"The text says A is a scaling factor, but no A appears in the displayed formula; if the standard 1/sqrt(d_k) scaling is intended, it should be written explicitly and the symbol A should be defined.","section":"Section III, Equation (1)"},{"comment":"The sentence following Equation (2) says 'Where A is the learnable projection matrix,' but the output projection is usually denoted W^O; the notation conflicts with the scaling factor A mentioned earlier.","section":"Section III, Equation (2)"},{"comment":"The preprocessing steps (deduplication, de-identification, code normalization, sentence segmentation) are listed but no details are given about how they were applied, how many samples were removed, or how the final cohort was constructed.","section":"Section IV-A"},{"comment":"The related-work section includes several references that are only loosely connected to clinical NLP, such as [17] on small-target detection and [21] on cache management, which makes the motivation for the proposed architecture diffuse.","section":"Section II"},{"comment":"Figure 1 is referenced in Section III as illustrating the model architecture, but the figure is not included in the manuscript; the authors should either include it or remove the reference.","section":"Section IV-B, Figure 1"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a desk reject. The paper's headline result is 77.8% accuracy on MIMIC-IV, but the table that supposedly shows superiority lists baselines from unrelated tasks, and the one named innovation, the context-aware semantic alignment mechanism, is never described. As written, the paper does not have a supportable empirical claim.\n\nWhat is actually on the page: a standard Transformer encoder plus a sigmoid multi-label BCE head trained on MIMIC-IV notes and ICD codes. Equations (1)-(4) are textbook. The preprocessing description is reasonable, and the sensitivity probes (learning rate, sample size, noise) are sensible things to do. I will give credit for that much. But standard components plus a sensitivity sweep is not a new contribution.\n\nThe soft spots are load-bearing. The abstract and conclusion credit a context-aware semantic alignment mechanism, but Section III never says what it is. There is no equation, no module diagram, no ablation isolating it. The model description is just a vanilla Transformer with multi-head attention and a linear-sigmoid classifier. The mechanism exists only as a phrase.\n\nTable 1 is worse. I checked references [22]-[26]. [22] is an LSTM for irregular sequential ICU data, not free-text diagnosis prediction. [23] is an IoT/CNN telehealth breast cancer model. [24] is a BiLSTM for blockchain cloud storage. [25] is an RFID access system from 2011. [26] is a genomics variant pathogenicity model. None of these is a clinical-note multi-label disease classifier on MIMIC-IV. Unless the authors reimplemented all of those systems under one shared protocol, which they never say, the 77.8% versus 68.4-74.6% comparison is not apples-to-apples. There are also no error bars, no seeds, no train/test split details, no hyperparameters beyond learning rate, and no code or data artifacts. The sensitivity figures are described in prose but not visible in the text I have, so I cannot verify them either.\n\nThe stress-test note is right: this is not mild under-reporting, it is a central claim that cannot be checked. The math that is shown is standard and fine, but the data reporting and citation pattern do not support the conclusions.\n\nWho is this paper for? Maybe someone wanting a rough engineering baseline number on MIMIC-IV, but they should not rely on it. This does not deserve a serious referee. Recommend desk reject; if the authors resubmit, they need to define the alignment mechanism, run all baselines on the same task and split, and make code and data available.","headline":"Desk reject: the 77.8% MIMIC-IV claim depends on Table 1 comparisons to unrelated tasks and a 'context-aware semantic alignment mechanism' that the methods never specify.","tokens_in":7454,"tokens_out":2915,"would_cite":false,"duration_ms":34386,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Transformer-based multi-label classifier with multi-head self-attention is claimed to outperform five baselines on MIMIC-IV disease prediction, reaching 77.8% accuracy.","keywords":["electronic health records","multi-label disease prediction","Transformer","attention mechanism","MIMIC-IV","clinical NLP","deep learning","context-aware semantic alignment"],"falsifier":"Re-run LSTM, CNN, BiLSTM, RF-MediSys, and DITTO on the exact MIMIC-IV subset, label set, and split used to produce Table 1; if any baseline matches or exceeds 77.8% accuracy under identical metric definitions, the claimed superiority fails.","tokens_in":6544,"feed_emoji":"🏥","tokens_out":6347,"duration_ms":66134,"temperature":0.7,"pith_summary":"This paper tries to establish that a Transformer-based multi-label classifier operating on raw intensive-care clinical notes can predict a patient's set of diagnoses more accurately than five established baselines. The authors report 77.8% accuracy, 75.9% precision, and 73.2% recall on MIMIC-IV, topping LSTM, CNN, BiLSTM, RF-MediSys, and DITTO, and they attribute the gain to multi-head self-attention capturing both medical entities and the co-occurrence structure of diseases. If the result holds, a single attention-based text model could serve as a practical backbone for extracting diagnostic information from EHR notes without hand-built features. The paper also claims the model degrades gracefully as training data shrinks and remains stable until roughly 10% injected noise.","feed_headline":"Transformer predicts ICU diseases from notes at 77.8% accuracy","feed_subtitle":"Multi-head attention on MIMIC-IV notes tops five baselines on accuracy, precision, and recall.","key_machinery":"The load-bearing mechanism is the multi-head self-attention encoder, which computes scaled dot-product attention $Attention(Q,K,V)=\\mathrm{softmax}(QK^T/\\sqrt{d_k})V$ and concatenates parallel heads to form a contextual representation of the note. That representation is fed to a sigmoid classifier for each disease label, with a multi-label cross-entropy loss that models labels independently while attention is expected to capture label co-occurrence. The abstract names this combination a 'context-aware semantic alignment mechanism,' and it is what the paper credits for the performance gains.","core_discovery":"The central claim is that multi-head self-attention over clinical text yields a better multi-label disease predictor than the five baselines in Table 1. The model embeds note text with a pre-trained medical language model, passes the embeddings through a Transformer encoder, and then applies a per-label sigmoid output layer trained with multi-label cross-entropy loss. The paper reports consistent gains in accuracy, precision, and recall, and interprets these as evidence that attention preserves contextual dependencies among co-occurring diagnoses while locating key medical entities. Sensitivity experiments show accuracy rising with training-sample fraction, a peak at learning rate 1e-4, and a sharp accuracy drop once injected noise exceeds 10%.","pith_inferences":["If the architecture truly encodes label co-occurrence, it should transfer to other multi-label clinical tasks such as procedure or medication prediction from the same notes.","The reported noise sensitivity suggests a concrete extension: training with injected noise or contrastive objectives should improve the 20%-noise regime.","A separate evaluation of information extraction (the paper's stated joint goal) would be needed to show that the unified modeling claim holds, since the experiments report only prediction."],"forward_implications":["A single Transformer encoder can replace separate feature-engineering pipelines for multi-label diagnosis prediction from EHR notes.","Accuracy keeps improving as training data grows and stabilizes past 80% of the sample, implying the model is data-hungry and saturates only with large datasets.","Learning rate near 1e-4 gives the best accuracy/recall balance, while higher rates improve recall at the cost of accuracy.","Accuracy stays roughly stable up to 10% input noise and then drops below 0.67, so real-world deployments would need denoising or semantic augmentation.","The architecture is presented as reusable across disease types and datasets, offering a generalized foundation for clinical NLP."],"supporting_citations":[{"why":"Provides the LSTM baseline compared in Table 1.","marker":"[22]"},{"why":"Provides the CNN baseline compared in Table 1.","marker":"[23]"},{"why":"Provides the BiLSTM baseline compared in Table 1.","marker":"[24]"},{"why":"Provides the RF-MediSys baseline compared in Table 1.","marker":"[25]"},{"why":"Provides the DITTO baseline compared in Table 1.","marker":"[26]"}],"fun_headline_variants":["Attention model predicts multiple ICU diseases from clinical notes","Transformer-based NLP improves multi-disease prediction on MIMIC-IV","Multi-head self-attention over notes tops five baselines in accuracy","Clinical text to disease labels: attention deep learning outperforms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The numbers in Table 1 are comparable: each baseline is assumed to have been evaluated on the same MIMIC-IV multi-label disease prediction task, the same data split, and the same preprocessing as the proposed model.","fun_headline_variants_meta":{"raw":{"variants":["Attention model predicts multiple ICU diseases from clinical notes","Transformer-based NLP improves multi-disease prediction on MIMIC-IV","Multi-head self-attention over notes tops five baselines in accuracy","Clinical text to disease labels: attention deep learning outperforms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1335,"prompt_tokens":876,"completion_tokens":459,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":390}},"tokens_in":492,"tokens_out":459,"duration_ms":5702,"temperature":1.0,"reasoning_tokens":390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:50:48.703141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run LSTM, CNN, BiLSTM, RF-MediSys, and DITTO on the exact MIMIC-IV subset, label set, and split used to produce Table 1; if any baseline matches or exceeds 77.8% accuracy under identical metric definitions, the claimed superiority fails.","supporting_citations":[{"cited_title":"Multi-way adaptive time aware LSTM for irregularly collected sequential ICU data,","cited_arxiv_id":null,"evidence_quote":"Provides the LSTM baseline compared in Table 1."},{"cited_title":"Advancements in telehealth: enhancing breast cancer detection and health automation through smart integration of IoT and CNN deep learning in residential and healthcare settings,","cited_arxiv_id":null,"evidence_quote":"Provides the CNN baseline compared in Table 1."},{"cited_title":"Blockchain-based secure cloud storage for IoT healthcare using BiGRU and BiLSTM models,","cited_arxiv_id":null,"evidence_quote":"Provides the BiLSTM baseline compared in Table 1."},{"cited_title":"RF-MediSys: a radio frequency identification-based electronic medical record system for improving medical information accessibility and services at point of care,","cited_arxiv_id":null,"evidence_quote":"Provides the RF-MediSys baseline compared in Table 1."},{"cited_title":"DITTO: an explainable machine-learning model for transcript-specific variant pathogenicity prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the DITTO baseline compared in Table 1."}],"review_version":1}