{"id":"56ea29bb-292e-4cc4-b291-5f1a413b6b04","arxiv_id":"2506.15330","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A set-transformer that embeds test names with GPT text vectors and predicts abnormal glucose, cholesterol, ferritin, and uric acid from incomplete lab panels, with AUC gains over MLP baselines.","lead":"This paper proposes a neural network that predicts abnormal lab results, such as high uric acid and glucose, from a patient's routine blood tests even when patients have different sets of tests. It reports AUC gains of up to 8 percentage points compared with a simple multilayer perceptron baseline on 1.17 million laboratory records from Russia.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 80/20 row-level split in Section II-E cannot rule out the same patient appearing in both train and test; if repeats exist, the reported AUC gains, especially uric acid (+8 pp), may reflect patient memorization rather than generalizable prediction. A patient-stratified split is needed.","rationale":"I agree with the reader's weakest_assumption: the random row split is the most load-bearing uncertainty. The paper has independent support that a conditional verdict is appropriate: the architecture is specified in enough detail, the GitHub link is given, and the comparison has per-run standard deviations from three training runs. The concern is therefore not that the results are impossible or fraudulent, but that their interpretation depends on an untested assumption about patient independence. I do not see a stronger internal objection: the attention mechanism is permutation invariant as claimed, the fixed GPT embeddings are parameter-free, and the code availability makes re-implementation feasible. A secondary gap is that the MLP baseline's handling of missing values is not described in Section II-E; this should be documented, but it is secondary to the split issue. The right action is to keep the CONDITIONAL verdict and require the patient-level split check before the central claim is accepted as stated.","tokens_in":10295,"tokens_out":9125,"duration_ms":97895,"concrete_test":"Ask the authors to provide anonymized patient identifiers or an equivalent grouping key and rerun the Section II-E protocol with GroupShuffleSplit stratified by patient. If Table IV AUCs, particularly uric acid (79.7 vs 71.5), do not survive the patient-level split, the row-level split is the cause. If identifiers cannot be shared, an alternative check is to deduplicate the dataset by quasi-identifiers (age, gender, date, and near-identical test vectors) and recompute the ULM-minus-MLP margin on the deduplicated test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest load-bearing assumption is the independence of the 80/20 row-level random split in Section II-E. The database is described as 1,173,403 'data rows' from commercial laboratories (Section II-A), and Table III treats each row as one laboratory report, but no patient identifier is used anywhere in the protocol. In a real LIS, the same patient contributes multiple reports over time, and repeated reports share stable, person-specific CBC and biochemistry values. If any patient appears on both sides of the split, a set-transformer with fixed feature embeddings can memorize the patient's joint value pattern rather than learning a general rule, artificially raising test AUC. The headline comparison (ULM vs MLP, up to +8 pp for uric acid in Table IV) is quantitatively dependent on this not happening. The manuscript neither reports unique-patient counts nor performs a group-aware split, so the evaluation rests on an unverified independence assumption. This is not a demonstrated flaw, but because the entire quantitative claim passes through it, it is the single most load-bearing point to check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Universal Laboratory Model (ULM), a set-transformer with fixed GPT-based embeddings for laboratory test names and values, designed to predict binary abnormalities of four target tests (glucose, cholesterol, ferritin, uric acid) from a variable-size set of available routine laboratory results. The authors evaluate ULM on a large commercial laboratory dataset (1,173,403 rows) and report AUC improvements of up to 8 percentage points over a three-layer MLP, with the largest gain for uric acid. The claimed contribution is a tabular-modeling formulation as set translation that handles missing values without explicit imputation.","tokens_in":10543,"tokens_out":4218,"duration_ms":42863,"significance":"If the reported results survive proper validation, the work addresses a practically important problem: predicting non-prescribed abnormal laboratory tests from routine CBC and biochemistry panels, which could enable earlier clinical warnings. The dataset is substantially larger than those used in prior studies, and the authors provide a public model and code. However, the current evidence is insufficient to support the broad claims of effectiveness: the evaluation uses only a single weak baseline (a three-layer MLP), the data split is row-level with no patient grouping, and the outcome definitions differ from prior work. The architectural idea of fixed GPT-based embeddings for both input and target features is interesting and worth further study.","major_comments":[{"comment":"The 80/20 random split is performed at the row level, and the manuscript states that each row is one laboratory report but does not report or use any patient identifier. In a real laboratory information system, the same patient contributes multiple reports over time, and repeated reports share stable person-specific values. If any patient appears on both sides of the split, a set-transformer can memorize patient-specific joint value patterns rather than learn a generalizable rule, which would inflate the reported test AUCs, especially the uric acid gain of 8 percentage points. The manuscript neither reports the number of unique patients nor performs a group-aware split. The authors must either stratify the split by patient or provide strong evidence (e.g., unique patient counts and an appropriate split) that this leakage does not occur.","section":"Section II-E"},{"comment":"The only comparator is a three-layer MLP with architecture 27 x 256 x 4. No classical machine learning baselines (e.g., random forest, gradient boosting) and no imputation-based methods are evaluated, even though the introduction and Table I emphasize comparisons to such methods. The headline claim of 'improvement up to 8% AUC' is therefore only a comparison to one specific MLP architecture; it does not establish that ULM is competitive with or superior to the standard methods used in the prior work cited. The authors should add reasonable baselines, including at least one tree-based method and one imputation-plus-classifier pipeline, on the same train and test sets to support the claim that ULM is a practically useful approach.","section":"Section III, Table IV"},{"comment":"The uric acid positive-class threshold used in Table IV (M: ≥0.48 mmol/L, F: ≥0.38 mmol/L) differs from the threshold used in the prior studies listed in Table I (M: ≥0.42 mmol/L, F: ≥0.36 mmol/L). The choice of threshold changes the prevalence and the difficulty of the classification task, so any implicit comparison of the reported AUC (79.7%) with the prior AUCs in Table I (0.78–0.85) is invalid. The authors should either adopt the thresholds from the prior literature for at least a subset of experiments, or explicitly state that the comparison is not apples-to-apples and discuss the clinical rationale for the higher thresholds.","section":"Table IV vs. Table I"},{"comment":"The confusion matrix counts for each target do not match the test set sizes implied by Table IV and the stated 80/20 split. For example, the glucose counts sum to 142,928, whereas 20% of the glucose sample size 680,502 is 136,100; for cholesterol the sum is 55,477 versus 49,761 expected; for ferritin 22,924 versus 21,186; for uric acid 19,906 versus 18,580. The discrepancies are too large to be rounding artifacts. The authors must correct these numbers or clarify what the sizes in Table IV represent and how the confusion matrices are computed.","section":"Table V"}],"minor_comments":[{"comment":"The notation in Eq. (3) should be clarified: GPT_i denotes a fixed embedding vector for feature i, and the operation 'GPT_i * v_i + B' is a scalar-vector product plus a (presumably vector) bias B. Please state the dimensions explicitly and explain how the resulting vector is incorporated into the encoder.","section":"Section II-C, Eq. (3)"},{"comment":"The sentence 'the sum of all WBC parts from a 5-DIFF analyzer ... must equal100±4%' has a formatting error (missing space before '100'), and the tolerance of 4% is an arbitrary free parameter. Please also state whether this condition is applied to the 3-DIFF analyzers (MID, GRA, LYMP) with the same tolerance.","section":"Section II-B"},{"comment":"There are typographical errors in Table I: 'Iron deficiancy anemia' should be 'Iron deficiency anemia', and 'Na ¨ıveBayes' should be 'Naive Bayes' with a space.","section":"Table I"},{"comment":"The manuscript refers to Figure 1 for the architecture, but the figure is not included in the text; please ensure the figure appears with its caption and is referenced correctly.","section":"Section II-C"},{"comment":"The limitations section does not mention the lack of a patient-stratified split or the absence of external validation. Given that the dataset comes from commercial laboratories, the authors should also discuss calibration and the potential for overfitting to the specific analyzer or laboratory population.","section":"Section IV"},{"comment":"The sentence 'The statistics in Table IV are the result of three different training runs for each variant' is helpful, but the paper does not report how the test set is held fixed across runs or whether the same random seed is used for the split. Please describe the experimental protocol for the random splits and runs.","section":"Section II-E"}],"recommendation":"major_revision","confidential_remarks":"The central architectural idea is interesting, but the evaluation is currently too weak to support the claims. The leakage risk from row-level splitting is the most serious concern, and the lack of any baseline other than a simple MLP makes the 'up to 8% AUC improvement' hard to interpret. The inconsistent confusion matrix counts also need correction. I would encourage the editor to seek a revision that addresses these points, with particular attention to a patient-stratified split and additional baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely appealing: treat a lab report as a set of (test, value) pairs, embed both the tests and the prediction targets with fixed GPT-style embeddings, and rely on a Set Transformer to handle missingness directly. That is a clean way to avoid imputation, and the fixed-embedding trick—avoiding matrix multiplication on the embeddings—is a thoughtful design choice. They also ship code and a live service, which earns real credit.\n\nThe evaluation, though, does not support the headline \"up to 8% AUC.\" The most serious issue is the split. The dataset is 1.17 million rows from commercial labs, and the paper says \"data rows,\" not \"patients.\" Real LIS records contain repeated patients, and if any patient appears on both sides of the 80/20 random split, the model can memorize person-level patterns instead of learning a generalizable rule. The paper gives no patient count and no group-aware split. That single issue could inflate the uric acid number by several points, and it is not checked anywhere in the manuscript.\n\nSecond, the baseline is only a three-layer MLP. The authors themselves note that deep models \"stagnate\" and are \"usually not better\" than RF or SVM, yet they never run RF, XGBoost, logistic regression, or any imputation-based baseline. So we don't know whether ULM wins because of the architecture or simply because the MLP is weak. For a paper claiming a \"universal\" model, that is a glaring gap.\n\nThird, the uric acid threshold differs from prior work: they use 0.48/0.38 mmol/L, while most cited studies use 0.42/0.36. That is not wrong per se, but it makes their AUC numbers non-comparable to the literature they cite, and they don't justify the choice. The small gains on ferritin (+0.8%) are plausible, but they report no significance tests.\n\nThe reproducibility story is mixed: the code is public, but the Yandex GPT embeddings are not fully specified (model, dimensionality, or version), which matters because the embeddings are load-bearing.\n\nIn short, the architecture is worth publishing, but the evaluation is not convincing yet. If the patient-split concern holds, the reported gains may partially evaporate; if not, with proper baselines and external validation, this could become a solid clinical ML paper. The right audience is researchers working on missing data in electronic health records and clinical informaticians interested in attention-based alternatives to imputation. I would send it to peer review—it is a serious, novel contribution with real flaws that a good reviewer can pinpoint and the authors can likely fix. It should not be accepted as-is.","headline":"A clever set-transformer for missing lab values, but the evaluation has a patient-leakage risk and too few baselines to believe the AUC gains.","tokens_in":11025,"tokens_out":3582,"would_cite":true,"duration_ms":34924,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a set-transformer with fixed GPT-style embeddings, called the Universal Laboratory Model, predicts low ferritin, high uric acid, glucose, and cholesterol levels from routine CBC and biochemistry tests, improving AUC…","keywords":["clinical laboratory tests","missing values","set transformer","GPT embeddings","complete blood count","multitask prediction","laboratory information system","uric acid prognosis"],"falsifier":"Re-run the same 80/20 split but group all rows belonging to the same patient into either the training or the test set (using whatever patient identifier exists in the source laboratory information system) and check whether the AUC advantage of ULM over the MLP survives; if the gap collapses to noise, the reported improvement is an artifact of patient leakage.","tokens_in":10120,"feed_emoji":"🩸","tokens_out":7147,"duration_ms":59984,"temperature":0.7,"pith_summary":"The paper argues that a patient's laboratory report can be treated as an unordered set of test-value pairs, and that a set-transformer encoder-decoder with fixed GPT-style test embeddings can predict abnormal values of tests that were not actually ordered. This formulation handles missing values without imputation because absent tests are simply absent. On 1,173,403 rows of routine laboratory records, the Universal Laboratory Model reports AUC gains of up to 8 percentage points over a standard MLP for low ferritin, high uric acid, glucose, and cholesterol. If the finding holds, laboratories could issue early warnings from a CBC and any available biochemistry tests.","feed_headline":"Set transformer flags four hidden lab results from routine tests","feed_subtitle":"ULM forecasts low ferritin, high uric acid, glucose, and cholesterol from a CBC.","key_machinery":"The central object is the Universal Laboratory Model (ULM), a set-transformer encoder-decoder that is permutation-invariant by construction. It encodes each available test as a pair of the test's fixed GPT embedding and its numerical value, applies multi-head attention in the encoder, and then lets the decoder attend to the encoded set using embeddings of the target tests as queries. Because the model operates on sets, reports of different sizes and compositions are accepted without imputation, and the GPT embeddings provide semantic context for what each test measures.","core_discovery":"In its own terms, the paper demonstrates that predicting abnormal laboratory outcomes reduces to a set-translation problem: the input is a set of (test embedding, value) pairs, and the output is a set of test embeddings for the predicted tests. The model, ULM, uses a multi-head attention encoder and decoder, where the decoder queries are the embeddings of the target tests, and the GPT-derived embeddings of test names are kept fixed and transformed only by scalar multiplication and a trainable bias. On held-out rows, the model achieves AUCs of 82.4% for glucose ≥7 mmol/L, 77.9% for cholesterol ≥5.2 mmol/L, 89.7% for ferritin ≤12 ng/mL, and 79.7% for uric acid at sex-specific thresholds, gains of 0.8 to 8 percentage points over a comparable MLP baseline.","pith_inferences":["We infer that the reported 8-point AUC advantage for uric acid is the strongest evidence that the set-transformer benefits from cross-test patterns, so a patient-level or temporal split would test whether that gain survives when repeated visits from the same patient are kept together.","We infer that the design choice to freeze GPT embeddings and avoid matrix multiplications on them, rather than learning column embeddings as in earlier tabular transformers, is the key ablation to try; if learned embeddings perform equally, the improvement would be attributed more to architecture than to semantic prior.","We infer that the model's current dichotomized outcomes could be extended to continuous value prediction or to percentiles, which would make the warnings more informative for clinicians."],"forward_implications":["If the central claim holds, a laboratory information system could automatically add a test predicted to be abnormal when biological material from the same patient is still available, as the paper proposes.","The architecture can be extended to other tests or demographic features without retraining on a fixed feature matrix.","The model's permutation invariance makes it applicable to any subset of the 31 available measurements, so the same trained ULM serves patients with different test panels.","The public deployment at https://ulm.roslis.ru would let any laboratory integrate the prognostic service into its workflow."],"supporting_citations":[{"why":"Supplies the set-transformer architecture with multi-head attention pooling that makes the model permutation-invariant.","marker":"[23]"},{"why":"Defines the multi-head attention mechanism used in the encoder and decoder blocks.","marker":"[24]"},{"why":"Provides the fixed GPT embeddings of test names used for both input features and target tests.","marker":"[25]"},{"why":"Earlier work that learned trainable column embeddings; the paper contrasts its fixed GPT embeddings with this approach.","marker":"[26]"},{"why":"Baseline for ferritin prediction (AUC 0.92) that the paper compares against with its own CBC-based ferritin model.","marker":"[4]"},{"why":"Baseline for uric acid prediction, a stacking ensemble model whose reported metrics are compared with ULM's uric acid AUC.","marker":"[9]"},{"why":"Baseline that imputes missing laboratory values before predicting low ferritin, the imputation-based approach ULM aims to replace.","marker":"[5]"},{"why":"Baseline for diabetes-related laboratory prediction (HbA1c) that frames the glucose prediction task.","marker":"[8]"}],"fun_headline_variants":["Set-transformer flags four hidden lab abnormalities from routine tests","Routine tests predict ferritin, uric acid, glucose, cholesterol","Set-translation lab model gains up to 8% AUC for four outcomes","GPT-based set translation reveals unmeasured lab results from routine panels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation randomly splits rows of the database into training and testing sets and treats each row as independent, so the same patient's repeated laboratory reports can appear in both parts of the split.","fun_headline_variants_meta":{"raw":{"variants":["Set-transformer flags four hidden lab abnormalities from routine tests","Routine tests predict ferritin, uric acid, glucose, cholesterol","Set-translation lab model gains up to 8% AUC for four outcomes","GPT-based set translation reveals unmeasured lab results from routine panels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001217,"raw_usage":{"total_tokens":4982,"prompt_tokens":894,"completion_tokens":4088,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":4012}},"tokens_in":510,"tokens_out":4088,"duration_ms":31343,"temperature":1.0,"reasoning_tokens":4012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:36:12.673943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 80/20 split but group all rows belonging to the same patient into either the training or the test set (using whatever patient identifier exists in the source laboratory information system) and check whether the AUC advantage of ULM over the MLP survives; if the gap collapses to noise, the reported improvement is an artifact of patient leakage.","supporting_citations":[{"cited_title":"Set transformer: A framework for attention-based permutation-invariant neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the set-transformer architecture with multi-head attention pooling that makes the model permutation-invariant."},{"cited_title":"Attention is All you Need,","cited_arxiv_id":null,"evidence_quote":"Defines the multi-head attention mechanism used in the encoder and decoder blocks."},{"cited_title":"Available: https://yandex.cloud/en-ru/services/yandexgpt","cited_arxiv_id":null,"evidence_quote":"Provides the fixed GPT embeddings of test names used for both input features and target tests."},{"cited_title":"Automated prediction of low ferritin concentrations using a machine learning algorithm,","cited_arxiv_id":null,"evidence_quote":"Baseline for ferritin prediction (AUC 0.92) that the paper compares against with its own CBC-based ferritin model."},{"cited_title":"Ensemble machine learning prediction of hyperuricemia based on a prospective health checkup population,","cited_arxiv_id":null,"evidence_quote":"Baseline for uric acid prediction, a stacking ensemble model whose reported metrics are compared with ULM's uric acid AUC."},{"cited_title":"Using machine learning to predict laboratory test results,","cited_arxiv_id":null,"evidence_quote":"Baseline that imputes missing laboratory values before predicting low ferritin, the imputation-based approach ULM aims to replace."},{"cited_title":"A comparative analysis of machine learning models for the detection of undiagnosed diabetes patients,","cited_arxiv_id":null,"evidence_quote":"Baseline for diabetes-related laboratory prediction (HbA1c) that frames the glucose prediction task."}],"review_version":2}