{"id":"0ad7e393-0558-4446-bcda-8c185d862939","arxiv_id":"2412.19696","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The study claims a PSO-plus-TabTransformer pipeline predicts IVF live birth with 99.5% accuracy, but the model uses outcome-derived features that make the result a leakage artifact rather than a clinical prediction.","lead":"This paper reports a machine learning pipeline that predicts whether an IVF cycle leads to a live birth, claiming 99.5% accuracy on UK fertility clinic data. A generalist should care because the reported performance is suspiciously high and the pipeline uses outcome-related features, like delivery dates and previous live birth counts, as inputs.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 99.5% accuracy/AUC claim is unsupported because the PSO-selected features in Table 6 include outcome-derived variables such as delivery dates, gestational weeks, and live-birth counts.","rationale":"The reader's weakest assumption correctly identifies outcome-derived features as the central problem. The paper's Table 6 explicitly lists features that are only known after a pregnancy has progressed to delivery, alongside the target variable live birth. The reported near-perfect performance is exactly what one expects when the label itself is included among the inputs. The reader's supporting points about patient-level leakage and missing ablations are secondary but real; the decisive issue is the presence of outcome features in the selected set. My concern lands on the same assumption, so the verdict should remain REJECT. The proposed test is a single reproducible experiment that would settle whether the headline result survives when the leakage features are removed.","tokens_in":14929,"tokens_out":1707,"duration_ms":19293,"concrete_test":"Rerun the PSO + TabTransformer pipeline on the same HFEA 2010-2018 cohort after removing every post-treatment/outcome feature listed in Table 6, specifically 'Heart one delivery date', 'Heart two delivery date', 'Heart one week's gestation', 'Heart two weeks gestation', 'Total number of live births - conceived through IVF', and 'Total number of live births - conceived through IVF or DI'. Also group cycles by patient before creating the 10 folds so no patient appears in both training and validation folds, and report accuracy and AUC for the leakage-free feature set. If accuracy and AUC drop materially below 99.5%/0.9996, the original claim is explained by leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the Abstract and Section 3.1, is that PSO + TabTransformer predicts live birth with 99.50% accuracy and 99.96% AUC. For that claim to hold, the model must be trained on features available before the outcome is known, and evaluation must not leak outcome information. Table 6 shows the PSO-selected feature set includes 'Heart one delivery date', 'Heart two delivery date', 'Heart one week's gestation', 'Heart two weeks gestation', and 'Total number of live births - conceived through IVF or DI'. These are post-conception outcome variables: a delivery date and gestational week of the baby are only known after a live birth, and live-birth counts directly encode the target. Including such features in the model makes near-perfect accuracy a mechanical consequence of label leakage, not a clinically meaningful prediction. The paper reports no experiment that removes these fields and no patient-level split; its random 10-fold cross-validation can also place cycles from the same patient in both training and test sets. The unstated assumption that outcome-derived registry fields are legitimate pre-treatment predictors is therefore the load-bearing weakness, and the headline performance has no stated basis without it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an AI pipeline for predicting live birth success after IVF from the 2010–2018 HFEA registry. It compares PCA- and PSO-based feature selection, combined with random forest, decision tree, a custom transformer, and a TabTransformer. The authors report that PSO + TabTransformer achieves 99.50% accuracy, 99.6% precision, 99.5% recall, 99.5% F1, and 0.99 AUC, and they interpret this as evidence that transformer-based models, when combined with optimization-based feature selection, can almost perfectly predict live birth from registry data. The central methodological steps are feature selection (PSO with a fitness based on logistic regression F1 minus a penalty), model training with 10-fold cross-validation, and evaluation with standard metrics. The authors also compare their results with prior work using the same HFEA dataset, positioning their accuracy improvement as the main contribution.","tokens_in":15207,"tokens_out":1654,"duration_ms":20759,"significance":"If the reported performance were valid, a registry-only model predicting live birth at 99.5% accuracy would be a substantial practical contribution, since most prior work on HFEA data reports accuracies in the range roughly 0.73–0.96 and AUCs of 0.63–0.79. The paper also makes a methodological claim, that combining PSO feature selection with TabTransformer is a novel and effective pipeline for tabular IVF outcome prediction. However, the central performance claim is not supported by the reported experiments because the feature sets include variables that are only observable after, or that directly encode, the outcome. The paper does not report any ablation excluding these outcome-derived fields, and it does not use a patient-level split, so the headline accuracy is most plausibly a mechanical artifact of label leakage rather than genuine predictive skill. The framework and model architecture themselves are reasonable to explore, but the experimental design in this manuscript does not provide evidence for the claimed clinical prediction capability.","major_comments":[{"comment":"The central claim of 99.50% accuracy and 0.99 AUC for PSO + TabTransformer is invalidated by feature leakage. Section 3.2 and Table 6 show that the PSO-selected feature set includes 'Heart one delivery date', 'Heart two delivery date', 'Heart one week's gestation', 'Heart two weeks gestation', 'Total number of live births - conceived through IVF', and 'Total number of live births - conceived through IVF or DI'. A delivery date and gestational weeks are known only after a live birth, and prior live-birth counts directly encode the outcome label. Feeding these variables to the model makes near-perfect accuracy a mechanical consequence of reading the answer from the features, not a clinically meaningful prediction. No experiment in the paper removes these fields, so the headline result has no stated basis as a live-birth prediction.","section":"Abstract and Section 3.1"},{"comment":"The evaluation uses 10-fold cross-validation, but the paper does not state that folds are split at the patient level. HFEA data contain multiple cycles per patient, and the outcome of repeated cycles from the same patient are strongly correlated. Without a patient-level split, cycles from the same patient can appear in both training and validation folds, inflating all reported metrics. The authors need to show a patient-stratified split and report results under that split, or explicitly justify why cycle-level independence is clinically appropriate for a prediction intended to support personalized fertility treatment.","section":"Section 2.4 and Section 3.1"},{"comment":"The feature-selection procedure itself, as described, does not exclude outcome-derived fields. The PSO fitness in Algorithm 1 and Equation (1) uses a logistic regression F1-score over the selected features, which will retain outcome-derived fields because they trivially predict the target. The paper provides no sensitivity analysis or ablation with a feature subset restricted to pre-treatment or pre-transfer information. Without such an ablation, the reported 99.50% accuracy cannot be attributed to the model's ability to learn meaningful clinical interactions; it is more directly attributable to the inclusion of post-outcome variables.","section":"Table 6 and Section 3.2"},{"comment":"The apparent monotone improvement from traditional models (91–93.5%) to transformer models (95.5–99.5%) is not informative about model quality because all methods use the same leakage-prone feature space. For example, PCA + Decision Tree also has access to the outcome-derived variables, and the fact that TabTransformer achieves 99.5% while PCA+DT achieves 91% likely reflects the capacity of the transformer to exploit the leaked variables, not a generalizable advantage in IVF outcome prediction. The paper should compare all models on a leakage-free feature set; otherwise, the comparative claim in Table 5 is not a fair test of either feature selection or model architecture.","section":"Section 3.1 and Table 5"}],"minor_comments":[{"comment":"There is an inconsistency between the transformer model input dimension described in Table 1 (45 selected features) and Table 2 (input_dim = 40 features). The paper should reconcile these numbers and state which feature count was actually used.","section":"Section 2.3.5 and Table 2"},{"comment":"The position update rule for the binary PSO is incompletely written: the text states \"Update position using a sigmoid function\" but does not explicitly state the thresholding step that maps the probability to a binary value, and the notation for the velocity update has a typo in the random coefficients (the text defines r2 twice instead of r1 and r2).","section":"Algorithm 1"},{"comment":"Table 6 contains two entries with the same feature name 'Total number of live births - conceived through IVF or DI' under different categories (Patient History and Outcome), and the typo 'mbryos stored' for 'Embryos stored'. The categories and feature names should be made consistent.","section":"Section 3.2 and Table 6"},{"comment":"The paper states that the model was built with 'balanced datasets to address class imbalance', but the methodology section does not describe any resampling, class weighting, or balancing step. This claim is unsupported by the presented methods.","section":"Abstract and Section 4"},{"comment":"The comparison in Table 7 reports the AUC for the proposed model as '99.96%', while Table 5 reports AUC = 0.99. The units and decimal places are inconsistent; the authors should report AUC consistently as a fraction or percentage.","section":"Table 7"},{"comment":"The text says the initial dataset has 94 features, but Section 2.3.5 and Table 1 refer to 45 selected features, while Table 2 says 40 features, and Section 2.3.6 says the input dimension is the number of selected numerical features. The paper should clarify the number of original features, the number retained after preprocessing, and the exact dimension given to each model.","section":"Section 2.1.1"}],"recommendation":"reject","confidential_remarks":"The paper's headline result is not credible because the feature set includes direct outcome indicators. The authors should be given the opportunity to resubmit a version that deletes all delivery date, gestational week, and live-birth-count fields, and that re-runs all experiments under patient-level cross-validation. If the updated results still support a strong method comparison, the paper could be of interest; in its current form, the methodological contribution is obscured by the leakage artifact."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is not salvageable as is. The headline accuracy of 99.50% is an artifact of label leakage: Table 6 lists 'Heart one delivery date', 'Heart one week's gestation', 'Heart two delivery date', 'Heart two weeks gestation', and 'Total number of live births - conceived through IVF or DI' among the PSO-selected features. These fields are only known after a live birth or directly encode the target. No experiment removes them, so the model is reading the answer from the input.\n\nWhat the paper does well: the writing is clear, the eight-way comparison of feature selectors and classifiers is easy to follow, and the use of the public HFEA registry is appropriate. The PSO setup is standard, the TabTransformer implementation is conventional, and Table 7's comparison to prior HFEA-based work is a useful reference point. If the leakage were removed, this would be a reasonable benchmark-style study.\n\nWhere it breaks: the leakage alone is fatal. But there are also softer issues: random 10-fold cross-validation likely mixes cycles from the same patient across folds; no patient-level split is reported; no ablation excludes outcome-derived features; and the paper contains internal inconsistencies (Table 1 says 45 input features, Table 2 says 40; the abstract reports AUC 99.96% while Table 5 reports 0.99). No code or preprocessing scripts are provided, which makes the leakage harder to detect but also prevents verification.\n\nBottom line: this is a cautionary example of target leakage, not a usable clinical model. It deserves a desk reject, not referee time. If the authors rerun everything without outcome-derived features and with patient-level splitting, the comparison might be worth a fresh look, but the current version does not support any of its predictive claims.","headline":"The 99.5% IVF live-birth accuracy is a label-leakage artifact: PSO-selected features include delivery dates and live-birth counts, so the paper's central claim is unsupported.","tokens_in":646,"tokens_out":1796,"would_cite":false,"duration_ms":39231,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A PSO-optimized TabTransformer predicts IVF live birth with 99.50% accuracy, the paper claims.","keywords":["in vitro fertilization","live birth prediction","particle swarm optimization","TabTransformer","feature selection","HFEA dataset","target leakage","transformer attention"],"falsifier":"Re-run the PSO plus TabTransformer pipeline on the same 115,012 records with all outcome-related fields removed, including delivery dates, gestational weeks, and live-birth counts, using the same 10-fold cross-validation; if accuracy and AUC fall far below 99.50% and 99.96%, the original result is explained by target leakage, while if they remain near those levels the claim survives.","tokens_in":14644,"feed_emoji":"👶","tokens_out":7727,"duration_ms":66548,"temperature":0.7,"pith_summary":"This paper aims to establish that an integrated pipeline—particle swarm optimization for feature selection followed by a TabTransformer classifier—can predict live birth after IVF almost perfectly from a national fertility registry. The reported result is 99.50% accuracy and 99.96% AUC on 115,012 HFEA treatment records from 2010 to 2018, beating random forests, decision trees, PCA-based inputs, and a custom transformer. If the claim held, it would mean registry-only data is sufficient for near-certain outcome prediction, a dramatic improvement over prior models that reached roughly 96% accuracy or AUCs in the 0.63–0.79 range. The paper's own feature table, however, shows that the optimized 45-feature set includes delivery dates, gestational weeks, and counts of previous live births—fields that record the outcome itself. That makes the load-bearing assumption the legitimacy of using outcome-derived registry fields as predictors.","feed_headline":"IVF live-birth model claims 99.5% accuracy","feed_subtitle":"A PSO-TabTransformer pipeline tops prior models, but the selected features include the outcome itself.","key_machinery":"The central mechanism is a two-stage pipeline. First, binary particle swarm optimization searches feature subsets by minimizing $C = -(F1 - P \\cdot N)$, where $F1$ is the F1-score of a logistic regression trained on the selected features, $P$ is a penalty weight, and $N$ is the number of selected features. Second, a TabTransformer embeds categorical variables, concatenates them with normalized numerical features, and applies multi-head self-attention with four heads followed by feed-forward layers to output a live-birth probability. The paper claims this combination captures feature interactions and yields the reported near-perfect metrics.","core_discovery":"The authors claim that combining binary particle swarm optimization with a TabTransformer produces a model that predicts live birth success in IVF with 99.50% accuracy, 99.6% precision, 99.5% recall, 99.5% F1-score, and an AUC of 99.96% (reported as 0.99 in their results table), outperforming all seven alternative pipelines tested on the same HFEA data. They attribute this to PSO's ability to select a compact, informative feature subset of 45 features and to the TabTransformer's learned categorical embeddings and multi-head self-attention, which capture interactions between clinical, demographic, and procedural variables that traditional classifiers miss. The paper positions this as the first application of transformer-based models to IVF live-birth prediction and as a clinically applicable tool for personalized fertility treatment.","pith_inferences":["Because the selected feature set includes 'Heart one delivery date', 'Heart two delivery date', weeks-of-gestation fields, and counts of prior live births, the near-perfect scores most plausibly reflect the model reading the outcome from the input; excluding those fields would likely drop performance to the range of earlier studies, not 99.5%.","A natural test of the pipeline's genuine predictive value is to rerun PSO inside the cross-validation loop on a feature set purged of all outcome-derived and post-treatment fields; the paper reports no such leakage-free experiment.","The attention weights of the TabTransformer could be audited to see whether the model is keying on delivery dates and live-birth histories, turning the claimed clinical tool into a diagnostic of data quality.","If the leakage-free accuracy falls to the clinical range, the transferable contribution is not the 99.5% number but the pipeline design—PSO selection plus TabTransformer—which could still be evaluated honestly on truly pre-treatment features."],"forward_implications":["If the accuracy of 99.50% holds, a registry-only model could counsel IVF patients on expected live-birth chances with near certainty.","The comparison suggests PSO-based feature selection is substantially better than PCA for this tabular prediction task, since every PSO classifier outperformed its PCA counterpart.","Transformer-based models beat random forest and decision tree on every metric, implying attention mechanisms add predictive value for IVF registry data.","A clinically deployed version of this pipeline would need only the 45 selected fields, making data collection and interpretation simpler than with the full 94-feature registry.","The result would set a new benchmark on the HFEA dataset, surpassing the 96.35% accuracy of the previous best model on the same 2010–2018 data."],"supporting_citations":[{"why":"Supplies the inclusion and exclusion criteria and the 96.35% accuracy baseline on the same HFEA data that the paper claims to beat.","marker":"[21]"},{"why":"Provides the hybrid PSO feature-selection method the pipeline adapts to IVF prediction.","marker":"[23]"},{"why":"Supplies the transformer and multi-head attention architecture underlying both deep models.","marker":"[29]"},{"why":"Supplies the TabTransformer design with categorical embeddings and self-attention for tabular data.","marker":"[30]"},{"why":"Provides a prior HFEA-based live-birth prediction model used as a benchmark in the comparison.","marker":"[15]"},{"why":"Gives the discrete-time logistic regression benchmark that motivates the need for improved live-birth prediction.","marker":"[16]"}],"fun_headline_variants":["IVF prediction hits 99.5% but uses the answer as a feature","99.5% IVF accuracy? The model saw the outcome in training","PSO-TabTransformer tops IVF but selects the outcome itself","IVF model's secret: it cheats with the ground truth","High accuracy, but IVF model leaks the outcome into features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline treats outcome-derived registry fields—delivery dates, gestational weeks, and counts of previous live births—as legitimate input features, even though they describe the birth event the model is supposed to predict.","fun_headline_variants_meta":{"raw":{"variants":["IVF prediction hits 99.5% but uses the answer as a feature","99.5% IVF accuracy? The model saw the outcome in training","PSO-TabTransformer tops IVF but selects the outcome itself","IVF model's secret: it cheats with the ground truth","High accuracy, but IVF model leaks the outcome into features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1319,"prompt_tokens":931,"completion_tokens":388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":547,"tokens_out":388,"duration_ms":339650,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:57:08.673074+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the PSO plus TabTransformer pipeline on the same 115,012 records with all outcome-related fields removed, including delivery dates, gestational weeks, and live-birth counts, using the same 10-fold cross-validation; if accuracy and AUC fall far below 99.50% and 99.96%, the original result is explained by target leakage, while if they remain near those levels the claim survives.","supporting_citations":[{"cited_title":"A., Khanjani, S., Javanmardi, S., Bayat, B., Naderi, Z., & Hajiyavand, A","cited_arxiv_id":null,"evidence_quote":"Supplies the inclusion and exclusion criteria and the 96.35% accuracy baseline on the same HFEA data that the paper claims to beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hybrid PSO feature-selection method the pipeline adapts to IVF prediction."},{"cited_title":"N., … Polosukhin, I","cited_arxiv_id":null,"evidence_quote":"Supplies the transformer and multi-head attention architecture underlying both deep models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TabTransformer design with categorical embeddings and self-attention for tabular data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a prior HFEA-based live-birth prediction model used as a benchmark in the comparison."}],"review_version":1}