{"id":"735c56e3-c788-4104-8381-bc86e5d9f5c4","arxiv_id":"2412.02801","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors report 96.5% accuracy for a PSO-optimized Transformer on a heart disease dataset, but the claim is unsupported by the missing Transformer baseline and missing experimental details.","lead":"A machine learning paper claims that a Transformer model tuned by particle swarm optimization predicts heart disease with 96.5% accuracy, beating random forest by 4.3 points. The paper provides no data, code, or comparison against a plain Transformer, so the claimed improvement cannot be verified.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No validation split and no non-PSO Transformer baseline are reported, so the 96.5% vs 92.2% result in Table 2 cannot establish that PSO improved the Transformer; the 4.3-point gap may reflect test-set overfitting or the Transformer architecture alone.","rationale":"The reader's REJECT verdict is appropriate. My stress-test pass focuses on the causal claim in the abstract and conclusion: PSO 'significantly improves Transformer performance.' The paper gives no evidence that isolates PSO's contribution. Section 4.4 describes PSO fitness evaluated by accuracy/loss but never defines a validation split, so test-set overfitting is not ruled out. Table 2 compares PSO-Transformer to decision tree, random forest, and XGBoost; none of these controls for the Transformer architecture itself. A non-PSO Transformer is absent, so the 96.5% could be achieved by the architecture and hyperparameter choice independent of PSO. The absence of code and data prevents an independent check, and the reference list contains placeholder and mismatched entries, but the decisive issue is the missing controls, not citation hygiene. There is no machine-checked proof or reproducible artifact to offset these gaps. I would keep the reader's REJECT verdict unchanged.","tokens_in":6271,"tokens_out":7926,"duration_ms":82510,"concrete_test":"Obtain the exact code, dataset, and split used by the authors. Rerun the pipeline with a fixed three-way partition (e.g., 80% train / 10% validation / 10% test), use only validation accuracy as the PSO fitness, and keep the test set untouched until the final evaluation. In the same run, train a vanilla Transformer with identical architecture and data but without PSO (e.g., fixed default hyperparameters), and compare test accuracies. If the held-out test accuracy falls below 96.5%, or if the vanilla Transformer reaches or exceeds the PSO-tuned accuracy, then the paper's attribution of the improvement to PSO is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that PSO improves Transformer accuracy on the heart-disease dataset. Two controls are required for that claim, and neither is present. First, Section 4.4 says PSO evaluates particles using metrics such as accuracy or loss, but no train/validation/test split is defined anywhere in the manuscript. If the same test set was used both to compute PSO fitness and to produce the Table 2 accuracy of 0.965, that number is a selection artifact, not an independent performance estimate. Second, even if the split were clean, the paper never trains a Transformer without PSO (or with default/random hyperparameters) under identical conditions. The closest comparisons are decision tree, random forest, and XGBoost (Table 2), which are different model classes and therefore cannot control for the Transformer architecture. Without a non-PSO Transformer control, the observed 96.5% does not demonstrate that PSO contributed anything; the improvement over random forest could be due to the Transformer itself or to chance. Because no code or dataset is provided, these controls cannot be recovered from the paper, and the headline empirical claim is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using particle swarm optimization (PSO) to tune hyperparameters of a Transformer model for heart disease prediction. The authors first train decision tree, random forest, and XGBoost models on an open-source dataset described as containing 1888 samples, reporting random forest as the best baseline with 92.2% accuracy. They then train a Transformer whose learning rate, number of layers, hidden dimension, and attention heads are optimized by PSO, and report 96.5% accuracy, concluding that PSO significantly improves Transformer performance. The paper includes no code, no dataset identifier, no train/validation/test split description, no non-PSO Transformer baseline, and no error bars or significance testing.","tokens_in":6556,"tokens_out":4108,"duration_ms":38708,"significance":"If the reported claim were fully supported, the paper would provide a modest but potentially useful demonstration that PSO-based hyperparameter optimization can improve Transformer accuracy on a small medical tabular dataset. However, the central claim is currently unverified because the experimental design lacks the two essential controls: a Transformer trained without PSO under identical conditions, and a clear separation between the data used for PSO fitness evaluation and the test set used for the reported accuracy. The paper also provides no reproducibility artifacts (code, data, or precise experimental protocol). Given these omissions, the paper in its current form does not establish that PSO contributed to the reported performance, and the contribution reduces to an accuracy number without scientific support.","major_comments":[{"comment":"The central claim that PSO improves Transformer performance is unsupported because the paper never trains a Transformer without PSO. The comparisons in Table 2 are against decision tree, random forest, and XGBoost, which are different model classes; any accuracy gap between the PSO-Transformer and these baselines could be due to the Transformer architecture alone, the PSO tuning, or both. An experiment with a Transformer trained using default or randomly chosen hyperparameters, evaluated on the same data split and with the same protocol, is necessary to isolate the effect of PSO. Without this control, the 96.5% accuracy does not demonstrate that PSO contributed anything.","section":"Section 4.4, Table 2"},{"comment":"The manuscript never describes a validation set. The text states that PSO evaluates each particle's fitness on 'specific tasks (e.g., accuracy, loss, etc.)' but does not specify whether the fitness accuracy is computed on a validation split, the test set, or the training set. If the same test set used for the final reported 96.5% in Table 2 is also used to compute PSO fitness, then the reported accuracy is a selection artifact caused by optimizing hyperparameters on the test set, not an independent estimate of generalization. The authors must explicitly define the train/validation/test split and confirm that PSO used only the validation split.","section":"Section 4.4"},{"comment":"The experimental protocol lacks cross-validation, repeated runs, and error bars. With a dataset of 1888 samples and a single reported accuracy per model, the 4.3 percentage point gap between the PSO-Transformer (0.965) and random forest (0.922) could be within run-to-run variability, especially for a stochastic PSO search and neural network training. The authors should report mean and standard deviation over multiple independent runs, or use k-fold cross-validation, and ideally perform a statistical significance test (e.g., McNemar's test) to support the claim of improvement.","section":"Section 2 and Section 4.4"},{"comment":"The dataset is not adequately described or correctly cited. The abstract says the target variable is coded 0 (no heart disease) and 1 (heart disease), but Section 2 says '1 in target indicates yes, and 2 indicates no', and Table 1 shows only rows with target equal to 1. The dataset is attributed to 'Vaswani et al. [2017a]', which is the 'Attention is all you need' Transformer paper, not a heart disease dataset. The authors must identify the exact dataset (e.g., the UCI Heart Disease dataset or a specific variant), report the number of classes, class balance, feature preprocessing, and how the 1888 samples were split into training, validation, and test sets. Without this information, the experimental results are not reproducible.","section":"Section 2, Table 1"}],"minor_comments":[{"comment":"The row label 'Textual model' appears to be a typo and should be 'Transformer model'.","section":"Table 2"},{"comment":"The claims that PSO with stacked sparse auto-encoders achieves 96.1% accuracy and that PSO has been used to optimize SVM are cited to Vaswani et al. [2017b] and Kazemnejad [2020], respectively; neither reference is related to those claims. Appropriate references for PSO-SAE and PSO-SVM should be provided.","section":"Section 4.1"},{"comment":"The phrase 'totaling 1888 pieces of information' should be '1888 samples' or '1888 patient records' for clarity.","section":"Section 2"},{"comment":"The PSO algorithm is described verbally but the velocity and position update equations are not given. Adding the standard PSO update equations would improve precision and reproducibility.","section":"Section 4.2"},{"comment":"The model parameter section lists fixed values for learning rate (0.001), hidden dimension (512), attention heads (8), and feedforward dimension (2048), but does not state whether these are the values found by PSO or the settings used for a non-PSO baseline. The authors should clarify the parameter search ranges and report the optimized hyperparameter values.","section":"Section 4.4"},{"comment":"The correlation analysis in Section 3 reports no quantitative results, and Figure 9's caption 'The comparison of model parameters' is vague; specify what is being compared (e.g., model size, training time, or hyperparameter values).","section":"Section 3 and Figure 9"},{"comment":"The conclusion states that PSO 'significantly improves' Transformer performance, but no statistical significance testing is reported; the wording should be tempered to match the evidence.","section":"Abstract and Conclusion"}],"recommendation":"reject","confidential_remarks":"The manuscript has several red flags beyond the missing controls: the dataset citation is to Vaswani et al., the target variable encoding is inconsistent between the abstract and Section 2, and Section 4.1 cites unrelated works for specific algorithmic claims. These suggest the manuscript was not carefully checked before submission. Given that the central empirical claim is unsupported and cannot be repaired without new experiments, I recommend rejection. If the authors were to resubmit, they would need to provide a proper dataset description, a clear train/validation/test protocol, a non-PSO Transformer baseline, and repeated-run statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper applies standard PSO to Transformer hyperparameter search on a public heart-disease dataset and reports 96.5% accuracy, about 4.3 points above random forest. The problem is that the experiment is missing the controls needed to support that headline. There is no validation split described anywhere, and the PSO fitness function is said to use accuracy or loss. If the same test set that produced the 96.5% was used to evaluate particles, that number is a selection artifact, not an independent measurement. Even if the split were clean, the paper never trains a Transformer without PSO, so the comparison to random forest cannot isolate the contribution of PSO. The gap could be due to the Transformer architecture itself or to chance.\n\nTo be fair, the paper does some things competently. It gives a legible walk-through of PSO and Transformer basics, uses a standard public dataset, reports confusion matrices, and lists hyperparameters. That is more than some arXiv submissions do. But the positives stop there. There is no cross-validation, no error bars, no code, and no dataset link. The references are messy: Vaswani is cited for the dataset and again for PSO with autoencoders, Kazemnejad's blog on positional encoding is cited for PSO+SVM, and there is a placeholder \"Doe et al.\" entry. The claim that PSO \"verifies\" effectiveness is not justified.\n\nThis looks like a routine application of a known optimizer to a small tabular problem, with nothing new in method or theory. The global-health framing in the abstract and conclusion is far out of proportion to the incremental result. I don't think this paper is important or solid enough to warrant referee time. A serious editor should desk reject it, though the authors might benefit from feedback on experimental design and citation hygiene.\n\nFor a reader wanting a simple example of PSO for Transformer tuning, this might illustrate the general idea, but the missing controls make the specific result unreliable. I would not cite it, and I would not bring it to a reading group.","headline":"PSO-tuned Transformer for heart disease data reports 96.5% accuracy, but missing validation split, absent Transformer baseline, and sloppy citations leave the central claim unsupported.","tokens_in":7021,"tokens_out":3419,"would_cite":false,"duration_ms":34097,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that using particle swarm optimization to tune a Transformer's hyperparameters achieves 96.5% classification accuracy on a heart disease dataset, beating random forest by 4.3 percentage points.","keywords":["heart disease prediction","particle swarm optimization","Transformer","hyperparameter optimization","classification","random forest","medical machine learning"],"falsifier":"Train the same PSO-Transformer pipeline with an explicit three-way split — training, validation for PSO fitness, and a held-out test set touched only once — and compare the final test accuracy with random forest on the same split. If the independently measured accuracy falls to roughly the random forest level or below, the paper's central claim of a PSO-driven improvement would be refuted; if it stays near 96.5%, the claim is supported.","tokens_in":6095,"feed_emoji":"🫀","tokens_out":7095,"duration_ms":57595,"temperature":0.7,"pith_summary":"This paper tries to show that particle swarm optimization (PSO) can tune a Transformer model to classify heart disease more accurately than standard machine learning baselines. On an open-source dataset of 1888 patient records, the PSO-optimized Transformer reportedly reaches 96.5% accuracy, versus 92.2% for random forest, 91.2% for decision tree, and 80.8% for XGBoost. If this holds, it would mean that a biologically inspired search algorithm can replace manual hyperparameter selection for Transformer models on medical tabular data. The result matters because accurate heart disease prediction could support earlier intervention and better use of health care resources.","feed_headline":"PSO-tuned Transformer hits 96.5% accuracy on heart disease","feed_subtitle":"Particle swarm optimization beats random forest by 4.3 points on the same data.","key_machinery":"The central object is the PSO-Transformer hybrid: particle swarm optimization treats hyperparameter combinations as particles moving through a multidimensional search space, with velocity updated from each particle's own best position and the swarm's global best, and fitness measured by model accuracy or loss during training. The Transformer itself is the base model, using multi-head self-attention and positional encoding to make predictions from tabular patient features. This machinery does the work of replacing manual hyperparameter search with an automated, swarm-driven search over model depth, width, learning rate, and attention head count.","core_discovery":"On its own terms, the paper's central claim is that PSO can optimize the architecture and training hyperparameters of a Transformer — specifically learning rate, number of layers, hidden layer dimension, and number of attention heads — so that the resulting model outperforms leading classical classifiers on the same heart disease prediction task. The claim is supported by a single comparison: after 100 PSO iterations with 30 particles, the optimized Transformer is reported to achieve 96.5% accuracy, 92.6% precision, 97.3% recall, and 96.2% F1 score, compared with 92.2% accuracy for random forest, the best of the three baselines. The paper concludes that the 4.3 percentage point improvement verifies the effectiveness of PSO in optimizing Transformer models for this task.","pith_inferences":["The paper evaluates only one dataset of 1888 records; whether the 4.3 point gain generalizes to other heart disease cohorts or to non-medical tabular datasets is an open question the paper does not address.","Because no validation split is described, a safer reading is that the headline accuracy may include selection on the test set; a nested cross-validation would clarify whether PSO's gain is real or an artifact.","An implicit comparison the paper does not make is against a PSO-tuned random forest or a default Transformer; without those, the improvement cannot be attributed specifically to PSO versus the Transformer architecture.","The PSO settings are fixed at 30 particles, 100 iterations, and inertia weight 0.9; testing sensitivity to these settings would show how robust the reported 96.5% accuracy is."],"forward_implications":["PSO can automate hyperparameter selection for Transformers, replacing manual tuning in this setting.","The optimized Transformer outperforms decision tree, random forest, and XGBoost on the same dataset, indicating that attention-based models with tuned hyperparameters are competitive on tabular medical data.","Because PSO also optimizes structural parameters such as layer count and attention heads, the method could be applied to other prediction tasks without task-specific architecture engineering.","The reported 4.3 percentage point gain over random forest is the concrete evidence for the paper's conclusion that PSO effectively optimizes Transformer performance."],"supporting_citations":[{"why":"Supplies the Transformer architecture whose learning rate, layers, hidden dimension, and attention heads are the target of PSO optimization.","marker":"Vaswani et al. [2017b]"},{"why":"Provides the particle swarm optimization algorithm description that defines the hyperparameter search procedure.","marker":"Freitas et al. [2020]"},{"why":"Background for the PSO fitness evaluation and velocity-position update equations used to score candidate hyperparameter combinations.","marker":"Hassan et al. [2004]"},{"why":"Cited as a preceding example of PSO optimizing SVM parameters, motivating the use of PSO to tune model parameters.","marker":"Kazemnejad [2020]"}],"fun_headline_variants":["PSO-tuned Transformer predicts heart disease with 96.5% accuracy","PSO boosts Transformer heart-disease accuracy to 96.5%","Transformer + PSO: 96.5% heart-disease prediction accuracy","PSO-optimized Transformer reaches 96.5% on heart disease","Heart disease prediction: PSO-tuned Transformer scores 96.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the accuracy used to score each PSO particle's fitness came from a validation set that was never used to select the final model; if the same test set that produced the 96.5% figure was used for fitness evaluation, the reported advantage could be an artifact of overfitting.","fun_headline_variants_meta":{"raw":{"variants":["PSO-tuned Transformer predicts heart disease with 96.5% accuracy","PSO boosts Transformer heart-disease accuracy to 96.5%","Transformer + PSO: 96.5% heart-disease prediction accuracy","PSO-optimized Transformer reaches 96.5% on heart disease","Heart disease prediction: PSO-tuned Transformer scores 96.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001074,"raw_usage":{"total_tokens":4487,"prompt_tokens":929,"completion_tokens":3558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":3458}},"tokens_in":545,"tokens_out":3558,"duration_ms":24868,"temperature":1.0,"reasoning_tokens":3458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:04:27.997178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same PSO-Transformer pipeline with an explicit three-way split — training, validation for PSO fitness, and a held-out test set touched only once — and compare the final test accuracy with random forest on the same split. If the independently measured accuracy falls to roughly the random forest level or below, the paper's central claim of a PSO-driven improvement would be refuted; if it stays near 96.5%, the claim is supported.","supporting_citations":[{"cited_title":"Particle swarm optimisation: A historical review up to the current developments","cited_arxiv_id":null,"evidence_quote":"Provides the particle swarm optimization algorithm description that defines the hyperparameter search procedure."},{"cited_title":"A comparison of particle swarm optimization and the genetic algorithm","cited_arxiv_id":null,"evidence_quote":"Background for the PSO fitness evaluation and velocity-position update equations used to score candidate hyperparameter combinations."},{"cited_title":"Understanding self-attention and positional encoding of the transformer architecture","cited_arxiv_id":null,"evidence_quote":"Cited as a preceding example of PSO optimizing SVM parameters, motivating the use of PSO to tune model parameters."}],"review_version":1}