{"id":"d97fc4f8-df34-4261-be66-0ee56a1ccb20","arxiv_id":"2411.17717","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Balancing harmonized multi-site EEG with propensity-score matching gives reported accuracies of 91-98% for classifying presymptomatic E280A carriers, but the supporting evidence has internal contradictions and unresolved demographic confounding.","lead":"This paper combines EEG preprocessing, multi-site harmonization, and propensity-score matching to classify people carrying the PSEN1-E280A Alzheimer's mutation, reporting 91-98% accuracy. A generalist might read it because it claims a practical template for getting reliable biomarkers from small, uneven clinical EEG datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PSM did not balance age/sex (Table 1), so the reported EEG accuracies may reflect demographic confounds rather than brain biomarkers; Fig 7/Table 2 metrics are also mutually inconsistent.","rationale":"The reader's verdict of REJECT is well supported by the evidence. The most load-bearing condition for the paper's central claim is that propensity score matching actually balanced the demographic covariates age and sex between ACr and HC. Table 1 directly contradicts this: in every ratio the ACr group remains older and more female than the HC group, with particularly stark differences at the 10:1 ratio. Because age and sex are both strong predictors of EEG characteristics and are associated with PSEN1-E280A carrier status, a classifier can achieve very high accuracy by exploiting these demographic differences alone. The paper does not provide a no-PSM baseline or a control model trained on demographics only, so the reported accuracies ranging from 0.92 to 0.96 cannot be causally attributed to EEG biomarkers. The internal inconsistency between the confusion matrices in Fig 7 and the metric table in Table 2 further indicates that the reported numbers are not reliable: for the 2:1 ratio the matrix yields precision about 97% and recall about 91%, while Table 2 reports 91% precision and 97% recall; for the 5:1 ratio the matrix yields recall about 97%, not the reported 100%. These are mechanical errors that undermine confidence in the results. The reader's weakest assumption was exactly the PSM balance failure, and my analysis agrees. Other issues, such as feature selection potentially using the full dataset, are also present, but the demographic-confounding problem is the most direct threat to the central claim. Thus the verdict should remain REJECT, and no adjustment to the reader's decision is needed.","tokens_in":11812,"tokens_out":4147,"duration_ms":39364,"concrete_test":"Recompute covariate balance after PSM for all three ratios using standardized mean differences (SMD) for age and sex between ACr and HC. Then train the same decision-tree pipeline using only age and sex as features (no EEG) with identical cross-validation, and compare its accuracy/AUC to the reported values. If |SMD| > 0.1 for either covariate or if the demographic-only model matches or exceeds the reported EEG classification performance, the central claim that EEG biomarkers drive the classification is unsupported. Independently, recompute accuracy, precision, recall, F1, and AUC from the TP/FP/FN/TN values in Fig 7 for each ratio and check them against Table 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PSM-based balancing improved EEG classification accuracy requires that matched groups be comparable on confounders. Section D states that PSM aimed to balance age and sex, but Table 1 shows it did not. At the 10:1 ratio, ACr subjects in UdeA1 are aged 41.86 +/- 2.35 years with 12 F / 2 M, and the single UdeA2 ACr is 43.0 years; all HC groups are aged 27-31 years with much higher male representation (e.g., CHBMP is 13 F / 25 M). Even at the 2:1 ratio, ACr groups are substantially older and more female than HC groups. Because age and sex are known to affect EEG measures and are associated with carrier status, a decision tree could achieve the reported 0.92-0.96 accuracies (Abstract) or 98% accuracy, 100% recall, 99% AUC (5:1, Table 2) by learning demographic differences alone. The paper provides no no-PSM baseline or age/sex-only control model, so the claimed improvement cannot be attributed to EEG biomarkers. In addition, Fig 7 confusion matrices are numerically inconsistent with Table 2: for 2:1, TP=31, FP=1, FN=3, TN=13 gives precision 31/32 = 96.9% and recall 31/34 = 91.2%, not 91% and 97%; for 5:1, TP=32, FP=0, FN=1, TN=5 gives recall 32/33 = 97.0%, not 100%. These discrepancies mean the reported performance metrics are not internally reproducible, further weakening the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a pipeline for EEG-based classification of asymptomatic PSEN1-E280A mutation carriers (ACr) versus healthy non-carriers (HC) by combining multi-site data harmonization (neuroHarmonize), propensity score matching (PSM) at 2:1, 5:1, and 10:1 ratios, feature selection on decision trees, and evaluation with an 80-20 train-test split. The authors report classification accuracies of 91-98% across ratios, with AUC up to 99%, and claim that balancing sample sizes via PSM significantly improves accuracy. The paper includes descriptions of preprocessing, feature extraction, harmonization, and PSM, with results presented in Tables 1-2 and Figures 2-7.","tokens_in":12213,"tokens_out":5212,"duration_ms":49447,"significance":"If the reported results were valid, the paper would offer a practical recipe for EEG-based Alzheimer's risk classification in small, multi-site cohorts, which is a topic of current interest. The authors make a positive effort to use public datasets (CHBMP, SRM) and to share code repositories, which is commendable and helps reproducibility in principle. However, the central claim that PSM improves classification accuracy is undermined by several internal inconsistencies in the reported data and metrics, and by the apparent failure of PSM to balance the demographic covariates that the authors themselves identify as confounders. The significance is therefore contingent on a thorough reanalysis, and the current evidence does not support the headline conclusions.","major_comments":[{"comment":"The PSM procedure aimed to balance age and sex between ACr and HC groups, but Table 1 shows that the matched groups remain strongly imbalanced on both covariates. For example, at the 5:1 ratio, UdeA1 ACr subjects have mean age 39.78±2.85 years and sex ratio 21/9 F/M, whereas UdeA1 HC subjects have mean age 30.45±4.81 years and sex ratio 47/30 F/M. Since age and sex are associated with both carrier status and EEG measures, the reported classification accuracies (Abstract: 0.92-0.96; Table 2: up to 98%) could be largely driven by demographic differences rather than by EEG biomarkers. The paper provides no no-PSM baseline and no age/sex-only control model, so the claimed improvement attributable to PSM is not demonstrated.","section":"Section D and Table 1"},{"comment":"The confusion matrices in Fig 7 are internally inconsistent with the performance metrics in Table 2 and with the stated dataset sizes. For the 2:1 ratio, TP=31, FP=1, FN=3, TN=13 gives precision 96.9% and recall 91.2%, which is the reverse of the reported precision 91% and recall 97%, indicating a labeling swap. For the 5:1 ratio, TP=32 and FN=1 imply 33 ACr subjects in the test set, but the 5:1 ACr group has only 31 subjects in total (Fig 2), and a 20% test split would contain roughly 6 ACr subjects. The 10:1 matrix is similarly impossible, with TP+FN=34 while the 10:1 ACr group contains only 15 subjects. These discrepancies make the reported performance metrics irreproducible from the information given.","section":"Section E, Fig 7, and Table 2"},{"comment":"The subject counts in Table 1 are inconsistent with the original cohort sizes described in the Methods. The Methods state that UdeA1 contains 27 ACr and 17 HC, and UdeA2 contains 22 ACr and 12 HC, yet Table 1 for the 2:1 ratio reports UdeA1 ACr count 68 and UdeA1 HC count 77, and UdeA2 ACr count 11. PSM can only remove or match existing subjects; it cannot increase the number of unique subjects from a single site. This suggests a serious data-handling error or a mislabeling of the table, and it means the true sample sizes underlying all subsequent analyses are unclear.","section":"Section A and Table 1"},{"comment":"The feature selection procedure uses an accuracy threshold T, but the value of T is not reported anywhere, and no sensitivity analysis is provided for this threshold. The step 'Assign weights to selected features' is not specified, and the decision tree hyperparameters (depth, splitting criterion, minimum samples) are not given. Together with the arbitrary choice of PSM ratios, these unspecified free parameters leave the reported accuracies without a clear bias-variance assessment and hinder replication.","section":"Section E, pseudocode"}],"minor_comments":[{"comment":"The Abstract reports accuracy ranging from 0.92 to 0.96, whereas Table 2 and the Results section report accuracies of 91%, 98%, and 96% for the 2:1, 5:1, and 10:1 ratios; these ranges should be reconciled.","section":"Abstract and Results"},{"comment":"The caption refers to '457 age- and sex-matched records,' but the three resulting datasets contain 237, 189, and 173 records, which sum to 599; it is unclear what the number 457 represents.","section":"Fig 2 caption"},{"comment":"The sentence 'The information used consists of 237 to 173 records' is grammatically unclear and should be rewritten to list the three dataset sizes explicitly.","section":"Section A"},{"comment":"The phrase 'computer precision' in the text and in the Table 2 title is unclear; the authors likely mean 'classification performance metrics.'","section":"Section E and Table 2"},{"comment":"References [19] and [20] cite Python and SciPy documentation for entropy and coherence, respectively; the authors should cite established methodological sources for these signal-processing measures.","section":"References"},{"comment":"The description of cross-validation is ambiguous: the text mentions an 80-20 train-test split and also 'ten-fold iterations,' but it is not clear whether the 10-fold cross-validation is applied only on the training portion or on the full dataset; this should be clarified.","section":"Section E"}],"recommendation":"reject","confidential_remarks":"The paper has multiple internal inconsistencies in subject counts and performance metrics that go beyond presentational issues and would require a complete reanalysis of the data. The failure of PSM to balance covariates and the absence of a demographic-only control model are particularly concerning, as they affect the core conclusion. The open code and use of public data are positive features, but they do not compensate for the inability to reproduce the reported results from the information provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is not ready for peer review. The workflow it describes is plausible and addresses a real bottleneck—pooling small EEG cohorts for Alzheimer's risk classification—but the reported results are internally inconsistent, and the central claim about PSM improving accuracy is not backed by the evidence they present.\n\nWhat's genuinely here: the authors assembled known tools (PREP/wavelet-ICA, gICA components, ComBat harmonization, Rosenbaum-Rubin PSM) into a single pipeline and applied it to a new dataset combination. The pipeline description is clear, sources are cited, and they posted code and links to public data. The Cohen's d feature analysis is a nice addition. But the novelty is thin—essentially three PSM ratios on a new pooling—and the scientific content is mostly derivative.\n\nThe soft spots are load-bearing. First, PSM was supposed to balance age and sex, but Table 1 shows it didn't: at the 10:1 ratio, ACr subjects are about 42 years old and mostly female, while HC groups are in their late 20s or early 30s and mostly male. With age and sex strongly associated with carrier status, a decision tree could be learning demographics, not brain signals. No no-PSM baseline or age/sex-only control is given, so the claimed accuracy gains are uninterpretable. Second, the confusion matrices are numerically impossible. With a 20% test split, the 5:1 matrix implies 33 ACr test records when the entire ACr group has 31; the 10:1 matrix implies 34 ACr test records when ACr totals 15. The 2:1 matrix also far exceeds the expected ACr test count. The metrics in Table 2 don't match the confusion matrices either (e.g., recall 97% vs. the computed 91.2% for 2:1). These aren't cosmetic typos—they indicate the numbers were not produced by the stated procedure. Third, the feature selection threshold T is never reported, so we can't judge how much test performance was tuned.\n\nI don't think this is deliberate; it reads like honest but careless work. But the internal contradictions make it incoherent on its own terms. I'd desk reject as is, or require major reanalysis before sending to referees: re-run PSM and show covariate balance, add a no-PSM control, fix the confusion matrices, and report the feature selection threshold. If the results survive that, there's a modest methodological contribution here.","headline":"Workflow is plausible but the reporting is internally contradictory—the confusion matrices cannot come from the stated split, and PSM didn't balance demographics, so the headline accuracies aren't supported.","tokens_in":12748,"tokens_out":3169,"would_cite":false,"duration_ms":33078,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining multi-site EEG harmonization with matching on age and sex lets a simple decision tree separate asymptomatic PSEN1-E280A Alzheimer's carriers from non-carriers with up to 98% accuracy, 100% recall, and 99%…","keywords":["Alzheimer's disease","EEG biomarkers","propensity score matching","data harmonization","PSEN1-E280A","decision tree classification","multi-site EEG","class imbalance"],"falsifier":"Hold the same matched train/test splits fixed, replace the EEG features with age and sex only, and rerun the decision-tree pipeline; if this demographic-only model reproduces the reported ~98% accuracy and 0.99 AUC, the EEG features are not the source of the classification.","tokens_in":11615,"feed_emoji":"🧠","tokens_out":12684,"duration_ms":116459,"temperature":0.7,"pith_summary":"This paper aims to show that small, imbalanced EEG datasets can still support reliable Alzheimer's-risk classification if the data are pooled across sites, harmonized, and then rebalanced. The authors combine four EEG databases, extract features from reproducible group independent components, correct for recording-site differences with a statistical harmonization step that preserves age and sex, and use propensity-score matching to make healthy-control groups two, five, or ten times larger than the carrier group. They report that this balancing substantially improves decision-tree classifiers, with the 5:1 ratio reaching 98% accuracy, 100% recall, and 99% AUC on held-out test data. The result matters because it suggests an EEG-based path to early Alzheimer's-risk identification in populations where mutation carriers are scarce.","feed_headline":"Age-sex matching lifts EEG Alzheimer's-risk accuracy to 98 percent","feed_subtitle":"Balanced cohorts and cross-site harmonization let simple decision trees flag at-risk carriers with 98% accuracy.","key_machinery":"The central object is propensity-score matching, a procedure that uses logistic regression on age and sex to estimate each subject's probability of carrying the mutation, then removes non-carrier subjects until the control group is two, five, or ten times the carrier group's size. This matching step is what creates the three balanced datasets whose performance the paper compares, and the paper's argument is that balance, not raw sample size, drives the accuracy. Supporting the workflow are a harmonization step that removes recording-site effects while preserving age and sex covariates, and group independent component analysis, which supplies reproducible spatial filters from which 967 candidate features are extracted before the decision tree selects the most informative ones.","core_discovery":"The paper's central claim is that after harmonization, the key obstacle to EEG-based discrimination between asymptomatic E280A carriers and healthy non-carriers is the imbalance between an abundant control group and a scarce carrier group, and that propensity-score matching on age and sex removes that obstacle. In the 5:1 matched sample, a decision-tree model reaches 98% accuracy, 97% precision, 100% recall, and an AUC of 99% on held-out test data; the 2:1 and 10:1 samples reach 91% and 96% accuracy. The authors interpret the 5:1 result as the best trade-off between enough control data and demographic comparability, so the classifier can learn brain-signal differences rather than site or sample-size artifacts. The workflow—standardized preprocessing, harmonization, matching, and decision-tree feature selection—is presented as the reason the model works in small cohorts.","pith_inferences":["The authors themselves flag representativeness and interpretability limits in their final sections; one concrete failure mode is that their Table 1 shows the matched carrier group is roughly a decade older than the controls, so demographic separation may be doing part of the classification work.","A direct extension of the paper's logic is to rerun the same matched splits with a classifier trained only on age and sex; if accuracy stays near 98%, the EEG features are not the source of the signal.","The optimal 5:1 ratio is an empirical property of this dataset, not a fixed law; other cohorts with different age structures or control availability would need to re-estimate the matching ratios.","Phase-scrambling the EEG before feature extraction would test whether the classification relies on oscillatory brain activity or on non-oscillatory artifacts and demographics; the paper does not report this control."],"forward_implications":["If the 5:1 result holds, resting-state EEG could identify asymptomatic PSEN1-E280A carriers with near-perfect recall in small, single-site cohorts.","The harmonize-then-match pipeline offers a template for other neurodegenerative conditions where one diagnostic group is far rarer than the other.","Because the 2:1 and 10:1 ratios also exceed 90% accuracy, the method does not depend on one specific class ratio, although 5:1 is reported as the best.","Multi-site pooling with harmonization could substitute for collecting one large homogeneous dataset, lowering the cost and time needed for EEG biomarker studies.","The reported AUC of 99% at the 5:1 ratio is the paper's evidence that the balancing step produces a clinically meaningful improvement over earlier resting-state EEG classifiers in this familial-AD setting."],"supporting_citations":[{"why":"Supplies the propensity-score matching method used to balance mutation carriers and non-carriers on age and sex.","marker":"[15]"},{"why":"Defines the standardized EEG preprocessing pipeline applied to all recordings before feature extraction.","marker":"[16]"},{"why":"Provides the reproducible group independent components used as spatial filters and feature sources.","marker":"[17]"},{"why":"Supplies the harmonization method used to remove site effects while preserving age and sex covariates.","marker":"[23]"},{"why":"Provides the batch-effect correction procedure that underlies the site-effect harmonization.","marker":"[24]"},{"why":"Validates the preprocessing pipeline the authors say they implement for EEG feature extraction.","marker":"[25]"},{"why":"Gives the prior resting-state EEG classification accuracy (83%) that this study's improved accuracy is compared against.","marker":"[32]"}],"fun_headline_variants":["Age-sex matching lifts EEG Alzheimer's-risk accuracy to 98%","EEG harmonization plus propensity matching: 98% Alzheimer's risk accuracy","Propensity score matching turns small EEG samples into 98% Alzheimer's accuracy","Balanced EEG cohorts: decision trees hit 98% accuracy on Alzheimer's risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the matching procedure actually makes the carrier and non-carrier groups comparable in age and sex; if the matched groups still differ sharply in age, the high EEG accuracy may just reflect that age difference rather than a brain-based Alzheimer's signal.","fun_headline_variants_meta":{"raw":{"variants":["Age-sex matching lifts EEG Alzheimer's-risk accuracy to 98%","EEG harmonization plus propensity matching: 98% Alzheimer's risk accuracy","Propensity score matching turns small EEG samples into 98% Alzheimer's accuracy","Balanced EEG cohorts: decision trees hit 98% accuracy on Alzheimer's risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1510,"prompt_tokens":960,"completion_tokens":550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":468}},"tokens_in":576,"tokens_out":550,"duration_ms":5144,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:45:03.000224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold the same matched train/test splits fixed, replace the EEG features with age and sex only, and rerun the decision-tree pipeline; if this demographic-only model reproduces the reported ~98% accuracy and 0.99 AUC, the EEG features are not the source of the classification.","supporting_citations":[{"cited_title":"Validation of EEG Pre- processing Pipeline by Test-Retest Reliability,","cited_arxiv_id":null,"evidence_quote":"Defines the standardized EEG preprocessing pipeline applied to all recordings before feature extraction."},{"cited_title":"Reproducible Neuronal Components found using Group Independent Component Analysis in Resting State Electroencephalographic Data,","cited_arxiv_id":null,"evidence_quote":"Provides the reproducible group independent components used as spatial filters and feature sources."},{"cited_title":"Harmonization of large MRI datasets for the analysis of brain imaging patterns throughout the lifespan,","cited_arxiv_id":null,"evidence_quote":"Supplies the harmonization method used to remove site effects while preserving age and sex covariates."},{"cited_title":"Tackling EEG Test-Retest Reliability with a Pre-Processing Pipeline Based on ICA and Wavelet-ICA","cited_arxiv_id":null,"evidence_quote":"Validates the preprocessing pipeline the authors say they implement for EEG feature extraction."},{"cited_title":"Automatic Classification of Subjects of the PSEN1-E280A Family at Risk of Developing Alzheimer’s Disease Using Machine Learning and Resting State Electroencephalography,","cited_arxiv_id":null,"evidence_quote":"Gives the prior resting-state EEG classification accuracy (83%) that this study's improved accuracy is compared against."}],"review_version":1}