{"id":"f5a719dc-4d9d-4f89-bf98-ef83dce569bd","arxiv_id":"2607.29493","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On 38 participants, LASSO-selected EEG features plus gaze variability gave AUC 0.78 for MCI vs. control, but feature selection leaked into the evaluation, making the gain an unverified upper bound.","lead":"This pilot study combined EEG brain-wave features with eye-gaze stability measures to distinguish people with mild cognitive impairment from healthy older adults, reporting that the combination (AUC 0.78) beats EEG alone. The numbers are self-admittedly upper-bound estimates because the feature choice saw all the data, so the result is a hypothesis to test properly, not a measured effect.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feature-selection leakage makes the reported LOOCV AUC gains—especially the 0.64→0.78 gaze-integration gain—untrustworthy as out-of-sample estimates; the paper discloses this, but the central complementarity claim is not yet established.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: feature selection was performed on the full dataset before LOOCV, making the reported AUCs preliminary upper bounds. The manuscript includes a clear disclosure in §3.4, which is credit to the authors, but the disclosure does not remove the threat to the central quantitative claim. The central claim is not that the method is useless or that the hypothesis is false; it is that integrating gaze variability improves discrimination. That claim rests on the 0.64→0.78 step, which is precisely the step most exposed to selection leakage because the two EEG features were chosen with full knowledge of all participant outcomes. The paper is exploratory and honest, but the statistical evidence for the incremental gaze benefit is not clean. I therefore agree with the reader's CONDITIONAL verdict: the complementarity hypothesis is plausible and worth testing, but the reported AUCs cannot be taken as reliable out-of-sample performance estimates. The concrete test—nested cross-validation with feature selection inside folds—would settle whether the gaze-integration gain survives when selection bias is removed. If the nested gain persists, the central claim gains real support; if not, the paper's conclusion should be weakened to a hypothesis-generating finding. The secondary issue of the unstated x-axis gaze feature choice is real but less central, as it affects the gaze feature pool rather than the leakage mechanism. No ad hominem is intended; the authors' explicit caveat strengthens the paper's credibility even while it underscores the limitation.","tokens_in":3902,"tokens_out":1851,"duration_ms":22944,"concrete_test":"Re-run the full pipeline with feature selection nested inside each LOOCV fold: in each fold, run LASSO on the 36 training participants, take the top two EEG features, then evaluate the trained logistic regression on the left-out participant. Compare this nested-LOOCV AUC for model (b) and model (c). If the nested AUC gain from adding R_std_x/L_std_x is small (e.g., <0.05) or confidence intervals overlap substantially, the reported 0.64→0.78 improvement is not established. Additionally, run a permutation test on the integrated model's AUC difference versus gaze-only to check whether the gain exceeds chance under label permutation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that adding gaze variability to LASSO-selected EEG features improves MCI discrimination from AUC 0.64 to 0.78—depends on the validity of the LOOCV AUCs. But the two EEG features (delta at O1, low-beta at T6) were selected by LASSO using the full dataset before any cross-validation split. During each LOOCV fold, the left-out participant's label was therefore available at feature-selection time. This creates a selection-leakage/selction-bias path: the features are chosen to fit the entire sample, so the subsequent LOOCV evaluation of those fixed features partially measures the fit of the feature selection to the test participant, not purely out-of-sample generalization. The paper's own §3.4 states this directly: 'Because feature selection and performance evaluation were not fully separated, the reported AUC values should be interpreted as preliminary upper-bound estimates.' That disclosure is honest but does not repair the statistic. The stepwise comparison (0.52 → 0.64 → 0.78) may be inflated in unknown magnitude, so the incremental gain attributed to gaze integration is not quantitatively established. A secondary fragility is the unstated choice of x-axis gaze standard deviations (R_std_x, L_std_x) in §3.4 when §3.3.2 computed both x and y; this feature-selection choice is also not cross-validated. These issues do not necessarily invalidate the qualitative complementarity hypothesis, but they make the headline AUC numbers unreliable as evidence for it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an exploratory comparison of three classification models for discriminating mild cognitive impairment (MCI) from healthy aging using EEG spectral power and gaze variability during a fixation task. Model (a) uses all EEG spectral-band features; model (b) uses two EEG features selected by LASSO (delta power at O1, low-beta power at T6); model (c) adds two gaze variability features (R_std_x, L_std_x). Classification is logistic regression with balanced class weights, evaluated by leave-one-out cross-validation (LOOCV) and AUC. Reported AUCs are 0.52, 0.64, and 0.78, respectively, and the authors interpret the stepwise increase, especially 0.64→0.78, as evidence that neural and behavioral variability measures carry complementary information.","tokens_in":4170,"tokens_out":2022,"duration_ms":27629,"significance":"If the reported effect is real, the combination of simple EEG spectral features and gaze variability could offer a low-cost, objective digital biomarker for early MCI screening. The idea that neural and behavioral variability during fixation tap complementary processes is plausible and clinically relevant. However, the central quantitative claim is not established by the current analysis because the headline AUCs are computed under feature-selection leakage, and the paper explicitly labels them as upper-bound estimates. The manuscript is honest about this limitation, but the admission does not repair the statistic. A properly nested cross-validation or feature-selection-within-training-folds design, plus uncertainty quantification, would be needed before the complementarity claim can be assessed.","major_comments":[{"comment":"Feature-selection leakage invalidates the reported LOOCV AUCs. LASSO was applied to the full dataset to select the two EEG features before any cross-validation split, so the label of the left-out participant was visible at feature-selection time. The LOOCV evaluation of the fixed two-feature model therefore partially measures the fit of feature selection to the test participant, not out-of-sample generalization. The paper's own §3.4 acknowledges this ('preliminary upper-bound estimates'), but the magnitude of the bias is unknown, so the stepwise comparison (0.52→0.64→0.78) is not a reliable estimate of discrimination. The authors should rerun the analysis with feature selection nested inside each training fold (or use a fully pre-specified feature set) and report AUC with confidence intervals.","section":"§3.4 and §3.3.1"},{"comment":"The choice of gaze features is not justified and is not cross-validated. §3.3.2 computes standard deviations in both x and y directions for each eye, but §3.4 selects only R_std_x and L_std_x for model (c). No rationale is given for discarding the y-direction features, and there is no indication whether this choice was made before seeing the outcome. If the gaze-feature subset was chosen after inspecting full-data results, the same leakage concern applies. At minimum, the feature set should be pre-specified in the methods, and the selection rule should be applied within cross-validation folds.","section":"§3.4 and §3.3.2"},{"comment":"No statistical inference accompanies the AUC comparisons. With only 38 participants, the difference between AUC = 0.64 and AUC = 0.78 may be within sampling variability, especially given the small number of MCI cases (n = 17). The authors should report confidence intervals for each AUC and a test comparing models (e.g., DeLong's test or bootstrap). Without these, the phrase 'suggesting a substantial improvement in classification performance' is unsupported.","section":"§4"}],"minor_comments":[{"comment":"The paper states LASSO identified the 'two most contributive features' but does not specify how the regularization strength was chosen. This matters because the number and identity of selected features depends on the path of λ; please report the criterion (e.g., cross-validated λ, fixed λ) and the coefficients.","section":"§3.3.1"},{"comment":"A gaze-only model is mentioned with AUC = 0.55, but this model is not described in the Methods. Its feature set, evaluation procedure, and whether it was subject to the same leakage issue should be specified if it is included in the stepwise comparison.","section":"§4"},{"comment":"The choice to analyze only the first 4 seconds of the 20-second trial is not justified. If this was an a priori decision, it should be stated; if it was data-driven, the implications for the reported AUCs should be discussed.","section":"§3.2"},{"comment":"Re-referencing by subtracting regional reference electrodes for different topographies is described briefly; please clarify whether this is a standard procedure or an ad hoc choice, and whether it affects comparability across channels.","section":"§3.3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its main methodological weakness, which is commendable. However, the central claim — that adding gaze variability improves discrimination beyond EEG features — rests on AUCs computed under feature-selection leakage. This is fixable in principle by redoing the evaluation with nested cross-validation or a pre-specified feature set, and by adding confidence intervals. Given the exploratory nature and short format, I would not reject outright, but the revised manuscript must demonstrate that the reported gains survive a leakage-free evaluation. The lack of any statistical comparison between models is an additional concern that should be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a 3-page ETRA pilot combining EEG spectral features with gaze variability to discriminate MCI from healthy controls (n=38). The specific combination is new relative to the papers it cites, and the writing is clear and appropriately modest. The catch is that the headline AUC sequence—0.52, 0.64, 0.78—is computed with feature-selection leakage. The authors ran LASSO on the full dataset to pick two EEG features, then did LOOCV on those fixed features, so the left-out participant's label was visible during feature selection. They disclose this in §3.4, calling the AUCs 'preliminary upper-bound estimates.' Credit for honesty, but it doesn't repair the statistic: the incremental gain attributed to gaze integration (0.64→0.78) is not a clean out-of-sample result.\n\nWhat's genuinely good: the empirical setup is new as far as the cited references show, and the interpretive frame—neural variability plus gaze variability may tap complementary processes—is plausible. The task is simple (4 seconds of fixation from a 20-second trial), so the approach could translate to a low-cost screening tool if the effect is real. The authors also keep their claims in check; the discussion lists small sample size, lack of external validation, and restricted gaze features as limitations.\n\nThe soft spot is the one above, and it's load-bearing. The solution is standard: perform feature selection inside each CV fold (or use a nested CV), and ideally validate on a separate cohort. A secondary, minor issue is that the gaze features R_std_x/L_std_x are chosen without explanation, while §3.3.2 computes both x and y standard deviations; that choice is also not cross-validated. The paper also omits some details needed for reproduction (LASSO penalty, exact band boundaries, channel count), which matters more in a 3-page format.\n\nWho should read this? Anyone working on digital biomarkers for MCI, mainly as a prompt to think about multimodal integration and about how easily CV can leak. It is not a reliable source for effect sizes.\n\nMy recommendation: send it to peer review with a specific request for a nested-CV re-analysis. The qualitative hypothesis is worth testing, the authors have been transparent, and a re-analysis could turn it into a solid pilot. As is, the numbers should be treated as preliminary upper bounds—which is exactly what the authors say.","headline":"Genuinely exploratory multimodal pilot whose headline AUCs are undermined by feature-selection leakage—the authors disclose it, but the gaze-complementarity claim is not yet established.","tokens_in":4824,"tokens_out":2676,"would_cite":false,"duration_ms":29481,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining a small set of EEG spectral features with gaze variability substantially improves discrimination of mild cognitive impairment, raising AUC from 0.64 to 0.78 in a preliminary study.","keywords":["mild cognitive impairment","EEG spectral features","gaze variability","eye tracking","LASSO","ROC AUC","fixation stability","multimodal biomarkers"],"falsifier":"Re-run the analysis with nested cross-validation or a completely held-out test set, performing LASSO feature selection only on the training folds each time. If the AUC for EEG-plus-gaze drops to near 0.64 or lower, the claim that gaze variability adds complementary information would be falsified.","tokens_in":3671,"feed_emoji":"🧠","tokens_out":2749,"duration_ms":30431,"temperature":0.7,"pith_summary":"The paper tries to establish that combining EEG spectral power features with gaze variability during a fixation task improves discrimination between mild cognitive impairment (MCI) and healthy aging. In a stepwise comparison on 38 participants, all EEG features alone gave an AUC of 0.52, two LASSO-selected EEG features gave 0.64, and adding two gaze variability features raised the AUC to 0.78. The authors interpret this jump as evidence that neural and behavioral variability measures carry complementary information. However, because feature selection was performed on the full dataset before cross-validation, the authors themselves caution that the reported AUCs are preliminary upper-bound estimates.","feed_headline":"Adding gaze variability lifts MCI discrimination AUC to 0.78","feed_subtitle":"EEG alone scores 0.52; adding two spectral features and gaze stability reaches 0.78.","key_machinery":"The central machinery is a stepwise classification pipeline: (1) high-dimensional EEG spectral power features across six frequency bands and all 10-20 electrodes; (2) L1-regularized logistic regression (LASSO) applied to the full dataset to select the two most contributive EEG features; (3) a logistic regression classifier with balanced class weights, trained under leave-one-out cross-validation; and (4) aggregation of held-out probabilities to compute the ROC AUC. The gaze features are the standard deviations of gaze position in the x-direction for the right and left eyes (R_std_x, L_std_x), computed after excluding saccade intervals, serving as fixation-stability measures.","core_discovery":"On the paper's own terms, the central claim is that dimensionality reduction of EEG features followed by integration of gaze variability yields a substantial gain in MCI discrimination. Specifically, delta-band power at electrode O1 and low-beta power at T6, selected via LASSO and concatenated with the standard deviations of the x-coordinate of gaze for the right and left eyes, produced a leave-one-out cross-validated AUC of 0.78, compared to 0.64 for the EEG features alone and 0.52 for the full EEG feature set. The authors suggest that the occipital delta and temporal low-beta rhythms reflect visual processing and attentional control, while gaze instability reflects a related but distinct b","pith_inferences":["If the gaze gain survives proper nested cross-validation, the complementarity between neural and behavioral variability might generalize to other cognitive assessments (e.g., memory or attention tasks), not just fixation.","The paper reports only x-axis gaze standard deviations in the final model even though both x and y were computed; y-axis variability might add further discrimination or substitute when x is unavailable.","The unseparated feature selection makes the 0.64-to-0.78 jump uncertain; a re-analysis with selection performed inside each cross-validation fold is the direct next test.","The proposed mechanism — that neural and gaze variability reflect different aspects of a common visual-attention process — is testable by correlating the two feature sets with independent cognitive scores."],"forward_implications":["If the result holds, multimodal indicators combining neural and behavioral variability could improve non-invasive MCI screening over either modality alone.","The specific selected features (occipital delta, temporal low-beta, gaze x-variability) point to concrete physiological targets for future validation.","The study suggests that even minimal gaze features, captured during a simple fixation task, can add value to EEG-based classification.","The stepwise improvement from 0.52 to 0.64 to 0.78 implies that feature selection and multimodal integration are both beneficial, not just one.","The approach is portable: the same pipeline could be applied to other cognitive tasks or larger cohorts to test generalizability."],"fun_headline_variants":["EEG plus gaze variability nudges MCI detection AUC from 0.52 to 0.78","Gaze variability, not more EEG features, boosts MCI discrimination to AUC 0.78","Selected EEG bands plus gaze variability push MCI discrimination AUC to 0.78","Combining two EEG features with gaze variability reaches 0.78 AUC for MCI","EEG feature selection plus gaze variability lifts MCI AUC to 0.78"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the leave-one-out cross-validated AUCs are valid out-of-sample estimates even though the two EEG features were selected using the full dataset before cross-validation; the paper itself states these should be interpreted as preliminary upper-bound estimates.","fun_headline_variants_meta":{"raw":{"variants":["EEG plus gaze variability nudges MCI detection AUC from 0.52 to 0.78","Gaze variability, not more EEG features, boosts MCI discrimination to AUC 0.78","Selected EEG bands plus gaze variability push MCI discrimination AUC to 0.78","Combining two EEG features with gaze variability reaches 0.78 AUC for MCI","EEG feature selection plus gaze variability lifts MCI AUC to 0.78"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001467,"raw_usage":{"total_tokens":5721,"prompt_tokens":709,"completion_tokens":5012,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":4897}},"tokens_in":453,"tokens_out":5012,"duration_ms":32633,"temperature":1.0,"reasoning_tokens":4897,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:54:46.722905+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the analysis with nested cross-validation or a completely held-out test set, performing LASSO feature selection only on the training folds each time. If the AUC for EEG-plus-gaze drops to near 0.64 or lower, the claim that gaze variability adds complementary information would be falsified.","supporting_citations":[],"review_version":1}