{"id":"88bf7da9-3c88-42d7-b6a2-16425f5bc3aa","arxiv_id":"1908.06319","paper_version":9,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using spatial Locally Linear Embedding with feature selection and linear classification is reported to distinguish schizophrenia, ADHD, and autism from controls above chance in small public fMRI datasets, although validation gaps remain.","lead":"One researcher reports that a nonlinear method for compressing brain scan data, called Locally Linear Embedding, can separate psychiatric patients from healthy controls in public fMRI datasets. The headline numbers are partly based on training-data accuracy, so the result needs independent validation before being used.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The >80% headline rests on training LOOCV accuracy after the same LOOCV loop selected d and diagnostic time points; independent holdout accuracies in Table I are 75%, 100%, 68.8%, 70%, and 80%, so three of five fall below 80% and no significance test supports 'better than chance'.","rationale":"The reader's weakest assumption is the same as mine: the abstract's >80% and 'better than chance' claims are anchored to training LOOCV accuracies that have been used for feature and hyperparameter selection, so they are optimistically biased. I confirm this from Sections II-A and II-C, including the paper's explicit but incorrect statement that the selection 'does not induce any biased sampling.' I also correct the reader's count: in the five main-table holdout rows, LLE accuracies are 75%, 100%, 68.8%, 70%, and 80%, so three, not two, fall below 80%. The absence of any significance test is material because small holdout samples cannot support 'significantly' better-than-chance: for example, 8 correct out of 10 gives a two-sided exact binomial p of about 0.11 against 50% chance, and against a majority-class baseline the comparison is weaker. The physical/conservation framing and the lack of code/data are secondary. I keep the reader's CONDITIONAL verdict: the method is specified well enough to re-evaluate, and some holdout numbers (COBRE 20/20, UM_2 8/10) are suggestive, but the current abstract claim is not supported. A revision that reports only nested or holdout accuracies with binomial tests and releases code and data would settle the concern.","tokens_in":17164,"tokens_out":9958,"duration_ms":90051,"concrete_test":"Run a nested LOOCV on each of the eleven datasets: the outer loop leaves out one subject; the inner loop selects d (log-scale 1..T) and SFS diagnostic time points using only the remaining training subjects; score only the outer held-out subject. Report outer accuracy per dataset with exact binomial 95% confidence intervals, a one-sided exact binomial test of accuracy greater than the majority-class proportion, and a McNemar test of LLE versus PCA and versus original fMRI on the same outer folds. Then count how many datasets have outer accuracy above 80% and a significant better-than-chance test. If that count is below ten of eleven (or below three of five in the main table), the abstract's '>80% on eleven datasets' and 'significantly better than chance' claims are not supported by unbiased estimates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline quantitative claim is the training LOOCV accuracy after the same LOOCV loop has selected both the embedding dimension d (searched on a log-scale from 1 to T, Section II-A) and a greedy set of diagnostic time points via Sequential Forward Selection (Section II-A). Section II-C states this explicitly: 'LOOCV is used on the training data to find both the best reconstruction parameter d, and the diagnostic volumes that produce the highest accuracy for this d.' The resulting 'training accuracy' is the maximum of an optimization over many model choices, not an unbiased estimate of generalization. The paper's assertion in Section II-A that this selection 'does not induce any biased sampling' is incorrect: selecting models by their LOOCV score on the same folds inflates the reported accuracy. The only unbiased numbers in Table I are the holdout rows. For the five datasets shown there, LLE holdout accuracy is 75% (Beijing), 100% (COBRE), 68.8% (MRN), 70% (Stanford), and 80% (UM_2): three of five fall below the advertised 80%. With holdout sizes of 10 to 20 subjects, these values are also too imprecise to support the claim of being 'significantly better than chance'; no binomial test, confidence interval, or McNemar comparison is reported. The central claim therefore rests on a biased training metric rather than on the independent holdout evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Locally Linear Embedding (LLE) as a nonlinear preprocessing step for fMRI-based psychiatric classification. The pipeline reconstructs each subject's BOLD time-series with LLE, selects the embedding dimension d and a set of diagnostic time points using LOOCV on the training partition together with sequential forward selection, and classifies with Fisher's linear discriminant. The authors report accuracies above 80% on eleven public datasets and claim significantly better-than-chance performance on ten of eleven datasets. The main text contains detailed results for five datasets, with six additional datasets deferred to a supplementary table.","tokens_in":17523,"tokens_out":11042,"duration_ms":99051,"significance":"If the advertised performance were supported, LLE would be a useful feature-extraction step for psychiatric fMRI classification, and the use of several public datasets is a strength. The paper makes falsifiable predictions and describes the pipeline in enough detail to be reimplemented. However, the central quantitative claim currently rests on training accuracies that are optimistically biased by model selection, and the independent holdout results are too few and too imprecise to support the abstract's assertions. The physical-conservation interpretation is not load-bearing for the classification claim and is not derived from the LLE equations.","major_comments":[{"comment":"The training accuracies reported in Table I are not unbiased estimates of generalization. Section II-C states that 'LOOCV is used on the training data to find both the best reconstruction parameter d, and the diagnostic volumes that produce the highest accuracy for this d.' The same LOOCV accuracy is then reported as the method's performance. Because d and the diagnostic time points are selected by maximizing this same score, the reported training accuracy is optimistically biased. The statement in Section II-A that the selection 'does not induce any biased sampling' is therefore incorrect. The authors should either use a nested cross-validation scheme, where model selection is performed inside each training fold, or base all primary claims on the independent holdout partition.","section":"II-C and Table I"},{"comment":"The headline '>80%' claim is contradicted by the independent holdout results in Table I: the LLE holdout accuracies are 75% (Beijing), 100% (COBRE), 68.8% (MRN), 70% (Stanford), and 80% (UM_2), so three of five fall below 80%. No confidence intervals, binomial tests, or comparisons against the majority-class baseline are provided for these holdout values, and the assertion of 'significantly better than chance classification' on ten of eleven datasets is therefore unsupported. The authors should report holdout accuracies with exact binomial confidence intervals and a test against the majority proportion (and, if relevant, against the PCA and original-data baselines), and revise the abstract to match the holdout evidence.","section":"Abstract and Table I"},{"comment":"The statement that 'discrimination performance on the holdout partition is determined in a one-time fashion, variance estimates are not applicable' is incorrect. A single holdout accuracy is a binomial proportion and has a well-defined sampling distribution. With holdout sizes of 10 to 20 subjects in Table I, the point estimates are highly variable; for example, 75% on 12 subjects has a 95% confidence interval of approximately 47% to 91%. The absence of holdout error bars is a major omission.","section":"Section III (Evaluation Criteria)"},{"comment":"The manuscript claims results on eleven datasets, but the main text reports results for only five, with the remaining six deferred to a supplementary table that is not included. The 'eleven datasets' claim cannot be verified from the manuscript. All dataset-level results and the supplementary tables and figures should be included, or the claim should be restricted to the datasets actually reported.","section":"Tables I and II"},{"comment":"For all five main-text datasets, the selected embedding dimension d equals the number of time points T (Beijing d=236, COBRE d=150, MRN d=177, Stanford d=60, UM_2 d=222). Thus the reported experiments never actually reduce the temporal dimension. This is inconsistent with the stated goal of 'nonlinear dimensionality reduction' and with the description in Section II of reducing a time-series of length T to a smaller number of modes. The authors should explain why d=T is selected and how this affects the interpretation of the method.","section":"Table II"}],"minor_comments":[{"comment":"The notation for the Modified LLE multiple weight vectors is difficult to follow; in particular, the definition of the local alignment matrix is not clear about how the multiple weight vectors are aggregated into Phi. Please clarify.","section":"II-B, Eqs. (2)-(4)"},{"comment":"The sentence 'Tables I and II demonstrate...' is misleading because Table II is a dataset summary, not a results table; the results sentence should refer only to Table I.","section":"Section III"},{"comment":"The procedure for constructing a 'representative sample' training partition is not specified. Please describe how the split was chosen, particularly for the unbalanced datasets.","section":"Section II-C"},{"comment":"The paper repeatedly invokes the Cauchy stress tensor, Gauss' Principle of Least Constraint, analytic capacity, and Theorema Egregium, but no derivation connects these concepts to the LLE equations. These interpretive claims are not needed for the classification pipeline and should either be substantiated with a formal argument or removed from the Methods section.","section":"Methods II-b and Appendix"},{"comment":"The manuscript contains numerous typographical and OCR-like artifacts (e.g., 'Index T erms', 'One hundred and fourty-eight years', and broken math symbols in the equations) and would benefit from a careful proofreading pass.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The selection-bias problem is serious enough that the paper should not be accepted without a reanalysis of the holdout results and appropriate statistical tests. The novelty beyond applying existing LLE software is limited, and the physical-conservation interpretation is idiosyncratic and not load-bearing for the classification claim. The citation list is broad but contains several references that do not appear to be used in the argument; the editor may wish to ask the author to trim irrelevant citations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the spatial-neighborhood variant of LLE for psychiatric fMRI classification is genuinely new and tested on an admirably large set of public datasets, but the headline >80% accuracy is computed on the training set after using the same cross-validation loop to select both the embedding dimension and the diagnostic time points. The independent holdout numbers in Table I are the only unbiased estimates, and they are lower and less consistent.\n\nWhat's new: defining LLE neighborhoods by spherical spatial neighborhoods rather than k-nearest in feature space, and combining that with sequential forward selection and LDA, does not appear in the cited literature, including the earlier LLE-fMRI work. Testing on eleven datasets is commendable in scope, and the author gives a plausible (if speculative) neurobiological reading of the resulting statistical maps.\n\nWhere it falls down: the evaluation is not sound. Section II-C says LOOCV on training data selects d and the diagnostic volumes; the reported training accuracy is therefore an optimistically biased estimate, not a measure of generalization. The paper even claims this selection 'does not induce any biased sampling,' which is incorrect. The holdout accuracies for the five main datasets are 75%, 100%, 68.8%, 70%, and 80%; three of these fall below the advertised threshold, and with 10-20 subjects each, the numbers are too imprecise to support 'significantly better than chance.' No confidence intervals, binomial tests, or McNemar comparisons are reported. The physics framing—LLE as conserving Cauchy stress tensors—is an interpretive overlay on standard LLE; it doesn't add empirical support or a derivation that would change the analysis.\n\nA revision that reports only independent holdout metrics with uncertainty, uses nested cross-validation, and releases code and data would be worth seeing. As posted, the central quantitative claim doesn't hold up. That said, the underlying idea of spatial-neighborhood LLE for fMRI has enough promise that I would send it to a serious referee after major revision, though I would not cite it as evidence of LLE's effectiveness in its current form.","headline":"Novel spatial-neighborhood LLE pipeline for fMRI classification, but the >80% claim rests on training accuracy after selection on the same folds; holdout numbers are weaker.","tokens_in":17989,"tokens_out":3121,"would_cite":false,"duration_ms":31288,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Locally Linear Embedding of fMRI time series yields above-80% accuracy on eleven psychiatric datasets.","keywords":["Locally Linear Embedding","fMRI","dimensionality reduction","psychiatric classification","leave-one-out cross-validation","feature selection","BOLD time-series","neuroimaging"],"falsifier":"Hold out a set of subjects completely untouched by any selection step, choose the embedding dimension $d$ and diagnostic volumes using nested cross-validation on the training set only, then measure LLE accuracy once on the held-out subjects. If the held-out accuracy drops to the majority-class baseline or no longer beats PCA on the same held-out subjects, the central claim fails.","tokens_in":16933,"feed_emoji":"🧠","tokens_out":11072,"duration_ms":103369,"temperature":0.7,"pith_summary":"Locally Linear Embedding (LLE) is proposed as a preprocessing step that turns each voxel's BOLD time-series into a low-dimensional spatial embedding before psychiatric classification. The paper reports leave-one-out cross-validation accuracies above 80% on the training partitions of eleven public fMRI datasets covering schizophrenia, ADHD, and autism, and above-chance classification on ten of the eleven, where chance is the majority-class proportion. Raw fMRI and PCA-reduced fMRI perform markedly worse on the same comparison. The motivation is clinical: if the embedding reliably preserves disorder-relevant brain-activity patterns, it would make fMRI-based diagnosis support more feasible and point to interpretable time points and regions.","feed_headline":"LLE-embedded fMRI clears 80% accuracy on eleven psychiatric datasets","feed_subtitle":"A nonlinear reduction step preserves activity patterns that separate schizophrenia, ADHD, and autism from controls.","key_machinery":"The load-bearing object is the per-voxel local covariance matrix $G_i = C_i^T C_i$, built from the differences between a voxel's BOLD waveform and the waveforms of neighbouring voxels on a sphere; solving $G_i w_i = 1$ through a minimum-norm least-squares step gives reconstruction weights that encode local geometry. Modified LLE adds multiple weight vectors when the local covariance is ill-conditioned. A global alignment matrix $\\Phi = \\sum_i \\hat{W}_i \\hat{W}_i^T$ is assembled from these weights, and the bottom $d+1$ eigenvectors of $\\Phi$ provide the low-dimensional embedding, with the last eigenvector discarded to centre the manifold. This machinery converts raw BOLD time series into low-dimensional spatial modes that the classifier then sees.","core_discovery":"The central claim is that a nonlinear, spatially-aware reconstruction of fMRI time series retains diagnostic information that the raw BOLD signal and its linear PCA projection discard. The pipeline embeds each subject's scan by LLE in a neighbourhood defined by the physical sphere around each voxel, solving the local reconstruction problem $G_i w_i = 1$ for each voxel and then finding global coordinates that preserve those local weights. A linear discriminant classifier, fed only the diagnostic volumes selected by sequential forward search, separates patients from controls. The paper reports this separation is above 80% accurate on all eleven training partitions, and significantly better than the majority-class baseline on ten of eleven, while raw and PCA versions often sit at chance.","pith_inferences":["The paper does not isolate how much of the gain comes from LLE's nonlinear geometry versus the sequential-forward-selection step; rerunning SFS on raw and PCA data with identical classifier settings would quantify that split.","Because the method selects only a few time points, the same pipeline could be tested on event-related task decoding, predicting stimulus condition rather than diagnosis, where sparse temporal selection could reveal when information appears.","The reported regional differences use uncorrected $p<0.05$ thresholds, so those maps are best treated as hypotheses to be confirmed rather than stand-alone evidence.","Comparing the stress-tensor sphere neighbourhood against generic nonlinear embeddings on the same datasets would show whether the spatial neighbourhood is the active ingredient or whether any nonlinear reduction would perform similarly."],"forward_implications":["If the embedding is as informative as reported, nonlinear manifold methods become a viable preprocessing alternative to PCA for fMRI-based diagnosis.","The selected diagnostic volumes map onto disorder-specific regions, giving concrete hypotheses about where and when patient and control brain activity differ.","The method's success on both resting-state and task-based scans suggests the preserved local geometry carries diagnostic signal beyond task-induced activation.","The contrast with PCA implies that simple linear preprocessing may discard the very structure that LLE retains for classification."],"supporting_citations":[{"why":"Supplies the LLE algorithm that reconstructs each point from neighbours and then embeds the data globally.","marker":"[54]"},{"why":"Supplies the theoretical basis for LLE and the local-linearity assumption the method relies on.","marker":"[56]"},{"why":"Supplies the modified LLE multiple-weight construction used when the local covariance is degenerate.","marker":"[65]"},{"why":"Supplies the sequential forward selection procedure that chooses diagnostic time points.","marker":"[63]"},{"why":"Supplies the linear discriminant classification rule applied to the embedded volumes.","marker":"[17]"},{"why":"Supplies the classification and cross-validation framework used to compute accuracy.","marker":"[26]"},{"why":"Provides the BOLD-to-neural coupling evidence that justifies treating fMRI time series as measures of local neuronal activity.","marker":"[39]"},{"why":"Documents an earlier LLE-to-fMRI application that discarded spatial neighbourhood structure, which this method explicitly avoids.","marker":"[44]"},{"why":"Documents challenges in fMRI disease classification and cautions about PCA, motivating the comparison pipeline.","marker":"[16]"}],"fun_headline_variants":["LLE fMRI hits 80% accuracy in psychiatric scans","Nonlinear embedding beats PCA for fMRI psychiatric diagnosis","LLE-embedded fMRI diagnostics top 80% on 11 datasets","Nonlinear fMRI feature selection beats linear PCA in psychiatry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracies come from the same leave-one-out cross-validation loop that chooses the embedding dimension and the diagnostic time points, so the numbers assume that selecting on the training folds does not inflate performance; an untouched test set for those choices is needed before the >80% figure can be taken as an unbiased estimate.","fun_headline_variants_meta":{"raw":{"variants":["LLE fMRI hits 80% accuracy in psychiatric scans","Nonlinear embedding beats PCA for fMRI psychiatric diagnosis","LLE-embedded fMRI diagnostics top 80% on 11 datasets","Nonlinear fMRI feature selection beats linear PCA in psychiatry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2591,"prompt_tokens":957,"completion_tokens":1634,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1569}},"tokens_in":573,"tokens_out":1634,"duration_ms":11905,"temperature":1.0,"reasoning_tokens":1569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:49:50.246877+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a set of subjects completely untouched by any selection step, choose the embedding dimension $d$ and diagnostic volumes using nested cross-validation on the training set only, then measure LLE accuracy once on the held-out subjects. If the held-out accuracy drops to the majority-class baseline or no longer beats PCA on the same held-out subjects, the central claim fails.","supporting_citations":[{"cited_title":"T., /a.sc/n.sc/d.sc S/a.sc/u.sc/l.sc, L","cited_arxiv_id":null,"evidence_quote":"Supplies the LLE algorithm that reconstructs each point from neighbours and then embeds the data globally."},{"cited_title":"K., /a.sc/n.sc/d.sc R/o.sc/w.sc/e.sc/i.sc/s.sc, S","cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical basis for LLE and the local-linearity assumption the method relies on."},{"cited_title":"Computers, IEEE Transactions on C-20 , 9 (sept","cited_arxiv_id":null,"evidence_quote":"Supplies the modified LLE multiple-weight construction used when the local covariance is degenerate."},{"cited_title":"Science 153 , 3736 (1966), 652–654","cited_arxiv_id":null,"evidence_quote":"Supplies the sequential forward selection procedure that chooses diagnostic time points."},{"cited_title":"P ., M/a.sc/g.sc/n.sc/o.sc/t.sc/t.sc/a.sc, V","cited_arxiv_id":null,"evidence_quote":"Supplies the linear discriminant classification rule applied to the embedded volumes."},{"cited_title":"Patavii: Typis heredum Pauli Frambotti, Italy, 1668","cited_arxiv_id":null,"evidence_quote":"Supplies the classification and cross-validation framework used to compute accuracy."},{"cited_title":"F., F/r.sc/i.sc/s.sc/t.sc/o.sc/n.sc, K","cited_arxiv_id":null,"evidence_quote":"Provides the BOLD-to-neural coupling evidence that justifies treating fMRI time series as measures of local neuronal activity."},{"cited_title":"Dimensionality reduction of fMRI time series data using locally linear embedding","cited_arxiv_id":null,"evidence_quote":"Documents an earlier LLE-to-fMRI application that discarded spatial neighbourhood structure, which this method explicitly avoids."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents challenges in fMRI disease classification and cautions about PCA, motivating the comparison pipeline."}],"review_version":1}