REVIEW 3 major objections 5 minor 26 references
Identification of relevant diffusion MRI metrics impacting cognitive functions using a novel feature selection method
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A crossover-augmented best-first search selects diffusion MRI features that predict working-memory scores with $r^2$ above 0.5 in most of nine cohort/test models.
desk verdict Incremental but real algorithm twist; the evaluation protocol has a likely leakage issue that undercuts the headline accuracy claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the wrapper feature-selection problem cast as a graph search over the boolean hypercube: each vertex is a binary vector indicating which of the 280 features are selected, edges flip one bit, and the edge weight is the difference in cross-validated $r^2$ between adjacent subsets. The paper's addition is a crossover operator that, after expanding the best node, forms the arithmetic combination $V_{\mathrm{cross}} = V_{\mathrm{first}} + V_{\mathrm{second}} - V_{\mathrm{parent}}$ from the two best children and inserts it into the priority queue, letting the search jump two bits at a time (merge, replace, or skip-up) rather than step through every intermediate child. Gradient boosting trees provide the regression scores, and repeated stratified 5-fold cross-validation scores each subset.
What would settle it
Run BFS with crossover inside a nested cross-validation loop, re-running the search from scratch on each training fold and evaluating only on the held-out fold; if the averaged validation $r^2$ falls substantially below the reported 0.38–0.64, the improvement over greedy BFS is largely an artifact of feature-selection leakage.
Extended reading notes
Core claim
On its own terms, the paper establishes that modifying greedy best-first search with a genetic-algorithm crossover operator improves cross-validated prediction of working-memory subtest scores from diffusion MRI features. Across nine models (three tests times control, mTBI, and combined cohorts), BFS with crossover yields the highest $r^2$ in every case, improving the DSB mTBI result from 0.5193 to 0.6005, and it finds feature subsets whose selected diffusion metrics are predominantly compartment-specific measures of axonal and extra-axonal microstructure. The paper concludes that optimizing feature selection over a large space with this heuristic reveals imaging features that relate to cognitive function and injury in mTBI.
Load-bearing premise
The reported $r^2$ values assume the feature-selection search never sees the validation folds used to compute them; the paper does not describe the nested cross-validation that would guarantee this, so the gains may be optimistically biased.
Editorial extensions
If this is right
- If the improvement holds, the BFS-with-crossover search is a drop-in replacement for greedy feature selection in any regression or classification problem with a large feature space and limited samples.
- The frequently selected metrics (AWF, DA, De-par, De-perp) implicate axonal and extra-axonal white-matter microstructure as the diffusional signature of working-memory function in both healthy and injured populations.
- Separate models for controls and mTBI patients predict better than a pooled model, indicating that brain–cognition relationships differ between the two populations.
- The graph-search perspective gives a unified account of forward, backward, floating, genetic, and best-first selection, so algorithmic improvements like crossover transfer across those families.
Reading between the lines
- A nested cross-validation replication—where the search runs only on training folds—would test whether the reported $r^2$ gains survive without leakage; the paper does not describe such a scheme.
- The same crossover arithmetic could be applied to other subset-search problems, such as biomarker discovery in genomics or feature selection for classification, where the boolean hypercube structure is identical.
- The feature-frequency table suggests a stability check: rerunning the search on bootstrap samples and measuring overlap of selected feature sets would show whether the chosen metrics are reproducible or specific to this cohort.
- Because pooled models underperform separate models, a natural next step is an interaction model (cohort x microstructural metric) rather than a single pooled predictor.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript proposes a wrapper feature selection method, best-first search with a genetic-algorithm-inspired crossover operator (BFS with crossover), and applies it to predict working memory performance (Digit Span Forward, Digit Span Backward, Letter-Number Sequencing) from 280 diffusion MRI features derived from 154 subjects (70 controls, 84 mTBI). The central claim is that BFS with crossover improves prediction accuracy over greedy BFS and other heuristic feature selection methods in 8 of 9 cohorts, with reported Pearson correlations above 0.7 and p-values below 0.002, and that the selected features are clinically interpretable.
Significance. If the reported accuracy is unbiased, the proposed method would be a useful contribution to feature selection for high-dimensional neuroimaging data with limited sample sizes, and the application to working memory prediction in mTBI is clinically relevant. The paper's strengths include a clinically meaningful problem, comparison against multiple feature selection baselines, and an interpretable analysis of which diffusion metrics are selected. However, the central empirical claim is not yet supported by the evidence as reported because the experimental protocol does not clearly prevent information leakage, and the reported statistics contain internal contradictions.
major comments (3)
- [Section II-C and II-D (Algorithms 1 and 2), Section III-A] The evaluation protocol appears to lack nested cross-validation. Algorithms 1 and 2 take the full dataset (X,y) and evaluate every candidate feature subset by calling child.crossvalidate(X,y) on the full data, while the final reported r2 values in Table II and Figure 3 are described as coming from 'validation samples in all five folds.' If the same cross-validation folds that guide feature selection are reused to compute the reported performance, the r2 values are optimistically biased. Because BFS with crossover explores more candidate subsets than greedy BFS, its apparent improvement could be an artifact of increased overfitting opportunity rather than better generalization. The authors must describe and implement a nested CV scheme in which feature selection is performed inside each training fold and the selected features are evaluated on a held-out test fold, and report results from that outer loop.
- [Table II and Section IV (Conclusion)] The reported summary statistics are internally inconsistent. The conclusion states Pearson correlation '> 0.7 in all cases' and 'p-value < 0.002,' but Table II lists DSF combine r=0.64 and DSB combine r=0.69, and several p-values exceed 0.002 (e.g., 0.0109, 0.0051, 0.0138, 0.0027). Furthermore, for a cohort of about 70 subjects, a Pearson correlation of 0.75 would yield a p-value many orders of magnitude smaller than 0.0109, suggesting either a different sample size, a different test, or an error in reporting. The authors should clarify the exact test used, the effective sample size, and correct the contradictory statements in the conclusion.
- [Section III-A and Abstract] The abstract and Section III-A claim that BFS with crossover achieves 'significantly more accurate predictions' and 'further improvement over greedy BFS in all cases,' but no statistical significance test for the improvement is provided, and no error bars or confidence intervals are reported. The r2 differences could be within fold-to-fold variability. The authors should report the standard deviation or confidence interval of r2 across repeated CV runs and perform a paired significance test (e.g., paired t-test or Wilcoxon signed-rank test across folds or repetitions) for the difference between BFS with crossover and greedy BFS.
minor comments (5)
- [Abstract and Section II-F] The abstract mentions 'repeated stratified cross-validation,' but Section II-F describes only a single stratified 5-fold cross-validation; please clarify whether the procedure was repeated and, if so, how many repetitions were performed.
- [Algorithm 2] In Algorithm 2, the last condition uses 'child /∈ close' where the previously defined variable is 'cross' (i.e., 'if cross /∈ open ⋀ cross /∈ close'); this appears to be a typo.
- [Section II-D and Figure 2] The three crossover operations (skip down, replace, skip up) are described in words and in Figure 2, but the single formula Vcross = Vchild1 + Vchild2 - Vparent is not explicitly mapped to each case; a short worked example for each operation would improve clarity.
- [Section III-A] The genetic algorithm comparator is mentioned but its hyperparameters (population size, number of generations, crossover and mutation rates) are not reported, which limits reproducibility of the comparison.
- [Table II] The table caption lists 'Columns 2-6 are r2 scores,' but 'Pearson Coefficient' and 'p-value' are also tabulated; the caption should be updated to describe all columns.
Circularity Check
No significant circularity: the central claim is an empirical algorithm comparison, and the paper's self-citations are background rather than load-bearing.
full rationale
The paper's central claim—that BFS with crossover improves prediction r2 over other wrapper feature selectors for working-memory regression—is an empirical evaluation of a search heuristic, not a derivation from a fitted constant or from the authors' prior theorems. The feature-selection graph formulation (Eqs. 1–5) is standard and the components (GBT, stratified K-fold CV, BFS) are cited to external references [14,19,20,24,25]. Self-citations (e.g., [5], [8], [10]) appear only as background or dataset motivation and are not used to force the algorithm choice or the reported accuracy. No 'uniqueness theorem' from prior work is imported to exclude alternatives, and no selected feature is defined in terms of the predicted score. Thus no prediction reduces by construction to its input. A legitimate methodological caveat exists—Algorithm 1 calls child.crossvalidate(X,y) on the full dataset and no nested CV is described, so the reported validation r2 may be optimistically biased—but this is a selection-bias/leakage soundness issue, not a circularity of the derivation; the paper's comparison could still be independently evaluated and corrected. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- patience =
25
- GBT number of trees =
100
- GBT tree depth =
searched 2 to 5
- label quantization bins =
5 bins over [-3,3]
assumptions (4)
- standard math Edge weights equal score differences, so the path sum telescopes to the score difference between endpoints.
- domain assumption The 8 diffusion metrics and 7 white matter regions contain the information relevant to working memory.
- ad hoc to paper The crossover operation Vcross = Vchild1 + Vchild2 - Vparent yields a valid binary feature subset.
- domain assumption Stratified 5-fold CV with 5 quantized bins provides unbiased performance estimates.
Cite this review
Pith. "Pith review of Identification of relevant diffusion MRI metrics impacting cognitive functions using a novel feature selection method." pith.science (2026). https://pith.science/paper/GHX5MXFH
@misc{pith2026190804752,
author = {Pith},
title = {Pith review of: Identification of relevant diffusion MRI metrics impacting cognitive functions using a novel feature selection method},
year = {2026},
howpublished = {\url{https://pith.science/paper/GHX5MXFH}},
note = {Machine review of arXiv:1908.04752}
}
read the original abstract
Mild Traumatic Brain Injury (mTBI) is a significant public health problem. The most troubling symptoms after mTBI are cognitive complaints. Studies show measurable differences between patients with mTBI and healthy controls with respect to tissue microstructure using diffusion MRI. However, it remains unclear which diffusion measures are the most informative with regard to cognitive functions in both the healthy state as well as after injury. In this study, we use diffusion MRI to formulate a predictive model for performance on working memory based on the most relevant MRI features. The key challenge is to identify relevant features over a large feature space with high accuracy in an efficient manner. To tackle this challenge, we propose a novel improvement of the best first search approach with crossover operators inspired by genetic algorithm. Compared against other heuristic feature selection algorithms, the proposed method achieves significantly more accurate predictions and yields clinically interpretable selected features.
Figures
Reference graph
Works this paper leans on
-
[1]
M. Faul, M. M. Wald, L. Xu, and V . G. Coronado, “Traumatic brain injury in the united states; emergency department visits, hospitalizations, and deaths, 2002-2006,” 2010
work page 2002
-
[2]
D. C. V oormolen, M. C. Cnossen, S. Polinder, N. V on Steinbuechel, P. E. V os, and J. A. Haagsma, “Divergent classification methods of post-concussion syndrome after mild traumatic brain injury: prevalence rates, risk factors, and functional outcome,” Journal of neurotrauma, vol. 35, no. 11, pp. 1233–1241, 2018
work page 2018
-
[3]
Mild traumatic brain injury: is diffusion imaging ready for primetime in forensic medicine?
E. J. Grossman, M. Inglese, and R. Bammer, “Mild traumatic brain injury: is diffusion imaging ready for primetime in forensic medicine?” Topics in magnetic resonance imaging: TMRI , vol. 21, no. 6, p. 379, 2010
work page 2010
-
[4]
M. E. Shenton, H. Hamoda, J. Schneiderman, S. Bouix, O. Paster- nak, Y . Rathi, M.-A. Vu, M. P. Purohit, K. Helmer, I. Koerte et al., “A review of magnetic resonance imaging and diffusion tensor imaging findings in mild traumatic brain injury,” Brain imaging and behavior, vol. 6, no. 2, pp. 137–192, 2012
work page 2012
-
[5]
White matter tract integrity: an indicator of axonal pathology after mild traumatic brain injury,
S. Chung, E. Fieremans, X. Wang, N. E. Kucukboyaci, C. J. Morton, J. Babb, P. Amorapanth, F.-Y . A. Foo, D. S. Novikov, S. R. Flanagan et al., “White matter tract integrity: an indicator of axonal pathology after mild traumatic brain injury,” Journal of neurotrauma, vol. 35, no. 8, pp. 1015–1020, 2018
work page 2018
-
[6]
S. Chung, X. Wang, E. Fieremans, R. Joseph, A. Prin, F. Farng- Yang A, C. Morton, N. Dmitry, F. Steven R, and Y . W. Lui, “Altered relationship between working memory and brain microstructure after mild traumatic brain injury,” American Journal of Neuroradiology , in press
-
[7]
Short-term dti predictors of cognitive dysfunction in mild traumatic brain injury,
L. Miles, R. I. Grossman, G. Johnson, J. S. Babb, L. Diller, and M. Inglese, “Short-term dti predictors of cognitive dysfunction in mild traumatic brain injury,” Brain injury, vol. 22, no. 2, pp. 115– 122, 2008
work page 2008
-
[8]
Classification algorithms using multiple mri features in mild trau- matic brain injury,
Y . W. Lui, Y . Xue, D. Kenul, Y . Ge, R. I. Grossman, and Y . Wang, “Classification algorithms using multiple mri features in mild trau- matic brain injury,”Neurology, vol. 83, no. 14, pp. 1235–1240, 2014
work page 2014
Show all 26 references
-
[9]
Prediction of longterm outcome of neuropsychological tests of mtbi patients using imaging features,
S. Minaee, Y . Wang, and Y . W. Lui, “Prediction of longterm outcome of neuropsychological tests of mtbi patients using imaging features,” in 2013 IEEE Signal Processing in Medicine and Biology Symposium (SPMB). IEEE, Conference Proceedings, pp. 1–6
2013
-
[10]
Mtbi identification from diffusion mr images using bag of adversarial visual features,
S. Minaee, Y . Wang, A. Aygar, S. Chung, X. Wang, Y . W. Lui, E. Fieremans, S. Flanagan, and J. Rath, “Mtbi identification from diffusion mr images using bag of adversarial visual features,” IEEE transactions on medical imaging , 2019
2019
-
[11]
Adversarial autoencoders,
A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey, “Adversarial autoencoders,”arXiv preprint arXiv:1511.05644, 2015
2015 arXiv
-
[12]
Un- supervised 3-d feature learning for mild traumatic brain injury,
P.-Y . Kao, E. Rojas, J. W. Chen, A. Zhang, and B. Manjunath, “Un- supervised 3-d feature learning for mild traumatic brain injury,” in International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Springer, 2016, pp. 282–290
2016
-
[13]
Statistical machine learning to identify traumatic brain injury (tbi) from structural disconnections of white matter networks,
J. Mitra, K.-k. Shen, S. Ghose, P. Bourgeat, J. Fripp, O. Salvado, K. Pannek, D. J. Taylor, J. L. Mathias, and S. Rose, “Statistical machine learning to identify traumatic brain injury (tbi) from structural disconnections of white matter networks,” NeuroImage, vol. 129, pp. 24...
2016
-
[14]
Wrappers for feature subset selection,
R. Kohavi and G. H. John, “Wrappers for feature subset selection,” Artificial intelligence, vol. 97, no. 1-2, pp. 273–324, 1997
1997
-
[15]
J. M. Sattler and J. J. Ryan, Assessment with the WAIS-IV. Jerome M Sattler Publisher, 2009
2009
-
[16]
White matter characterization with diffusional kurtosis imaging,
E. Fieremans, J. H. Jensen, and J. A. Helpern, “White matter characterization with diffusional kurtosis imaging,” Neuroimage, vol. 58, no. 1, pp. 177–188, 2011
2011
-
[17]
Evaluating kurtosis-based diffusion mri tissue models for white matter with fiber ball imaging,
J. H. Jensen, E. T. McKinnon, G. R. Glenn, and J. A. Helpern, “Evaluating kurtosis-based diffusion mri tissue models for white matter with fiber ball imaging,” NMR in Biomedicine, vol. 30, no. 5, p. e3689, 2017
2017
-
[18]
A survey on feature selection methods,
G. Chandrashekar and F. Sahin, “A survey on feature selection methods,” Computers and Electrical Engineering , vol. 40, no. 1, pp. 16–28, 2014
2014
-
[19]
A wrapper approach for feature selection based on bat algorithm and optimum-path forest,
D. Rodrigues, L. A. Pereira, R. Y . Nakamura, K. A. Costa, X.-S. Yang, A. N. Souza, and J. P. Papa, “A wrapper approach for feature selection based on bat algorithm and optimum-path forest,” Expert Systems with Applications , vol. 41, no. 5, pp. 2250–2258, 2014
2014
-
[20]
Experiments with the graph traverser program,
J. E. Doran and D. Michie, “Experiments with the graph traverser program,” Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences , vol. 294, no. 1437, pp. 235– 259, 1966
1966
-
[21]
Optimal column subset selection by a-star search,
H. Arai, C. Maung, and H. Schweitzer, “Optimal column subset selection by a-star search,” in Twenty-ninth AAAI conference on artificial intelligence, 2015
2015
-
[22]
Floating search methods in feature selection,
P. Pudil, J. Novovi ˇcov´a, and J. Kittler, “Floating search methods in feature selection,” Pattern recognition letters , vol. 15, no. 11, pp. 1119–1125, 1994
1994
-
[23]
Feature subset selection using a ge- netic algorithm,
J. Yang and V . Honavar, “Feature subset selection using a ge- netic algorithm,” in Feature extraction, construction and selection . Springer, 1998, pp. 117–136
1998
-
[24]
Hastie, R
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction , ser. Springer Series in Statistics. Springer New York, 2013. [Online]. Available: https://books.google.com/books?id=yPfZBwAAQBAJ
2013
-
[25]
Greedy function approximation: a gradient boosting machine,
J. H. Friedman, “Greedy function approximation: a gradient boosting machine,” Annals of statistics , pp. 1189–1232, 2001
2001
-
[26]
Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation,
Y .-D. Zhang, Z.-J. Yang, H.-M. Lu, X.-X. Zhou, P. Phillips, Q.- M. Liu, and S.-H. Wang, “Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation,” IEEE Access , vol. 4, pp. 8375–8385, 2016
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.