Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Identification of relevant diffusion MRI metrics impacting cognitive functions using a novel feature selection method

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A crossover-augmented best-first search selects diffusion MRI features that predict working-memory scores with $r^2$ above 0.5 in most of nine cohort/test models.

desk verdict Incremental but real algorithm twist; the evaluation protocol has a likely leakage issue that undercuts the headline accuracy claim. read the letter →

arxiv 1908.04752 v2 pith:GHX5MXFH submitted 2019-08-10 eess.IV cs.LGq-bio.QMstat.ML

classification eess.IVcs.LGq-bio.QMstat.ML
keywords featureselectionbest-firstsearchcrossoveroperatordiffusionMRImildtraumaticbraininjuryworkingmemorygradientboostingtreewhitemattermicrostructure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a crossover-augmented best-first search (BFS with crossover) is a more accurate feature selection method for predicting working memory scores from diffusion MRI than greedy forward, greedy backward, genetic algorithm, or greedy best-first search. Applying it to 280 statistics from eight diffusion metrics in seven white-matter regions, it reports $r^2$ above 0.5 in most of nine test/cohort models, Pearson correlations above 0.7, and $p<0.002$. The selected features converge on compartment-specific white-matter metrics such as axonal water fraction and intra-axonal diffusivity, which the paper reads as evidence that axonal microstructure underlies working-memory performance. If correct, the method would give clinicians a quantitative, interpretable way to tie cognitive complaints in mTBI to specific tissue injury and to track recovery.

What carries the argument

The carrying object is the wrapper feature-selection problem cast as a graph search over the boolean hypercube: each vertex is a binary vector indicating which of the 280 features are selected, edges flip one bit, and the edge weight is the difference in cross-validated $r^2$ between adjacent subsets. The paper's addition is a crossover operator that, after expanding the best node, forms the arithmetic combination $V_{\mathrm{cross}} = V_{\mathrm{first}} + V_{\mathrm{second}} - V_{\mathrm{parent}}$ from the two best children and inserts it into the priority queue, letting the search jump two bits at a time (merge, replace, or skip-up) rather than step through every intermediate child. Gradient boosting trees provide the regression scores, and repeated stratified 5-fold cross-validation scores each subset.

What would settle it

Run BFS with crossover inside a nested cross-validation loop, re-running the search from scratch on each training fold and evaluating only on the held-out fold; if the averaged validation $r^2$ falls substantially below the reported 0.38–0.64, the improvement over greedy BFS is largely an artifact of feature-selection leakage.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that modifying greedy best-first search with a genetic-algorithm crossover operator improves cross-validated prediction of working-memory subtest scores from diffusion MRI features. Across nine models (three tests times control, mTBI, and combined cohorts), BFS with crossover yields the highest $r^2$ in every case, improving the DSB mTBI result from 0.5193 to 0.6005, and it finds feature subsets whose selected diffusion metrics are predominantly compartment-specific measures of axonal and extra-axonal microstructure. The paper concludes that optimizing feature selection over a large space with this heuristic reveals imaging features that relate to cognitive function and injury in mTBI.

Load-bearing premise

The reported $r^2$ values assume the feature-selection search never sees the validation folds used to compute them; the paper does not describe the nested cross-validation that would guarantee this, so the gains may be optimistically biased.

Editorial extensions

If this is right

  • If the improvement holds, the BFS-with-crossover search is a drop-in replacement for greedy feature selection in any regression or classification problem with a large feature space and limited samples.
  • The frequently selected metrics (AWF, DA, De-par, De-perp) implicate axonal and extra-axonal white-matter microstructure as the diffusional signature of working-memory function in both healthy and injured populations.
  • Separate models for controls and mTBI patients predict better than a pooled model, indicating that brain–cognition relationships differ between the two populations.
  • The graph-search perspective gives a unified account of forward, backward, floating, genetic, and best-first selection, so algorithmic improvements like crossover transfer across those families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A nested cross-validation replication—where the search runs only on training folds—would test whether the reported $r^2$ gains survive without leakage; the paper does not describe such a scheme.
  • The same crossover arithmetic could be applied to other subset-search problems, such as biomarker discovery in genomics or feature selection for classification, where the boolean hypercube structure is identical.
  • The feature-frequency table suggests a stability check: rerunning the search on bootstrap samples and measuring overlap of selected feature sets would show whether the chosen metrics are reproducible or specific to this cohort.
  • Because pooled models underperform separate models, a natural next step is an interaction model (cohort x microstructural metric) rather than a single pooled predictor.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript proposes a wrapper feature selection method, best-first search with a genetic-algorithm-inspired crossover operator (BFS with crossover), and applies it to predict working memory performance (Digit Span Forward, Digit Span Backward, Letter-Number Sequencing) from 280 diffusion MRI features derived from 154 subjects (70 controls, 84 mTBI). The central claim is that BFS with crossover improves prediction accuracy over greedy BFS and other heuristic feature selection methods in 8 of 9 cohorts, with reported Pearson correlations above 0.7 and p-values below 0.002, and that the selected features are clinically interpretable.

Significance. If the reported accuracy is unbiased, the proposed method would be a useful contribution to feature selection for high-dimensional neuroimaging data with limited sample sizes, and the application to working memory prediction in mTBI is clinically relevant. The paper's strengths include a clinically meaningful problem, comparison against multiple feature selection baselines, and an interpretable analysis of which diffusion metrics are selected. However, the central empirical claim is not yet supported by the evidence as reported because the experimental protocol does not clearly prevent information leakage, and the reported statistics contain internal contradictions.

major comments (3)
  1. [Section II-C and II-D (Algorithms 1 and 2), Section III-A] The evaluation protocol appears to lack nested cross-validation. Algorithms 1 and 2 take the full dataset (X,y) and evaluate every candidate feature subset by calling child.crossvalidate(X,y) on the full data, while the final reported r2 values in Table II and Figure 3 are described as coming from 'validation samples in all five folds.' If the same cross-validation folds that guide feature selection are reused to compute the reported performance, the r2 values are optimistically biased. Because BFS with crossover explores more candidate subsets than greedy BFS, its apparent improvement could be an artifact of increased overfitting opportunity rather than better generalization. The authors must describe and implement a nested CV scheme in which feature selection is performed inside each training fold and the selected features are evaluated on a held-out test fold, and report results from that outer loop.
  2. [Table II and Section IV (Conclusion)] The reported summary statistics are internally inconsistent. The conclusion states Pearson correlation '> 0.7 in all cases' and 'p-value < 0.002,' but Table II lists DSF combine r=0.64 and DSB combine r=0.69, and several p-values exceed 0.002 (e.g., 0.0109, 0.0051, 0.0138, 0.0027). Furthermore, for a cohort of about 70 subjects, a Pearson correlation of 0.75 would yield a p-value many orders of magnitude smaller than 0.0109, suggesting either a different sample size, a different test, or an error in reporting. The authors should clarify the exact test used, the effective sample size, and correct the contradictory statements in the conclusion.
  3. [Section III-A and Abstract] The abstract and Section III-A claim that BFS with crossover achieves 'significantly more accurate predictions' and 'further improvement over greedy BFS in all cases,' but no statistical significance test for the improvement is provided, and no error bars or confidence intervals are reported. The r2 differences could be within fold-to-fold variability. The authors should report the standard deviation or confidence interval of r2 across repeated CV runs and perform a paired significance test (e.g., paired t-test or Wilcoxon signed-rank test across folds or repetitions) for the difference between BFS with crossover and greedy BFS.
minor comments (5)
  1. [Abstract and Section II-F] The abstract mentions 'repeated stratified cross-validation,' but Section II-F describes only a single stratified 5-fold cross-validation; please clarify whether the procedure was repeated and, if so, how many repetitions were performed.
  2. [Algorithm 2] In Algorithm 2, the last condition uses 'child /∈ close' where the previously defined variable is 'cross' (i.e., 'if cross /∈ open ⋀ cross /∈ close'); this appears to be a typo.
  3. [Section II-D and Figure 2] The three crossover operations (skip down, replace, skip up) are described in words and in Figure 2, but the single formula Vcross = Vchild1 + Vchild2 - Vparent is not explicitly mapped to each case; a short worked example for each operation would improve clarity.
  4. [Section III-A] The genetic algorithm comparator is mentioned but its hyperparameters (population size, number of generations, crossover and mutation rates) are not reported, which limits reproducibility of the comparison.
  5. [Table II] The table caption lists 'Columns 2-6 are r2 scores,' but 'Pearson Coefficient' and 'p-value' are also tabulated; the caption should be updated to describe all columns.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical algorithm comparison, and the paper's self-citations are background rather than load-bearing.

full rationale

The paper's central claim—that BFS with crossover improves prediction r2 over other wrapper feature selectors for working-memory regression—is an empirical evaluation of a search heuristic, not a derivation from a fitted constant or from the authors' prior theorems. The feature-selection graph formulation (Eqs. 1–5) is standard and the components (GBT, stratified K-fold CV, BFS) are cited to external references [14,19,20,24,25]. Self-citations (e.g., [5], [8], [10]) appear only as background or dataset motivation and are not used to force the algorithm choice or the reported accuracy. No 'uniqueness theorem' from prior work is imported to exclude alternatives, and no selected feature is defined in terms of the predicted score. Thus no prediction reduces by construction to its input. A legitimate methodological caveat exists—Algorithm 1 calls child.crossvalidate(X,y) on the full dataset and no nested CV is described, so the reported validation r2 may be optimistically biased—but this is a selection-bias/leakage soundness issue, not a circularity of the derivation; the paper's comparison could still be independently evaluated and corrected. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three hand-chosen hyperparameters, the diffusion feature set assumed from prior work, a possibly ill-specified crossover arithmetic, and an evaluation protocol whose nested CV structure is not described.

free parameters (4)
  • patience = 25
    Termination threshold for BFS search; chosen by hand, affects the number of feature subsets evaluated and therefore the final selected features and reported accuracy.
  • GBT number of trees = 100
    Number of boosting trees fixed for all experiments; a hyperparameter influencing model capacity and prediction accuracy.
  • GBT tree depth = searched 2 to 5
    Tree depth is chosen from a small range; the specific depth for each model is data-dependent and not reported.
  • label quantization bins = 5 bins over [-3,3]
    Continuous test scores are quantized into 5 bins to enable stratified splitting; the number of bins and boundaries are chosen without justification and can affect the fold composition.
assumptions (4)
  • standard math Edge weights equal score differences, so the path sum telescopes to the score difference between endpoints.
    Equations 1-4 define edge weight as score(Vj)-score(Vi), making the path length exactly score(Vj)-score(Vi). This is a definition, not an empirical assumption.
  • domain assumption The 8 diffusion metrics and 7 white matter regions contain the information relevant to working memory.
    The feature space is restricted to these metrics and regions based on prior mTBI literature (Section II-A); other metrics or regions might improve prediction but are excluded.
  • ad hoc to paper The crossover operation Vcross = Vchild1 + Vchild2 - Vparent yields a valid binary feature subset.
    Algorithm 2 introduces this arithmetic without specifying clamping or set operations; with standard integer arithmetic it can produce 2 or -1 for some bits, so the operation is not well-defined for binary vectors.
  • domain assumption Stratified 5-fold CV with 5 quantized bins provides unbiased performance estimates.
    Section II-F describes the stratification but does not state that feature selection is nested within the training folds; if selection sees validation data, the performance estimate is biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Identification of relevant diffusion MRI metrics impacting cognitive functions using a novel feature selection method." pith.science (2026). https://pith.science/paper/GHX5MXFH

@misc{pith2026190804752,
  author       = {Pith},
  title        = {Pith review of: Identification of relevant diffusion MRI metrics impacting cognitive functions using a novel feature selection method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHX5MXFH}},
  note         = {Machine review of arXiv:1908.04752}
}
read the original abstract

Mild Traumatic Brain Injury (mTBI) is a significant public health problem. The most troubling symptoms after mTBI are cognitive complaints. Studies show measurable differences between patients with mTBI and healthy controls with respect to tissue microstructure using diffusion MRI. However, it remains unclear which diffusion measures are the most informative with regard to cognitive functions in both the healthy state as well as after injury. In this study, we use diffusion MRI to formulate a predictive model for performance on working memory based on the most relevant MRI features. The key challenge is to identify relevant features over a large feature space with high accuracy in an efficient manner. To tackle this challenge, we propose a novel improvement of the best first search approach with crossover operators inspired by genetic algorithm. Compared against other heuristic feature selection algorithms, the proposed method achieves significantly more accurate predictions and yields clinically interpretable selected features.

Figures

Figures reproduced from arXiv: 1908.04752 by the authors.

Figure 1
Figure 1. An example of 4 feature graph with crossover operator, each [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. 3 types of cross over operations over the best and second best [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison between ground truth label values and predicted [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 23 canonical work pages

  1. [1]

    Traumatic brain injury in the united states; emergency department visits, hospitalizations, and deaths, 2002-2006,

    M. Faul, M. M. Wald, L. Xu, and V . G. Coronado, “Traumatic brain injury in the united states; emergency department visits, hospitalizations, and deaths, 2002-2006,” 2010

  2. [2]

    Divergent classification methods of post-concussion syndrome after mild traumatic brain injury: prevalence rates, risk factors, and functional outcome,

    D. C. V oormolen, M. C. Cnossen, S. Polinder, N. V on Steinbuechel, P. E. V os, and J. A. Haagsma, “Divergent classification methods of post-concussion syndrome after mild traumatic brain injury: prevalence rates, risk factors, and functional outcome,” Journal of neurotrauma, vol. 35, no. 11, pp. 1233–1241, 2018

  3. [3]

    Mild traumatic brain injury: is diffusion imaging ready for primetime in forensic medicine?

    E. J. Grossman, M. Inglese, and R. Bammer, “Mild traumatic brain injury: is diffusion imaging ready for primetime in forensic medicine?” Topics in magnetic resonance imaging: TMRI , vol. 21, no. 6, p. 379, 2010

  4. [4]

    A review of magnetic resonance imaging and diffusion tensor imaging findings in mild traumatic brain injury,

    M. E. Shenton, H. Hamoda, J. Schneiderman, S. Bouix, O. Paster- nak, Y . Rathi, M.-A. Vu, M. P. Purohit, K. Helmer, I. Koerte et al., “A review of magnetic resonance imaging and diffusion tensor imaging findings in mild traumatic brain injury,” Brain imaging and behavior, vol. 6, no. 2, pp. 137–192, 2012

  5. [5]

    White matter tract integrity: an indicator of axonal pathology after mild traumatic brain injury,

    S. Chung, E. Fieremans, X. Wang, N. E. Kucukboyaci, C. J. Morton, J. Babb, P. Amorapanth, F.-Y . A. Foo, D. S. Novikov, S. R. Flanagan et al., “White matter tract integrity: an indicator of axonal pathology after mild traumatic brain injury,” Journal of neurotrauma, vol. 35, no. 8, pp. 1015–1020, 2018

  6. [6]

    Altered relationship between working memory and brain microstructure after mild traumatic brain injury,

    S. Chung, X. Wang, E. Fieremans, R. Joseph, A. Prin, F. Farng- Yang A, C. Morton, N. Dmitry, F. Steven R, and Y . W. Lui, “Altered relationship between working memory and brain microstructure after mild traumatic brain injury,” American Journal of Neuroradiology , in press

  7. [7]

    Short-term dti predictors of cognitive dysfunction in mild traumatic brain injury,

    L. Miles, R. I. Grossman, G. Johnson, J. S. Babb, L. Diller, and M. Inglese, “Short-term dti predictors of cognitive dysfunction in mild traumatic brain injury,” Brain injury, vol. 22, no. 2, pp. 115– 122, 2008

  8. [8]

    Classification algorithms using multiple mri features in mild trau- matic brain injury,

    Y . W. Lui, Y . Xue, D. Kenul, Y . Ge, R. I. Grossman, and Y . Wang, “Classification algorithms using multiple mri features in mild trau- matic brain injury,”Neurology, vol. 83, no. 14, pp. 1235–1240, 2014

Show all 26 references
  1. [9]

    Prediction of longterm outcome of neuropsychological tests of mtbi patients using imaging features,

    S. Minaee, Y . Wang, and Y . W. Lui, “Prediction of longterm outcome of neuropsychological tests of mtbi patients using imaging features,” in 2013 IEEE Signal Processing in Medicine and Biology Symposium (SPMB). IEEE, Conference Proceedings, pp. 1–6

  2. [10]

    Mtbi identification from diffusion mr images using bag of adversarial visual features,

    S. Minaee, Y . Wang, A. Aygar, S. Chung, X. Wang, Y . W. Lui, E. Fieremans, S. Flanagan, and J. Rath, “Mtbi identification from diffusion mr images using bag of adversarial visual features,” IEEE transactions on medical imaging , 2019

  3. [11]

    Adversarial autoencoders,

    A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey, “Adversarial autoencoders,”arXiv preprint arXiv:1511.05644, 2015

  4. [12]

    Un- supervised 3-d feature learning for mild traumatic brain injury,

    P.-Y . Kao, E. Rojas, J. W. Chen, A. Zhang, and B. Manjunath, “Un- supervised 3-d feature learning for mild traumatic brain injury,” in International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Springer, 2016, pp. 282–290

  5. [13]

    Statistical machine learning to identify traumatic brain injury (tbi) from structural disconnections of white matter networks,

    J. Mitra, K.-k. Shen, S. Ghose, P. Bourgeat, J. Fripp, O. Salvado, K. Pannek, D. J. Taylor, J. L. Mathias, and S. Rose, “Statistical machine learning to identify traumatic brain injury (tbi) from structural disconnections of white matter networks,” NeuroImage, vol. 129, pp. 24...

  6. [14]

    Wrappers for feature subset selection,

    R. Kohavi and G. H. John, “Wrappers for feature subset selection,” Artificial intelligence, vol. 97, no. 1-2, pp. 273–324, 1997

  7. [15]

    J. M. Sattler and J. J. Ryan, Assessment with the WAIS-IV. Jerome M Sattler Publisher, 2009

  8. [16]

    White matter characterization with diffusional kurtosis imaging,

    E. Fieremans, J. H. Jensen, and J. A. Helpern, “White matter characterization with diffusional kurtosis imaging,” Neuroimage, vol. 58, no. 1, pp. 177–188, 2011

  9. [17]

    Evaluating kurtosis-based diffusion mri tissue models for white matter with fiber ball imaging,

    J. H. Jensen, E. T. McKinnon, G. R. Glenn, and J. A. Helpern, “Evaluating kurtosis-based diffusion mri tissue models for white matter with fiber ball imaging,” NMR in Biomedicine, vol. 30, no. 5, p. e3689, 2017

  10. [18]

    A survey on feature selection methods,

    G. Chandrashekar and F. Sahin, “A survey on feature selection methods,” Computers and Electrical Engineering , vol. 40, no. 1, pp. 16–28, 2014

  11. [19]

    A wrapper approach for feature selection based on bat algorithm and optimum-path forest,

    D. Rodrigues, L. A. Pereira, R. Y . Nakamura, K. A. Costa, X.-S. Yang, A. N. Souza, and J. P. Papa, “A wrapper approach for feature selection based on bat algorithm and optimum-path forest,” Expert Systems with Applications , vol. 41, no. 5, pp. 2250–2258, 2014

  12. [20]

    Experiments with the graph traverser program,

    J. E. Doran and D. Michie, “Experiments with the graph traverser program,” Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences , vol. 294, no. 1437, pp. 235– 259, 1966

  13. [21]

    Optimal column subset selection by a-star search,

    H. Arai, C. Maung, and H. Schweitzer, “Optimal column subset selection by a-star search,” in Twenty-ninth AAAI conference on artificial intelligence, 2015

  14. [22]

    Floating search methods in feature selection,

    P. Pudil, J. Novovi ˇcov´a, and J. Kittler, “Floating search methods in feature selection,” Pattern recognition letters , vol. 15, no. 11, pp. 1119–1125, 1994

  15. [23]

    Feature subset selection using a ge- netic algorithm,

    J. Yang and V . Honavar, “Feature subset selection using a ge- netic algorithm,” in Feature extraction, construction and selection . Springer, 1998, pp. 117–136

  16. [24]

    Hastie, R

    T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction , ser. Springer Series in Statistics. Springer New York, 2013. [Online]. Available: https://books.google.com/books?id=yPfZBwAAQBAJ

  17. [25]

    Greedy function approximation: a gradient boosting machine,

    J. H. Friedman, “Greedy function approximation: a gradient boosting machine,” Annals of statistics , pp. 1189–1232, 2001

  18. [26]

    Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation,

    Y .-D. Zhang, Z.-J. Yang, H.-M. Lu, X.-X. Zhou, P. Phillips, Q.- M. Liu, and S.-H. Wang, “Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation,” IEEE Access , vol. 4, pp. 8375–8385, 2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.