{"id":"be72d2a8-f270-45b3-955e-b49286deba9f","arxiv_id":"1908.04752","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A best-first search with crossover improves diffusion-MRI-based prediction of working memory, selecting compartment-specific white matter metrics most often.","lead":"This paper proposes a new feature selection algorithm, best-first search with a genetic-style crossover step, and tests it on predicting working memory scores from diffusion MRI in healthy controls and mild traumatic brain injury patients. The method beats standard search heuristics in eight of nine experiments, and the most selected brain measurements are compartment-specific white matter diffusion metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feature selection appears to use the full dataset (Algorithm 1), and the reported p-values contradict Table II, so the claimed r2 improvement is not yet supported.","rationale":"The reader's weakest assumption coincides with the principal threat: no nested CV is described and Algorithm 1's signature is consistent with full-data feature selection. I agree this is load-bearing. I additionally noticed an objective inconsistency in the reported p-values that strengthens the need for revision: with NC/mTBI cohort sizes of 70/84, the p-values in Table II are incompatible with the corresponding Pearson r values, and the conclusion's summary thresholds are not met by all rows. This does not by itself prove the method fails, but it means the central claim as stated is not credible without rerunning. Since the paper could in principle be fixed by adding an outer CV loop and recomputing statistics, the conditional verdict stands; no change to the reader's verdict is needed. The concrete test that settles the leakage question is to re-run the pipeline with nested CV and compare held-out r2 to Table II; if the improvement persists, the method remains interesting.","tokens_in":9120,"tokens_out":9537,"duration_ms":109222,"concrete_test":"Run a nested 5-fold CV exactly as the paper would need: for each outer fold, run Algorithms 1 and 2 and forward selection on the training folds only, then evaluate the GBT with the chosen features on the held-out fold. Compare the average held-out r2 and the Pearson p-values (computed with the actual subject count) to Table II. If the crossover gain shrinks or disappears, or if BFS-with-crossover gives above-zero r2 on permuted labels while forward selection does not, the reported improvement is selection leakage rather than generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithms 1 and 2 take the full X,y and score every candidate subset with child.crossvalidate(X,y). The final r2, Pearson r, and p-values are then reported for 'validation samples in all five folds' (Fig. 3), but no outer/nested cross-validation loop is described. If the same CV folds that guide feature selection are reused to compute the reported performance, the r2 values are optimistically biased. Because BFS-with-crossover evaluates more candidate subsets than Greedy BFS or forward selection, its apparent advantage (e.g., DSB mTBI 0.5193 to 0.6005) could be an artifact of greater overfitting opportunity rather than better generalization. The paper's statistical summary is also internally inconsistent: Table II lists r=0.75, p=0.0109 for DSF NC; with n about 70 the correct two-tailed p is many orders of magnitude smaller, and several rows have p>0.002 or r<0.7, contradicting the conclusion's 'p<0.002' and 'r>0.7 in all cases'. These two issues together mean the central accuracy claim is not supported by the evidence as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes a wrapper feature selection method, best-first search with a genetic-algorithm-inspired crossover operator (BFS with crossover), and applies it to predict working memory performance (Digit Span Forward, Digit Span Backward, Letter-Number Sequencing) from 280 diffusion MRI features derived from 154 subjects (70 controls, 84 mTBI). The central claim is that BFS with crossover improves prediction accuracy over greedy BFS and other heuristic feature selection methods in 8 of 9 cohorts, with reported Pearson correlations above 0.7 and p-values below 0.002, and that the selected features are clinically interpretable.","tokens_in":9352,"tokens_out":3053,"duration_ms":33624,"significance":"If the reported accuracy is unbiased, the proposed method would be a useful contribution to feature selection for high-dimensional neuroimaging data with limited sample sizes, and the application to working memory prediction in mTBI is clinically relevant. The paper's strengths include a clinically meaningful problem, comparison against multiple feature selection baselines, and an interpretable analysis of which diffusion metrics are selected. However, the central empirical claim is not yet supported by the evidence as reported because the experimental protocol does not clearly prevent information leakage, and the reported statistics contain internal contradictions.","major_comments":[{"comment":"The evaluation protocol appears to lack nested cross-validation. Algorithms 1 and 2 take the full dataset (X,y) and evaluate every candidate feature subset by calling child.crossvalidate(X,y) on the full data, while the final reported r2 values in Table II and Figure 3 are described as coming from 'validation samples in all five folds.' If the same cross-validation folds that guide feature selection are reused to compute the reported performance, the r2 values are optimistically biased. Because BFS with crossover explores more candidate subsets than greedy BFS, its apparent improvement could be an artifact of increased overfitting opportunity rather than better generalization. The authors must describe and implement a nested CV scheme in which feature selection is performed inside each training fold and the selected features are evaluated on a held-out test fold, and report results from that outer loop.","section":"Section II-C and II-D (Algorithms 1 and 2), Section III-A"},{"comment":"The reported summary statistics are internally inconsistent. The conclusion states Pearson correlation '> 0.7 in all cases' and 'p-value < 0.002,' but Table II lists DSF combine r=0.64 and DSB combine r=0.69, and several p-values exceed 0.002 (e.g., 0.0109, 0.0051, 0.0138, 0.0027). Furthermore, for a cohort of about 70 subjects, a Pearson correlation of 0.75 would yield a p-value many orders of magnitude smaller than 0.0109, suggesting either a different sample size, a different test, or an error in reporting. The authors should clarify the exact test used, the effective sample size, and correct the contradictory statements in the conclusion.","section":"Table II and Section IV (Conclusion)"},{"comment":"The abstract and Section III-A claim that BFS with crossover achieves 'significantly more accurate predictions' and 'further improvement over greedy BFS in all cases,' but no statistical significance test for the improvement is provided, and no error bars or confidence intervals are reported. The r2 differences could be within fold-to-fold variability. The authors should report the standard deviation or confidence interval of r2 across repeated CV runs and perform a paired significance test (e.g., paired t-test or Wilcoxon signed-rank test across folds or repetitions) for the difference between BFS with crossover and greedy BFS.","section":"Section III-A and Abstract"}],"minor_comments":[{"comment":"The abstract mentions 'repeated stratified cross-validation,' but Section II-F describes only a single stratified 5-fold cross-validation; please clarify whether the procedure was repeated and, if so, how many repetitions were performed.","section":"Abstract and Section II-F"},{"comment":"In Algorithm 2, the last condition uses 'child /∈ close' where the previously defined variable is 'cross' (i.e., 'if cross /∈ open ⋀ cross /∈ close'); this appears to be a typo.","section":"Algorithm 2"},{"comment":"The three crossover operations (skip down, replace, skip up) are described in words and in Figure 2, but the single formula Vcross = Vchild1 + Vchild2 - Vparent is not explicitly mapped to each case; a short worked example for each operation would improve clarity.","section":"Section II-D and Figure 2"},{"comment":"The genetic algorithm comparator is mentioned but its hyperparameters (population size, number of generations, crossover and mutation rates) are not reported, which limits reproducibility of the comparison.","section":"Section III-A"},{"comment":"The table caption lists 'Columns 2-6 are r2 scores,' but 'Pearson Coefficient' and 'p-value' are also tabulated; the caption should be updated to describe all columns.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The central claim rests on an evaluation protocol that is not fully described and may be leaking information from validation folds into feature selection. The statistical inconsistencies in Table II and the conclusion need to be resolved before the paper can be considered. This is potentially fixable with additional experiments and corrected reporting, so I recommend major revision rather than rejection. There is also a mismatch between the strong language in the abstract and the absence of significance tests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the BFS-with-crossover idea is a genuine incremental twist on an existing wrapper, and the feature-frequency results are worth a look, but the evaluation protocol as described has a leakage problem and the reported statistics don't match the table. Net: needs revision before the accuracy claim is believable.\n\nWhat's actually new: combining the genetic-algorithm crossover with best-first search on the feature-subset graph is not something I've seen, and the paper is careful to define the arithmetic (child1+child2-parent) and the three geometric cases. That is a real, small algorithmic contribution, and the comparison against forward, backward, GA, and plain BFS is the right set.\n\nWhat it does well: the design of the feature space is reasonable for the question—280 features from 7 regions x 8 metrics x 5 statistics is defensible. GBT as the regressor is a sensible choice, and they give a preliminary comparison against SVM and NN. The selected-feature analysis (Table III) is clearly presented, and the observation that compartment-specific metrics (AWF, DA, De-par, De-perp) dominate in the mTBI and NC models is biologically plausible and aligns with prior work. That part is the most useful output.\n\nSoft spots, in order of severity. First: the cross-validation protocol. Algorithms 1 and 2 both call child.crossvalidate(X,y) on the full dataset during search, and the final r2 is computed on \"validation samples in all five folds\" from what appears to be the same fold assignment. There is no outer loop. That means the reported r2 values are likely optimistically biased, and the advantage of BFS-with-crossover (which evaluates more subsets) over plain BFS could be partly an artifact of more search. This is the load-bearing issue. Second: the p-values and the \"r>0.7, p<0.002\" conclusion do not match Table II. DSF NC has p=0.0109, DSF combine has r=0.64, and only three of nine rows have r>=0.7. Either the table or the conclusion is wrong. Third: no error bars or significance tests for the r2 differences across methods, so \"improvement in 8 of 9\" has no stated uncertainty.\n\nMinor: the description of label stratification into 5 bins is fine, but it is never stated that feature selection is nested inside the CV folds. Also, the crossover \"cross\" node is added with same priority, but Algorithm 2's condition \"if cross & cross.score > current.score\" appears to assume a score has already been computed—the initial crossover-score computation is missing. That's a reproducibility gap.\n\nBottom line: the clinical motivation and the feature ranking are interesting; the accuracy claim is not supported by the evidence as reported. A serious referee should see it, because the core idea and the dataset are worth a closer look, but it needs a nested CV evaluation and corrected statistics.\n\nRecommendation: send to peer review with a request for major revision focused on the evaluation protocol.","headline":"Incremental but real algorithm twist; the evaluation protocol has a likely leakage issue that undercuts the headline accuracy claim.","tokens_in":9873,"tokens_out":1885,"would_cite":false,"duration_ms":19672,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A crossover-augmented best-first search selects diffusion MRI features that predict working-memory scores with $r^2$ above 0.5 in most of nine cohort/test models.","keywords":["feature selection","best-first search","crossover operator","diffusion MRI","mild traumatic brain injury","working memory","gradient boosting tree","white matter microstructure"],"falsifier":"Run BFS with crossover inside a nested cross-validation loop, re-running the search from scratch on each training fold and evaluating only on the held-out fold; if the averaged validation $r^2$ falls substantially below the reported 0.38–0.64, the improvement over greedy BFS is largely an artifact of feature-selection leakage.","tokens_in":8933,"feed_emoji":"🧠","tokens_out":6858,"duration_ms":60241,"temperature":0.7,"pith_summary":"The paper claims that a crossover-augmented best-first search (BFS with crossover) is a more accurate feature selection method for predicting working memory scores from diffusion MRI than greedy forward, greedy backward, genetic algorithm, or greedy best-first search. Applying it to 280 statistics from eight diffusion metrics in seven white-matter regions, it reports $r^2$ above 0.5 in most of nine test/cohort models, Pearson correlations above 0.7, and $p<0.002$. The selected features converge on compartment-specific white-matter metrics such as axonal water fraction and intra-axonal diffusivity, which the paper reads as evidence that axonal microstructure underlies working-memory performance. If correct, the method would give clinicians a quantitative, interpretable way to tie cognitive complaints in mTBI to specific tissue injury and to track recovery.","feed_headline":"Crossover search predicts working memory from diffusion MRI","feed_subtitle":"The crossover jump helps best-first search beat greedy forward, backward, and genetic feature selection.","key_machinery":"The carrying object is the wrapper feature-selection problem cast as a graph search over the boolean hypercube: each vertex is a binary vector indicating which of the 280 features are selected, edges flip one bit, and the edge weight is the difference in cross-validated $r^2$ between adjacent subsets. The paper's addition is a crossover operator that, after expanding the best node, forms the arithmetic combination $V_{\\mathrm{cross}} = V_{\\mathrm{first}} + V_{\\mathrm{second}} - V_{\\mathrm{parent}}$ from the two best children and inserts it into the priority queue, letting the search jump two bits at a time (merge, replace, or skip-up) rather than step through every intermediate child. Gradient boosting trees provide the regression scores, and repeated stratified 5-fold cross-validation scores each subset.","core_discovery":"On its own terms, the paper establishes that modifying greedy best-first search with a genetic-algorithm crossover operator improves cross-validated prediction of working-memory subtest scores from diffusion MRI features. Across nine models (three tests times control, mTBI, and combined cohorts), BFS with crossover yields the highest $r^2$ in every case, improving the DSB mTBI result from 0.5193 to 0.6005, and it finds feature subsets whose selected diffusion metrics are predominantly compartment-specific measures of axonal and extra-axonal microstructure. The paper concludes that optimizing feature selection over a large space with this heuristic reveals imaging features that relate to cognitive function and injury in mTBI.","pith_inferences":["A nested cross-validation replication—where the search runs only on training folds—would test whether the reported $r^2$ gains survive without leakage; the paper does not describe such a scheme.","The same crossover arithmetic could be applied to other subset-search problems, such as biomarker discovery in genomics or feature selection for classification, where the boolean hypercube structure is identical.","The feature-frequency table suggests a stability check: rerunning the search on bootstrap samples and measuring overlap of selected feature sets would show whether the chosen metrics are reproducible or specific to this cohort.","Because pooled models underperform separate models, a natural next step is an interaction model (cohort x microstructural metric) rather than a single pooled predictor."],"forward_implications":["If the improvement holds, the BFS-with-crossover search is a drop-in replacement for greedy feature selection in any regression or classification problem with a large feature space and limited samples.","The frequently selected metrics (AWF, DA, De-par, De-perp) implicate axonal and extra-axonal white-matter microstructure as the diffusional signature of working-memory function in both healthy and injured populations.","Separate models for controls and mTBI patients predict better than a pooled model, indicating that brain–cognition relationships differ between the two populations.","The graph-search perspective gives a unified account of forward, backward, floating, genetic, and best-first selection, so algorithmic improvements like crossover transfer across those families."],"supporting_citations":[{"why":"Supplies the greedy best-first search (Greedy BFS) algorithm that the proposed crossover operator modifies.","marker":"[14]"},{"why":"Provides the genetic-algorithm crossover concept that the BFS-with-crossover operator adapts to graph search.","marker":"[23]"},{"why":"Provides gradient boosting, the regression model whose cross-validated $r^2$ scores drive feature subset evaluation.","marker":"[25]"},{"why":"Supplies the repeated stratified k-fold cross-validation scheme used to score each candidate feature subset.","marker":"[26]"},{"why":"Defines sequential floating feature selection, which the paper identifies as a special case of greedy BFS, anchoring the theoretical comparison.","marker":"[22]"},{"why":"Supplies the compartment-specific diffusion metrics (AWF, DA, De-par, De-perp) that form the study's feature set and ground the axonal-injury interpretation.","marker":"[5]"},{"why":"Links diffusion metrics to cognitive dysfunction in mTBI, motivating the choice of working-memory tests as prediction targets.","marker":"[7]"},{"why":"Earlier feature-selection application to mTBI classification that the paper positions its method against.","marker":"[8]"}],"fun_headline_variants":["Crossover-boosted search finds key MRI metrics for memory","Best-first search with crossover improves memory prediction","Crossover jump enhances diffusion MRI-based memory prediction","Genetic crossover in feature selection finds memory-linked MRI metrics","Crossover-augmented feature selection ties MRI to working memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported $r^2$ values assume the feature-selection search never sees the validation folds used to compute them; the paper does not describe the nested cross-validation that would guarantee this, so the gains may be optimistically biased.","fun_headline_variants_meta":{"raw":{"variants":["Crossover-boosted search finds key MRI metrics for memory","Best-first search with crossover improves memory prediction","Crossover jump enhances diffusion MRI-based memory prediction","Genetic crossover in feature selection finds memory-linked MRI metrics","Crossover-augmented feature selection ties MRI to working memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001123,"raw_usage":{"total_tokens":4614,"prompt_tokens":827,"completion_tokens":3787,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":3710}},"tokens_in":443,"tokens_out":3787,"duration_ms":27629,"temperature":1.0,"reasoning_tokens":3710,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:06:11.841879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BFS with crossover inside a nested cross-validation loop, re-running the search from scratch on each training fold and evaluating only on the held-out fold; if the averaged validation $r^2$ falls substantially below the reported 0.38–0.64, the improvement over greedy BFS is largely an artifact of feature-selection leakage.","supporting_citations":[{"cited_title":"Feature subset selection using a ge- netic algorithm,","cited_arxiv_id":null,"evidence_quote":"Provides the genetic-algorithm crossover concept that the BFS-with-crossover operator adapts to graph search."},{"cited_title":"Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation,","cited_arxiv_id":null,"evidence_quote":"Supplies the repeated stratified k-fold cross-validation scheme used to score each candidate feature subset."},{"cited_title":"Floating search methods in feature selection,","cited_arxiv_id":null,"evidence_quote":"Defines sequential floating feature selection, which the paper identifies as a special case of greedy BFS, anchoring the theoretical comparison."},{"cited_title":"White matter tract integrity: an indicator of axonal pathology after mild traumatic brain injury,","cited_arxiv_id":null,"evidence_quote":"Supplies the compartment-specific diffusion metrics (AWF, DA, De-par, De-perp) that form the study's feature set and ground the axonal-injury interpretation."},{"cited_title":"Short-term dti predictors of cognitive dysfunction in mild traumatic brain injury,","cited_arxiv_id":null,"evidence_quote":"Links diffusion metrics to cognitive dysfunction in mTBI, motivating the choice of working-memory tests as prediction targets."},{"cited_title":"Classification algorithms using multiple mri features in mild trau- matic brain injury,","cited_arxiv_id":null,"evidence_quote":"Earlier feature-selection application to mTBI classification that the paper positions its method against."}],"review_version":1}