{"id":"d2d667a6-d0c8-40fc-bd2d-2cc7941b3408","arxiv_id":"2608.12185","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-task deep learning system trained on 49,246 brain MRIs with 17 clinical labels produces transferable brain features that improve downstream prediction accuracy and sample efficiency.","lead":"GenFAR trains a modular set of deep learning models on 49,246 brain MRIs, using 17 clinical tasks to learn general brain features. The paper reports that these features transfer to new tasks and improve accuracy and sample efficiency compared with training directly on MRIs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LTO transfer numbers are inflated by model selection: donor-score and sequence-length analyses used the very tasks later held out, so the held-out tasks are not truly unseen.","rationale":"The reader identified the same load-bearing concern: the fixed base sequence, sequence length, and donor exclusions were selected using all 17 tasks, including the tasks later held out in the LTO evaluation. This is a form of task-level selection leakage: the held-out task's labels influence the architecture even though no held-out subject is used in training. The concern is concrete and specifically affects Table 2, which is the main evidence for the strongest claim that sequential learned features are comparable or superior to direct training in 16 of 17 unseen tasks. The external MESA evaluation provides some independent support, but it covers only four tasks and still inherits the same donor-selected architecture, so it does not fully resolve the concern. The paper's own report that optimization-based sequence search overfit the validation set underscores how sensitive sequential configuration is to selection; using the same tasks for selection and evaluation risks optimistic estimates. I therefore agree with the CONDITIONAL verdict and recommend no change: the internal LTO numbers should not be taken at face value until a genuinely untouched set of tasks is evaluated, but the contribution is likely partially real and the external evidence justifies a conditional rather than a rejection verdict.","tokens_in":22707,"tokens_out":8005,"duration_ms":74797,"concrete_test":"Select a random subset of 5 tasks and remove them entirely from all model-selection analyses: do not include their labels in the 5,000-sequence length comparison, the Donor Score computation, or the donor-exclusion decision. With the architecture fixed from the remaining 12 tasks, retrain the sequential GenFAR and run LTO on the 5 untouched tasks using the same secondary-encoder protocol and direct baselines as Table 2. If the untouched tasks still show sequential-feature gains comparable to Table 2 (e.g., in 4 of 5 tasks), the circularity concern is refuted; if gains shrink, reverse, or concentrate in tasks related to the donor set, the headline claim should be weakened to exclude the internal LTO numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The leave-task-out evaluation is not a clean test of transfer to unseen tasks because the architecture evaluated on each held-out task was selected using that task's labels. The Donor Score analysis (Methods, 'Donor Score'; Results, 'Task ordering') computed each task's average benefit as a donor across all 17 tasks and then chose Age, AD/MCI, MMSE, Hypertension, and Hyperlipidemia as the fixed base; the sequence-length analysis (Results, Figure 2; Methods, 'Sequence length selection') compared 5,000 random sequences using all tasks. When a task is later 'held out', the base tasks, their order, the length-6/slot-in architecture, and the exclusion of Total-Tau and Smoking have all been chosen partly because they improved that very task's performance in the sequence experiments. The LTO sampling strategy prevents subject-level leakage (Methods, 'Leave-Task-Out Cross-Validation'), but it does not prevent task-label leakage through model selection. Hence Table 2's 16-of-17 claim measures transfer plus favorable selection, so the held-out tasks are not genuinely unseen. The MESA results (Table 3) are less affected but still use the same donor-selected architecture, and only four tasks are evaluated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GenFAR trains modular 3D SE-ResNet channels on 49,246 T1-weighted MRIs from 11 cohorts using 17 classification and regression tasks (age, AD/MCI, MMSE, hypertension, hyperlipidemia, smoking, BMI, CSF biomarkers, and others). Two frameworks are compared: an independent set of 17 parallel channels and a sequential framework in which each task concatenates the frozen feature outputs of preceding tasks. From 5,000 random task sequences, the authors select sequence length six and a fixed base of five donor tasks via a proposed Donor Score, exclude two negative-donor tasks, and slot the remaining tasks as a 64-dimensional sixth task; the frozen channels then feed lightweight secondary predictors. The paper reports that sequential features beat or match direct training on 16 of 17 held-out tasks in leave-task-out (LTO) evaluation, large sample-efficiency gains as secondary-training data shrink to 100 subjects, and consistent gains on four tasks in the external MESA cohort. Model weights and CPU/GPU inference pipelines are publicly released.","tokens_in":22876,"tokens_out":17289,"duration_ms":145113,"significance":"If the claims hold, GenFAR is a useful community resource: a publicly released, clinically anchored brain-MRI feature extractor trained at unusually large scale (49,246 subjects, 11 cohorts, 17 diverse endpoints), with concrete evidence of low-data transfer gains and a genuinely external cross-cohort validation in MESA. The machine-checkable reproducibility elements — open weights, inference pipelines, and the web portal — are real strengths, as is the honesty of reporting channels that do not transfer well (e.g., BNT) rather than selecting only favorable tasks. The consistency of the transfer advantage across most tasks and its persistence in the external cohort make the central qualitative finding credible. However, the quantitative headline (16 of 17 tasks; 'unseen' tasks) is inflated by the model-selection protocol, because the donor sequence and sequence length were chosen using the same tasks that are later held out; the magnitude of the transfer benefit therefore needs re-estimation under a nested or pre-registered selection scheme.","major_comments":[{"comment":"The leave-task-out evaluation does not produce genuinely unseen tasks, because the architecture evaluated on each held-out task was selected using that task's labels. The final configuration — the five-task base (Age, AD/MCI, MMSE, Hypertension, Hyperlipidemia), the length-6 design with a 64-dimensional slotted sixth task, the exclusion of Total-Tau and Smoking, and the choice of which tasks provide features — was derived from the same 5,000-sequence transfer experiments whose per-task results constitute the LTO outcome (Results 'Task ordering'; Methods 'Donor Score'). The sequence-length analysis (Methods 'Sequence length selection'; Figure 2) likewise compared model test performance across all tasks before any task was 'held out.' When task r is later held out, the model evaluated on r was chosen in part by optimizing the average transfer performance across all receivers, including r. This is not equivalence-by-definition, and the large, consistent gains in Table 2 indicate that genuine transfer exists, but the reported numbers measure transfer plus favorable selection, so the 16-of-17 claim is optimistic and the held-out tasks are not truly unseen. The manuscript itself notes that optimization-based sequence search overfit the validation set, yet the hand-picked donor and length selection from the same experiments is subject to the same bias in milder form. The MESA evaluation (Table 3) is a valid cross-cohort test because the data are external, but its four tasks also participated in the selection analysis, so it does not validate task-unseen transfer. Please restructure the evaluation so that donor set, order, and length are selected on a development subset disjoint from the transfer-test tasks (or pre-registered), or disclose the selection-dependence and provide a sensitivity analysis with architectures chosen without each held-out task.","section":"Methods (Donor Score, Sequence length selection, LTO-CV); Results (Task ordering); Table 2"},{"comment":"The statistical significance analysis underlying the donor ranking is not valid as presented. The Welch t-tests compare mean performance across thousands of random task sequences that are far from independent: any two sequences of the same length share most of their tasks, so the effective number of comparisons is far smaller than the number of sequences, and the FDR-corrected p-values in Figure 3 are anti-conservative through pseudo-replication. In addition, the Donor Score pools sequences of lengths 3-9 even though Figure 2 shows strong length-dependent variation in test performance; since a donor's inclusion probability increases with sequence length, sequences containing a given donor are enriched for longer, generally worse-performing sequences, shifting the absolute scores and making the -0.1 threshold for 'negative donors' (Total-Tau, Smoking) hard to interpret without conditioning on length. A permutation test over task orderings, or a mixed-effects model with sequence length as a covariate and sequence as a random effect, would be more appropriate; at minimum the analysis should show that the donor ranking is stable when conditioning on sequence length.","section":"Methods 'Donor Score'; Figure 3"},{"comment":"The sample-efficiency evaluation, one of the paper's central claims, is under-specified. The text says the analysis began with 4,000 subjects and decreased to 100, but the Methods do not state which tasks contribute to the aggregate curves in Figure 4, how many random subsample replicates are performed at each size, whether the overlap-avoidance sampling of the LTO protocol is applied to the secondary-training subjects, or whether confidence intervals or significance tests support the claim that sequential features are superior 'across all sample size ranges.' The exact crossover thresholds (e.g., Age below 2,000 samples; Abeta at 1,000; BNT and Total-Tau at 500) should be backed by per-task curves with variability estimates. Please add the full protocol and per-task or error-barred results.","section":"Results (sample-efficiency subsection); Figure 4"}],"minor_comments":[{"comment":"The reported sequence lengths are inconsistent: the Results describe 5,000 sequences of varying lengths 3-16 and claim superiority over 'Length 1,' while the Methods state that models were trained on lengths 3-9 with additional samples for 12 and 16, and Figure 2's caption lists lengths 1, 3, 6, 9, 12, and 16. Please reconcile these descriptions.","section":"Results; Methods 'Sequence length selection'; Figure 2 caption"},{"comment":"The '16 of 17 comparable or superior' count treats correlation differences below 0.03 as 'comparable,' but several such cases (e.g., Digit Span Forward: 0.413 vs 0.406 with MAE 1.31 vs 1.27) are statistical ties; the headline should be reported together with per-task effect sizes rather than as a binary count.","section":"Results; Table 2"},{"comment":"The abstract's '49,246 individuals across 11 cohorts' and the Discussion's '50,302 T1 brain MRIs across 12 studies' refer to different sets (primary training set vs. including the 1,056 MESA evaluation subjects); this should be clarified to avoid an apparent inconsistency.","section":"Abstract; Discussion"},{"comment":"The paper positions GenFAR against BrainIAC, Triad, SimCLR-based models, and BrainAge but provides no empirical comparison with any of them; adding at least one such comparison (e.g., on the MESA tasks) would considerably strengthen the transfer claims.","section":"Introduction; Discussion"},{"comment":"The phrase 'various tasks beyond those included in the training set' is misleading because all 17 tasks participate in the sequence-length and Donor Score analyses; recommend wording such as 'tasks not used in feature training.'","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the central transfer result is probably real, but the framing of LTO as a test on 'unseen' tasks is the weakest point and will draw immediate criticism; the revision should either restructure the evaluation or clearly disclose the selection dependence. The paper is a strong resource contribution (scale, public release, external validation) and fits a medical-imaging or applied ML venue; for a cs.CV venue, reviewers may also expect comparison with self-supervised foundation models, which is currently missing. I also note that the Donor Score analysis is used to make exclusion decisions whose statistical basis is fragile (pseudo-replication); the authors should treat the excluded tasks as hypotheses to be confirmed on external tasks rather than as settled facts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is worth serious refereeing, not because the headline LTO claim is clean, but because the underlying resource and the sequential mechanism are genuinely useful. The real contribution is a supervised multi-task T1-MRI feature extractor trained on ~49k scans across 11 cohorts and 17 clinical endpoints, with code and weights released. The Donor Score idea—quantifying a task's average marginal benefit to downstream tasks from 5,000 random sequences—is a practical addition, and the sequential architecture (later tasks concatenate previous features) is a reasonable way to let weak tasks borrow structure. The external MESA evaluation, though only four tasks, gives independent evidence that the features generalize across studies and help in low-data settings. That is real value.\n\nNow the soft spot, and it is load-bearing for the headline number. The leave-task-out protocol is supposed to make each task 'unseen.' But the five-task donor base and the length-6 architecture were chosen from the same 5,000-sequence experiments that covered all 17 tasks, including the ones later held out. The Donor Score for each task is computed as average benefit across all receiver tasks; then the fixed sequence is built from the tasks that most often helped. When a task is later left out, the base sequence, its order, and the slot-in architecture were selected partly because they improved that task in the sequence experiments. So Table 2's 16-of-17 is transfer plus favorable selection, not a clean prediction. The subject-level split prevents sample leakage, but not task-label leakage through model selection. This does not collapse the paper, because MESA is genuinely external and the gains are consistent, but the numbers will shrink under a pre-specified design. The BNT deficit and modest losses on Age, Abeta, and Total-Tau are honest counterexamples, and the paper reports them.\n\nOne more minor note: the Donor Score stats use Welch's t-tests over sequences and FDR; fine as a descriptive tool, but those significance asterisks should not be read as causal evidence about task relationships.\n\nWho is this for? Neuroimaging labs that want a ready-made feature extractor for T1 MRI and do not have large local training sets. I would use the released model myself. I would send it to peer review: the contribution deserves referee time, and the circular selection point is fixable with a truly held-out task family or a pre-registered donor set. Just ask the authors to report both the selected and a conservative variant.","headline":"A useful, openly released neuroimaging feature extractor with real transfer gains, but the headline leave-task-out numbers are inflated because the same tasks' labels were used to select the donor sequence.","tokens_in":23616,"tokens_out":2466,"would_cite":true,"duration_ms":23754,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modular network trained on 17 tasks and 49,246 brain MRIs learns a representation whose features match or beat directly trained models on 16 of 17 held-out tasks.","keywords":["brain MRI","foundation model","transfer learning","multi-task learning","sequential learning","sample efficiency","leave-task-out evaluation","neuroimaging"],"falsifier":"Repeat the leave-task-out evaluation with the architecture chosen without any use of the held-out tasks: fix the five base tasks and the six-task sequence length by Donor Score analysis on a development set of tasks, then evaluate on a disjoint set of tasks whose labels never entered any design decision. If sequential features then beat direct training on only a minority of the untouched tasks, the 16-of-17 result depends on the selection step rather than on transfer; the released model and pipeline make this experiment runnable.","tokens_in":22449,"feed_emoji":"🧠","tokens_out":17606,"duration_ms":133690,"temperature":0.7,"pith_summary":"This paper sets out to show that a single modular network trained on many clinical prediction tasks at once can learn a general representation of brain structure that transfers to new tasks. The authors train GenFAR on 49,246 T1-weighted MRIs from 11 cohorts across 17 classification and regression tasks, then freeze the learned features and attach lightweight predictors for tasks the network never trained on directly. In leave-task-out evaluation, the sequentially learned features matched or beat models trained directly on the full 3D images in 16 of 17 tasks, with the clearest gains in classification and in low-data settings. The claim matters because most neuroimaging studies are small; if features learned from large, diverse cohorts really transfer this well, small studies could build accurate predictors on frozen features instead of training from scratch.","feed_headline":"Shared brain features beat task-specific models on 16 of 17 tasks","feed_subtitle":"Features from 49,246 MRIs match direct training on new tasks, and win most clearly when data are scarce.","key_machinery":"The load-bearing mechanism is sequential feature chaining. Each task channel is a 3D Squeeze-and-Excitation ResNet (a 3D convolutional network whose feature maps are recalibrated channel-wise), and in the sequential framework later channels take the concatenated 512- or 64-dimensional outputs of earlier channels as extra inputs, so knowledge flows forward without retraining earlier channels. Two design choices make the chain work: sequence length, set to six because analysis of 5,000 random sequences showed longer chains accumulate noise and overfit, and task order, set by the Donor Score, which compares the average performance improvement a receiver task gets from sequences that contain a candidate donor with sequences that do not. The five strong donors (age, AD/MCI, MMSE, hypertension, hyperlipidemia) form the base sequence, with each remaining task slotted into the sixth position, and this structure is what lets a frozen 3,200-feature representation serve as the foundation for secondary predictors.","core_discovery":"GenFAR's central claim is that clinically supervised multi-task learning over many endpoints yields a general brain representation. The network is modular: each of 17 prediction channels is a 3D Squeeze-and-Excitation ResNet — a 3D convolutional network with channel-wise recalibration — followed by a task-specific head, and the channels' outputs are concatenated into a shared feature vector (8,704 features in the independent framework, 3,200 in the sequential framework). In the sequential framework, every task after the first receives both the brain scan and the concatenated features of all preceding tasks, so later tasks bootstrap from stronger earlier ones. The authors analysed 5,000 random task sequences, found that six-task chains transfer best while longer chains overfit, and introduced a Donor Score, $\\operatorname{DonorScore}(d,r)=\\mathrm{avg}[P(r\\mid s):d\\in s]-\\mathrm{avg}[P(r\\mid s):d\\notin s]$, measuring how much a donor task's presence in a sequence raises a receiver task's performance; five tasks (age, AD/MCI, MMSE, hypertension, hyperlipidemia) emerged as consistently strong donors and form the fixed base of the final model. Freezing those channels and training a small head on a held-out task matched or beat full-image direct training in 16 of 17 tasks (the Boston Naming Test was the sole exception), and beat direct training on all four tasks tested on a separate external cohort.","pith_inferences":["The paper's leave-task-out protocol withholds a held-out task's labels from the prediction channels, but those same labels entered the Donor Score and sequence-length analyses that fixed the base sequence; a stricter test would exclude held-out tasks from every design decision, and the 16-of-17 result could shrink under it.","The finding that age and AD/MCI are the strongest donors suggests that tasks tied to broad, whole-brain structural change are the best sources of general features; the same donor-score machinery could be used to pick anchor tasks for other modalities, such as diffusion or functional MRI.","Because the paper's closest self-supervised counterparts learn robust, acquisition-invariant features without labels, combining a contrastively pre-trained encoder with GenFAR's clinically anchored supervision is a natural untested next step that might improve both transfer and robustness."],"forward_implications":["Frozen GenFAR features can stand in for full 3D image training on a new task: secondary predictors built on them matched or beat direct training in 16 of 17 leave-task-out experiments and on all four external-cohort tasks.","The advantage is largest where data are scarce: sequential features overtook direct training below roughly 2,000 training samples even for tasks where direct training won at full sample size (age, the CSF biomarkers, and the Boston Naming Test).","Which tasks are learned together matters: excluding negative-donor tasks (smoking, total-tau CSF) and fixing the chain at six tasks avoided the overfitting seen in longer sequences.","The released model and inference pipeline let outside researchers extract the same features from their own scans, so the transfer results can be checked on any cohort without retraining the foundation."],"supporting_citations":[{"why":"It supplies the sequential ensemble-growing scheme the paper adapts, where later models receive earlier models' outputs as inputs.","marker":"[29]"},{"why":"It provides the representation-learning rationale that informative feature spaces improve sample efficiency and transfer to new tasks.","marker":"[10]"},{"why":"It is the closest prior generalized brain-MRI feature extractor using contrastive self-supervision, and serves as the reference point for low-data transfer claims.","marker":"[21]"},{"why":"It shows that a single clinical endpoint (age) yields transferable features, the proof of concept that the paper extends to many endpoints.","marker":"[23]"},{"why":"It is the prior multi-phenotype framework that reuses task outputs as features for new targets, the main comparison for sharing intermediate representations.","marker":"[20]"},{"why":"It provides the external cohort, unused in training, on which cross-study generalization of the features is tested.","marker":"[30]"},{"why":"It defines the squeeze-and-excitation convolutional block used in every task channel of the architecture.","marker":"[53]"}],"fun_headline_variants":["49,246 MRIs train one brain model that beats task-specific nets","Multi-task MRI model wins on 16 of 17 new tasks, boosts small data","Five donor tasks power a general brain representation from 49k MRIs","GenFAR: one model, 17 tasks, 49k brains - beats direct training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The leave-task-out comparison is only a fair test of pure transfer if the held-out tasks' labels had no influence on the design, but those same labels were used to pick the five base tasks and the six-task sequence length, so the held-out tasks were not fully unseen when the architecture was fixed.","fun_headline_variants_meta":{"raw":{"variants":["49,246 MRIs train one brain model that beats task-specific nets","Multi-task MRI model wins on 16 of 17 new tasks, boosts small data","Five donor tasks power a general brain representation from 49k MRIs","GenFAR: one model, 17 tasks, 49k brains - beats direct training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000849,"raw_usage":{"total_tokens":3762,"prompt_tokens":1082,"completion_tokens":2680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":2593}},"tokens_in":698,"tokens_out":2680,"duration_ms":20289,"temperature":1.0,"reasoning_tokens":2593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:13:14.014644+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the leave-task-out evaluation with the architecture chosen without any use of the held-out tasks: fix the five base tasks and the six-task sequence length by Donor Score analysis on a development set of tasks, then evaluate on a disjoint set of tasks whose labels never entered any design decision. If sequential features then beat direct training on only a minority of the untouched tasks, the 16-of-17 result depends on the selection step rather than on transfer; the released model and pipeline make this experiment runnable.","supporting_citations":[{"cited_title":"Representation learning: A review and new perspectives.IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013","cited_arxiv_id":null,"evidence_quote":"It provides the representation-learning rationale that informative feature spaces improve sample efficiency and transfer to new tasks."},{"cited_title":"Brainiac: A foundation model for generalized brain mri analysis.medRxiv,","cited_arxiv_id":null,"evidence_quote":"It is the closest prior generalized brain-MRI feature extractor using contrastive self-supervision, and serves as the reference point for low-data transfer claims."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It shows that a single clinical endpoint (age) yields transferable features, the proof of concept that the paper extends to many endpoints."},{"cited_title":"Meta-matching as a simple framework to translate phenotypic predictive models from big to small data.Nature Neuroscience, 25(6):795–804, 2022","cited_arxiv_id":null,"evidence_quote":"It is the prior multi-phenotype framework that reuses task outputs as features for new targets, the main comparison for sharing intermediate representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the external cohort, unused in training, on which cross-study generalization of the features is tested."},{"cited_title":"Squeeze-and-excitation networks","cited_arxiv_id":null,"evidence_quote":"It defines the squeeze-and-excitation convolutional block used in every task channel of the architecture."}],"review_version":1}