{"id":"73446805-b938-43e0-89b6-b4ce1aefb11c","arxiv_id":"2411.09618","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A benchmark challenge shows that voxel-wise MLP harmonization, RISH mapping, and NeSH resampling reduce cross-protocol bias in tractometry and connectomics, with the MLP requiring paired same-subject data.","lead":"This paper reports results from a competition where teams harmonized brain diffusion MRI scans from two acquisition protocols on the same scanner. The winning method, a voxel-wise machine learning correction, best reduced cross-acquisition bias in tractometry and connectomics, but it requires paired same-subject scans.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sub1's MLP may take acquisition B as input during inference (Sec 3.3.2), leaking target information and inflating its ICC; the paper never clarifies test-time procedure.","rationale":"The paper's central claim is that specific harmonization methods, especially a voxel-wise MLP, reduce cross-acquisition bias. The evaluation ranks submissions by average ICC across acquisitions. For this ranking to be meaningful, each harmonized acquisition must be produced independently from that acquisition's data. Section 3.3.2 describes Submission 1's MLP as taking concatenated data from both sites as input. If this description reflects inference, then the harmonized A is computed using B, which is the very target of comparison. This is a form of label leakage: the model can copy B, yielding artificially high ICC. The paper never clarifies that inference uses only A, nor does it report the output generation procedure in enough detail. The reader's weakest_assumption concerned pipeline validity and the sufficiency of average ICC; while related to evaluation quality, it did not identify this potential leakage, which is a more direct threat to the validity of the top result. If the concern is confirmed, the paper's main conclusion would be unsupported. A concrete check—reading the public code—can settle it. Until then, the appropriate verdict is conditional acceptance pending clarification. Even if the code shows A-only inference, this ambiguity should be fixed in the manuscript because it is essential to the central claim.","tokens_in":15522,"tokens_out":15821,"duration_ms":146628,"concrete_test":"Inspect the public repository (github.com/nancynewlin-masi/QuantConn) and determine the exact inference procedure for the MLP in Submission 1. Specifically, check whether producing the harmonized acquisition-A image for a test subject requires the corresponding acquisition-B image as input (e.g., via concatenation or as a conditioning signal). If it does, re-run the evaluation by generating harmonized A from acquisition A alone (feeding B as zero or omitting that channel) and recompute the cross-acquisition ICC. If the Conn ICC drops substantially below 0.98 (e.g., below 0.85), the top ranking is an artifact of leakage. Also run an ablation: compare outputs of the network with A-only vs A+B inputs on the same test subjects; if they differ significantly, the use of B at inference is material.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.3.2 states that for Submission 1 (The Harmonizers 1), the MLP input is 'the subject’s data from both sites were concatenated along the channel dimension per voxel for input.' If this concatenation is performed at inference, then the harmonized acquisition-A image for a test subject is a function of that same subject's acquisition-B data. The evaluation computes ICC with 'rater=acquisition,' requiring two independent measurements per subject. If the harmonized A is produced with access to B, the model can trivially reproduce B, making the reported Conn ICC of 0.98 an artifact of information leakage rather than evidence of harmonization. The paper nowhere states that inference uses only acquisition A; the phrase 'output format was chosen based on acquisition B' is compatible with a fusion of both acquisitions. This ambiguity is load-bearing because the paper's central claim that a voxel-wise MLP is the most effective harmonization method depends entirely on Sub1's ranking. If the method leaks B into the harmonized A, it is not a genuine cross-acquisition harmonization, and the comparison to other submissions is unfair. The CoV analysis for preserved biological variation also becomes meaningless because the output is partly derived from the target.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports the design and outcome of the MICCAI-CDMRI 2023 QuantConn challenge, in which 103 subjects (25 held-out test subjects) were scanned with two different diffusion MRI acquisitions on the same scanner. Nine submitted harmonization pipelines were evaluated by feeding harmonized data through a fixed downstream pipeline consisting of tensor fitting, CSD-based tractography, RecoBundlesX bundle segmentation, and an 84-node Desikan-Killiany parcellation, and then measuring cross-acquisition agreement with ICC (rater=acquisition) for 12 connectomics measures, 6 bundles x 4 microstructure measures, and 6 bundles x 6 macrostructure measures. Bias is further characterized with Cohen's d, Wilcoxon rank-sum tests, and coefficient of variation. The paper's central finding is that the top-performing method, Submission 1 ('The Harmonizers 1'), which combines PreQual preprocessing, cross-acquisition registration, and a voxel-wise MLP mapping, best reduces acquisition bias while preserving biological variation, with RISH mapping (Submission 2) and NeSH resampling (Submission 3) also effective.","tokens_in":15761,"tokens_out":5795,"duration_ms":59152,"significance":"If the reported ranking is valid, this is a valuable community resource and a meaningful step forward for diffusion MRI harmonization evaluation. The challenge releases a relatively large traveling-subject dataset (206 scans, 103 pairs), publicly shares code and data, uses a properly held-out test split, and is, to my knowledge, the first harmonization challenge to evaluate tractometry and connectomics as downstream outcomes. The paper also makes a concrete, falsifiable claim: that a voxel-wise MLP trained on co-registered same-subject pairs is the most effective harmonization method among those tested. The main strength is the clarity of the evaluation protocol and the availability of the challenge infrastructure. However, the validity of the headline result depends on a test-time input ambiguity in the winning method and on adequate disclosure of the relationship between the organizers and the submissions.","major_comments":[{"comment":"The description of Submission 1 is ambiguous about what data are used as input to the MLP at inference time. The text states that 'the subject's data from both sites were concatenated along the channel dimension per voxel for input' and that 'the output format was chosen based on acquisition B.' If this two-site concatenation is also performed for the 25 held-out test subjects, then the harmonized acquisition-A output for a subject is a function of that same subject's acquisition-B image. Because the primary evaluation computes ICC with 'rater=acquisition' to compare measurements from acquisitions A and B, a model that can access B during inference can trivially reproduce B and inflate the reported Conn ICC of 0.98. The CoV analysis in Section 4.4 would also be invalid under this reading, since the harmonized output would be partially derived from the target acquisition. The authors must state explicitly which inputs are used at test time. If both acquisitions are used, the evaluation must be rerun with an inference-time variant that uses only acquisition A, or the ranking must be revised to exclude Submission 1 as not a genuine cross-acquisition harmonization. The same ambiguity applies to Submission 7 in Section 3.3.8.","section":"Section 3.3.2, Section 4.1"},{"comment":"The significance testing used to support the claim that Submissions 1, 2, and 3 removed significant acquisition effects in all 12 connectomics measures relies on Wilcoxon rank-sum tests at p<0.05 without any correction for multiple comparisons. The paper reports many tests across 9 submissions, 12 network measures, 6 bundles, and multiple microstructure and macrostructure features, so dozens of false positives are expected at alpha=0.05. The authors should report the total number of tests performed, apply a false-discovery-rate or family-wise error control procedure, or explicitly justify why per-comparison inference is appropriate for the conclusions drawn. This issue is load-bearing because the claim that specific methods fully remove acquisition bias is stated categorically in the abstract and Section 6.","section":"Section 3.4, Section 4.1"},{"comment":"The manuscript does not disclose which submitted methods, if any, were developed by the challenge organizers. Submission 1, the winning method, is built from PreQual preprocessing and registration components closely associated with the organizing group, and the challenge organizers had full access to the training data and designed the downstream evaluation pipeline. This creates a material evaluation-design interdependence that should be transparently disclosed. The authors should state which submissions came from independent teams, whether any organizer-affiliated team competed, and provide a sensitivity analysis of the ranking when organizer-authored submissions are excluded. At present, the reader cannot assess whether the top ranking reflects a genuinely independent community result or an organizer-advantaged entry.","section":"Section 3.3, Section 6"},{"comment":"The central conclusion that a particular harmonization method is 'most effective' is conditioned on a single fixed downstream pipeline: WLS tensor fitting, constrained spherical deconvolution, 10-million-streamline tractography, RecoBundlesX segmentation, and the Desikan-Killiany parcellation. The paper does not test whether the rankings generalize to other tractography algorithms, bundle segmentation methods, or parcellation schemes, even though the stated goal is 'robust quantitative connectivity.' The conclusion in Section 6 should be explicitly qualified as holding for this pipeline, or the authors should validate the ranking with at least one alternative downstream pipeline before making the broader claim.","section":"Section 3.2, Section 6"}],"minor_comments":[{"comment":"The text for Submission 9 states that it 'used the same methods as Submission 1,' but Table 1 and Section 3.3.9 indicate that Submission 9 is a spherical-harmonic coefficient mapping method like Submission 8 with the target and reference reversed; this cross-reference is incorrect and should be fixed.","section":"Section 3.3.10"},{"comment":"The acquisition B description contains a typo: '23s cm FOV' should read '23 cm FOV.'","section":"Section 3.1.3"},{"comment":"The atlas name is spelled inconsistently as 'Desikan-Killany' and 'Deskian-Killany'; the correct spelling is 'Desikan-Killiany.' The reference for FreeSurfer also appears malformed.","section":"Section 3.2"},{"comment":"The sentence 'All but submission 1 failed to remove acquisition bias from all measures for bilateral arcuate fasciculus' is ambiguous: it is unclear whether Submission 1 removed bias for all measures in both arcuate fasciculi or only for the bilateral average. Please rephrase.","section":"Section 4.2"},{"comment":"The terms 'Wilcoxon ranksum test' and 'Mann-Whitney U-test' are used interchangeably; please use one consistent name throughout.","section":"Section 3.4"},{"comment":"Reporting ICC scores as mean +/- SD across measures conveys variability across measures rather than sampling uncertainty of the ranking. Please include bootstrap confidence intervals or explicitly point to the ChallengeR ranking-stability analysis in the main text.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue for me is the test-time input ambiguity for Submission 1. If the MLP uses both acquisitions at inference, the Conn ICC of 0.98 is not evidence of harmonization, and the entire ranking collapses. This is fixable by a clear statement and, if necessary, a re-evaluation, so I do not recommend rejection at this stage. The lack of disclosure about organizer participation in the winning submission is also a serious editorial concern; I would ask the handling editor to require a clear declaration of which teams were independent and which, if any, were affiliated with the organizing group."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The QuantConn challenge is the first harmonization benchmark to evaluate tractography-based bundles and connectomics, on 103 paired subjects—ten times the prior MUSHAC cohort. The dataset, code, and the analysis of which downstream measures are most acquisition-sensitive (surface area, FA, assortativity, etc.) are genuinely useful, and the held-out test split is clean. I'd point a student to this paper as a solid example of a challenge report.\n\nBut the headline result is shaky. Submission 1, the winning 'Harmonizers 1' method, is described as concatenating the subject's data from both acquisitions along the channel dimension per voxel. The paper never states what happens at test time. If the harmonized A image is generated with access to acquisition B, then the ICC with 'rater=acquisition' is not measuring reproducibility—it's measuring how well the model reproduces B from B. That would make the reported Conn ICC of 0.98 an artifact, not a harmonization result. The limitation paragraph mentions the method requires paired, co-registered scans, but that addresses training requirements, not test-time leakage. This is not a peripheral detail: the paper's central claim that a voxel-wise MLP is the most effective harmonizer depends entirely on this submission's ranking. The authors can fix this with one sentence in the methods, but as written the ambiguity is load-bearing.\n\nThere's also a governance issue: the winning submission appears to come from the organizers, and the paper does not disclose that interdependence. That is a fair concern for a challenge report, regardless of the method's actual merit.\n\nThe other soft spots are minor in context. The analysis runs many unadjusted statistical tests, but the ChallengeR ranking and per-measure breakdowns in the supplementary offer some protection. The evaluation uses one downstream pipeline, so the rankings are about harmonization under this pipeline, not harmonization in general—the authors mostly acknowledge this.\n\nOn balance, the benchmark is worth taking seriously, but the top submission's status is unresolved. A serious referee should ask for a clear test-time procedure for Submission 1 and a re-analysis that excludes it or includes a single-input variant. If the method is single-input at inference, this is a strong paper. If not, the ranking needs rework. Either way, the field benefits from the data and the benchmark itself.","headline":"A genuinely useful harmonization benchmark whose winning method has an unclarified test-time input—if it leaks acquisition B, the headline ranking doesn't hold.","tokens_in":16406,"tokens_out":3839,"would_cite":true,"duration_ms":35443,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":["87.61.-c"],"model":"deepseek-v4-flash","headline":"Voxel-wise MLP removes acquisition bias from brain connectivity measures","keywords":["Diffusion MRI","harmonization","tractometry","connectomics","tractography","multilayer perceptron","multi-site imaging","QuantConn challenge"],"falsifier":"Re-run all nine harmonization methods through a different but equally standard downstream pipeline—for example, a different tractography algorithm or a different cortical parcellation—and check whether the winning MLP method still removes all significant acquisition effects and keeps the same ranking. If the ranking changes or significant biases reappear under a plausible alternative pipeline, the paper's conclusion is specific to its evaluation pipeline rather than general.","tokens_in":15380,"feed_emoji":"🧠","tokens_out":9380,"duration_ms":80403,"temperature":0.7,"pith_summary":"This paper reports a challenge in which 103 people were scanned twice on the same scanner using two deliberately different diffusion-MRI protocols, and nine teams were asked to preprocess the raw data so that downstream measurements would no longer depend on which protocol was used. The central claim is that acquisition protocol biases tractometry and connectomics substantially, and that the bias can be largely removed by the right harmonization: the winning submission corrected distortions, registered the two acquisitions, and learned a voxel-wise signal mapping with a multilayer perceptron, reducing cross-acquisition differences to small effect sizes across all twelve complex network measures and across bundle microstructure and macrostructure. Rotation-invariant spherical-harmonic mapping and a neural implicit resampling method also performed well. If this holds, multi-site studies that pool diffusion-MRI data can trust their connectivity and white-matter measurements more, which matters because these measures are increasingly used as disease biomarkers.","feed_headline":"A learned voxel map erased scan-protocol bias in brain measures","feed_subtitle":"In a 103-subject paired-scan challenge, one machine-learning harmonization method beat eight others on connectivity and bundle measures.","key_machinery":"The central mechanism is a paired evaluation design with a fixed downstream pipeline. Each subject is scanned under both protocol A (anisotropic resolution, 27 directions) and protocol B (isotropic resolution, 94 directions); after each team's harmonization, all data pass through the same processing chain—tensor fitting, fiber orientation distribution estimation, whole-brain tractography, bundle segmentation, tractometry, and connectome construction with a standard cortical atlas—and the outputs are scored by intra-class correlation across acquisitions, Cohen's d effect sizes, and median comparisons. This makes reproducibility of the measurements researchers actually use the target quantity, rather than voxel-level signal similarity.","core_discovery":"The paper's claim, stated in its own terms, is that harmonization should be evaluated on downstream bundle and connectome measures, not just on voxel-level signal, and that when evaluated this way, voxel-wise machine-learning correction is the most effective strategy. In the un-harmonized reference data, the two protocols produced large and statistically significant differences in bundle surface area, fractional anisotropy, and eight of the twelve complex network measures—assortativity, betweenness centrality, edge count, modularity, nodal strength, participation coefficient, and related measures—while AD, MD, RD, bundle length, density, efficiency, and path length were comparatively stable. The top submission combined distortion correction, boundary-based registration of one acquisition onto the other, and an MLP trained to predict one acquisition's voxel signal from the other; it removed all significant acquisition effects in the twelve network measures and in the bundle measures studied, and it did not collapse inter-subject variation. Notably, two submissions that simply mapped spherical-harmonic coefficients across protocols introduced significant biases where the baseline had none, which the paper takes as evidence that naive coefficient mapping is not a safe harmonization strategy.","pith_inferences":["My inference: if the same paired-subject, downstream-scored evaluation were applied to multi-scanner and multi-site data, it could become a standard benchmark for harmonization methods, since the current experiment isolates protocol differences while holding scanner constant.","My inference: the winning method's requirement that the same subjects be scanned under both protocols and co-registered limits its use to retrospective traveling-subject cohorts; a natural extension is to test whether the learned voxel mapping transfers to new subjects scanned under only one protocol.","My inference: because the ranking averages intra-class correlation across many heterogeneous measures, a method could rank highly by succeeding on easy-to-harmonize measures while failing on a clinically important one; re-scoring with per-measure weights or pre-specified primary outcomes could change the winner."],"forward_implications":["With the winning MLP-based harmonization, connectome and tractometry measures from the two protocols become comparable, so data collected under different protocols on the same scanner can be pooled for analysis.","Bundle surface area, fractional anisotropy, assortativity, betweenness centrality, edge count, modularity, nodal strength, and participation coefficient should be treated as acquisition-sensitive in multi-site diffusion-MRI studies unless harmonization is applied.","AD, MD, RD, bundle length, density, efficiency, and path length are relatively stable across the two protocols and can be compared with less correction.","Methods that only map spherical-harmonic coefficients from one protocol to another can worsen cross-acquisition bias, so harmonization methods need to be validated on downstream measurements rather than on signal statistics alone."],"supporting_citations":[{"why":"Supplies the 103 same-subject two-acquisition diffusion-MRI pairs that the challenge harmonizes and evaluates.","marker":"Strike et al. (2023)"},{"why":"The PreQual pipeline used by the top submission and several others to remove motion, eddy-current, and EPI distortions.","marker":"Cai et al. (2021)"},{"why":"The RISH feature-mapping harmonization used by the second-ranked submission.","marker":"Karayumak et al. (2019)"},{"why":"The NeSH neural implicit representation used by the third-ranked submission to resample spatial and angular resolution.","marker":"Hendriks et al. (2023)"},{"why":"RecoBundlesX segmentation that yields the six bundles whose microstructure and macrostructure are evaluated.","marker":"Garyfallidis et al. (2018)"},{"why":"BUAN, the bundle analysis framework that produces tractometry profiles and shape-similarity scores.","marker":"Chandio et al. (2020)"},{"why":"The brain connectivity toolbox defining the twelve complex network measures used to score connectome harmonization.","marker":"Rubinov and Sporns (2010)"},{"why":"MRtrix3 tools used for tractography and preprocessing across submissions.","marker":"Tournier et al. (2019)"}],"fun_headline_variants":["AI harmonization erases scan-protocol bias in brain measures","Neural net beats 8 rivals in brain MRI harmonization test","ML mapping best for cross-protocol brain metrics, challenge finds","Harmonizing brain scans: ML wins, naive methods can backfire","Voxel-wise AI correction tops 8 methods for brain connectivity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the assumption that the fixed measurement pipeline—tensor fitting, tractography, bundle segmentation, and the cortical atlas—is itself accurate and unbiased across the two acquisitions, so that any remaining cross-acquisition differences can be blamed on the harmonization method.","fun_headline_variants_meta":{"raw":{"variants":["AI harmonization erases scan-protocol bias in brain measures","Neural net beats 8 rivals in brain MRI harmonization test","ML mapping best for cross-protocol brain metrics, challenge finds","Harmonizing brain scans: ML wins, naive methods can backfire","Voxel-wise AI correction tops 8 methods for brain connectivity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000756,"raw_usage":{"total_tokens":3434,"prompt_tokens":1094,"completion_tokens":2340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":2250}},"tokens_in":710,"tokens_out":2340,"duration_ms":15937,"temperature":1.0,"reasoning_tokens":2250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:26:35.678506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run all nine harmonization methods through a different but equally standard downstream pipeline—for example, a different tractography algorithm or a different cortical parcellation—and check whether the winning MLP method still removes all significant acquisition effects and keeps the same ranking. If the ranking changes or significant biases reappear under a plausible alternative pipeline, the paper's conclusion is specific to its evaluation pipeline rather than general.","supporting_citations":[{"cited_title":"Retrospective harmonization of multi-site diffusion mri data acquired with different acquisition parameters","cited_arxiv_id":null,"evidence_quote":"The RISH feature-mapping harmonization used by the second-ranked submission."},{"cited_title":"Neural spherical harmonics for structurally coherent continuous representation of diffusion mri signal","cited_arxiv_id":null,"evidence_quote":"The NeSH neural implicit representation used by the third-ranked submission to resample spatial and angular resolution."},{"cited_title":"Recognition of white matter bundles using local and global streamline-based registration and clustering","cited_arxiv_id":null,"evidence_quote":"RecoBundlesX segmentation that yields the six bundles whose microstructure and macrostructure are evaluated."},{"cited_title":"Bundle analytics, a computational framework for investigating the shapes and profiles of brain pathways across populations","cited_arxiv_id":null,"evidence_quote":"BUAN, the bundle analysis framework that produces tractometry profiles and shape-similarity scores."},{"cited_title":"Complex network measures of brain connectivity: uses and interpretations","cited_arxiv_id":null,"evidence_quote":"The brain connectivity toolbox defining the twelve complex network measures used to score connectome harmonization."}],"review_version":1}