{"id":"4949abcb-f149-4bd8-9b1f-1aa01fd32b23","arxiv_id":"2506.14176","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A one-shot NAS method with similarity-constrained initialization and MMD domain adaptation reports slightly higher F1 than SPOS and random search on BRACS breast histology images.","lead":"This paper adds a diversity-preserving initialization and a domain-adaptation loss to one-shot neural architecture search for breast pathology images. On the BRACS dataset the best found network edges out prior NAS baselines, though the gains are small and the comparison is not fully controlled.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Epoch-count confound invalidates the performance attribution: DAOS-A/B train 3000/4000 epochs vs SPOS's 2000, and Table 3's equal-time claim contradicts §4.2.2; no MMD-ablated control exists.","rationale":"The paper's central claim is that the proposed NSDI initialization and MMD-based domain adaptation yield superior classification performance and feature localization. The most load-bearing assumption underlying this claim is that any performance difference between DAOS variants and the SPOS/random baselines is caused by these components rather than by unequal training budgets or search protocols. This assumption is not tested. §4.2.2 specifies 2000 epochs for the SPOS baseline, then adds 1000 epochs of MMD fine-tuning for DAOS-A and another 1000 for DAOS-B; Table 3 nevertheless reports identical 48 GPU-hours for SPOS and DAOS-A. This internal inconsistency aside, Table 4 compares architectures whose supernets received different amounts of optimization, and DAOS also differs from SPOS in using NSDI for population initialization. No ablation removes the MMD loss while keeping epochs and initialization fixed, and no baseline extends SPOS to 3000 epochs. As a result, the best-F1 gap (61.41 vs 59.54) and the improved validation-test correlation (0.7096 vs 0.1794) could be artifacts of extra training rather than domain adaptation. The reader's concern about using the validation split as both adaptation target and selection set is real but secondary: it threatens the correlation metric's independence, while the epoch confound directly undermines the headline performance comparison. The proposed framework is modular and the CAM analysis is suggestive, but the central quantitative claim is currently unsubstantiated. An epoch-matched MMD-ablation is a minimal, decisive check.","tokens_in":11713,"tokens_out":5021,"duration_ms":52177,"concrete_test":"Run an epoch-matched ablation: train the SPOS supernet for 3000 epochs (classification loss only, with the LR schedule extended to match DAOS-A) and perform the same NSDI-based EA search and top-10 retraining; compare test F1 against DAOS-A. Also train DAOS-A for only 2000 epochs (i.e., omit the final 1000-epoch MMD fine-tuning). If the 3000-epoch SPOS matches or exceeds DAOS-A's 61.41% F1, or if DAOS-A without MMD achieves the same F1, the central performance claim is not supported. Additionally, audit Table 3 by recomputing GPU-hours from the epoch counts and batch sizes in §4.2.2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DAOS outperforms existing approaches is not supported because the comparison is not epoch-matched. §4.2.2 states the SPOS baseline trains for 2000 epochs, while DAOS-A adds 1000 epochs of MMD fine-tuning and DAOS-B adds 2000 (1000 MMD + 1000 frozen-classifier fine-tuning). Yet Table 3 lists identical 48 GPU-hours for SPOS and DAOS-A, contradicting the training schedule. Consequently, the F1 differences in Table 4 (61.41 vs 59.54 best; 59.28 vs 58.43 mean) could be entirely due to additional training (and to NSDI initialization, which is also not ablated) rather than to the proposed domain-adaptation loss. The paper reports no control consisting of SPOS trained for 3000 epochs, nor a version of DAOS-A without the MMD term. Without such an ablation, the superior classification performance and the validation–test correlation improvement cannot be attributed to the proposed mechanisms.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DAOS, a one-shot neural architecture search framework for breast cancer histopathology image classification. DAOS combines two ideas: a Network Similarity Directed Initialization (NSDI) strategy that constrains the average pairwise similarity of the initial evolutionary-search population, and a Maximum Mean Discrepancy (MMD) domain-adaptation loss added to the supernet training objective to align training and validation feature distributions. The authors evaluate on the BRACS dataset with a ShuffleNetV2-based search space and report that DAOS improves top-1 F1 score over SPOS and random search, improves validation-to-test ranking correlation, and yields more clinically relevant class activation maps. The central claims are that NSDI stabilizes search and that domain adaptation improves supernet evaluation reliability.","tokens_in":11970,"tokens_out":4141,"duration_ms":41613,"significance":"If the empirical claims were fully controlled, the contribution would be modest but useful: the combination of MMD-based domain adaptation with one-shot supernet training is a sensible idea for pathology data with stain variation, and the NSDI heuristic is a practical way to diversify the initial population in evolutionary NAS. The paper's strengths include the explicit formulation of the MMD objective in Eq. (10), the 10-seed search experiments in Table 4, and the direct comparison of validation-test correlation across training schemes in Figure 6. However, the paper does not release code or trained models, and the central performance claims currently rest on uncontrolled comparisons and on a validation split that is used both as the adaptation target and as the ranking set. As a result, the demonstrated significance is substantially weaker than the abstract claims.","major_comments":[{"comment":"The training schedules are not matched across methods. The text states that the SPOS baseline is trained for 2000 epochs, DAOS-A adds 1000 epochs of MMD fine-tuning, and DAOS-B adds a further 1000 epochs of frozen-classifier fine-tuning, yet Table 3 reports identical 48 GPU-hours for SPOS and DAOS-A. This internal contradiction, together with the absence of an epoch-matched SPOS control (e.g., SPOS trained for 3000 epochs without MMD) and a DAOS-A variant without the MMD term, means that the F1 differences in Table 4 (61.41 vs 59.54 best; 59.28 vs 58.43 mean) and the correlation gains in §4.3.2 cannot be attributed to the proposed domain-adaptation loss.","section":"§4.2.2 and Table 3"},{"comment":"The validation split is used as the unlabeled target distribution D_t during supernet training, and the same validation split is later used to rank architectures in the search (§4.2.3, Eq. (2)). The reported improvement in validation-test Pearson correlation from 0.1794 to 0.6985/0.7096 is therefore not an independent measure of ranking quality: the supernet has been explicitly trained to align its features to the validation split, making the validation metric a partially self-fulfilling ranking signal. The authors should either use a separate unlabeled set for the MMD target or report the correlation between validation-ranked and test-ranked architectures on an untouched holdout.","section":"§3.3, Eq. (10) and §4.3.2"},{"comment":"The statement that DAOS-A achieves the highest F1 score among all methods relies on Table 5, which reports no error bars or number of trials, while Table 4 reports 10-seed results whose mean and standard deviation overlap across methods (e.g., DAOS-A 59.28±0.97 vs SPOS 58.43±0.64; DAOS-B 59.35±0.73). Overlapping standard deviations do not establish a statistically significant improvement, and the 'Best' row is a best-of-seed statistic that inflates apparent gains. Please provide paired per-seed comparisons, confidence intervals, or significance tests, and report error bars in Table 5.","section":"§4.3.3, Table 5"},{"comment":"The contribution of NSDI is not isolated from the domain-adaptation changes. Table 2 compares EA(NSDI) with EA(RI) only on validation F1 (mean 68.89 vs 68.37, std 0.69 vs 0.82) with no significance test and no test-set follow-up, while the final comparisons in Table 4 change both the initialization method (RI→NSDI) and the supernet objective (CL→CL+DAL) simultaneously. The paper needs a 2×2 ablation (initialization method × objective) on the final test metric to support the separate claims that NSDI stabilizes search and that domain adaptation improves evaluation reliability.","section":"§4.3.1 and Table 2"}],"minor_comments":[{"comment":"Equation (9) uses the kernel function k(x, x') without specifying the kernel type or bandwidth; since this is a free parameter of the MMD estimator that directly enters Eq. (10), it should be reported.","section":"§3.3, Eq. (9)"},{"comment":"The text states that the search space contains '420 possible architectures'; this appears to be a typo for 4^20 (approximately 1.1×10^12), and the search-space size should be corrected.","section":"§4.2.1"},{"comment":"Tables 4 and 5 use inconsistent reporting conventions: Table 4 gives mean±std over 10 seeds, while Table 5 gives only a single F1 value with no indication of which run or how many runs were used; this should be clarified.","section":"Tables 4 and 5"},{"comment":"Repeated 'PRIME AI paper' headers appear between sections; if these are a typesetting artifact, they should be removed before publication.","section":"General formatting"},{"comment":"The red-box annotation highlighting under-represented operators is difficult to discern in a black-and-white rendering; please use color or add explicit layer labels to make the point visible.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is best suited to a specialist medical-imaging or AutoML venue; the single-dataset evaluation and the lack of controlled ablations would be a concern for a general computer-vision journal. I would not reject on novelty grounds, because MMD in one-shot NAS is a reasonable and underexplored combination, but the authors should be required to address the epoch-matching, validation-leakage, and statistical-significance issues before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this as a plausible modular extension of one-shot NAS, not a new framework. The genuinely interesting part is using an MMD loss during supernet training to align training and validation feature distributions, and then showing that validation–test correlation jumps from 0.18 to about 0.70. If that holds, it is a useful trick for NAS under distribution shift. The NSDI initialization is a minor diversity heuristic; it helps a bit in the search-stability experiments, but it is not the core contribution.\n\nThe paper is honest about its single-dataset scope, but the evidence for the central performance claim does not hold together. DAOS-A is trained for 3000 epochs (2000 base plus 1000 MMD fine-tuning) while SPOS gets 2000, and Table 3 lists identical 48 GPU-hours for both. That contradiction alone means the F1 gains in Table 4 cannot be attributed to the method. You need an epoch-matched SPOS baseline and an ablated version of DAOS-A without the MMD term before you can say anything about classification performance. Table 5 has no error bars, and the best-of-seed numbers overlap with the baseline anyway.\n\nI also have a methodological worry about the correlation claim. The validation set is used both as the unlabeled MMD target during supernet training and as the ranking set during search, so the improved validation–test Pearson r is not an independent measure of generalization. The test set is genuinely held out, so the correlation improvement is still suggestive, but it is weaker than it looks.\n\nThe claims of being 'first' to incorporate domain adaptation into one-shot NAS are likely overbroad, though I did not chase every citation. The paper releases no code, which makes the empirical results hard to check.\n\nWho should read this: researchers working on one-shot NAS robustness or cross-domain pathology classification. It is worth a serious referee, but only if the authors are pushed to fix the training schedule comparison, add the missing ablations, and get a second dataset. I would not cite it yet, but I would keep an eye on the correlation result if it survives a cleaner test.","headline":"Solid idea, undercut by epoch mismatch: the MMD alignment idea is worth testing, but the F1 claims are confounded.","tokens_in":12457,"tokens_out":1402,"would_cite":false,"duration_ms":19753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain adaptation added to one-shot neural architecture search makes validation scores predict test performance on breast pathology images.","keywords":["neural architecture search","one-shot NAS","domain adaptation","maximum mean discrepancy","computational pathology","breast cancer classification","evolutionary algorithm","population diversity"],"falsifier":"Retrain the DAOS-A supernet with the MMD loss but evaluate validation-to-test correlation on a held-out split that was never used as MMD target or for architecture selection; if the Pearson correlation drops back toward the 0.18 baseline, the central claim of improved ranking reliability fails.","tokens_in":11503,"feed_emoji":"🔬","tokens_out":6450,"duration_ms":63847,"temperature":0.7,"pith_summary":"This paper argues that one-shot neural architecture search (NAS) for pathological image classification is held back by two problems: evolutionary search starts from poorly diversified populations, and supernet rankings from training-set features do not transfer to the test distribution. It proposes DAOS, which adds a Network Similarity Directed Initialization (NSDI) that forces the initial architectures to be diverse, and a Maximum Mean Discrepancy (MMD) loss during supernet training that aligns labeled training features with unlabeled validation features. On the BRACS breast histology dataset, the method reports a best F1 of 61.41% versus 59.54% for the SPOS baseline, and raises the Pearson correlation between validation and test F1 from 0.1794 to roughly 0.70. If correct, this means that stable ranking in medical-image NAS does not require extra search cost, and that domain adaptation can be grafted onto existing one-shot pipelines.","feed_headline":"Domain-adapted NAS lifts breast pathology F1 to 61.4 percent","feed_subtitle":"Adding an MMD loss to supernet training raises validation-test correlation from 0.18 to 0.71 on BRACS.","key_machinery":"The load-bearing machinery is a pair of mechanisms in the DAOS pipeline. First, Network Similarity Directed Initialization (NSDI) encodes each candidate architecture as a binary vector, defines pairwise similarity $SS(v_i,v_j)=\\sum_{k=1}^{N} v_{i,k}\\odot v_{j,k}$ with $\\odot$ the XNOR operation, and constrains the Average Population Similarity to stay below a threshold $APS_{\\max}$ while sampling, with a timeout that relaxes the threshold if sampling stalls. This forces the initial population to cover the discrete search space instead of clustering. Second, the supernet training objective augments the classification cross-entropy with an MMD term $\\lambda\\sum_{\\gamma\\in\\{q,l\\}}\\hat{d}_{\\mathcal{H}}(p,\\gamma)$ over unlabeled validation features, using the unbiased kernel estimator of Equation (9). The MMD term is what aligns training and validation distributions, and it is this alignment that the paper credits for the improved validation–test correlation.","core_discovery":"The central claim is that injecting domain adaptation into one-shot NAS jointly fixes search stability and evaluation reliability on histopathology images. With NSDI, the initial population is chosen so that no architecture is too similar to another, measured by an XNOR overlap of their operation encodings, avoiding the unbalanced exploration that random initialization produces under small population budgets. With the MMD loss, the supernet is trained to produce features that are indistinguishable between the training and validation splits, so that validation F1 becomes a trustworthy predictor of test F1. The paper reports that DAOS-A achieves 61.41% F1 on BRACS, exceeding SPOS (59.54%) and the random-search baseline, and that the validation–test Pearson correlation rises from 0.1794 to 0.6985 (DAOS-A) and 0.7096 (DAOS-B). It further claims this is the first incorporation of domain adaptation into one-shot NAS and the first explicit quantification of population diversity during initialization.","pith_inferences":["Beyond the paper's own evidence, the same validation split serving as MMD target and architecture-selection set means the 0.71 correlation likely overstates how well DAOS predicts performance on a truly unseen site; a multi-site or multi-scanner evaluation would be needed to confirm the mechanism.","NSDI's diversity measure is defined on operation choices alone and ignores weight-sharing interactions between candidate subnets, so its benefit may shrink when supernet ranking noise dominates.","A natural stress test the paper does not run is replacing MMD with a stronger alignment method such as adversarial domain discrimination to see if the validation-test correlation improves further.","The t-SNE and CAM evidence suggests feature alignment is the active ingredient, predicting the method should transfer to other stain-variant histology tasks using the same search space."],"forward_implications":["If the ranking improvement is real, one-shot NAS becomes usable for histopathology without paying extra retraining cost, because the same supernet training budget (48 GPU-hours for DAOS-A) yields both better accuracy and stabler selection.","The MMD regularization should generalize to any one-shot NAS pipeline with a labeled source and unlabeled target split, making the method modular as the paper states.","The searched architectures stay within a roughly 1700M FLOP budget, so edge deployment of pathology classifiers becomes plausible.","Higher validation–test correlation means the top-ranked architecture chosen during search is more likely to remain the best after retraining, reducing wasted retraining runs."],"supporting_citations":[{"why":"Defines the one-shot ShuffleNetV2 search space and the SPOS supernet training baseline that DAOS extends.","marker":"[33]"},{"why":"Supplies the force-directed placement analogy that motivates the similarity-constrained initialization.","marker":"[34]"},{"why":"Provides the kernel two-sample test and unbiased MMD estimator used as the domain adaptation loss.","marker":"[35]"},{"why":"The BRACS dataset on which all experiments and comparisons are run.","marker":"[36]"},{"why":"Provides the regularized evolutionary search framework used to mutate and crossover candidates.","marker":"[14]"}],"fun_headline_variants":["Domain-adapted NAS lifts breast pathology F1 to 61.4%","One-shot NAS gains domain adaptation, 4x better search reliability","Pathology NAS: domain adaptation fixes search stability and trust","NSDI + MMD: stable one-shot NAS for breast cancer images","DAOS beats SPOS on BRACS with domain-adapted supernet"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the BRACS training and validation splits really do differ in staining and scale in a way that MMD alignment corrects, and that using the validation split both as the unlabeled adaptation target and as the selector of architectures does not make the reported correlation improvement a measure of the method's ability to predict unseen test performance independently.","fun_headline_variants_meta":{"raw":{"variants":["Domain-adapted NAS lifts breast pathology F1 to 61.4%","One-shot NAS gains domain adaptation, 4x better search reliability","Pathology NAS: domain adaptation fixes search stability and trust","NSDI + MMD: stable one-shot NAS for breast cancer images","DAOS beats SPOS on BRACS with domain-adapted supernet"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3164,"prompt_tokens":861,"completion_tokens":2303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":2210}},"tokens_in":477,"tokens_out":2303,"duration_ms":17765,"temperature":1.0,"reasoning_tokens":2210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:18:51.944215+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the DAOS-A supernet with the MMD loss but evaluate validation-to-test correlation on a held-out split that was never used as MMD target or for architecture selection; if the Pearson correlation drops back toward the 0.18 baseline, the central claim of improved ranking reliability fails.","supporting_citations":[{"cited_title":"Single path one-shot neural architecture search with uniform sampling","cited_arxiv_id":null,"evidence_quote":"Defines the one-shot ShuffleNetV2 search space and the SPOS supernet training baseline that DAOS extends."},{"cited_title":"Graph drawing by force-directed placement","cited_arxiv_id":null,"evidence_quote":"Supplies the force-directed placement analogy that motivates the similarity-constrained initialization."},{"cited_title":"Bracs: A dataset for breast carcinoma subtyping in h&e histology images","cited_arxiv_id":null,"evidence_quote":"The BRACS dataset on which all experiments and comparisons are run."},{"cited_title":"Regularized evolution for image classifier architecture search","cited_arxiv_id":null,"evidence_quote":"Provides the regularized evolutionary search framework used to mutate and crossover candidates."}],"review_version":1}