{"id":"d9b21f12-1e36-44fe-a802-67956b4b2ba6","arxiv_id":"2607.26940","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Dual GAT surrogates estimating accuracy and latent diversity guide greedy selection of diverse high-performing neural architectures for ensembles on standard image benchmarks.","lead":"The paper trains two graph-network surrogates—one for accuracy, one for diversity via triplet loss on prediction agreement—to pick neural-network ensembles without training every candidate. It reports competitive or better accuracy than Deep Ensembles and random search on FashionMNIST, CIFAR-10 and CIFAR-100.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"The capacity/training-time transfer of the diversity latent space is the load-bearing untested assumption behind the greedy selection claim.","rationale":"The Reader correctly isolates the transfer assumption as the weakest link. Everything else in the pipeline (GAT accuracy regression, triplet construction on agreement matrix D, greedy max-average-distance selection) is standard and internally consistent; the empirical tables show small but real gains. Because the paper never closes the loop from cheap latent geometry to full-model disagreement, the causal attribution “diversity surrogate \to better ensembles” remains unproven. A single pairwise rank-correlation check on matched cheap/full pairs would settle it. No stronger internal contradiction or mathematical error is present, so the verdict stays CONDITIONAL rather than moving to REJECT; tightening that experiment (or showing the correlation holds) would be enough to support ACCEPT.","tokens_in":10958,"tokens_out":499,"duration_ms":10445,"concrete_test":"Hold out ~200 architecture pairs that appear in both the surrogate dataset and a fully-trained higher-capacity counterpart. Compute (i) latent Euclidean distance from the trained f_div and (ii) true prediction-disagreement (or 1-agreement) on the same validation set for both the cheap and the full models. Report Spearman rank correlation between cheap-latent distances and full-model disagreements; if it falls below ~0.3 or loses statistical significance, the transfer assumption fails and the diversity-guidance claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Table 4 superiority via dual-surrogate NES) rests on the diversity GAT producing Euclidean distances that still rank true predictive disagreement after the models are scaled from the surrogate regime (reduced cells/width, fixed partial epochs; Table 1 and §4.1) to the final ensemble regime (more cells, larger width, full training). The paper never measures this transfer: Spearman/Recall@K for the diversity surrogate (Table 2) and the component ablation (Table 3) are computed only inside the cheap regime, while final ensembles are selected by latent distance then evaluated only after full training. If the ranking of pairwise distances collapses or inverts under capacity/epoch scaling, the greedy step in Fig. 3 no longer optimizes the diversity that actually appears in the reported ensembles, and the modest gains over DeepEns become attributable to accuracy filtering alone rather than explicit diversity modeling.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a dual-objective surrogate-guided Neural Ensemble Search (NES) framework. Candidate architectures from a reduced DARTS-style cell space are encoded as DAGs and scored by two independently trained Graph Attention Network surrogates: an accuracy regressor (MSE) and a diversity embedder trained with triplet loss so that Euclidean distances reflect predictive disagreement on a validation set. A two-phase procedure (accuracy thresholding of a random candidate pool, then greedy farthest-point selection in the diversity latent space; Fig. 3) produces the final ensemble. The method is evaluated against Deep Ensembles (repeated training of one DARTS architecture) and Random Search on FashionMNIST, CIFAR-10, and CIFAR-100 (Table 4), with ablations on surrogate data size and component contribution (Tables 2–3). The authors also release ~3,000 trained architecture/weight pairs per dataset.","tokens_in":11221,"tokens_out":1455,"duration_ms":35188,"significance":"Explicit surrogate modeling of ensemble diversity via a metric-learned latent space is a clear and useful idea for NES, where joint architecture-and-composition search is otherwise intractable. The dual-surrogate separation, the GAT+triplet construction, the released model datasets, and the multi-metric evaluation (accuracy, NLL, calibration, disagreement, adversarial AUCs) are concrete contributions. If the claimed transfer of diversity structure from cheap partial models to full ensembles holds, the approach offers a practical route to diversity-aware NES without training every candidate. The empirical gains over DeepEns are modest and mixed across metrics, so the significance is incremental rather than transformative, but the direction and the public datasets are valuable to the community.","major_comments":[{"comment":"Section 4.1 and Table 1 vs. final ensembles: the diversity surrogate is trained exclusively on reduced-capacity, partially trained cells (n=5 nodes, fixed epochs, smaller width), while final ensembles use more cells, larger width, and fuller training. Table 2 Spearman/Recall@K and Table 3 ablations are computed only inside the cheap regime. The paper never reports whether latent Euclidean rankings still correlate with true predictive disagreement after capacity and epoch scaling. This transfer is the load-bearing assumption behind the claim that greedy selection in Fig. 3 optimizes the diversity that appears in Table 4. A direct measurement (e.g., Spearman between cheap-regime distances and full-model disagreement on a held-out set of scaled architectures) is needed; without it, the modest Top-1 gains could be explained by accuracy filtering alone.","section":"§4.1, Table 1, Table 2, Fig. 3"},{"comment":"Table 4 baseline construction: DeepEns is “a single architecture obtained via DARTS with different random initializations.” This confounds the comparison. The proposed method selects heterogeneous architectures; DeepEns does not. A fairer architecture-level baseline would be an ensemble of distinct high-accuracy architectures chosen without the diversity surrogate (or NES methods that already vary architecture). As written, superiority over DeepEns partly reflects architecture diversity per se rather than the specific dual-surrogate procedure. On CIFAR-100, DeepEns already matches or beats Surrogate Ens on NLL (0.682 vs 0.692), average model accuracy, Brier, and ECE, which weakens the “competitive or superior” claim when diversity is the selling point.","section":"§4.4, Table 4"},{"comment":"Fig. 3 Phase 2 selection rule: after the accuracy threshold λ and the single most accurate anchor, subsequent members are chosen solely by maximum average latent distance, with no further accuracy term. Combined with free parameters (λ, margin m, quantile thresholds for D, N1/N2 and top-10% refinement), it is unclear how sensitive Table 4 is to these choices. A short sensitivity study or a joint accuracy–diversity selection score would make the dual-objective claim more convincing and show that diversity is not purchased at uncontrolled accuracy cost.","section":"§3.3, Fig. 3"}],"minor_comments":[{"comment":"Eq. (3): the product formula for |A_cell| with n=5, m=10, |O|=7 yielding ~10^9 should be checked or expanded; the combinatorial accounting of “exactly two incoming and two outgoing edges” is not fully spelled out.","section":"§3.1, Eq. (3)"},{"comment":"Notation drift: f_θ_acc / f_acc and f_θ_div / f_div are used interchangeably; y_i is both prediction vector and (in places) label-like. Unify notation early in §3.2.","section":"§3.2"},{"comment":"Table 3 “Pred. Disagreement” for Accuracy+Diversity (0.411) is lower than Random (0.430) and Diversity-only (0.419), yet the text emphasizes diversity benefits. Briefly reconcile why lower disagreement can still help Top-1.","section":"§4.3, Table 3"},{"comment":"Fig. 1 caption and body refer to “surrogate diversity function” mapping similar predictive behavior close together; ensure the figure’s d1 ≤ d2 labeling matches the triplet inequality in Eq. (8).","section":"Fig. 1, Eq. (8)"},{"comment":"Related work: cite additional NES/diversity baselines beyond Zaidi et al. and Chen et al. (e.g., explicit diversity regularizers or disagreement-based ensemble selection) so the “one of the first” claim in contribution 2 is better scoped.","section":"§1"},{"comment":"Typos/consistency: “F unction” spacing in headings (§3.2, §4.2); “simularity” in appendix Fig. 10 caption; arXiv date “29 Jul 2026” looks like a placeholder.","section":"Throughout / Appendix"}],"recommendation":"major_revision","confidential_remarks":"The transfer gap identified by the stress test is real and should block acceptance until addressed; it is fixable with a targeted experiment and does not require redesigning the method. Gains over DeepEns are small enough that a stronger heterogeneous-architecture baseline could flip the narrative on CIFAR-100. The public 3k-model datasets are a genuine plus for the venue. Fit for a solid ML systems/NAS venue is reasonable after major revision; borderline for a top general ML journal without clearer gains and the transfer check."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this is a competent dual-surrogate NES pipeline (accuracy GAT + triplet-loss diversity GAT on prediction agreement) that ships ~3k trained architectures per dataset and gets small but real gains over DeepEns and random search on FashionMNIST/CIFAR-10/100. It does not invent the pieces—GAT predictors, FaceNet-style embeddings, greedy diversity selection, and NES itself are all in the cited work—but the joint system and the artifact are new enough to be useful.\n\nWhat it does well is practical and honest. The ablations (Tables 2–3) show both surrogates improve with data and that accuracy+diversity beats either alone. They report means±std across several metrics, not just Top-1. They flag the compute bottleneck and release the model pool so others can skip the three-day A100 burn. The math is standard (MSE + triplet loss, greedy max average Euclidean distance); no circular scoring of the final ensembles on surrogate outputs.\n\nThe soft spots are real but proportionate. DeepEns is one DARTS architecture with different seeds, so the comparison is somewhat friendly to any multi-architecture method. More important is the stress-test point: diversity structure is learned on reduced cells, partial epochs, and smaller width, then used to pick full-capacity models. Spearman/Recall for the diversity surrogate and the component ablation live only in the cheap regime; there is no direct check that latent rankings still track predictive disagreement after scaling. If they do not, the modest Table 4 edge is mostly accuracy filtering. That is a load-bearing untested assumption, not a fatal hole—the paper still shows a working pipeline—but it should be tightened before anyone treats the diversity story as settled.\n\nWho it is for: people already doing NAS/ensemble search who want cheaper diversity-aware selection and a ready evaluation pool. Not a must-read outside that niche. I would send it to peer review; it is formally grounded enough and evidentially sharp enough for referee time, with the transfer experiment and a fairer multi-architecture DeepEns as the obvious revision asks. Engage if you work in NES; otherwise skim the dataset release and move on.","headline":"Solid incremental NES systems paper with a useful released dataset; the diversity-transfer claim is the real soft spot, not the overall framing.","tokens_in":11835,"tokens_out":539,"would_cite":false,"duration_ms":17391,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Two surrogate models—one for accuracy, one for diversity—let neural ensemble search pick strong, complementary architectures without training every candidate.","keywords":["neural ensemble search","ensemble diversity","surrogate function","triplet loss","graph attention network","neural architecture search","DARTS"],"falsifier":"Retrain the final selected ensembles and a matched set of random high-accuracy architectures to full capacity, then check whether pairwise prediction disagreement still correlates with the surrogate latent distances; if the correlation collapses or the surrogate ensembles lose their accuracy edge, the transfer assumption fails.","tokens_in":11810,"feed_emoji":"🧠","tokens_out":771,"duration_ms":21111,"temperature":0.7,"pith_summary":"Building strong neural ensembles requires both accurate individual models and genuine diversity among them, yet jointly searching architectures and ensemble membership is exponentially harder than ordinary architecture search. This paper shows that the bottleneck can be broken by training two cheap graph-based surrogates on a modest pool of partially trained cells: one predicts validation accuracy, the other embeds architectures so that Euclidean distance tracks how differently the models predict. A simple greedy selection then keeps only high-accuracy candidates that sit far apart in that embedding. The resulting ensembles match or beat standard Deep Ensembles and random search on FashionMNIST, CIFAR-10 and CIFAR-100 while avoiding the cost of training thousands of full-scale networks during search. The practical payoff is a workable route to diversity-aware ensembles when exhaustive evaluation is impossible.","feed_headline":"Surrogates pick diverse neural ensembles without full training","feed_subtitle":"Accuracy and diversity predictors guide search that matches or beats Deep Ensembles on CIFAR","key_machinery":"The diversity surrogate: a graph-attention network that maps each architecture DAG into a continuous latent vector trained with triplet loss so that Euclidean distance approximates predictive disagreement; greedy selection then maximises average distance from already chosen members inside an accuracy-filtered pool.","core_discovery":"A dual-objective surrogate framework—accuracy regression plus a triplet-loss latent space built from prediction agreement—can guide neural ensemble search so that the selected set is both individually strong and collectively diverse, yielding competitive or superior Top-1 accuracy and negative log-likelihood versus Deep Ensembles and Random Search on three standard image benchmarks.","pith_inferences":["If latent distances transfer reliably, the same embedding could be reused for online ensemble pruning or continual learning without re-training the surrogate.","Replacing the binary agreement matrix with a continuous divergence (KL, total variation) might yield smoother gradients and better-calibrated diversity scores.","The accuracy-threshold-plus-greedy-distance recipe is architecture-agnostic and could be tried on transformer or NAS-Bench spaces with only a change of graph encoder."],"forward_implications":["Ensemble search can be performed largely training-free once a few thousand cheap proxy models have been evaluated.","Explicit diversity modelling becomes more valuable as task difficulty rises (CIFAR-100 gains exceed FashionMNIST).","Released proxy datasets of ~3000 trained cells per benchmark let others test new ensemble methods without repeating the expensive data-collection step.","The same dual-surrogate pattern can be swapped into other discrete search spaces that admit graph encodings."],"fun_headline_variants":["Dual surrogates estimate accuracy and diversity for NES","Accuracy plus triplet-loss surrogates guide ensemble search","Surrogate pair finds strong diverse nets without full trains","DAG surrogates cut cost of neural ensemble architecture search","Two surrogates yield ensembles matching Deep Ensembles on CIFAR"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The diversity geometry learned on small, only partially trained cells still ranks true predictive disagreement correctly once those cells are scaled up and trained to completion for the final ensemble.","fun_headline_variants_meta":{"raw":{"variants":["Dual surrogates estimate accuracy and diversity for NES","Accuracy plus triplet-loss surrogates guide ensemble search","Surrogate pair finds strong diverse nets without full trains","DAG surrogates cut cost of neural ensemble architecture search","Two surrogates yield ensembles matching Deep Ensembles on CIFAR"]},"model":"grok-4.5","effort":"low","cost_usd":0.004614,"raw_usage":{"total_tokens":1225,"prompt_tokens":684,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":46140000,"prompt_tokens_details":{"text_tokens":684,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":477,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":684,"tokens_out":64,"duration_ms":8526,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T16:20:08.742744+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Retrain the final selected ensembles and a matched set of random high-accuracy architectures to full capacity, then check whether pairwise prediction disagreement still correlates with the surrogate latent distances; if the correlation collapses or the surrogate ensembles lose their accuracy edge, the transfer assumption fails.","supporting_citations":[],"review_version":1}