Pith. sign in

REVIEW 3 major objections 6 minor 18 references

Surrogate assisted diversity estimation in neural ensemble search

T0 review · 3 major / 6 minor · reviewed 2026-07-30 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two surrogate models—one for accuracy, one for diversity—let neural ensemble search pick strong, complementary architectures without training every candidate.

desk verdict Solid incremental NES systems paper with a useful released dataset; the diversity-transfer claim is the real soft spot, not the overall framing. read the letter →

arxiv 2607.26940 v1 pith:NH7ZS43C submitted 2026-07-29 cs.LG

Alexandr Udeneev , Petr Babkin , Oleg Bakhteev This is my paper · ORCID
classification cs.LG
keywords neuralensemblesearchdiversitysurrogatefunctiontripletlossgraphattentionnetworkarchitectureDARTS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Building strong neural ensembles requires both accurate individual models and genuine diversity among them, yet jointly searching architectures and ensemble membership is exponentially harder than ordinary architecture search. This paper shows that the bottleneck can be broken by training two cheap graph-based surrogates on a modest pool of partially trained cells: one predicts validation accuracy, the other embeds architectures so that Euclidean distance tracks how differently the models predict. A simple greedy selection then keeps only high-accuracy candidates that sit far apart in that embedding. The resulting ensembles match or beat standard Deep Ensembles and random search on FashionMNIST, CIFAR-10 and CIFAR-100 while avoiding the cost of training thousands of full-scale networks during search. The practical payoff is a workable route to diversity-aware ensembles when exhaustive evaluation is impossible.

What carries the argument

The diversity surrogate: a graph-attention network that maps each architecture DAG into a continuous latent vector trained with triplet loss so that Euclidean distance approximates predictive disagreement; greedy selection then maximises average distance from already chosen members inside an accuracy-filtered pool.

What would settle it

Retrain the final selected ensembles and a matched set of random high-accuracy architectures to full capacity, then check whether pairwise prediction disagreement still correlates with the surrogate latent distances; if the correlation collapses or the surrogate ensembles lose their accuracy edge, the transfer assumption fails.

Watch

Extended reading notes

Core claim

A dual-objective surrogate framework—accuracy regression plus a triplet-loss latent space built from prediction agreement—can guide neural ensemble search so that the selected set is both individually strong and collectively diverse, yielding competitive or superior Top-1 accuracy and negative log-likelihood versus Deep Ensembles and Random Search on three standard image benchmarks.

Load-bearing premise

The diversity geometry learned on small, only partially trained cells still ranks true predictive disagreement correctly once those cells are scaled up and trained to completion for the final ensemble.

Editorial extensions

If this is right

  • Ensemble search can be performed largely training-free once a few thousand cheap proxy models have been evaluated.
  • Explicit diversity modelling becomes more valuable as task difficulty rises (CIFAR-100 gains exceed FashionMNIST).
  • Released proxy datasets of ~3000 trained cells per benchmark let others test new ensemble methods without repeating the expensive data-collection step.
  • The same dual-surrogate pattern can be swapped into other discrete search spaces that admit graph encodings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If latent distances transfer reliably, the same embedding could be reused for online ensemble pruning or continual learning without re-training the surrogate.
  • Replacing the binary agreement matrix with a continuous divergence (KL, total variation) might yield smoother gradients and better-calibrated diversity scores.
  • The accuracy-threshold-plus-greedy-distance recipe is architecture-agnostic and could be tried on transformer or NAS-Bench spaces with only a change of graph encoder.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a dual-objective surrogate-guided Neural Ensemble Search (NES) framework. Candidate architectures from a reduced DARTS-style cell space are encoded as DAGs and scored by two independently trained Graph Attention Network surrogates: an accuracy regressor (MSE) and a diversity embedder trained with triplet loss so that Euclidean distances reflect predictive disagreement on a validation set. A two-phase procedure (accuracy thresholding of a random candidate pool, then greedy farthest-point selection in the diversity latent space; Fig. 3) produces the final ensemble. The method is evaluated against Deep Ensembles (repeated training of one DARTS architecture) and Random Search on FashionMNIST, CIFAR-10, and CIFAR-100 (Table 4), with ablations on surrogate data size and component contribution (Tables 2–3). The authors also release ~3,000 trained architecture/weight pairs per dataset.

Significance. Explicit surrogate modeling of ensemble diversity via a metric-learned latent space is a clear and useful idea for NES, where joint architecture-and-composition search is otherwise intractable. The dual-surrogate separation, the GAT+triplet construction, the released model datasets, and the multi-metric evaluation (accuracy, NLL, calibration, disagreement, adversarial AUCs) are concrete contributions. If the claimed transfer of diversity structure from cheap partial models to full ensembles holds, the approach offers a practical route to diversity-aware NES without training every candidate. The empirical gains over DeepEns are modest and mixed across metrics, so the significance is incremental rather than transformative, but the direction and the public datasets are valuable to the community.

major comments (3)
  1. [§4.1, Table 1, Table 2, Fig. 3] Section 4.1 and Table 1 vs. final ensembles: the diversity surrogate is trained exclusively on reduced-capacity, partially trained cells (n=5 nodes, fixed epochs, smaller width), while final ensembles use more cells, larger width, and fuller training. Table 2 Spearman/Recall@K and Table 3 ablations are computed only inside the cheap regime. The paper never reports whether latent Euclidean rankings still correlate with true predictive disagreement after capacity and epoch scaling. This transfer is the load-bearing assumption behind the claim that greedy selection in Fig. 3 optimizes the diversity that appears in Table 4. A direct measurement (e.g., Spearman between cheap-regime distances and full-model disagreement on a held-out set of scaled architectures) is needed; without it, the modest Top-1 gains could be explained by accuracy filtering alone.
  2. [§4.4, Table 4] Table 4 baseline construction: DeepEns is “a single architecture obtained via DARTS with different random initializations.” This confounds the comparison. The proposed method selects heterogeneous architectures; DeepEns does not. A fairer architecture-level baseline would be an ensemble of distinct high-accuracy architectures chosen without the diversity surrogate (or NES methods that already vary architecture). As written, superiority over DeepEns partly reflects architecture diversity per se rather than the specific dual-surrogate procedure. On CIFAR-100, DeepEns already matches or beats Surrogate Ens on NLL (0.682 vs 0.692), average model accuracy, Brier, and ECE, which weakens the “competitive or superior” claim when diversity is the selling point.
  3. [§3.3, Fig. 3] Fig. 3 Phase 2 selection rule: after the accuracy threshold λ and the single most accurate anchor, subsequent members are chosen solely by maximum average latent distance, with no further accuracy term. Combined with free parameters (λ, margin m, quantile thresholds for D, N1/N2 and top-10% refinement), it is unclear how sensitive Table 4 is to these choices. A short sensitivity study or a joint accuracy–diversity selection score would make the dual-objective claim more convincing and show that diversity is not purchased at uncontrolled accuracy cost.
minor comments (6)
  1. [§3.1, Eq. (3)] Eq. (3): the product formula for |A_cell| with n=5, m=10, |O|=7 yielding ~10^9 should be checked or expanded; the combinatorial accounting of “exactly two incoming and two outgoing edges” is not fully spelled out.
  2. [§3.2] Notation drift: f_θ_acc / f_acc and f_θ_div / f_div are used interchangeably; y_i is both prediction vector and (in places) label-like. Unify notation early in §3.2.
  3. [§4.3, Table 3] Table 3 “Pred. Disagreement” for Accuracy+Diversity (0.411) is lower than Random (0.430) and Diversity-only (0.419), yet the text emphasizes diversity benefits. Briefly reconcile why lower disagreement can still help Top-1.
  4. [Fig. 1, Eq. (8)] Fig. 1 caption and body refer to “surrogate diversity function” mapping similar predictive behavior close together; ensure the figure’s d1 ≤ d2 labeling matches the triplet inequality in Eq. (8).
  5. [§1] Related work: cite additional NES/diversity baselines beyond Zaidi et al. and Chen et al. (e.g., explicit diversity regularizers or disagreement-based ensemble selection) so the “one of the first” claim in contribution 2 is better scoped.
  6. [Throughout / Appendix] Typos/consistency: “F unction” spacing in headings (§3.2, §4.2); “simularity” in appendix Fig. 10 caption; arXiv date “29 Jul 2026” looks like a placeholder.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: surrogate-guided selection is evaluated by independent full training and external test metrics

full rationale

This is an empirical NES methods paper. The dual GAT surrogates (accuracy MSE regression; diversity triplet loss on prediction-agreement labels) are fit on a pool of partially trained, reduced-capacity cells, then used only to filter and greedily rank new discrete candidates (Fig. 3). Final ensembles are retrained at higher capacity and scored by Top-1, NLL, Brier, ECE, and attack AUCs on held-out data (Table 4) against DeepEns and Random Search. Those reported numbers are not algebraic rearrangements of the surrogate losses, nor are they forced by any fitted parameter of f_acc or f_div. Ablations (Tables 2–3) likewise measure Spearman/Recall and ensemble metrics on held-out or fully trained models, not on the training objectives themselves. Citations are standard external NAS/ensemble references; the authors’ own dataset release [15] is data, not a load-bearing uniqueness theorem. The capacity/epoch transfer assumption is a validity risk, not a circular reduction. No step collapses claim to input by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on standard NAS/ensemble assumptions plus several modeling choices that are not derived: the reduced DARTS cell, the particular discretization of prediction agreement into a ternary similarity matrix, the accuracy threshold λ, the triplet margin, and the unproven transfer from cheap surrogate models to expensive final models. No new physical entities are invented; the free parameters are algorithmic knobs fitted or set by hand.

free parameters (5)
  • accuracy threshold λ
    Used in Phase 1 of ensemble construction (Fig. 3) to prune the candidate pool; value chosen by the authors, directly affects which architectures enter the diversity stage.
  • triplet loss margin m
    Appears in Eq. (8) and the training procedure (Fig. 2); controls separation in the diversity latent space and is a standard but free hyper-parameter of the metric-learning objective.
  • quantile thresholds for similarity matrix D
    Prediction-agreement matrix C is discretized into D ∈ {-1,0,1} via unspecified quantiles (Section 3.2); these cut-offs define positive/negative pairs and therefore shape the entire latent space.
  • surrogate training set sizes N1=1000, N2=2000 and top-10% refinement = N1=1000, N2=2000
    Iterative sampling schedule (Section 4.1) that determines the data distribution seen by both surrogates; chosen for compute budget, not derived.
  • reduced cell configuration (n=5 nodes, m=10 edges, partial epochs) = n=5, m=10, epochs 125/200
    Architectures used to train surrogates are deliberately smaller and not fully converged (Table 1, Section 4.1); the mapping from this regime to full final models is an untested modeling choice.
assumptions (4)
  • ad hoc to paper Euclidean distance in a triplet-loss embedding of architectures is a faithful proxy for predictive disagreement of the corresponding fully-trained networks.
    Core modeling hypothesis of Section 3.2 and Fig. 1; supported by Spearman correlations on the surrogate set but not proven to transfer to final ensemble members.
  • domain assumption DARTS cell-based search space with discrete random sampling adequately represents the architectures worth ensembling.
    Inherited from Liu et al. 2018 and used throughout Sections 2–3; standard in the NAS literature but still an unproven restriction of the true architecture space.
  • domain assumption Greedy forward selection on average latent distance yields near-optimal diverse ensembles of size K.
    Algorithm in Fig. 3; common heuristic in diversity-based selection, not shown to be optimal or even approximately optimal for the NES objective (Eq. 2).
  • domain assumption Validation-set prediction agreement (indicator equality averaged over K examples) is the right notion of diversity for ensemble performance.
    Defines the similarity matrix C in Section 3.2; alternative information-theoretic or gradient-based diversity measures are acknowledged only as future work.
invented entities (1)
  • dual-objective surrogate pair (f_acc, f_div) with GAT backbone and triplet-loss diversity head
    purpose: Enable training-free estimation of both accuracy and pairwise diversity so that NES becomes tractable.
    The specific pairing and the use of a dedicated diversity GAT whose output is Euclidean distance are introduced by the paper; components exist separately in prior work but the joint construct is new to this manuscript.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Surrogate assisted diversity estimation in neural ensemble search." pith.science (2026). https://pith.science/paper/NH7ZS43C

@misc{pith2026260726940,
  author       = {Pith},
  title        = {Pith review of: Surrogate assisted diversity estimation in neural ensemble search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NH7ZS43C}},
  note         = {Machine review of arXiv:2607.26940}
}
read the original abstract

Ensembles are a standard way to improve the performance and robustness of deep neural networks, but their effectiveness crucially depends on both the quality and the diversity of individual models. Most neural architecture search (NAS) methods are computationally expensive. Extending them to neural ensemble search (NES), which requires joint optimization of individual architectures and their ensemble composition, leads to an exponential growth of the search space and makes the problem computationally intractable. To address this, we introduce a dual-objective surrogate-guided ensemble search: candidate architectures are represented as directed acyclic graphs, and two surrogate models are trained independently to estimate predictive accuracy and diversity potential. Their combined estimates guide an NES framework that efficiently identifies architectures that are both individually strong and collectively diverse. Our final ensemble achieves competitive or superior performance compared to standard baselines such as Deep Ensembles and Random Search on FashionMNIST, CIFAR-10, and CIFAR-100.

Figures

Figures reproduced from arXiv: 2607.26940 by the authors.

Figure 1
Figure 1. Illustration of the surrogate diversity function. Neural architectures are [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Training procedure for the diversity surrogate function. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Surrogate-Assisted Ensemble Construction. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Construction of the Model Evaluation Dataset [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Combined normal and reduced cells. The red vertices belong to the [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Accuracy of the surrogate ensemble under FGSM, BIM, and PGD attacks [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Conversion of an architecture to the NAS-Bench-101 format. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 9
Figure 9. Figure 9: Distribution of inter-model diver [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Training curve of the surrogate model for diversity [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 3 canonical work pages

  1. [1]

    Neural network ensembles.IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993–1001, 1990.https://doi.org/10.1109/34.58871

    Lars Kai Hansen and Peter Salamon. Neural network ensembles.IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993–1001, 1990.https://doi.org/10.1109/34.58871

    doi:10.1109/34.58871 1990

  2. [2]

    Ye Ren, Le Zhang, and P. N. Suganthan. Ensemble classification and regression–recent developments, applications and future directions.IEEE Computational Intelligence Magazine, 11(1):41–53, February 2016. ISSN 1556-603X.https://doi.org/10.1109/MCI.2015.2471235

    arXiv 2016

  3. [3]

    AdaNet: Adaptive structural learning of artificial neural net- works

    Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. AdaNet: Adaptive structural learning of artificial neural net- works. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pages 874–883. PMLR, 06–11 Aug 2017. UR...

    2017

  4. [4]

    Holmes, Frank Hutter, and Yee Whye Teh

    Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris C. Holmes, Frank Hutter, and Yee Whye Teh. Neural ensemble search for uncertainty estimation and dataset shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems 34, pages 7898–7911, 2021

    2021

  5. [5]

    Surrogate-assisted Multi-objective Neural Architecture Search for Real-time Semantic Segmentation

    Zhichao Lu, Ran Cheng, Shihua Huang, Haoming Zhang, Changxiao Qiu, and Fan Yang. Surrogate-assisted multi-objective neural architecture search for real-time semantic segmentation.CoRR, abs/2208.06820, 2022.https: //doi.org/10.48550/ARXIV.2208.06820

    work page Pith review arXiv doi:10.48550/arxiv.2208.06820 2022

  6. [6]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,NeurIPS, volume 30, pages 6405–6416, 2017

    2017

  7. [7]

    One-shot neural ensemble architecture search by diversity-guided search space shrinking

    Minghao Chen, Jianlong Fu, and Haibin Ling. One-shot neural ensemble architecture search by diversity-guided search space shrinking. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16530–16539, June 2021

    2021

  8. [8]

    Neural predictor for neural architecture search

    Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors,ECCV, pages 660–676. Springer, 2020.https://doi.org/10.1007/978-3-030-58580-8_39

    work page doi:10.1007/978-3-030-58580-8_39 2020

Show all 18 references
  1. [9]

    Bananas: Bayesian optimization with neural architectures for neural architecture search

    Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10293–10301, 2021

    2021

  2. [10]

    Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy.Elec- tronic Research Archive, 32(2):1017–1043, 2024

    Yu Xue, Zhenman Zhang, and Ferrante Neri. Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy.Elec- tronic Research Archive, 32(2):1017–1043, 2024. ISSN 2688-1594.https: //doi.org/10.3934/era.2024050. 14 Alexandr Udeneev, Petr Babk...

    2024 doi

  3. [11]

    Graph attention networks.CoRR, abs/1710.10903, 2017.https://doi.org/10.48550/arxiv.1710.10903

    Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks.CoRR, abs/1710.10903, 2017.https://doi.org/10.48550/arxiv.1710.10903

    2017 doi arXiv

  4. [12]

    Facenet: A unified embedding for face recognition and clustering

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. InCVPR, pages 815–823, 2015.https://doi.org/10.1109/CVPR.2015.7298682

    2015

  5. [13]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017

    Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  6. [14]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  7. [15]

    NES surrogate architectures, 2026

    Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev. NES surrogate architectures, 2026. URL https://huggingface.co/datasets/Demoren/ nes-surrogate-architectures. Revision b6b3f3c

    2026

  8. [16]

    Darts: Differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. InICLR. OpenReview.net, 2018

    2018

  9. [17]

    Graphnorm: A principled approach to accelerating graph neural network training

    Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-yan Liu, and Liwei Wang. Graphnorm: A principled approach to accelerating graph neural network training. InInternational Conference on Machine Learning, pages 1204–1215. PMLR, 2021

    2021

  10. [18]

    Random search and reproducibility for neural architecture search

    Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In J. Peters and D. Sontag, editors,UAI, volume 124, pages 367–377. PMLR, 2020. Surrogate assisted diversity estimation in neural ensemble search 15 A NAS-Bench formats Fig.5: Combin...

    2020

Pith tools

Reviewed July 30, 2026 · model on record in the stance chip above.