ASNN: Learning to Suggest Neural Architectures from Performance Distributions

Jinwook Hong

REVIEW 4 major objections 4 minor 1 cited by

ASNN: Learning to Suggest Neural Architectures from Performance Distributions

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A neural network that maps accuracy vectors to layer sizes can propose MLP architectures that outperform the best architecture in its training grid.

desk verdict Interesting framing but the data augmentation destroys the label-architecture pairing, so the headline results don't support the central claim; still worth a hard-nosed referee for the idea and the honest disclosure. read the letter →

arxiv 2507.20164 v1 pith:ZUHZUJMF submitted 2025-07-27 cs.LG cs.AI

Jinwook Hong This is my paper · ORCID

classification cs.LGcs.AI

keywords neuralarchitecturesearchinverseregressionmulti-layerperceptronperformancepredictioniterativeretrainingoptimizationaccuracydistributionrandom

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes the Architecture Suggesting Neural Network (ASNN), a small neural network trained to do the reverse of the usual mapping: instead of predicting test accuracy from a network's layer sizes, it takes a vector of ten accuracy measurements as input and outputs suggested numbers of nodes in each layer. The authors train this inverse model on a coarse grid of MLP architectures (25 two-layer and 64 three-layer configurations, each trained ten times), then repeatedly feed it an all-100 input that stands for maximum scaled accuracy to obtain new architectures, measure those, and add them to the training set. They report that in both the two-layer and three-layer settings the predicted architectures achieved mean test accuracies above the best mean accuracy in the original grid, with the best two-layer suggestion reaching 0.9838 and the best three-layer suggestion reaching 0.9831. If these results hold, ASNN would be a cheap, iterative alternative to random search for architecture optimization, needing only a few proposed architectures to find better configurations than the grid contained.

What carries the argument

The central object is the inverse accuracy-to-architecture map learned by ASNN, written $ASNN: \mathbb{R}^{10} \to \mathbb{R}^{2}$ (or $\mathbb{R}^{3}$): a vector of ten test-accuracy trials maps to layer sizes. The mechanism is an iterative closed loop: train ASNN, feed the all-100 vector, round the predicted node counts, train and evaluate that architecture ten times, append the new (accuracy-vector, architecture) pairs to the dataset, and retrain. The all-100 input is what carries the extrapolation: it is outside the range of any real measured accuracy, so the model is being asked to invert its learned mapping at a point corresponding to perfect performance.

What would settle it

Train two ASNN models on the same grid data: one with the true accuracy-architecture pairs and one with the same accuracy values randomly reassigned to architectures (the paper's augmentation). If the shuffled-label model proposes architectures whose mean test accuracy is statistically indistinguishable from the true-pair model's proposals, then the reported improvement does not come from learning the performance-structure relationship. A second check: compare each predicted architecture's ten-run mean against the best mean among all original grid architectures using a paired test over trial seeds; if the difference is within run-to-run noise, the claim of outperforming the grid is not established.

Watch

Extended reading notes

Core claim

The paper's claim is that an inverse regression model can generalize the accuracy-to-architecture relationship beyond the evaluated grid. ASNN is trained with accuracy vectors as inputs and architecture parameters (layer sizes) as outputs, on data where the raw accuracy values are randomly shuffled to create about 10,000 training samples. At inference, the fixed input $(100,100,\ldots,100)$ encodes the goal of maximal accuracy, and the output is rounded to node counts. The discovered architectures—(448,65), (313,72), and others in the two-layer case; (339,184,66) and others in the three-layer case—were trained from scratch and their averaged ten-run test accuracies are reported. The paper claims this shows ASNN has learned the performance-structure relationship and provides an efficient alternative to random search.

Load-bearing premise

The load-bearing premise is that randomly shuffling accuracy values across architectures to expand the dataset preserves enough of the true accuracy-to-architecture relationship for the trained model, and that feeding an all-100 input extrapolates that relationship to genuinely better architectures; if either fails, the reported improvements do not demonstrate learning.

Editorial extensions

If this is right

If ASNN works as claimed, a user can start from a small grid of a few hundred training runs and obtain improved MLP layer sizes with only a handful of extra evaluations.
The iterative retraining loop turns architecture search into a fixed-point problem: each new architecture's measured accuracies are fed back, so later suggestions reflect the expanded region of the search space.
Because the method does not rely on gradients through the evaluated network or on a separate controller, it could apply to any architecture family whose performance can be summarized as a vector of repeated trial scores.
The two-layer and three-layer experiments suggest the benefit persists as depth grows from two to three layers, at least for the small MLPs used.
The reported margin over the best grid point is small—around 0.0005 to 0.0014 in mean accuracy—so even a modest gain would need to be weighed against the cost of the extra evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the reported gains could be produced by the iterative expansion alone, because adding the measured high-scoring architectures to the training set biases later predictions toward their neighborhood; a control that adds random architectures to the dataset would isolate the contribution of the learned inverse map.
Editorial inference: a direct test of the learning claim would be to hold out entire grid cells, train ASNN only on the rest, and see whether its suggested architectures beat the best held-out cell—something the paper does not report.
Editorial inference: the use of a fixed $(100,\ldots,100)$ input is equivalent to asking the model for an infinite extrapolation; the approach would be easier to interpret if the input were reparameterized as a high but realistic quantile of the observed accuracy distribution, for example the observed maximum.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: the paper has an unusual and genuinely interesting framing — train a network to map a vector of measured accuracies to the architecture that produced them, then iterate. That inversion is not common in the NAS literature, and the experimental tables are detailed and reproducible enough that the empirical claims are checkable. The author also discloses ChatGPT assistance, which is honest. But the load-bearing flaw the stress-test note flags is real, and it is not minor.

Section 4.3 says the 250 or 640 true accuracy vectors were 'augmented by randomly shuffling the accuracy values' to get roughly 10,000 samples. If that means what it says, almost every training pair becomes false: architecture A gets accuracies that were measured for other architectures. Training an inverse model on that can't learn the architecture→performance relationship; at best it learns marginals. The paper treats this as a diversity-boosting trick, but it destroys the very signal the method is supposed to exploit. I don't see an intact-label control anywhere. That alone guts the claim that ASNN 'learned' to suggest better architectures.

Second, the headline improvements are tiny: 0.98363 vs the original best 0.98310 in 2-layer, and 0.98313 vs 0.98171 in 3-layer. With 10 trials, standard errors on those means are on the order of 0.0003–0.0005, so the differences are around one or two standard errors — not conclusive even if the labels were clean. And the 'efficient alternative to random search' claim is never tested: there is no random-search baseline with the same evaluation budget, which is a strange omission given that the abstract and discussion lean on it.

Worth noting as credit: each suggested architecture is actually trained and evaluated; the numbers in the tables are not fabricated. The iterative loop is also transparent, and the paper is short enough that a reader can see exactly what was done. But the method section is so under-specified (no ASNN architecture, no hyperparameters, no dataset/task named, just 'TensorFlow-based models' on a classification task) that independent replication would be hard.

Who is this for? Someone curious about inverse performance-to-architecture models as a thought experiment might get value, but as a scientific claim it is not supported. I would not cite it. I would still send it to a referee if I were an editor — the idea is worth a rigorous look, and the flaws are the kind a sharp referee could catch and the author could fix with a re-run. But my own verdict is that it needs major revision, likely a full redo of the augmentation and a real baseline comparison, before it carries weight.

Regards.

Referee Report

4 major / 4 minor

Summary. The paper proposes an Architecture Suggesting Neural Network (ASNN) that learns an inverse mapping from measured test-accuracy vectors to MLP architecture parameters. Training data are collected for 2-layer (5x5 grid, 250 measurements) and 3-layer (4x4x4 grid, 640 measurements) networks, each architecture evaluated with 10 trials. Section 4.3 describes augmenting these data to about 10,000 samples by randomly shuffling accuracy values across architectures. Algorithm 1 then iteratively trains ASNN, feeds the scaled perfect-accuracy input x=(100,...,100), rounds the predicted architecture to integers, evaluates it for 10 trials, and adds the new measurements to the training set. The paper reports that suggested architectures such as (448,65) and (339,184,66) achieve mean accuracies above the best grid point (0.98363 vs 0.98310 in the 2-layer case, 0.98313 vs 0.98171 in the 3-layer case) and claims that ASNN is an efficient alternative to random search.

Significance. If the central claim were established, an inverse-regression approach to architecture search would be a useful, lightweight complement to existing NAS methods, and the paper's honest evaluation of each suggested architecture with repeated trials is a strength. The complete tables of per-trial accuracies are also valuable for reproducibility. However, the current evidence does not support the claim: the augmentation procedure described in Section 4.3 destroys the pairing between accuracies and architectures, so the reported suggestions cannot be attributed to a learned performance-structure relationship; the query point in Algorithm 1 is an unvalidated extrapolation; and no random-search or other baseline is provided. The significance is therefore conditional on a substantial revision that re-establishes the training data integrity and adds the missing comparisons.

major comments (4)

[Section 4.3] The augmentation step 'randomly shuffling the accuracy values' breaks the pairing between each accuracy vector and its architecture. With only 250 (2-layer) or 640 (3-layer) true pairs, almost every row of the expanded training set is a false pairing. A model trained on such data can learn at best marginal statistics of accuracies and architectures, not a mapping between the two. Consequently, the reported predictions in Section 5 cannot be taken as evidence that ASNN generalized the performance-structure relationship. The authors must retrain on the intact pairs or on a pairing-preserving augmentation (for example, adding small noise to the true accuracy vectors) and re-run the iterative search.
[Algorithm 1, line 3] The input x=(100,...,100) is an extrapolation beyond the observed accuracy range: the best mean accuracy in the original data is 0.9831, which scales to 98.31, not 100. Feeding a perfect accuracy vector that never occurs in the training data assumes that the inverse mapping behaves monotonically and smoothly in this extrapolation region, which is not established. The suggested architectures such as (448,65) and (339,184,66) may be artifacts of this extrapolation. The authors should justify the choice, or better, query a target accuracy within the observed support and report sensitivity to the chosen target.
[Section 5] The claimed advantage over random search is asserted but never tested. The only comparison is against the best point of the original coarse grid (0.98310 for 2-layer, 0.98171 for 3-layer), with no random-search baseline that evaluates the same number of architectures under the same budget. Additionally, the reported improvements (0.98363 vs 0.98310, and 0.98313 vs 0.98171) are small relative to the trial-to-trial spread visible in Tables 1 and 2, where individual runs for the same architecture differ by more than 0.003. The paper should report standard errors or confidence intervals and, if possible, a statistical test comparing ASNN suggestions with random search.
[Section 5.2] The text states that 'each was evaluated over five independent runs,' but Tables 8 through 12 report ten trials labeled E1 through E10, and Section 4.2 states that each configuration was evaluated over 10 trials. This inconsistency makes the reported mean accuracies ambiguous and should be corrected, along with a statement of whether the 3-layer predictions were evaluated 5 or 10 times.

minor comments (4)

[Section 4.3] The sentence 'the accuracy values used as output labels' contradicts the method description in Section 3, where test accuracy is the input and architecture parameters are the output; the wording should be fixed.
[Section 3/4] The benchmark dataset is never named; the paper should state which classification task the accuracies refer to (e.g., MNIST or similar), as this is essential for interpreting the absolute accuracy values.
[Algorithm 1] The stopping condition 'while desired test accuracy not obtained' is never quantified, and the number of iterations is not specified in the algorithm; the paper should state the actual budget (apparently five iterations in each case).
[Section 4.3] The ASNN's own architecture and training hyperparameters (layer sizes, activation, optimizer, epochs, and the exact number of augmented samples) are not reported, which prevents replication of the method.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: ASNN suggestions are independently evaluated and reported gains do not reduce to training inputs by construction.

full rationale

The derivation chain is: collect (architecture, 10-trial accuracy) pairs; train an inverse regression from accuracy vectors to architectural parameters; query the fixed perfect-accuracy vector (100,...,100); evaluate the suggested architecture; add the new measured pair; repeat. Each reported mean (e.g., 0.98363 for (448,65), 0.98313 for (339,184,66)) comes from actually training and testing TensorFlow models with those architectures, so the headline numbers are empirical measurements rather than fitted parameters relabeled as predictions. Algorithm 1 is a standard active-search loop: retraining on previously evaluated candidates does not make the next suggested architecture equal to a previous training example by construction, and the final evaluation is a fresh measurement. There are no self-citations, uniqueness theorems, or ansatz-by-citation steps carrying the argument. The Section 4.3 random-shuffle augmentation ('augmented by randomly shuffling the accuracy values') is a serious threat to the validity of the learned mapping and would make the claim that ASNN learned the accuracy-to-architecture relationship unsupported, but that is an experimental-design/correctness problem rather than a circularity pattern: it does not make the reported prediction equal to its inputs by definition. The absence of a random-search baseline similarly undermines the efficiency claim without being circular. Under the hard rule that circularity requires quoting a specific reduction, no such reduction is exhibited in the paper.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small number of hand choices, an invalid augmentation assumption, and an extrapolation at the perfect-accuracy query point. The ASNN itself is an under-specified fitted model, so its learned parameters act as hidden free parameters.

free parameters (4)

Input scale factor for accuracy values = 100
Chosen in Section 4.3 to 'correct for differences in numerical scale'; the query point (100, ..., 100) in Algorithm 1 depends on this scaling.
Augmented dataset size = approximately 10,000
Chosen in Section 4.3; the training distribution is dominated by randomly shuffled, partly corrupted samples.
ASNN architecture and hyperparameters = not reported
The hidden layer sizes, learning rate, epochs, and optimizer for ASNN are never given, leaving the fitted model as a black box.
Number of training epochs for evaluated networks = 50
Section 3 says the original code's 10 epochs were increased to 50 for all measured networks, which affects every accuracy value.

assumptions (4)

standard math Universal Approximation Theorem justifies using a neural network as the inverse mapping function.
Invoked via references [3,4] in Section 2 to justify ASNN as a function approximator.
domain assumption Accuracy values are independent and identically distributed.
Stated in Section 4.3 to justify random shuffling before training.
ad hoc to paper Randomly shuffling accuracy values across architectures preserves enough of the relationship to improve learning.
Assumed in Section 4.3; random reassignment actually destroys the architecture-accuracy pairing.
ad hoc to paper Feeding a perfect accuracy score of 100 extrapolates the inverse mapping to high-performing architectures.
Algorithm 1 line 3 queries the model at (100, ..., 100), far outside the observed accuracy range, and assumes the output points to a good architecture.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ASNN: Learning to Suggest Neural Architectures from Performance Distributions." pith.science (2026). https://pith.science/paper/ZUHZUJMF

@misc{pith2026250720164,
  author       = {Pith},
  title        = {Pith review of: ASNN: Learning to Suggest Neural Architectures from Performance Distributions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZUHZUJMF}},
  note         = {Machine review of arXiv:2507.20164}
}

read the original abstract

The architecture of a neural network (NN) plays a critical role in determining its performance. However, there is no general closed-form function that maps between network structure and accuracy, making the process of architecture design largely heuristic or search-based. In this study, we propose the Architecture Suggesting Neural Network (ASNN), a model designed to learn the relationship between NN architecture and its test accuracy, and to suggest improved architectures accordingly. To train ASNN, we constructed datasets using TensorFlow-based models with varying numbers of layers and nodes. Experimental results were collected for both 2-layer and 3-layer architectures across a grid of configurations, each evaluated with 10 repeated trials to account for stochasticity. Accuracy values were treated as inputs, and architectural parameters as outputs. The trained ASNN was then used iteratively to predict architectures that yield higher performance. In both 2-layer and 3-layer cases, ASNN successfully suggested architectures that outperformed the best results found in the original training data. Repeated prediction and retraining cycles led to the discovery of architectures with improved mean test accuracies, demonstrating the model's capacity to generalize the performance-structure relationship. These results suggest that ASNN provides an efficient alternative to random search for architecture optimization, and offers a promising approach toward automating neural network design. "Parts of the manuscript, including text editing and expression refinement, were supported by OpenAI's ChatGPT. All content was reviewed and verified by the authors."

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
cs.LG 2026-07 conditional novelty 7.0 of 10

Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.

Reference graph

Works this paper leans on

11 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

Do deep nets really need to be deep?

J. Ba and R. Caruana, “Do deep nets really need to be deep?” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 27, 2014

work page 2014
[2]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , pp. 770–778, 2016

work page 2016
[3]

Approximation by superpositions of a sigmoidal function,

G. Cybenko, “Approximation by superpositions of a sigmoidal function,”Mathematics of Control, Signals and Systems , vol. 2, no. 4, pp. 303–314, 1989

work page 1989
[4]

Approximation capabilities of multilayer feedforward networks,

K. Hornik, “Approximation capabilities of multilayer feedforward networks,” Neural Networks, vol. 4, no. 2, pp. 251–257, 1991

work page 1991
[5]

Deep neural networks motivated by partial differential equations,

L. Ruthotto and E. Haber, “Deep neural networks motivated by partial differential equations,” Journal of Mathematical Imaging and Vision , vol. 62, no. 3, pp. 352–364, 2020

work page 2020
[6]

DGM: A deep learning algorithm for solving partial differential equations,

J. Sirignano and K. Spiliopoulos, “DGM: A deep learning algorithm for solving partial differential equations,” Journal of Computational Physics , vol. 375, pp. 1339–1364, 2018

work page 2018
[7]

Human-level control through deep rein- forcement learning,

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep rein- forcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015

work page 2015
[8]

Continuous control with deep reinforcement learning,

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,”arXiv preprint arXiv:1509.02971 , 2015

arXiv 2015

Show all 11 references

[9]

Proximal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017. 9

2017 arXiv
[10]

Neural architecture search with reinforcement learning,

B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” inInternational Conference on Learning Representations (ICLR) , 2017

2017
[11]

DARTS: Differentiable architecture search,

H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” inInternational Conference on Learning Representations (ICLR) , 2019. 10

2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Do deep nets really need to be deep?

J. Ba and R. Caruana, “Do deep nets really need to be deep?” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 27, 2014

work page 2014

[2] [2]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , pp. 770–778, 2016

work page 2016

[3] [3]

Approximation by superpositions of a sigmoidal function,

G. Cybenko, “Approximation by superpositions of a sigmoidal function,”Mathematics of Control, Signals and Systems , vol. 2, no. 4, pp. 303–314, 1989

work page 1989

[4] [4]

Approximation capabilities of multilayer feedforward networks,

K. Hornik, “Approximation capabilities of multilayer feedforward networks,” Neural Networks, vol. 4, no. 2, pp. 251–257, 1991

work page 1991

[5] [5]

Deep neural networks motivated by partial differential equations,

L. Ruthotto and E. Haber, “Deep neural networks motivated by partial differential equations,” Journal of Mathematical Imaging and Vision , vol. 62, no. 3, pp. 352–364, 2020

work page 2020

[6] [6]

DGM: A deep learning algorithm for solving partial differential equations,

J. Sirignano and K. Spiliopoulos, “DGM: A deep learning algorithm for solving partial differential equations,” Journal of Computational Physics , vol. 375, pp. 1339–1364, 2018

work page 2018

[7] [7]

Human-level control through deep rein- forcement learning,

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep rein- forcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015

work page 2015

[8] [8]

Continuous control with deep reinforcement learning,

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,”arXiv preprint arXiv:1509.02971 , 2015

arXiv 2015