{"id":"b64a5af1-63f4-4e57-ac29-0b0954be4c19","arxiv_id":"2507.20164","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An inverse-regression model maps test accuracy distributions to layer sizes and suggests architectures that, in a small MLP study, beat the best accuracy in the original training grid.","lead":"A small study trains a neural network to guess a network's layer sizes from its test accuracies, then feeds it a perfect score to propose new architectures. On two tiny grids of fully-connected networks, the proposed architectures sometimes beat the best accuracy in the training data, but no random-search baseline or code is provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.3's 'randomly shuffling the accuracy values' creates mostly false training pairs, so the reported architecture suggestions cannot be attributed to a learned accuracy-to-architecture mapping; the claimed random-search advantage is untested.","rationale":"The paper does provide concrete raw tables, repeated trials, and specific predicted architectures, which is useful transparency. The central claim, however, depends on the ASNN learning a real mapping from accuracy vectors to architecture parameters. The augmentation described in Section 4.3 randomly reassigns accuracy values across architectures, which destroys that mapping in most of the 10,000 training samples. If the text is read literally, the training set is mostly false, so the reported suggestions cannot demonstrate generalization of a performance-structure relationship. Even if the augmentation were intended as a benign row-order shuffle, the wording is seriously misleading and the experimental protocol remains incomplete: there is no random-search baseline, no named task or dataset, and no analysis of whether the small mean differences exceed run-to-run variability. Algorithm 1's query at the scaled maximum (100,...,100) is an extrapolation beyond the observed accuracy range, and it is especially unsupported when trained on shuffled labels. The reader's verdict of REJECT is appropriate because the main empirical evidence is invalidated by the described procedure rather than merely missing a comparison. A controlled ablation with intact labels and a random-search baseline could in principle rescue the claim, but the current manuscript does not provide it.","tokens_in":10931,"tokens_out":5371,"duration_ms":59784,"concrete_test":"Retrain ASNN on the original unshuffled 250/640 pairs and on the shuffled 10,000 pairs using the same architecture and hyperparameters, then evaluate each model's top suggested architecture with 10 fresh trials. Compare both against a random-search baseline that evaluates the same number of architectures (five suggestions times ten trials) from the same node-count ranges. If the shuffled model does not underperform the intact model, or if either fails to beat the random-search baseline, the reported advantage is unsupported. Report per-suggestion standard errors so differences of +0.0005 to +0.0014 can be judged against noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the augmentation described in Section 4.3: 'the original datasets (250 for 2-layer, 640 for 3-layer) were augmented by randomly shuffling the accuracy values, expanding the training dataset to approximately 10,000 samples.' As written, this breaks each (architecture, accuracy-vector) pair. With only 250 or 640 true pairs, random recombination makes almost every training row a false pairing. An inverse model trained on such labels cannot recover a performance-structure relationship; it can only learn marginal statistics. The fact that ASNN then 'suggests' (448,65), (885,20), and (339,184,66) is therefore not evidence that the model generalized the mapping. In addition, Algorithm 1 line 3 queries x=(100,...,100) after the ×100 scaling; that point lies above every observed accuracy vector (best grid mean 0.9831 scaled to 98.31), so the suggestion is an unvalidated extrapolation from corrupted data. The claimed improvements (0.98363 vs 0.98310 in the 2-layer case; 0.98313 vs 0.98171 in the 3-layer case) are small relative to trial noise, and the paper never compares against a random-search baseline with the same evaluation budget. Without an intact-label control or a random-search comparison, the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Architecture Suggesting Neural Network (ASNN) that learns an inverse mapping from measured test-accuracy vectors to MLP architecture parameters. Training data are collected for 2-layer (5x5 grid, 250 measurements) and 3-layer (4x4x4 grid, 640 measurements) networks, each architecture evaluated with 10 trials. Section 4.3 describes augmenting these data to about 10,000 samples by randomly shuffling accuracy values across architectures. Algorithm 1 then iteratively trains ASNN, feeds the scaled perfect-accuracy input x=(100,...,100), rounds the predicted architecture to integers, evaluates it for 10 trials, and adds the new measurements to the training set. The paper reports that suggested architectures such as (448,65) and (339,184,66) achieve mean accuracies above the best grid point (0.98363 vs 0.98310 in the 2-layer case, 0.98313 vs 0.98171 in the 3-layer case) and claims that ASNN is an efficient alternative to random search.","tokens_in":11358,"tokens_out":4571,"duration_ms":49636,"significance":"If the central claim were established, an inverse-regression approach to architecture search would be a useful, lightweight complement to existing NAS methods, and the paper's honest evaluation of each suggested architecture with repeated trials is a strength. The complete tables of per-trial accuracies are also valuable for reproducibility. However, the current evidence does not support the claim: the augmentation procedure described in Section 4.3 destroys the pairing between accuracies and architectures, so the reported suggestions cannot be attributed to a learned performance-structure relationship; the query point in Algorithm 1 is an unvalidated extrapolation; and no random-search or other baseline is provided. The significance is therefore conditional on a substantial revision that re-establishes the training data integrity and adds the missing comparisons.","major_comments":[{"comment":"The augmentation step 'randomly shuffling the accuracy values' breaks the pairing between each accuracy vector and its architecture. With only 250 (2-layer) or 640 (3-layer) true pairs, almost every row of the expanded training set is a false pairing. A model trained on such data can learn at best marginal statistics of accuracies and architectures, not a mapping between the two. Consequently, the reported predictions in Section 5 cannot be taken as evidence that ASNN generalized the performance-structure relationship. The authors must retrain on the intact pairs or on a pairing-preserving augmentation (for example, adding small noise to the true accuracy vectors) and re-run the iterative search.","section":"Section 4.3"},{"comment":"The input x=(100,...,100) is an extrapolation beyond the observed accuracy range: the best mean accuracy in the original data is 0.9831, which scales to 98.31, not 100. Feeding a perfect accuracy vector that never occurs in the training data assumes that the inverse mapping behaves monotonically and smoothly in this extrapolation region, which is not established. The suggested architectures such as (448,65) and (339,184,66) may be artifacts of this extrapolation. The authors should justify the choice, or better, query a target accuracy within the observed support and report sensitivity to the chosen target.","section":"Algorithm 1, line 3"},{"comment":"The claimed advantage over random search is asserted but never tested. The only comparison is against the best point of the original coarse grid (0.98310 for 2-layer, 0.98171 for 3-layer), with no random-search baseline that evaluates the same number of architectures under the same budget. Additionally, the reported improvements (0.98363 vs 0.98310, and 0.98313 vs 0.98171) are small relative to the trial-to-trial spread visible in Tables 1 and 2, where individual runs for the same architecture differ by more than 0.003. The paper should report standard errors or confidence intervals and, if possible, a statistical test comparing ASNN suggestions with random search.","section":"Section 5"},{"comment":"The text states that 'each was evaluated over five independent runs,' but Tables 8 through 12 report ten trials labeled E1 through E10, and Section 4.2 states that each configuration was evaluated over 10 trials. This inconsistency makes the reported mean accuracies ambiguous and should be corrected, along with a statement of whether the 3-layer predictions were evaluated 5 or 10 times.","section":"Section 5.2"}],"minor_comments":[{"comment":"The sentence 'the accuracy values used as output labels' contradicts the method description in Section 3, where test accuracy is the input and architecture parameters are the output; the wording should be fixed.","section":"Section 4.3"},{"comment":"The benchmark dataset is never named; the paper should state which classification task the accuracies refer to (e.g., MNIST or similar), as this is essential for interpreting the absolute accuracy values.","section":"Section 3/4"},{"comment":"The stopping condition 'while desired test accuracy not obtained' is never quantified, and the number of iterations is not specified in the algorithm; the paper should state the actual budget (apparently five iterations in each case).","section":"Algorithm 1"},{"comment":"The ASNN's own architecture and training hyperparameters (layer sizes, activation, optimizer, epochs, and the exact number of augmented samples) are not reported, which prevents replication of the method.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is far from publishable in its current form because the central evidence is invalidated by the destructive data augmentation and the missing random-search baseline. However, the underlying problem is not intractable: if the authors retrain on intact pairs or a pairing-preserving augmentation, restrict the query to the observed accuracy support, and add a proper baseline with confidence intervals, the central claim could become testable. I do not see grounds for rejection based on the idea itself, but the current version is not close to meeting the journal's standards."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper has an unusual and genuinely interesting framing — train a network to map a vector of measured accuracies to the architecture that produced them, then iterate. That inversion is not common in the NAS literature, and the experimental tables are detailed and reproducible enough that the empirical claims are checkable. The author also discloses ChatGPT assistance, which is honest. But the load-bearing flaw the stress-test note flags is real, and it is not minor.\n\nSection 4.3 says the 250 or 640 true accuracy vectors were 'augmented by randomly shuffling the accuracy values' to get roughly 10,000 samples. If that means what it says, almost every training pair becomes false: architecture A gets accuracies that were measured for other architectures. Training an inverse model on that can't learn the architecture→performance relationship; at best it learns marginals. The paper treats this as a diversity-boosting trick, but it destroys the very signal the method is supposed to exploit. I don't see an intact-label control anywhere. That alone guts the claim that ASNN 'learned' to suggest better architectures.\n\nSecond, the headline improvements are tiny: 0.98363 vs the original best 0.98310 in 2-layer, and 0.98313 vs 0.98171 in 3-layer. With 10 trials, standard errors on those means are on the order of 0.0003–0.0005, so the differences are around one or two standard errors — not conclusive even if the labels were clean. And the 'efficient alternative to random search' claim is never tested: there is no random-search baseline with the same evaluation budget, which is a strange omission given that the abstract and discussion lean on it.\n\nWorth noting as credit: each suggested architecture is actually trained and evaluated; the numbers in the tables are not fabricated. The iterative loop is also transparent, and the paper is short enough that a reader can see exactly what was done. But the method section is so under-specified (no ASNN architecture, no hyperparameters, no dataset/task named, just 'TensorFlow-based models' on a classification task) that independent replication would be hard.\n\nWho is this for? Someone curious about inverse performance-to-architecture models as a thought experiment might get value, but as a scientific claim it is not supported. I would not cite it. I would still send it to a referee if I were an editor — the idea is worth a rigorous look, and the flaws are the kind a sharp referee could catch and the author could fix with a re-run. But my own verdict is that it needs major revision, likely a full redo of the augmentation and a real baseline comparison, before it carries weight.\n\nRegards.","headline":"Interesting framing but the data augmentation destroys the label-architecture pairing, so the headline results don't support the central claim; still worth a hard-nosed referee for the idea and the honest disclosure.","tokens_in":11707,"tokens_out":685,"would_cite":false,"duration_ms":9417,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural network that maps accuracy vectors to layer sizes can propose MLP architectures that outperform the best architecture in its training grid.","keywords":["neural architecture search","inverse regression","multi-layer perceptron","performance prediction","iterative retraining","architecture optimization","accuracy distribution","random search"],"falsifier":"Train two ASNN models on the same grid data: one with the true accuracy-architecture pairs and one with the same accuracy values randomly reassigned to architectures (the paper's augmentation). If the shuffled-label model proposes architectures whose mean test accuracy is statistically indistinguishable from the true-pair model's proposals, then the reported improvement does not come from learning the performance-structure relationship. A second check: compare each predicted architecture's ten-run mean against the best mean among all original grid architectures using a paired test over trial seeds; if the difference is within run-to-run noise, the claim of outperforming the grid is not established.","tokens_in":10695,"feed_emoji":"🧠","tokens_out":6741,"duration_ms":65256,"temperature":0.7,"pith_summary":"This paper proposes the Architecture Suggesting Neural Network (ASNN), a small neural network trained to do the reverse of the usual mapping: instead of predicting test accuracy from a network's layer sizes, it takes a vector of ten accuracy measurements as input and outputs suggested numbers of nodes in each layer. The authors train this inverse model on a coarse grid of MLP architectures (25 two-layer and 64 three-layer configurations, each trained ten times), then repeatedly feed it an all-100 input that stands for maximum scaled accuracy to obtain new architectures, measure those, and add them to the training set. They report that in both the two-layer and three-layer settings the predicted architectures achieved mean test accuracies above the best mean accuracy in the original grid, with the best two-layer suggestion reaching 0.9838 and the best three-layer suggestion reaching 0.9831. If these results hold, ASNN would be a cheap, iterative alternative to random search for architecture optimization, needing only a few proposed architectures to find better configurations than the grid contained.","feed_headline":"Inverse model finds MLP architectures that beat its training grid","feed_subtitle":"A vector of ten accuracy scores feeds a small net that predicts layer sizes that beat the best grid results.","key_machinery":"The central object is the inverse accuracy-to-architecture map learned by ASNN, written $ASNN: \\mathbb{R}^{10} \\to \\mathbb{R}^{2}$ (or $\\mathbb{R}^{3}$): a vector of ten test-accuracy trials maps to layer sizes. The mechanism is an iterative closed loop: train ASNN, feed the all-100 vector, round the predicted node counts, train and evaluate that architecture ten times, append the new (accuracy-vector, architecture) pairs to the dataset, and retrain. The all-100 input is what carries the extrapolation: it is outside the range of any real measured accuracy, so the model is being asked to invert its learned mapping at a point corresponding to perfect performance.","core_discovery":"The paper's claim is that an inverse regression model can generalize the accuracy-to-architecture relationship beyond the evaluated grid. ASNN is trained with accuracy vectors as inputs and architecture parameters (layer sizes) as outputs, on data where the raw accuracy values are randomly shuffled to create about 10,000 training samples. At inference, the fixed input $(100,100,\\ldots,100)$ encodes the goal of maximal accuracy, and the output is rounded to node counts. The discovered architectures—(448,65), (313,72), and others in the two-layer case; (339,184,66) and others in the three-layer case—were trained from scratch and their averaged ten-run test accuracies are reported. The paper claims this shows ASNN has learned the performance-structure relationship and provides an efficient alternative to random search.","pith_inferences":["Editorial inference: the reported gains could be produced by the iterative expansion alone, because adding the measured high-scoring architectures to the training set biases later predictions toward their neighborhood; a control that adds random architectures to the dataset would isolate the contribution of the learned inverse map.","Editorial inference: a direct test of the learning claim would be to hold out entire grid cells, train ASNN only on the rest, and see whether its suggested architectures beat the best held-out cell—something the paper does not report.","Editorial inference: the use of a fixed $(100,\\ldots,100)$ input is equivalent to asking the model for an infinite extrapolation; the approach would be easier to interpret if the input were reparameterized as a high but realistic quantile of the observed accuracy distribution, for example the observed maximum."],"forward_implications":["If ASNN works as claimed, a user can start from a small grid of a few hundred training runs and obtain improved MLP layer sizes with only a handful of extra evaluations.","The iterative retraining loop turns architecture search into a fixed-point problem: each new architecture's measured accuracies are fed back, so later suggestions reflect the expanded region of the search space.","Because the method does not rely on gradients through the evaluated network or on a separate controller, it could apply to any architecture family whose performance can be summarized as a vector of repeated trial scores.","The two-layer and three-layer experiments suggest the benefit persists as depth grows from two to three layers, at least for the small MLPs used.","The reported margin over the best grid point is small—around 0.0005 to 0.0014 in mean accuracy—so even a modest gain would need to be weighed against the cost of the extra evaluations."],"supporting_citations":[{"why":"provides the universal approximation theorem the paper cites to justify using a neural network as the ASNN function approximator.","marker":"[3]"},{"why":"extends universal approximation to multilayer feedforward networks, the class ASNN itself belongs to.","marker":"[4]"},{"why":"motivates that deeper is not always better, supporting the need for architecture suggestion.","marker":"[1]"},{"why":"demonstrates architectural changes can be decisive for performance, the premise of the search.","marker":"[2]"},{"why":"supplies the deep reinforcement-learning lookup-table-to-function-approximator analogy used to frame ASNN.","marker":"[7]"},{"why":"defines neural architecture search with reinforcement learning, the line of work ASNN offers as a simpler alternative.","marker":"[10]"},{"why":"represents differentiable architecture search, an alternative NAS strategy that ASNN avoids.","marker":"[11]"}],"fun_headline_variants":["Inverse model suggests MLPs that beat its training grid","ASNN inverse mapping finds superior layer sizes from accuracy vectors","Neural net designs better neural nets than its own training data","Shuffled accuracy scores teach a net to suggest winning architectures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that randomly shuffling accuracy values across architectures to expand the dataset preserves enough of the true accuracy-to-architecture relationship for the trained model, and that feeding an all-100 input extrapolates that relationship to genuinely better architectures; if either fails, the reported improvements do not demonstrate learning.","fun_headline_variants_meta":{"raw":{"variants":["Inverse model suggests MLPs that beat its training grid","ASNN inverse mapping finds superior layer sizes from accuracy vectors","Neural net designs better neural nets than its own training data","Shuffled accuracy scores teach a net to suggest winning architectures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1602,"prompt_tokens":966,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":567}},"tokens_in":582,"tokens_out":636,"duration_ms":7296,"temperature":1.0,"reasoning_tokens":567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:47:49.555205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two ASNN models on the same grid data: one with the true accuracy-architecture pairs and one with the same accuracy values randomly reassigned to architectures (the paper's augmentation). If the shuffled-label model proposes architectures whose mean test accuracy is statistically indistinguishable from the true-pair model's proposals, then the reported improvement does not come from learning the performance-structure relationship. A second check: compare each predicted architecture's ten-run mean against the best mean among all original grid architectures using a paired test over trial seeds; if the difference is within run-to-run noise, the claim of outperforming the grid is not established.","supporting_citations":[{"cited_title":"Approximation by superpositions of a sigmoidal function,","cited_arxiv_id":null,"evidence_quote":"provides the universal approximation theorem the paper cites to justify using a neural network as the ASNN function approximator."},{"cited_title":"Approximation capabilities of multilayer feedforward networks,","cited_arxiv_id":null,"evidence_quote":"extends universal approximation to multilayer feedforward networks, the class ASNN itself belongs to."},{"cited_title":"Do deep nets really need to be deep?","cited_arxiv_id":null,"evidence_quote":"motivates that deeper is not always better, supporting the need for architecture suggestion."},{"cited_title":"Deep residual learning for image recognition,","cited_arxiv_id":null,"evidence_quote":"demonstrates architectural changes can be decisive for performance, the premise of the search."},{"cited_title":"Human-level control through deep rein- forcement learning,","cited_arxiv_id":null,"evidence_quote":"supplies the deep reinforcement-learning lookup-table-to-function-approximator analogy used to frame ASNN."},{"cited_title":"Neural architecture search with reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"defines neural architecture search with reinforcement learning, the line of work ASNN offers as a simpler alternative."},{"cited_title":"DARTS: Differentiable architecture search,","cited_arxiv_id":null,"evidence_quote":"represents differentiable architecture search, an alternative NAS strategy that ASNN avoids."}],"review_version":1}