{"id":"c08e71d8-dff0-4acc-be77-9ba25f064e35","arxiv_id":"1908.06886","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ASED uses an estimation-of-distribution algorithm to search over a probability matrix of layer types, discovering non-regular CNN architectures for USPS and CIFAR-100.","lead":"This paper presents ASED, a neural architecture search method that represents possible network structures as a probability matrix and iteratively tunes it by sampling and evaluating candidate networks. It finds irregular, non-repeating architectures for image classification that are competitive with other automated search methods while using a simpler and more interpretable framework.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Brief-training ranking distortion, admitted in Sec. IV-C, plus random-baseline parity in Table 2, means the EDA update's selection signal is unvalidated; the central claim is not yet fully supported.","rationale":"The reader's weakest assumption correctly identifies the brief-training ranking distortion, and the paper itself flags the problem in Section IV-C. My stress-test concurs and sharpens it with Table 2's random-baseline parity: the full ASED baseline at 256 channels (0.7483) barely matches a single 1000-network uniform-random sample (0.7499), so the EDA update's contribution is not empirically demonstrated. The best variants add inversion or shortcut mechanisms, which could be responsible for the gains, but the core selection mechanism remains unvalidated. This supports the reader's CONDITIONAL verdict rather than reversing it: the concern is genuine, concrete, and addressable, but it does not refute the paper's more modest claim that useful irregular architectures can be found. The proposed concrete test—rank correlation between brief and full training, plus a rerun with longer candidate training at fixed budget—would settle whether the update signal is reliable and whether ASED outperforms random search. No independent formal verification or reproducible code check was assessed, so the empirical evaluation is decisive. Overall, the verdict should remain CONDITIONAL: the central claim is plausible and honestly reported, but the load-bearing selection mechanism needs stronger validation before the claim is accepted as established.","tokens_in":20811,"tokens_out":5269,"duration_ms":63421,"concrete_test":"On CIFAR-100, sample a fixed set of roughly 200 architectures from a mid-search ASED prototype and train each for both 20 epochs (brief) and 200 epochs (full) with identical hyperparameters. Compute the Spearman rank correlation between brief and full validation accuracies separately for each prototype depth used in the search (5 to 16 layers). If the correlation is below about 0.5, the top-K update in Eq. (2) is selecting on noise. Then rerun the baseline ASED search with 50-epoch candidate training using the same total GPU budget (fewer evaluations), and compare the final full-trained architecture with (a) the 20-epoch search's final architecture and (b) the best architecture from the 1000-network uniform-random sample. If the final architecture changes materially or does not exceed the random sample, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the ASED prototype update (Eq. 2) tunes a probability matrix toward high-performance architectures by ranking briefly trained candidates. The authors themselves falsify the necessary ranking assumption in Section IV-C: 'this condition does not strictly hold,' and the baseline variant has the highest 20-epoch validation accuracy yet the weakest full-trained accuracy (Fig. 3 vs. Table 2). This is not a cosmetic caveat: Table 2 shows that a one-shot sample of 1000 networks from a uniform 16-layer prototype reaches 0.7499 at 256 channels, essentially matching the full ASED baseline (0.7483), while the best ASED variants exceed it by only about 2.3 points and do so after adding exploration mechanisms (inversion, shortcuts). Consequently, the observed results do not demonstrate that the EDA selection signal is what drives performance; they are consistent with the search acting as an expensive random sampler plus heuristic perturbations. The strongest claim about discovering competitive irregular architectures may still hold, but the mechanism claimed as the contribution is unvalidated, and no repeated runs or error bars are provided to rule out seed luck.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ASED, an architecture search method that represents a CNN by a matrix of independent per-layer probabilities over ten layer types, updates the matrix with a UMDA-style re-estimation from the best Ks of K briefly trained candidates, and progressively adds layers. It reports experiments on USPS and CIFAR-100, comparing variants with probability capping, prototype inversion, and residual/semi-dense shortcut patterns, and it claims to discover irregular architectures competitive in accuracy and compute with cell-based NAS methods. The paper also includes an initialization baseline and a random-uniform architecture baseline.","tokens_in":21178,"tokens_out":6947,"duration_ms":70256,"significance":"If the empirical claims hold, ASED makes a useful contribution by demonstrating that a simple probability-matrix representation can search a non-cell, continuously growing architecture space, with interpretable state and easy parallelization. The authors are honest about limitations and provide source code. The reported best architectures exhibit genuinely non-repeating layer patterns that cannot be represented by identical cells, which is valuable. However, the strength of the causal claim that the EDA update drives search improvement is currently not established, and several comparisons lack the statistical and procedural controls needed to justify the word 'competitive.'","major_comments":[{"comment":"The paper explicitly states in Section IV-C that the assumption that brief-training rankings match full-training rankings 'does not strictly hold,' and the baseline variant has the highest brief-training validation accuracy but the weakest final accuracy. Since Eq. (2) re-estimates the prototype from the top-Ks briefly trained candidates, the central mechanism—that the prototype is tuned toward high-performance models—is not supported unless the selection signal is validated. I request a quantitative analysis of the rank correlation between 20-epoch and 200-epoch validation performance for a sample of architectures, or, alternatively, a reframing of the contribution as a search heuristic whose improvements come from the inversion/shortcut perturbations rather than from the EDA update.","section":"Section IV-C, Fig. 3, Table 2"},{"comment":"Every ASED variant is reported as a single best architecture from one stochastic search, with no repeated runs, confidence intervals, or significance tests. At 256 channels the ASED baseline (0.7483) is essentially matched by a single sample of 1000 networks from a uniform 16-layer prototype (0.7499), so the result does not demonstrate that iterative prototype updates improve over random sampling. The full-inversion variant's 0.7729 is promising, but without multiple seeds it is not possible to rule out seed luck. Please report repeated independent searches (mean and standard deviation, or at least best-of-k with error bars), and compare against random sampling under a matched computational budget.","section":"Table 2"},{"comment":"The comparison to published NAS methods in Table 3 is not made under a unified training setup; the text acknowledges that the results were obtained under non-matching environments and that some numbers are borrowed from other papers (e.g., PNAS/ENAS/DARTS/NAONet). Because final accuracy is highly sensitive to training schedule, regularization, and preprocessing, the claim that ASED is 'competitive both in accuracy and computational cost' is not established by this table. Please evaluate the final ASED architecture under the same training protocol as at least one strong competitor, or use a standardized benchmark such as NAS-Bench, or restrict the competitive claim to the internal baselines and non-cell-based methods.","section":"Table 3"},{"comment":"Algorithm 1 returns the prototype P and says the final architecture is the one with highest probability, but the experiments report the 'best discovered architectures' and the 'best-performing network structures from each algorithm variant.' These are different selection rules: choosing the highest-probability architecture is a model-based output, while choosing the best validation-scored sampled network is best-of-search and inflates performance via selection bias. The paper must specify which rule was used and, if best-of-search, analyze the validation-gap/selection bias or use a hold-out selection procedure.","section":"Algorithm 1 and Section IV-A"}],"minor_comments":[{"comment":"The normalization formula is incomplete as written; define all symbols and specify how the row-wise scaling is applied when multiple entries are capped.","section":"Equation (4)"},{"comment":"'PReLu' should be 'PReLU'; also 'We choose the follow' is a typo for 'We choose to follow.'","section":"Section IV-A"},{"comment":"The caption references red dotted and green dashed lines but the text should state which colors correspond to max and median, and the y-axis label is missing units (accuracy).","section":"Figure 3"},{"comment":"The row 'ASED (best)' should identify the variant (full inversion), the channel count (256), and the source of the 20-GPU-day search-cost estimate.","section":"Table 3"},{"comment":"The related-work discussion would benefit from a direct comparison of ASED to PARSEC in the experiments or at least a clear statement of why PARSEC was not included in Table 3.","section":"Section II"},{"comment":"The shortcut patterns are fixed rules rather than learned structures; this should be stated more prominently in the abstract or contributions so that readers do not over-interpret 'non-linear connectivity' as optimized connectivity.","section":"Section III-D"}],"recommendation":"major_revision","confidential_remarks":"I am recommending major revision rather than rejection: the paper is transparent about its main weakness, the proposed representation and search variants are interesting, and the missing repeated-run and correlation analyses are obtainable within the manuscript's scope. The editor may want to ensure that the revised version reports multiple seeds and a proper selection-signal validation before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nBottom line: this is a novel NAS paper with an honest evaluation, but the central selection mechanism is not validated by the reported experiments. The authors' own numbers show a one-shot uniform random sample nearly matching the baseline ASED, which undercuts the claim that the EDA update is what drives performance. Still, the work deserves a serious referee.\n\nWhat's actually new: a probability matrix over whole networks rather than cells, with UMDA/PBIL-style updates, dynamic depth growth, prototype inversion, and fixed shortcut patterns. The representation is interpretable and parallel-friendly, the code is on GitHub, and the USPS results show competitive accuracy at much smaller model sizes. The CIFAR-100 architectures in Fig. 4 are genuinely non-repeating, which supports the core claim about discovering irregular structures outside cell-based spaces.\n\nWhat the paper does well: it openly discusses the brief-training ranking distortion and the lack of unified comparison settings, and it includes random-uniform and initialization baselines. That transparency is rare.\n\nThe soft spots, in proportion:\nFirst, the selection signal is unvalidated. Section IV-C admits that 20-epoch rankings 'do not strictly hold' and that the baseline variant had the highest brief-training validation accuracy yet the weakest final accuracy. That is a direct hit on the update rule.\nSecond, Table 2 shows a best-of-1000 uniform random sample at depth 16 reaching 0.7499, essentially matching the ASED baseline (0.7483). The variants that clearly beat it all add inversion or shortcuts. So the gains are at least as attributable to those exploration mechanisms as to the EDA update. A controlled comparison of ASED's update against random sampling with matched depth and channel counts is missing.\nThird, there are no repeated runs or error bars; each result is a single stochastic search, so seed luck cannot be ruled out.\nFourth, the accuracy gap to cell-based SOTA is 5-7 points, which the authors attribute to training setup, but it remains a limitation.\n\nNone of this is fatal to the paper's stated aim of discovering irregular competitive architectures—the paper is more cautious than its abstract. But the mechanism claimed as the contribution is not yet demonstrated.\n\nRecommendation: send to a serious referee. The idea is worth engaging, and the main question—does the EDA update beat random sampling with matched budget—is empirically answerable. I would not cite it until that is settled.","headline":"Novel representation and honest limitations, but the EDA update's value over random sampling is unshown; worth refereeing, not citing yet.","tokens_in":21552,"tokens_out":4471,"would_cite":false,"duration_ms":42627,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Irregular neural network architectures can be found with a probability matrix alone.","keywords":["neural architecture search","estimation of distribution algorithms","probabilistic network representation","convolutional neural networks","CIFAR-100","USPS","architecture search without cells"],"falsifier":"Take a sample of architectures from a late-stage prototype, train each for the 20-epoch brief regime and again for the 200-epoch full regime, and compute the rank correlation between the two accuracy orderings; if the correlation is near zero, the search's selection signal is dominated by training-speed artifacts rather than final model quality.","tokens_in":20649,"feed_emoji":"🧠","tokens_out":5556,"duration_ms":50799,"temperature":0.7,"pith_summary":"This paper introduces ASED, a neural architecture search method that represents a whole feedforward convolutional network as a matrix of layer-type probabilities instead of as a stack of repeated cells or blocks. Each row of the matrix is an independent categorical distribution over ten operations, and the search iterates by sampling networks, briefly training them, re-estimating the matrix from the best candidates, and appending new rows to grow deeper. The paper claims that this representation can reach irregular architectures that cell-based spaces cannot express, and that the discovered models are competitive in accuracy and computational cost. On USPS, the found architectures roughly match a heavily modified ResNet with far fewer parameters. On CIFAR-100, the best ASED model reaches 77.3% accuracy with 16.9 million parameters after 20 GPU-days of search.","feed_headline":"Probability matrix finds irregular neural networks without block constraints","feed_subtitle":"ASED samples networks from a layer-type distribution and grows depth, matching block-based search on CIFAR-100 and USPS.","key_machinery":"The prototype matrix $P$ is the load-bearing object: a discrete probability distribution over layer types for every position in a growing feedforward network. The update rule (Eq. 2) re-estimates $P$ from the empirical layer choices of the best $K_s$ sampled networks, which is the same marginal-re-estimation step used by univariate estimation of distribution algorithms. Around this core, the paper adds three mechanisms: probability capping (clamping row entries to $[p_{\\min}, p_{\\max}]$), prototype inversion (replacing high probabilities with low ones when the mean row $L^2$-norm crosses a threshold, to escape premature convergence), and fixed shortcut patterns—residual or semi-dense—that are applied to every sampled network so deeper candidates train more reliably.","core_discovery":"The paper's central claim is that a single probability matrix—a prototype—can stand in for an entire population of network architectures and can be optimized to produce useful, non-regular CNNs. Under the assumption that each layer type is chosen independently, the prototype $P$ has one row per current layer and one column per operation in the layer library; sampling a network from $P$ gives a concrete feedforward architecture. After $K$ candidates are briefly trained and ranked, the top $K_s$ are used to set $P_{ij}=\\frac{1}{|K_s|}\\sum_{k=1}^{K_s} x^k_{ij}$, the empirical frequency of operation $j$ at layer $i$ among the selected models, and this is followed by appending newly initialized rows. The authors show that the resulting search discovers architectures without repeating operation sequences, such as a CIFAR-100 net dominated by large 7x7 and dilated 5x5 convolutions in later layers, and they report that these architectures are competitive with existing methods even though the search uses no weight sharing and only 20-epoch candidate training.","pith_inferences":["The brief-training rank distortion the authors document suggests a promising extension: replacing the fixed 20-epoch evaluation with adaptive budgets, such as spending more epochs on promising candidates, could improve final architectures without scaling cost linearly.","Prototype inversion behaves like a tabu-style diversification mechanism; one could test whether inverting only the most certain rows (partial inversion) versus all rows is better on deeper searches, especially when combined with shortcut patterns.","The same prototype representation could be applied to other structure-selection tasks, such as choosing operations in recurrent cells or transformer layers, where the independence assumption is even more approximate but may still provide a useful prior."],"forward_implications":["The independence assumption plus a simple sampling loop is enough to find competitive CNNs, so the search space itself—not gradient-based architecture optimization or reinforcement learning—can carry much of the work.","Irregular architectures that mix large kernels, dilated convolutions, and pooling at varying depths are reachable and can be competitive, so cell-based spaces may be leaving useful designs undiscovered.","Because candidate networks are trained independently, the search parallelizes almost linearly with the number of GPUs.","The algorithm's low-complexity bias means it naturally favors simpler, faster-to-train models, which is useful when a compact deployment target is more important than peak accuracy.","Probability capping can stall the search, while inversion and shortcuts improve final accuracy, showing that convergence-control choices materially change the outcome."],"supporting_citations":[{"why":"Supplies the population-based incremental learning update idea that ASED adapts.","marker":"[32]"},{"why":"Supplies the univariate marginal distribution re-estimation rule used in Eq. 2.","marker":"[33]"},{"why":"The closest prior probabilistic structure-distribution NAS method, which ASED contrasts by modeling the whole network rather than cells.","marker":"[52]"},{"why":"Supplies evidence that weight sharing distorts candidate rankings and that random search is competitive in cell spaces, motivating ASED's design choices.","marker":"[31]"},{"why":"Introduced the cell-based search space that ASED explicitly avoids.","marker":"[26]"},{"why":"Provides the residual connection pattern and the vanishing-gradient motivation behind shortcut variants.","marker":"[2]"},{"why":"Provides the dense connectivity pattern that inspires the semi-dense shortcut variant.","marker":"[3]"},{"why":"Supplies the strong modified ResNet baseline on USPS that ASED is compared against.","marker":"[62]"},{"why":"Supplies re-evaluated CIFAR-100 numbers for several comparison methods in Table 3.","marker":"[68]"}],"fun_headline_variants":["One probability matrix replaces a whole population of networks","Sample from a layer-type matrix to find irregular CNNs","Distribution-based NAS discovers non-block architectures","Search by layer-type distributions yields competitive irregular nets","Matrix of probabilities finds high-performance irregular networks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The search assumes that the ranking of candidates after only 20 training epochs is accurate enough to guide re-estimation, and the paper concedes in Section IV-C that this condition does not strictly hold, with the baseline variant scoring best under brief training while ultimately performing worst.","fun_headline_variants_meta":{"raw":{"variants":["One probability matrix replaces a whole population of networks","Sample from a layer-type matrix to find irregular CNNs","Distribution-based NAS discovers non-block architectures","Search by layer-type distributions yields competitive irregular nets","Matrix of probabilities finds high-performance irregular networks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1564,"prompt_tokens":1009,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":485}},"tokens_in":625,"tokens_out":555,"duration_ms":6958,"temperature":1.0,"reasoning_tokens":485,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:31:04.636916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of architectures from a late-stage prototype, train each for the 20-epoch brief regime and again for the 200-epoch full regime, and compute the rank correlation between the two accuracy orderings; if the correlation is near zero, the search's selection signal is dominated by training-speed artifacts rather than final model quality.","supporting_citations":[{"cited_title":"Population-Based Incremental Learning: A Method for In- tegrating Genetic Search Based Function Optimization and Competitive Learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the population-based incremental learning update idea that ASED adapts."},{"cited_title":"The Equation for Response to Selection and Its Use for Prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the univariate marginal distribution re-estimation rule used in Eq. 2."},{"cited_title":"Probabilistic Neural Architecture Search","cited_arxiv_id":"1902.05116","evidence_quote":"The closest prior probabilistic structure-distribution NAS method, which ASED contrasts by modeling the whole network rather than cells."},{"cited_title":"Learning Transferable Architectures for Scalable Image Recognition,","cited_arxiv_id":null,"evidence_quote":"Introduced the cell-based search space that ASED explicitly avoids."},{"cited_title":"Deep Residual Learning for Image Recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the residual connection pattern and the vanishing-gradient motivation behind shortcut variants."},{"cited_title":"Densely connected convolutional networks,","cited_arxiv_id":null,"evidence_quote":"Provides the dense connectivity pattern that inspires the semi-dense shortcut variant."},{"cited_title":"Improving the Capacity of Very Deep Networks with Maxout Units,","cited_arxiv_id":null,"evidence_quote":"Supplies the strong modified ResNet baseline on USPS that ASED is compared against."},{"cited_title":"Neural Architecture Optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies re-evaluated CIFAR-100 numbers for several comparison methods in Table 3."}],"review_version":1}