{"id":"b895c89d-0b15-494e-bfa4-e9d70db89d63","arxiv_id":"2506.02623","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SiamNAS uses an ensemble of Siamese network blocks to predict dominance relations between architectures, replacing costly evaluations during multi-objective neural architecture search.","lead":"An ensemble of Siamese networks is trained to predict which of two neural network designs is better, and is used inside an evolutionary search to find good trade-offs between accuracy, size, and speed. The method, called SiamNAS, finds top architectures on a standard benchmark while spending little search time, though it still needs hundreds of pre-computed evaluations to train the predictor.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported success may be explained by search-space coverage rather than the surrogate: with 2,000 generations of population 50 on NAS-Bench-201's 15,625 architectures, the algorithm can cheaply visit almost every candidate, so a random comparator with the same budget may match the result.","rationale":"The reader's weakest assumption was pairwise consistency, which is a valid theoretical concern about Algorithm 3. I do not choose it as the load-bearing concern because a proof-of-concept can still be empirically successful with an inconsistent comparator; moreover, the reported 92% accuracy suggests that most pairwise comparisons are correct, so transitivity violations may be rare in practice. The train-accuracy-versus-test-error mismatch is also serious, but it can be remedied by a correlation check. The most decisive issue is whether the surrogate matters at all given the budget: with 100k cheap evaluations over a 15,625-architecture space, near-exhaustive coverage can trivially produce the reported result. The paper compares against methods with much smaller true-evaluation budgets and does not report unique-architecture coverage or a trivial comparator. If the control experiment I propose reproduces SiamNAS's performance, the central claim collapses; if it does not, the concern is refuted and the conditional acceptance is justified. Thus I keep the reader's conditional verdict.","tokens_in":16355,"tokens_out":9722,"duration_ms":109297,"concrete_test":"Run a controlled ablation on NAS-Bench-201: keep N_s=600, N=50, T=2000, and the final evaluation phase, but replace the learned surrogate in Algorithm 3's non-dominated sort with (1) a random pairwise comparator and (2) a comparator that always prefers the architecture with more parameters; repeat 10 times with the same random seeds. Record the best test-error architecture in the final population and the fraction of the 15,625 architectures visited by the surrogate/offspring stream. If either control finds the same best architecture, or if the final front error matches SiamNAS within noise, the surrogate is not the cause of the headline result and the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the Siamese dominance surrogate is what enables near-optimal results. The paper does not provide a control that isolates the surrogate. The NAS-Bench-201 search space has 5^6 = 15,625 architectures. Algorithm 1 runs for T=2000 generations with population N=50, producing 50 offspring per generation, so roughly 100,000 candidates are processed by the surrogate. This is more than six times the total search space, so over a run the algorithm can cheaply visit essentially all architectures, with selection pressure supplied only by the surrogate plus the parameter-count heuristic in Algorithm 3. The 600 true evaluations used to train the surrogate are excluded from the 0.01 GPU-day runtime, even though in real NAS those evaluations are the dominant cost. With no budget-matched baseline—e.g., a control using the same 600 true evaluations and the same 100k cheap comparisons but a random or parameter-count-only comparator—the reported discovery of the oracle CIFAR-10 architecture does not establish that the learned dominance relation is doing the work. This is the load-bearing condition for the paper's main claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SiamNAS, a multi-objective neural architecture search framework in which an ensemble of Siamese MLP blocks is trained to predict pairwise dominance between candidate architectures. The surrogate is trained on 600 architectures from NAS-Bench-201 using true objective values, and is then used inside an NSGA-II-style evolutionary loop to perform tournament selection and non-dominated sorting, while survivor selection replaces crowding distance with a heuristic preference for architectures with more parameters. Experiments on NAS-Bench-201 report that SiamNAS finds the oracle CIFAR-10 architecture, obtains competitive results on CIFAR-100 and ImageNet-16-120, and runs in 0.01 GPU days when true evaluations are excluded.","tokens_in":16601,"tokens_out":5500,"duration_ms":57939,"significance":"If the central claim is established, a dominance-relation surrogate that needs only a few hundred true evaluations would be a useful ingredient for multi-objective NAS, and the idea of replacing objective-function approximation with pairwise comparison is interesting and clearly presented. The paper includes pseudocode for all components, ablation studies over ensemble size and training-set size, and comparison against 26 algorithms. However, the current evidence does not isolate the surrogate's contribution: the evolutionary budget is large enough to cover the entire search space many times, no budget-matched control is provided, and the surrogate's low F1 score raises doubts about the reliability of the non-dominated sorting that drives selection. These issues are fixable with additional experiments, so the work is a promising proof of concept rather than a fully supported claim as it stands.","major_comments":[{"comment":"The central claim that the Siamese surrogate enables efficient search is not isolated by any budget-matched baseline. With N=50 and T=2000, Algorithm 1 generates roughly 100,000 offspring in a search space of 5^6=15,625 architectures, so the evolutionary loop can visit every valid architecture many times. The fact that the final population contains the oracle CIFAR-10 architecture could therefore be due to coverage of the entire space rather than to the learned dominance relation. I request control runs with the same 600 true evaluations and the same 100k cheap comparisons but (a) a random comparator, (b) a parameter-count-only comparator, and (c) an oracle comparator using true objectives; these controls determine whether the surrogate is doing the work.","section":"§5.2, Table 4, Algorithm 1"},{"comment":"The surrogate's positive-class F1 score is only 0.70–0.79 on all datasets (Table 1). Algorithm 3 performs non-dominated sorting by thresholding these pairwise predictions; because each pair is classified independently, the resulting relation need not be asymmetric or transitive. Cycles or intransitivities would make the front assignment in lines 1–2 arbitrary and could alter the final population. Please report the fraction of cyclic triples on a held-out set, and/or compare Algorithm 3's front assignment against true non-dominated sorting on the validation set. Without this, the search's selection pressure is not characterized.","section":"Table 1, Algorithm 3 lines 1–2"},{"comment":"The final survivor selection within the last non-dominated front is not made by the surrogate at all: it is a heuristic that keeps the architectures with the largest number of trainable parameters. This heuristic is justified in Section 4.3 only by the empirical observation that larger models often perform better. Because this tie-break is applied after every generation, the reported results are confounded between the surrogate and the parameter-count heuristic. An ablation that replaces the surrogate-based front assignment with a random ordering while keeping the parameter-count tie-break is needed to separate these two contributions.","section":"Section 4.3, Algorithm 3 lines 8–12"},{"comment":"The runtime comparison is incomplete: Table 4 reports 0.01 GPU days for SiamNAS, but that excludes Phase 1 and Phase 3 true-objective evaluations (up to N_s + N = 650 evaluations by the paper's own accounting in Section 4.1). On NAS-Bench-201 those evaluations are free lookups, but in a real NAS setting they are the dominant cost. The comparison with methods such as BANANAS and ReNAS therefore does not support the abstract's 'within 0.01 GPU days' claim, and the total true-evaluation budget should be reported consistently (600 in Table 3 vs. up to 650 in Algorithm 1).","section":"§4.1, Algorithm 1 line 13, Table 4"}],"minor_comments":[{"comment":"The notation x1⪯x2 is nonstandard and potentially confusing: in most multi-objective optimization texts x⪯y means y dominates x, whereas the paper writes 'x1⪯x2, i.e. x1 dominates x2'. Please define the symbol explicitly and use a consistent convention.","section":"Section 3.3"},{"comment":"Equation (2) writes f_e = 1−acc, which suggests a value in [0,1], but Figure 4 reports values such as 5.627 and 3.078×10^1, which are evidently percentages. Specify the units consistently throughout the text and figures.","section":"Section 5, Figure 4"},{"comment":"NASWOT [31] appears twice with different values (7.04±0.81 on CIFAR-10 and 7.19±0.99 on CIFAR-10); one of these rows is likely a different variant or a typo and should be corrected or labeled.","section":"Table 3"},{"comment":"The line 'M ← build ensemble by trained Siamese surrogate models m1...m_R' refers to m_R, but the loop variable is r; this should be m_{N_m} or m_r for consistency.","section":"Algorithm 2, line 20"},{"comment":"The 'Random Search [13]' baseline is cited to a surrogate-assisted NAS paper rather than to the original random search approach; please ensure the citation matches the method being compared.","section":"Section 5.2"},{"comment":"The 'SiamNAS-transfer' experiment is not described in the experimental setup. Specify whether the same 600 CIFAR-10 architectures are used to train the surrogate, whether the search is then run on the target dataset, and whether the final Phase 3 evaluation uses the target dataset's true objectives.","section":"Section 5.2, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for GECCO and the core idea is interesting, but the missing budget-matched control and the low surrogate F1 are load-bearing issues that need to be addressed before the claims can be accepted. The requested controls and consistency analysis are feasible within the manuscript's scope, so I would be willing to reconsider after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful thing: the paper defines a pairwise dominance comparator as a Siamese MLP ensemble with three outputs (A dominates B, B dominates A, incomparable). That is a clean idea and a natural fit for multi-objective NAS, where you only need ordering to run non-dominated sorting. The integration into NSGA-II with a parameter-count tie-breaker is a legitimate engineering choice, and the experiments on NAS-Bench-201 are extensive: twenty-six baselines, ablations on ensemble size and training sample count, ten runs. Credit where due: the reported CIFAR-10 result exactly matches the oracle, which is a strong signal even on a small space.\n\nThe soft spots are the usual ones for this genre, but they matter here because the headline claim is about cost. The 0.01 GPU days excludes the 600 true evaluations used to train the surrogate. In real NAS those evaluations are the dominant expense; on NAS-Bench-201 they are cheap table lookups, but the claim as written hides that. More importantly, there is no budget-matched baseline. The search runs 2,000 generations with population 50 — about 100,000 offspring processed, versus a search space of only 15,625 architectures. A control that uses the same 600 true evaluations and the same number of cheap comparisons, but with a random or parameter-count-only comparator, could plausibly match the result. Without that control, the learned dominance relation is not shown to be doing the work.\n\nTwo smaller issues. The surrogate is trained on train accuracy, not test accuracy, yet the final results are reported as test error. That mismatch should be stated and justified. And the dominance relation is treated as transitive and asymmetric in the non-dominated sorting, but it is just an independently trained binary classifier; nothing enforces consistency, and cycles in the predictions would make the front assignment arbitrary. A quick consistency check on a validation set would address this.\n\nThe paper is honest about being a proof-of-concept, and the comparator idea is worth building on. But the central efficiency claim needs a control experiment, and the budget accounting needs to be explicit. I'd send it for review with major revisions rather than desk-reject, though I wouldn't cite it yet.","headline":"A clean comparator idea for multi-objective NAS, but the efficiency claim is undermined by missing budget-matched baselines and a search space small enough to be covered by the evolutionary budget.","tokens_in":17139,"tokens_out":2622,"would_cite":false,"duration_ms":25409,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a Siamese-network surrogate that predicts pairwise dominance between architectures can replace true objective evaluation throughout an evolutionary multi-objective NAS search, finding the best NAS-Bench-201…","keywords":["Neural Architecture Search","Multi-objective Optimization","Surrogate Model","Siamese Network","Dominance Relation","Pareto Front","NAS-Bench-201","Evolutionary Algorithm"],"falsifier":"Query the trained surrogate on a sample of triples (A, B, C) from NAS-Bench-201 and test whether A-vs-B, B-vs-C, and C-vs-A predictions form a cycle; a nontrivial cycle rate would show that Algorithm 3's front assignment is arbitrary, while a near-zero cycle rate would confirm the consistency assumption on which the method rests.","tokens_in":16150,"feed_emoji":"🤖","tokens_out":6146,"duration_ms":55914,"temperature":0.7,"pith_summary":"This paper proposes SiamNAS, a multi-objective neural architecture search method built around a surrogate that compares two architectures and predicts which dominates the other, avoiding true evaluations during the search loop. The claim is that a lightweight ensemble of Siamese MLP blocks, trained on 600 pre-evaluated architectures from NAS-Bench-201, is accurate enough (about 92 percent) to drive tournament selection and non-dominated sorting. The framework replaces crowding distance with a simple bias toward larger models, then evaluates only the final front with true metrics. If correct, this makes multi-objective NAS dramatically cheaper: the paper reports finding the best CIFAR-10 architecture and the second-best ImageNet-16-120 architecture among all compared methods in about 0.01 GPU days.","feed_headline":"Siamese surrogate finds near-optimal architectures in 0.01 GPU days","feed_subtitle":"A pairwise dominance predictor replaces costly evaluations, hitting the best NAS-Bench-201 error on CIFAR-10.","key_machinery":"The load-bearing object is the Siamese surrogate ensemble. Each block shares weights between two MLP encoders, forms the difference vector d = b1 - b2, passes it through a two-layer MLP with sigmoid output, and rounds the result to a 0/1 dominance vote. An ensemble of seven such blocks makes a prediction, and a two-direction majority vote yields one of three outputs. This comparator replaces every dominance test in the evolutionary loop, including tournament selection and the efficient non-dominated sort, while the crowding-distance step is replaced by a heuristic that keeps larger models within a front.","core_discovery":"The central discovery is that a pairwise dominance classifier can stand in for the expensive objective functions in evolutionary multi-objective NAS. Instead of predicting accuracy, parameter count, or FLOPs, the surrogate takes two one-hot encoded architectures, computes an embedding difference through shared MLPs, and outputs a rounded scalar indicating whether the first architecture dominates the second. An ensemble of such blocks and an asymmetric two-direction check produce three outcomes—first dominates, second dominates, or non-dominated—which feed directly into the non-dominated sorting of a modified NSGA-II-style loop. On NAS-Bench-201, the resulting SiamNAS found the theoretical best architecture for CIFAR-10 (5.63 percent test error) and near-best for CIFAR-100 and ImageNet-16-120, with all true evaluations confined to the initial 600 training architectures and the final front.","pith_inferences":["The consistency assumption is left unverified: testing the trained surrogate for cycles on triples of architectures would directly show whether its pairwise dominance relation is transitive enough for non-dominated sorting to be meaningful.","The heuristic that favors larger models inside a front may itself drive much of the search quality; a controlled comparison against random survivor selection within a front would isolate the surrogate's contribution.","The same pairwise comparator could be applied to any multi-objective evolutionary loop with discrete encodings, not just NAS, whenever cheap pairwise dominance estimates are needed; the paper gestures toward this generality in its SOS discussion.","The proof-of-concept results are limited to NAS-Bench-201's 15,625 architectures; scaling to larger search spaces would require retesting the one-hot encoding and the 600-sample training budget."],"forward_implications":["During the search phase no true objective values are needed; only 600 architectures are evaluated up front and a final front once, bounding total true evaluations by 650 with a population size of 50.","A surrogate trained on CIFAR-10 transfers to CIFAR-100 and ImageNet-16-120, yielding the second-best ImageNet error among all compared methods.","The runtime excluding true evaluations is about 0.01 GPU days, far below gradient-based methods (4 GPU days) and Bayesian-optimization-based neural predictors (2 GPU days).","Increasing the ensemble size beyond seven Siamese blocks gives diminishing returns, with prediction accuracy saturating around 92–93 percent."],"supporting_citations":[{"why":"Provides the NAS-Bench-201 search space and the true objective values used to train the surrogate and evaluate the final front.","marker":"[11]"},{"why":"Supplies the base NSGA-II survivor-selection mechanism that SiamNAS modifies with the biased selection rule.","marker":"[7]"},{"why":"Establishes the dominance-prediction approach and the class-imbalance handling for pairwise ranking surrogates in multi-objective NAS.","marker":"[30]"},{"why":"Supplies the ranking-logic inspiration for using cheap isomorphic comparisons rather than objective approximation in surrogate-assisted NAS.","marker":"[16]"},{"why":"Provides the efficient non-dominated sorting method used inside Algorithm 3 to build fronts from surrogate comparisons.","marker":"[52]"},{"why":"Serves as a runtime and performance baseline, with 2 GPU days versus SiamNAS's 0.01 GPU days.","marker":"[45]"},{"why":"Serves as a neural-predictor baseline with 0.001 GPU days that SiamNAS compares against in runtime.","marker":"[48]"},{"why":"Provides the Adam optimizer used to train each Siamese surrogate block.","marker":"[17]"}],"fun_headline_variants":["Siamese dominance predictor cuts NAS cost to 0.01 GPU days","Pairwise dominance surrogate finds near-optimal NAS in 0.01 GPU days","SiamNAS predicts dominance to reach best architecture in 0.01 GPU days","92% accurate dominance surrogate speeds NAS to 0.01 GPU days"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the surrogate's pairwise dominance predictions are consistent enough—asymmetric and transitive, with no cycles—that non-dominated sorting over those predictions yields a meaningful front ordering.","fun_headline_variants_meta":{"raw":{"variants":["Siamese dominance predictor cuts NAS cost to 0.01 GPU days","Pairwise dominance surrogate finds near-optimal NAS in 0.01 GPU days","SiamNAS predicts dominance to reach best architecture in 0.01 GPU days","92% accurate dominance surrogate speeds NAS to 0.01 GPU days"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000782,"raw_usage":{"total_tokens":3460,"prompt_tokens":958,"completion_tokens":2502,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2419}},"tokens_in":574,"tokens_out":2502,"duration_ms":16594,"temperature":1.0,"reasoning_tokens":2419,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:20:15.726983+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Query the trained surrogate on a sample of triples (A, B, C) from NAS-Bench-201 and test whether A-vs-B, B-vs-C, and C-vs-A predictions form a cycle; a nontrivial cycle rate would show that Algorithm 3's front assignment is arbitrary, while a near-zero cycle rate would confirm the consistency assumption on which the method rests.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NAS-Bench-201 search space and the true objective values used to train the surrogate and evaluate the final front."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the dominance-prediction approach and the class-imbalance handling for pairwise ranking surrogates in multi-objective NAS."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ranking-logic inspiration for using cheap isomorphic comparisons rather than objective approximation in surrogate-assisted NAS."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the efficient non-dominated sorting method used inside Algorithm 3 to build fronts from surrogate comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as a runtime and performance baseline, with 2 GPU days versus SiamNAS's 0.01 GPU days."},{"cited_title":"Kingma and Jimmy Ba","cited_arxiv_id":null,"evidence_quote":"Provides the Adam optimizer used to train each Siamese surrogate block."}],"review_version":1}