{"id":"55ad65a2-95d6-4511-ba54-220ee9c00aef","arxiv_id":"2411.19819","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GradAlign scores architectures by per-sample gradient agreement at initialization and beats most existing training-free NAS metrics on NAS-Bench-101, NAS-Bench-201, and NDS.","lead":"The paper proposes GradAlign, a way to rank untrained neural network architectures by measuring how much each training example's gradient points in the same direction. It shows that this cheap score often predicts final accuracy better than several established zero-cost neural architecture search proxies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GradAlign scores are unnormalized by parameter dimension, so on variable-size NDS search spaces the reported rankings may be dominated by model size rather than per-sample gradient conflict.","rationale":"The reader's weakest assumption was that sign-binarized gradient directions at initialization carry a reliable signal about final accuracy, with the Section 4 theorem only analyzing a single step. My concern is more specific and more directly tied to the proposed scores: both GradAlign variants use unnormalized sign vectors, whose inner products and Gram determinants scale with the number of parameters. Because NDS explicitly includes architectures of varying width and depth, and because the paper itself reports that GradAlign's advantage shrinks on the fixed-size DARTS-fix-w-d space, the empirical ranking could be largely explained by parameter count. This is not a critique of the authors' integrity; it is a missing control in the evaluation protocol. The theoretical mismatch the reader identified is real but secondary: the theorem concerns full per-sample gradients and one gradient step, whereas the method uses sign-binarized gradients and different aggregate statistics. However, even a correct theory about gradient conflict would not rescue the method if the implemented scores are dominated by d. The proposed test—normalizing by sqrt(d) and comparing to a parameter-count baseline—would settle whether the claimed signal exists. The paper also lacks error bars and released code, but the normalization/confound check is the specific load-bearing issue. Verdict should remain CONDITIONAL, with the explicit condition that the authors rule out the parameter-count confound; if the test fails, the central claim should be rejected.","tokens_in":12377,"tokens_out":10846,"duration_ms":97797,"concrete_test":"On NDS, compute Kendall's tau between a trivial parameter-count proxy and final validation accuracy in each search space. Then recompute GradAlign-I and GradAlign-II after normalizing each sign gradient by sqrt(d), or by replacing the Gram matrix with cosine similarities, and re-rank architectures within DARTS, DARTS-fix-w-d, NASNet, and Amoeba. If the normalized GradAlign tau on variable-size spaces is no better than the parameter-count proxy's tau, or if it stays near zero on DARTS-fix-w-d while parameter count is fixed, the reported scores are measuring model size rather than per-sample gradient conflict.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"GradAlign-I and GradAlign-II are computed on unnormalized sign gradients. For a network with d parameters, sign(g) is in {±1}^d. In GradAlign-II's Gram matrix (Eq. 12), every diagonal entry is exactly d, and off-diagonal entries are O(sqrt(d)) under near-random alignment, so log det G is approximately N_c log d plus a lower-order correction. Averaged over classes, GradAlign-II is essentially (N/C) log d, a monotone function of parameter count. GradAlign-I's per-term dot products also scale with d. The paper never divides by d, centers the score, or compares against a trivial parameter-count baseline. On NDS, where widths and depths vary, GradAlign-II obtains high Kendall's tau on DARTS (0.540) and Amoeba (0.257), but drops to 0.148 on DARTS-fix-w-d, where size variation is removed; this is exactly the pattern expected if the metric is a proxy for model size rather than gradient conflict. This confound is unaddressed and makes the central empirical claim unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GradAlign, a training-free neural architecture search (NAS) proxy that ranks architectures by measuring per-sample gradient conflicts at random initialization. Two scoring strategies are proposed: GradAlign-I, which averages the alignment of sign-binarized per-sample gradients with the sign of the mean gradient, and GradAlign-II, which computes the log-determinant of the Gram matrix of sign-binarized per-sample gradients. The authors claim a theoretical result (Theorem 1) linking conflicting per-sample gradients to slower convergence, and they evaluate GradAlign on NAS-Bench-101, NAS-Bench-201, and NDS, reporting Kendall's tau correlations with final accuracy. They also argue that the number of linear regions is an unreliable training-free proxy because it is sensitive to parameter perturbations. The central claim is that GradAlign achieves better overall ranking performance than existing training-free NAS methods.","tokens_in":12641,"tokens_out":6201,"duration_ms":56695,"significance":"If the empirical claims hold, GradAlign would provide a simple and inexpensive proxy for architecture performance that could be useful in NAS pipelines, especially in settings where training is prohibitive. The paper includes a comparison against a wide set of established baselines on standard benchmarks, which is a strength. The attempt to give a theoretical motivation for gradient-conflict-based scores is also valuable, and the negative result on the sensitivity of linear region counts is a useful caution. However, the theoretical foundation is currently flawed: the proof of Theorem 1 contains an algebraic error, and the result is not connected to the actual sign-binarized scores used in the method. In addition, the empirical evaluation lacks error bars and does not control for a plausible parameter-count confound on the NDS search spaces. These issues make the central claim not yet established, though they appear addressable with additional analysis.","major_comments":[{"comment":"The algebraic step from Eq. (9) to Eq. (10) is incorrect. In Eq. (9) the cross term is 2 g1·g2, which equals 2 cosβ ||g1|| ||g2||, not cosβ ||g1||^2 ||g2||^2. Consequently, the bound in Eq. (3), L(θ) − L(θ+) ≥ (λ/2)(2G + cosβ G^2), does not follow from the proof. The expression also mixes quantities of different orders: with ||g_i|| ≤ √G, the correct bound would involve G(1+cosβ) after a proper manipulation, not G and G^2. As written, Theorem 1 is unproven.","section":"Section 4, Eq. (10) and Theorem 1"},{"comment":"The theoretical result analyzes raw per-sample gradients g1 and g2 and a single full-batch gradient step, whereas GradAlign-I and GradAlign-II are defined on sign-binarized gradients and do not correspond to a gradient descent update. No argument is given for why replacing gi by sign(gi) preserves the ordering predicted by the theorem, nor why the Gram-matrix log-determinant or the mean-alignment score should behave like the bound in Eq. (3). The statement in the abstract that the authors 'theoretically show' the connection is therefore not supported by the presented theory. Either a formal bridge should be provided, or the theory should be reframed as a motivating heuristic.","section":"Sections 4 and 5"},{"comment":"The GradAlign scores are not normalized by the number of parameters, which introduces a plausible confounding factor on NDS where architecture widths and depths vary. For sign-binarized gradients in {±1}^d, the diagonal entries of the Gram matrix in Eq. (12) equal d and off-diagonal entries are of order O(√d) under near-random alignment, so log det G for a class of N_c samples is approximately N_c log d plus lower-order terms; after averaging over classes, GradAlign-II is essentially (N_total/C) log d. GradAlign-I's inner products also scale linearly with d. On DARTS-fix-w-d, where this size variation is removed, GradAlign-II drops from 0.540 to 0.148 and GradAlign-I from 0.545 to 0.189, a pattern consistent with a size-driven proxy. The paper should add a parameter-count baseline (e.g., Kendall's tau of d alone), normalize the scores, or otherwise show that the NDS rankings are not dominated by model size.","section":"Section 6.3 and Table 6"},{"comment":"All reported Kendall's tau values are point estimates without error bars or statistical significance. Several differences that support the main claim are very small, for example on DARTS GradAlign-I (0.545) vs. GradSign (0.541), and on Amoeba GradAlign-I (0.274) vs. GradSign (0.260). To substantiate the claim of better overall performance, the authors should report mean and standard deviation over multiple random initializations and probe batches, and perform a significance test (e.g., Wilcoxon signed-rank) for the pairwise differences.","section":"Section 6.4, Tables 2–6"}],"minor_comments":[{"comment":"The statement 'our empirical findings indicate its comparatively limited efficacy' refers to experiments that are not shown; please include these results or replace the claim with a citation.","section":"Section 5"},{"comment":"The phrase 'reduced to 0.60s by leveraging functools 3 for computing per-sample gradients in parallel' is unclear; functools is a standard library and not a parallelization tool, so please clarify the implementation detail.","section":"Section 7"},{"comment":"There is a typo in the abstract: 'during at initialization' should be 'during initialization'. Also, in Section 6.4 the NDS paragraph repeats 'the highest the highest'.","section":"Abstract and text"},{"comment":"The caption contains a typo: 'seperately' should be 'separately'.","section":"Table 8 caption"},{"comment":"The scatter plots lack axis labels and a description of which search space and dataset they correspond to; please add these details.","section":"Figures 2 and 3"},{"comment":"The claim that 'both GradAlign-I and GradAlign-II identify a well-performing network that is in closer proximity to the best-performing network' is not uniformly supported by Table 7; for example, on NASNet and PNAS, NASWOT achieves higher top-ranked accuracy (94.250 and 94.670) than GradAlign-I (92.630 and 93.380). Please qualify this statement.","section":"Section 6.4, NDS top-1 results"}],"recommendation":"major_revision","confidential_remarks":"The paper is a modest incremental contribution over GradSign, with class-wise averaging and two aggregation strategies as the main novelties. The strongest risk is the parameter-count confound on NDS; the authors should be required to add a parameter-count baseline and normalization analysis. The broken theorem is fixable in principle, but as written it undermines the claimed theoretical contribution. The empirical results on NAS-Bench-201 are promising and worth preserving."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GradAlign is a plausible new zero-cost NAS proxy, but the paper's theoretical backing is broken and the main empirical claim is currently unsupported because the scores are unnormalized and probably tracking parameter count.\n\nWhat's actually new: the two gradient-based scores—sign alignment with the mean gradient (GradAlign-I) and log-determinant of the sign-gradient Gram matrix (GradAlign-II)—are new in the training-free NAS literature. The evaluation on NAS-Bench-101, NAS-Bench-201, and NDS is broad, and GradAlign-I's Kendall's tau beats most baselines on those benchmarks. The method is cheap, and the running-time table shows it's competitive. The point about linear-region count being fragile under parameter perturbation is reasonable and consistent with earlier observations.\n\nThe soft spots are serious. Theorem 1 is algebraically wrong: Equation (10) replaces 2 g1·g2 with cosβ ||g1||^2 ||g2||^2, which is dimensionally inconsistent and not a valid step from Equation (9). Even if that were fixed, the theorem bounds one step of full-batch gradient descent; it says nothing about the sign-binarized, class-averaged scores that GradAlign actually computes. So the theory is decorative, not load-bearing.\n\nThere is also an internal sign inconsistency: the text says a lower log-det (more concentrated gradients) should predict faster convergence, but the reported Kendall's tau values for GradAlign-II are positive, meaning higher scores correlate with higher accuracy. Unless the implementation flips the sign, the paper contradicts itself.\n\nThe biggest problem is the parameter-count confound, and the stress-test note is right. Both scores are unnormalized: sign vectors have norm sqrt(d), so GradAlign-I terms scale with d, and GradAlign-II's log-det is roughly (N/C) log d plus lower-order terms. Without dividing by d or comparing against a trivial parameter-count baseline, the rankings on NDS—which drop from ~0.54 on DARTS to ~0.15 on DARTS-fix-w-d—are exactly what a size-proxy would produce. The paper never reports a parameter-count baseline, so the central claim that per-sample gradient conflict drives the correlation is not established.\n\nThe paper would benefit from a serious referee, but it needs major revision: either correct the theory or drop it, normalize the scores or at least add a parameter-count baseline, fix the sign logic, report error bars, and release code. As is, I would not cite it for the claim about gradient conflict. Readers working on zero-cost NAS may find the empirical comparison useful as a starting point.","headline":"A plausible new zero-cost NAS proxy whose empirical claim is undermined by an unaddressed parameter-count confound and a broken theoretical derivation.","tokens_in":13078,"tokens_out":6217,"would_cite":false,"duration_ms":55455,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Gradient clashes at step zero forecast final model accuracy","keywords":["training-free NAS","gradient alignment","per-sample gradients","model performance inference","zero-cost proxies","architecture ranking","initialization","Kendall's tau"],"falsifier":"Train the top-ranked and bottom-ranked architectures selected by GradAlign on a held-out search space over multiple random seeds; if the bottom-ranked architecture matches or beats the top-ranked one, or if Kendall's tau between GradAlign scores and final accuracy is not positive, the central claim is falsified.","tokens_in":12206,"feed_emoji":"⚡","tokens_out":5671,"duration_ms":46188,"temperature":0.7,"pith_summary":"This paper argues that a network's final accuracy can be predicted before training by measuring how much its per-sample gradients disagree at initialization. It introduces GradAlign, which scores architectures by the alignment of sign-binarized gradient directions across samples in a batch. The paper proves a one-step gradient-descent bound showing that conflicting per-sample gradients slow convergence, and empirically shows GradAlign-I and GradAlign-II rank architectures better than most existing training-free proxies on standard NAS benchmarks. If true, this gives a near-instant, training-free signal for neural architecture search and suggests that linear-region counting is fragile.","feed_headline":"Gradient clashes at step zero forecast final model accuracy","feed_subtitle":"A two-second, training-free score ranks architectures by per-sample gradient alignment, beating most NAS proxies.","key_machinery":"The carrying object is the set of sign-binarized per-sample gradients $\\{\\operatorname{sign}(g_i)\\}_{i=1}^{N}$ computed at initialization, grouped by class. GradAlign-I computes the mean gradient direction $\\tilde{g} = \\operatorname{sign}(\\frac{1}{N}\\sum_i g_i)$ and scores $s = \\frac{1}{N}\\sum_i \\operatorname{sign}(g_i)\\cdot \\tilde{g}$; GradAlign-II forms the Gram matrix of these sign vectors and uses $s = \\log\\det G$. The theoretical backing is a one-step descent inequality under an $M$-Lipschitz gradient and step size $\\lambda \\le 1/M$, which shows that the guaranteed loss reduction grows with the cosine similarity $\\cos\\beta$ between per-sample gradients, so lower conflict gives a larger reduction guarantee.","core_discovery":"The central claim is that conflicting per-sample gradients at a random initialization hinder convergence and therefore predict worse final performance. GradAlign quantifies this conflict two ways: GradAlign-I averages the agreement between each sample's sign-binarized gradient and the mean gradient direction, and GradAlign-II takes the log-determinant of the Gram matrix of sign-binarized gradients to measure their volume. The paper proves a Lipschitz-gradient bound for one gradient step showing that a larger cosine angle between two per-sample gradients reduces the guaranteed loss decrease, and treats this as evidence for preferring low-conflict architectures. Empirically, on NAS-Bench-201 and NDS, GradAlign-I attains the highest Kendall's tau among compared training-free methods; on NAS-Bench-101 it is second to Zen-NAS. The paper also demonstrates that the number of linear regions is sensitive to small parameter perturbations, arguing that this widely used proxy is not dependable.","pith_inferences":["The same gradient-conflict idea could be used to measure dataset difficulty or to select which training samples are hardest to reconcile, since low per-sample alignment may flag examples that pull the update in opposing directions.","GradAlign's success at initialization hints that other trajectory-independent statistics, such as gradient norms or Hessian approximations at step zero, may also rank architectures.","A natural extension is to apply GradAlign to transfer learning or fine-tuning, where the pretrained initialization is fixed and per-sample gradient conflicts could predict downstream performance.","The theoretical bound is for a single step; extending it to multi-step or stochastic gradient descent would strengthen the link between initialization statistics and final accuracy."],"forward_implications":["Architecture search can rank candidates in seconds, since GradAlign needs only one forward-backward pass on a single batch.","Gradient conflict at initialization becomes a principled training-free signal, outperforming linear-region counting on most benchmarks.","The demonstrated sensitivity of linear-region counts suggests existing proxies based on activation patterns should be re-examined.","GradAlign can serve as a cheap pre-filter before expensive training, or be combined with other zero-cost proxies to improve ranking stability.","The class-wise grouping makes the score less dependent on inter-class relationships, aiding transfer across datasets."],"supporting_citations":[{"why":"Defines the evaluation protocol (Kendall's tau between initialization scores and final accuracy) and provides the NASWOT baseline.","marker":"[33]"},{"why":"Provides the TE-NAS baseline combining linear-region counts with NTK trainability, which GradAlign compares against.","marker":"[11]"},{"why":"Introduces GradSign, the closest gradient-based training-free method whose assumptions and scores GradAlign extends and improves.","marker":"[56]"},{"why":"Supplies the NTK theory that earlier training-free methods rely on and that the paper argues is unreliable for finite networks.","marker":"[22]"},{"why":"Cites evidence that deep networks do not converge to minimal-complexity solutions, motivating GradAlign's shift from per-sample minima to per-sample gradients.","marker":"[2]"},{"why":"Provides the NAS-Bench-201 benchmark with CIFAR-10, CIFAR-100, and ImageNet16-120 accuracies used to compute Kendall's tau.","marker":"[14]"},{"why":"Provides the NAS-Bench-101 benchmark used for architecture ranking evaluation.","marker":"[54]"},{"why":"Provides the NDS benchmark with six search spaces used for the most extensive comparison.","marker":"[38]"}],"fun_headline_variants":["Training-free NAS scores models by gradient conflict at init","Gradient alignment at start predicts model accuracy","No training needed: gradient clashes reveal architecture quality","GradAlign: measure gradient conflicts to rank networks instantly","Why linear regions fail: gradient conflict is a better NAS proxy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking signal only works if the directions of per-sample gradients at the random start reliably predict final accuracy after full training, even though training is non-convex and these directions change substantially.","fun_headline_variants_meta":{"raw":{"variants":["Training-free NAS scores models by gradient conflict at init","Gradient alignment at start predicts model accuracy","No training needed: gradient clashes reveal architecture quality","GradAlign: measure gradient conflicts to rank networks instantly","Why linear regions fail: gradient conflict is a better NAS proxy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000682,"raw_usage":{"total_tokens":3095,"prompt_tokens":940,"completion_tokens":2155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":2079}},"tokens_in":556,"tokens_out":2155,"duration_ms":14078,"temperature":1.0,"reasoning_tokens":2079,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:46:50.420936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the top-ranked and bottom-ranked architectures selected by GradAlign on a held-out search space over multiple random seeds; if the bottom-ranked architecture matches or beats the top-ranked one, or if Kendall's tau between GradAlign scores and final accuracy is not positive, the central claim is falsified.","supporting_citations":[{"cited_title":"In: ICML","cited_arxiv_id":null,"evidence_quote":"Defines the evaluation protocol (Kendall's tau between initialization scores and final accuracy) and provides the NASWOT baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cites evidence that deep networks do not converge to minimal-complexity solutions, motivating GradAlign's shift from per-sample minima to per-sample gradients."},{"cited_title":"In: International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"Provides the NAS-Bench-101 benchmark used for architecture ranking evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NDS benchmark with six search spaces used for the most extensive comparison."}],"review_version":1}