{"id":"979e3ab4-a833-4c55-97ef-aceaf29af424","arxiv_id":"2506.04001","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CARL improves NAS performance predictors by separating critical from redundant architecture features and training with latent-space interventions.","lead":"This paper introduces CARL, a method that splits a neural network architecture into critical and redundant parts in a learned latent space and trains the performance predictor to rely mainly on the critical part by pairing critical representations with random redundant ones. It reports improved accuracy and ranking for architecture performance prediction on five neural architecture search benchmark spaces.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim rests on an untested assumption: the random-pair intervention does not provably cut the backdoor path, and the original loss still permits reliance on redundant features.","rationale":"The paper's novelty and the interpretability claim rest on causal separation: CARL claims to discover critical features that cause performance, not merely correlate. If the latent intervention does not actually enforce invariance to redundant features, then the central mechanism is unverified, and the method reduces to a heuristic feature-swap regularizer whose gains may come from augmentation, not from causal adjustment. The reader's weakest_assumption points to this same issue, but I sharpen it: the loss does not directly measure invariance, and the original disentanglement has no identifiability guarantee, so the concern is not just theoretical but testable. The proposed test directly measures whether predictions change when redundant representations are swapped, which is the operational meaning of the backdoor adjustment. If the test fails, the causal claims in the abstract and Section 4.3 should be withdrawn, and the paper's contribution would need re-framing. If it passes, the causal interpretation gains support. This is more load-bearing than the missing error bars because it concerns the validity of the proposed method, not just the strength of the evidence.","tokens_in":15136,"tokens_out":6319,"duration_ms":62134,"concrete_test":"Train CARL on NAS-Bench-201 with the 0.5% training split (matching Table 1). After training, freeze the model. For 100 held-out architectures, extract the critical representation ZC_i. For each i, sample 20 redundant representations ZR_j uniformly from architectures in the search space that were not used in training. Compute predicted performance for the concatenated inputs (ZC_i, ZR_j). Measure the standard deviation of predictions across j for each i. If the average standard deviation exceeds, say, 5% of the standard deviation of true accuracies in the held-out set, the predictor is not invariant to redundant features, contradicting the claimed backdoor adjustment. Report the median and 90th percentile of these per-architecture standard deviations, and compare with a control using a randomly initialized (untrained) regressor to calibrate the scale of non-invariance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CARL's central claim is that its latent-space intervention in Section 4.3 implements a backdoor adjustment that removes spurious dependence of performance on redundant features, so the learned critical representation is causal. This requires that (i) the substructure extractor provably separates causal from redundant information, leaving no residual redundancy in ZC, and (ii) the interventional loss ensures the regressor's prediction is invariant to the redundant representation ZR. Neither is established. The extractor is a softmax-weighted mask trained end-to-end; it has no identifiability guarantee, so ZC may still contain redundant information. The loss LI (Eq. 5) is a ranking loss on mixed pairs; it penalizes ranking errors but does not directly penalize prediction variance under ZR substitution. A regressor can keep the correct ranking on average while still responding to ZR, and the absence of a variance penalty means invariance is only an emergent, untested property. Moreover, the pairing draws ZR from the same mini-batch; if the mini-batch is small or the redundant representations are clustered, the interventional distribution differs little from the original, so the backdoor path is not effectively cut. The paper does not report mini-batch size, the diversity of sampled redundant representations, or any diagnostic showing that predictions are unchanged when ZR is replaced. Without such evidence, the 'causal' interpretation is unsupported; the observed gains could come from the regularizing effect of feature swapping rather than from a genuine backdoor adjustment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes CARL, an architecture performance predictor for NAS that splits GCN-based latent representations of a neural architecture into 'critical' and 'redundant' substructures via softmax-weighted masks, generates interventional training samples by pairing critical representations with redundant representations drawn from other mini-batch architectures, and trains with a combined loss consisting of two hinge ranking losses and a redundant-feature MSE loss. The authors evaluate CARL on NAS-Bench-101, NAS-Bench-201, TransNAS-Bench-101, NAS-Bench-NLP, and DARTS, reporting Kendall's Tau rankings and predictor-guided search results, and claim state-of-the-art accuracy and interpretability, for example 97.67% CIFAR-10 top-1 accuracy with a DARTS search space.","tokens_in":15469,"tokens_out":4639,"duration_ms":45230,"significance":"The empirical scope is substantial: five search spaces, ranking and search protocols, ablations of encoder choice, node/edge disentanglement, and loss weights, plus a qualitative visualization that aligns with earlier motif-based analyses. If the causal-interpretation claim were supported, CARL would be a notable step toward more generalizable and interpretable NAS predictors. However, the main advertised contribution—causality-guided separation and backdoor adjustment at the representation level—is not established by the current theory or diagnostics, and several headline numbers lack variance information. The empirical gains are still valuable as an engineering contribution, but the paper as written overclaims the causal mechanism.","major_comments":[{"comment":"The central claim that the random-pair interventional loss implements a backdoor adjustment is unsupported. A backdoor adjustment requires a formal adjustment set and an identifiability condition; pairing Z_C from one architecture with a randomly drawn Z'_R from another architecture in the same mini-batch changes the joint distribution, but there is no argument that predictions computed from such pairs equal the causal effect of C on Y, nor is there a condition under which the estimator is consistent. If the mini-batch is small or the redundant representations are clustered, the intervention differs little from the observational distribution. To keep the causal claim, the authors should either provide a formal identification result or substantially weaken the claim to 'a heuristic that encourages robustness to redundant representation substitution' and add diagnostics such as prediction variance under Z_R replacement.","section":"Section 4.3, Eq. (5)"},{"comment":"The disentanglement procedure cannot guarantee that Z_C is free of redundant information. The masks are softmax outputs of an MLP trained end-to-end; without an independence or identifiability constraint, residual dependency between Z_C and R can persist. The loss L_R penalizes deviation of the redundant-branch prediction from the mean accuracy, but it does not penalize reliance on redundant information inside Z_C. Consequently, the 'critical features' may be merely the features that are easiest for the ranking loss to exploit, making the causal-interpretation claim partly circular. A concrete remedy is to add an independence or adversarial penalty and to evaluate whether predictions are invariant to substituting Z_R.","section":"Section 4.2, Eqs. (2) and (4)"},{"comment":"The headline DARTS results are not accompanied by any measure of uncertainty. Table 5 reports 97.67% versus PINAT's 97.58%, a 0.09 percentage point margin, and Table 4 reports 76.1% versus PRE-NAS's 76.0%, a 0.1 point margin. It is not stated whether these are single runs or averages, nor the number of seeds; such differences are typically within run-to-run variation for DARTS retraining. Please report mean and standard deviation over at least 3-5 independent search runs and the retraining seed protocol. This is necessary to support the state-of-the-art claim.","section":"Tables 4 and 5"},{"comment":"The ranking experiments also lack error bars, and the conclusion that CARL 'takes the lead across six data splits' is based on small Kendall's Tau differences, for example 0.683 versus 0.679 on NAS-Bench-101 at 0.02% training data. The paper does not state the number of seeds or the variance of the reported Ktau values. In addition, the training setup omits several hyperparameters that the method depends on: the mini-batch size (which directly affects the diversity of redundant representations used in Eq. (5)), the margin m, the GCN depth and hidden dimensions, and the specific values of lambda_1 and lambda_2 used in the main experiments. No code is released, which further impedes verification.","section":"Section 5.1, Table 1"}],"minor_comments":[{"comment":"The notation \\hat{y}_i^c and \\hat{y}_i^I is used without definition; please clarify that these are predictions from the critical-only and interventional branches, respectively.","section":"Eqs. (3) and (5)"},{"comment":"The text refers to 'PINAT [30]' while the reference list and other parts of the paper use '[29]' for PINAT; please unify the citation numbering.","section":"Section 5.2"},{"comment":"The sentence containing 'crtical features' has a typo, and the abstract contains nonstandard spacing in 'C ausality-guided A rchitecture Representation L earning'; please fix these formatting issues.","section":"Section 5.4"},{"comment":"The caption says 'edge types represent operations and edge direction represent the computation flow'; please correct the grammar and clarify how node importance scores are visualized, if at all.","section":"Figure 5 caption"},{"comment":"The reader cannot tell which axis corresponds to lambda_1 and which to lambda_2; please state this explicitly in the caption.","section":"Table 9"},{"comment":"The ImageNet result is obtained by transferring an architecture searched on CIFAR-10; please state this in the caption or main text for clarity.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"This is a competent empirical paper, but the causal framing currently exceeds what is demonstrated. I would encourage the editor to invite a revision that either supplies identification or invariance evidence, or reframes the method as a causally motivated regularization, while also adding uncertainty reporting for the main results. The missing code is a concern given the small margins in several comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2506.04001. First, the empirical work is real and substantial: CARL is evaluated across five NAS search spaces, including skeleton-based macro and NLP benchmarks, and its ranking results on NAS-Bench-101/201 are consistently at or near the top across training splits. Second, the causal framing is doing heavy lifting that the evidence does not support. The paper calls the latent feature-swap a 'causal intervention' that cuts the backdoor path from redundant features to performance, but there is no identifiability result for the disentanglement and no distributional condition under which the random-pairing loss provably severs the spurious dependence. The stress-test note gets this right. The intervention is a regularizer that encourages invariance; it may work empirically, but the paper does not show it works by cutting a backdoor path.\n\nWhat is actually new: the specific pipeline—softmax node/edge masks to split representations into critical and redundant parts, plus a hinge loss on pairs where the redundant half is randomly drawn from other architectures in the mini-batch—is not in the cited predictor literature. The ablation shows both disentanglement and the interventional loss matter on NAS-Bench-201. The visualization results (good architectures favor convs, poor ones avg-pool near input) match prior motif analyses, which is a nice sanity check.\n\nSoft spots, in order of severity. (1) The causal claim is unsupported as stated. No proof that ZC contains no redundant information, no diagnostic that predictions are invariant when ZR is replaced, and no mini-batch diversity analysis. The paper should soften 'causal' to 'causal-inspired' and add an invariance diagnostic. (2) Reproducibility: no code, no seed counts for the main ranking table, no error bars. The DARTS CIFAR-10 gain over PINAT is 0.09 percentage points; without multiple runs that is within noise. (3) The hyperparameters λ1 and λ2 used in the main experiments are not stated; the ablation table gives a grid but not the chosen operating point. That is a minor fix.\n\nThe math is mostly standard GCN + MLP + hinge losses; no internal contradictions. Citation pattern is appropriate and includes recent predictor papers. Overall the empirical contribution is likely real and the method is worth understanding. I would send it to a serious referee, but the referee should request code, error bars, and a rewrite of the causal section. I wouldn't cite it in my own work until the code is out and the invariance claim is checked.\n\nVerdict: conditionally accept after revision; not desk-reject.","headline":"Solid empirical NAS predictor with an overclaimed causal story; deserves review but needs reproducibility fixes and a softer causal claim.","tokens_in":15975,"tokens_out":1882,"would_cite":false,"duration_ms":18387,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A NAS performance predictor that splits architectures into causal and redundant substructures and trains on re-paired representations generalizes better across five search spaces, reaching 97.67% top-1 accuracy on CIFAR-10.","keywords":["neural architecture search","performance predictor","causal intervention","representation learning","spurious correlation","graph neural network","interpretability","NAS-Bench"],"falsifier":"Train CARL and a standard predictor on a NAS benchmark whose training split is deliberately biased so that one redundant feature (for example, the number of average-pooling operations) is perfectly correlated with accuracy in the training set but uncorrelated in the test set. If CARL's ranking correlation on the test set stays high while the standard predictor collapses, the intervention is removing a real confound; if both drop equally, the causal mechanism is not doing the work. A second check is to freeze the learned masks and re-run prediction with redundant representations drawn from a different search space: invariance to that swap would confirm the backdoor path is cut.","tokens_in":14900,"feed_emoji":"🧠","tokens_out":6949,"duration_ms":62139,"temperature":0.7,"pith_summary":"The paper argues that neural architecture search (NAS) performance predictors fail mainly because they learn spurious correlations: with only a tiny, biased training sample, predictors latch onto redundant architectural features that happen to correlate with good performance in that sample but not in the broader search space. To fix this, the authors propose CARL, which splits each architecture's latent representation into critical (causal) and redundant parts, then trains the predictor on re-paired samples in which a critical representation is combined with a random redundant representation from another architecture. The central claim is that this representation-level intervention removes the backdoor path from redundant features to performance, forcing the predictor to rely on the critical substructures that actually determine accuracy. If correct, CARL gives NAS a predictor that stays accurate with very small training sets and that also exposes which operations matter, with reported results including 97.67% top-1 accuracy on CIFAR-10 and 76.1% on ImageNet from DARTS-based search.","feed_headline":"Causal split yields sharper NAS performance predictors","feed_subtitle":"By re-pairing critical architecture features with random redundant ones, CARL stays accurate with tiny training sets.","key_machinery":"The load-bearing object is the representation-level causal intervention. The paper models architecture performance prediction with a structural causal model in which architecture data $X$ is produced by critical features $C$ and redundant features $R$, and $C \\rightarrow Y$ is the true performance link while $C \\leftrightarrow R$ creates a backdoor path $R \\leftrightarrow C \\rightarrow Y$. To cut this path without synthesizing invalid architectures, CARL operates in latent space: a substructure extractor (an MLP over GCN node and edge embeddings) produces masks $\\alpha^C_i, \\alpha^R_i$ with $\\alpha^C_i + \\alpha^R_i = 1$, yielding $Z^C$ and $Z^R$; interventional samples $Z^I$ are formed by pairing $Z^C$ with a random $Z'^R$ from another architecture. The intervention-based loss $L = L_C + \\lambda_1 L_R + \\lambda_2 L_I$ then trains the regressor so that ranking is determined by $Z^C$ regardless of which redundant representation it is paired with.","core_discovery":"CARL's central discovery is that a performance predictor can be made to generalize across the distribution shift between a small training sample and a diverse search space by explicitly disentangling architecture representations into critical and redundant substructures and intervening on the redundant part. The intervention is performed in latent space: after a GCN encoder produces node and edge representations, a substructure extractor learns soft masks that split each architecture into a critical substructure $X^C$ and a redundant substructure $X^R$, embedded as $Z^C$ and $Z^R$. During training, each critical representation is paired with a randomly drawn redundant representation from another architecture in the mini-batch, producing interventional samples whose ranking target is the critical part's ground-truth performance. A hinge-ranking loss on critical samples, a mean-squared-error loss pulling all redundant predictions toward a constant, and a hinge-ranking loss on interventional samples together encourage the regressor to base predictions on $Z^C$ alone. The paper demonstrates on five search spaces that this yields better Kendall's Tau rankings, better searched architectures, and importance maps consistent with known performance-relevant motifs.","pith_inferences":["The authors do not test whether the learned redundant representation is truly redundant on a distribution different from the training split; an explicit check would be to swap $Z^R$ with representations from a different search space and measure prediction invariance.","The same disentangle-and-repair recipe could be carried over to other graph-structured regression tasks, such as molecule or program performance prediction, where the training distribution is similarly biased.","If the causal reading is right, CARL's importance scores could be used to prune search spaces before searching, discarding architectures whose critical substructure is already known to be poor.","A sharper test of the interpretability claim would compare the discovered critical substructures against the motifs found by Bayesian-optimization-based interpretable NAS, rather than only by visual inspection."],"forward_implications":["Predictor-based NAS can find competitive or better architectures with fewer queried architectures: on NAS-Bench-201, CARL matches the optimal accuracy on CIFAR-10 and CIFAR-100 with only 80 queries.","The same predictor transfers across tasks: on TransNAS-Bench-101 micro and macro spaces it achieves the best average rank across seven vision tasks, and on NAS-Bench-NLP it reaches 4.572 log perplexity.","With only 0.02% of NAS-Bench-101 as training data, CARL reaches a Kendall's Tau of 0.683, outperforming predictors trained on twice as much data, showing the method matters most when samples are scarce.","The learned importance scores are interpretable: critical features concentrate near the input node and convolution-heavy motifs, consistent with known good motifs, while excessive average pooling marks poor architectures.","CARL's search on DARTS costs about 0.25 GPU days, less than most predictor-based and one-shot baselines, while producing a 97.67% CIFAR-10 and 76.1% ImageNet architecture."],"supporting_citations":[{"why":"supplies the structural causal model and the definition of causal effect that the paper's intervention targets.","marker":"[37]"},{"why":"supports treating the probabilistic dependence between critical and redundant features as a backdoor path to be cut.","marker":"[38]"},{"why":"provides the causal intervention concept that motivates generating interventional samples.","marker":"[39]"},{"why":"shows that not all operations contribute equally to performance, motivating critical-substructure extraction.","marker":"[5]"},{"why":"argues that only a subset of motifs dominates architecture performance, which the disentanglement step operationalizes.","marker":"[43]"},{"why":"provides evidence on redundancy and diversity in cell-based search spaces that justifies separating critical from redundant features.","marker":"[49]"},{"why":"supplies the GCN encoder whose node and edge embeddings carry the representation being disentangled.","marker":"[19]"},{"why":"introduces the ranking-based loss that CARL adapts for critical and interventional samples.","marker":"[36]"}],"fun_headline_variants":["Causality-guided split makes NAS predictors more generalizable","Intervening on redundant features sharpens NAS predictors","Disentangling causal and redundant traits boosts NAS prediction","Only critical features matter: a causal NAS predictor","CARL: learning causal architecture features for robust NAS prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that swapping a critical representation with a random redundant representation from another architecture genuinely breaks the spurious link between redundant features and performance; if the mini-batch lacks redundant-feature diversity, or if the critical representation still encodes hidden shortcuts, the predictor will simply learn a different spurious correlation.","fun_headline_variants_meta":{"raw":{"variants":["Causality-guided split makes NAS predictors more generalizable","Intervening on redundant features sharpens NAS predictors","Disentangling causal and redundant traits boosts NAS prediction","Only critical features matter: a causal NAS predictor","CARL: learning causal architecture features for robust NAS prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1851,"prompt_tokens":958,"completion_tokens":893,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":816}},"tokens_in":574,"tokens_out":893,"duration_ms":7879,"temperature":1.0,"reasoning_tokens":816,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:49:32.218823+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CARL and a standard predictor on a NAS benchmark whose training split is deliberately biased so that one redundant feature (for example, the number of average-pooling operations) is perfectly correlated with accuracy in the training set but uncorrelated in the test set. If CARL's ranking correlation on the test set stays high while the standard predictor collapses, the intervention is removing a real confound; if both drop equally, the causal mechanism is not doing the work. A second check is to freeze the learned masks and re-run prediction with redundant representations drawn from a different search space: invariance to that swap would confirm the backdoor path is cut.","supporting_citations":[{"cited_title":"Models, reasoning and inference","cited_arxiv_id":null,"evidence_quote":"provides the causal intervention concept that motivates generating interventional samples."},{"cited_title":"Causality","cited_arxiv_id":null,"evidence_quote":"supplies the structural causal model and the definition of causal effect that the paper's intervention targets."},{"cited_title":"Not all operations contribute equally: Hier- archical operation-adaptive predictor for neural architecture search","cited_arxiv_id":null,"evidence_quote":"shows that not all operations contribute equally to performance, motivating critical-substructure extraction."},{"cited_title":"Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels","cited_arxiv_id":null,"evidence_quote":"argues that only a subset of motifs dominates architecture performance, which the disentanglement step operationalizes."},{"cited_title":"On redundancy and diversity in cell-based neural architecture search","cited_arxiv_id":null,"evidence_quote":"provides evidence on redundancy and diversity in cell-based search spaces that justifies separating critical from redundant features."},{"cited_title":"Semi-supervised classi- fication with graph convolutional networks","cited_arxiv_id":null,"evidence_quote":"supplies the GCN encoder whose node and edge embeddings carry the representation being disentangled."},{"cited_title":"Ta-gates: An encoding scheme for neu- ral network architectures","cited_arxiv_id":null,"evidence_quote":"introduces the ranking-based loss that CARL adapts for critical and interventional samples."}],"review_version":1}