{"id":"d657582b-1de9-4cb4-82b1-d8189bcec81a","arxiv_id":"2501.03782","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A 3000-architecture benchmark shows ViT OoD accuracy varies widely with architecture and that embedding dimension is the strongest structural correlate of OoD robustness.","lead":"This paper releases a large benchmark of 3000 vision transformer architectures measured on 8 out-of-distribution image datasets. It reports that architecture choices matter for OoD robustness, and that simply increasing the embedding dimension often helps more than other design changes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's OoD rankings come from weight-sharing supernet subnets, not independently trained ViTs; without validating that ranking transfer, the central insights may be artifacts of shared weights.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing issue: the benchmark treats weight-sharing subnet performance as a faithful proxy for independently trained ViT architectures, citing only ID-based evidence. This is the right primary concern because it underpins all headline claims. The 11.85% OoD spread, the claim that architecture design matters, the ID-OoD decoupling observation, the training-free NAS comparison, and the embedding-dimension heuristic are all computed on the same inherited-weights evaluations. If subnet OoD rankings do not match standalone-training rankings, the scientific conclusions could be artifacts of the AutoFormer supernet's shared weights rather than properties of ViT architecture design. The paper does include some complementary evidence, notably the Appendix 8 human-designed ViT study and the frequency analysis, which are useful supporting resources. However, that evidence does not validate the 3,000-architecture benchmark itself: Appx. 8 uses only a handful of hand-modified ViT-B configurations trained on IN-100, and the frequency analysis still uses the shared-weight subnets. The proposed concrete test directly settles the transfer question on a small but representative sample, which is feasible given the paper already reports roughly 3,900 GPU-hours for evaluation. If the test passes, the benchmark is much stronger; if it fails, the paper would need to be reframed as a study of one-shot subnet OoD behavior rather than ViT architecture OoD generalization. The conditional verdict remains appropriate, and no change to the reader's judgment is needed.","tokens_in":25363,"tokens_out":2251,"duration_ms":24441,"concrete_test":"Independently train from scratch a stratified sample of 30-50 subnets spanning low, middle, and high inherited OoD accuracy, using the same AutoFormer training recipe (same epochs, augmentation, and resolution). Evaluate each standalone model on IN-R, IN-C, IN-A, and IN-Sketch, then compute Kendall tau between the inherited-subnet OoD ranking and the standalone-training OoD ranking for each dataset. If tau is high (e.g., > 0.7), the weight-sharing proxy is supported; if tau is low or negative, the benchmark's central insights would need reanalysis on independently trained models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 3 Protocol states that sampled subnets inherit weights from the AutoFormer supernet and that their performance is comparable to or even superior to architectures trained alone, citing [20, 6]. Those citations support ID accuracy; no experiment in this paper tests whether subnet OoD accuracy rankings transfer to independently trained ViTs. Because all 3,000 benchmark entries and every downstream claim (the 11.85% OoD range, ID-OoD decoupling, training-free NAS failure, embedding-dimension benefit, and the HFC analysis) use the same shared weights, a divergence in ranking would invalidate the insights, not merely add noise. The danger is concrete: one-shot supernets are known to rank subnets imperfectly even for ID accuracy, and OoD generalization is more sensitive to optimization details. Capacity-correlated proxies such as #Param/#Flops could win simply because larger subnets obtain better-fitting shared weights in the supernet. The paper should either validate the transfer empirically or explicitly reframe all conclusions as one-shot subnet findings. The Appendix's human-designed ViT experiment (Appx. 8) only varies one attribute at a time on IN-100 and does not provide the missing validation for the 3,000-subnet benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OoD-ViT-NAS, a benchmark of 3,000 ViT architectures sampled from the AutoFormer one-shot supernets (Tiny/Small/Base) and evaluated on 8 ImageNet-scale OoD datasets (IN-C, IN-P, IN-A, IN-O, IN-R, IN-Sketch, Stylized-IN, IN-D). Using this benchmark, the authors investigate (i) the range of OoD accuracy across architectures, (ii) the correlation between ID and OoD accuracy, (iii) the ability of nine training-free NAS proxies to predict OoD accuracy versus simple #Param/#Flops, and (iv) the impact of ViT architectural attributes, concluding that embedding dimension is the most important attribute and that increasing it generally improves OoD generalization. The authors also present a frequency-based explanation and a small human-designed ViT study in the appendix to support the embedding-dimension finding.","tokens_in":25732,"tokens_out":6417,"duration_ms":59381,"significance":"If the benchmark's ranking of architectures is trustworthy, this would be a valuable community resource: it is the first large-scale ViT NAS benchmark targeting OoD generalization, and the released data/code are likely to catalyze future work. The paper also makes concrete, falsifiable claims about the limits of training-free NAS proxies for OoD prediction and about the importance of embedding dimension. However, the benchmark's reliance on weight-sharing supernet subnets, without validation that OoD rankings transfer to independently trained ViTs, places a major asterisk on every insight. The statistical evidence for the training-free NAS comparison is also weaker than the text suggests. The human-designed ViT experiment (Appx 8) provides some partial support for the embedding-dimension conclusion, but it is on a different scale and does not cover the full benchmark.","major_comments":[{"comment":"The benchmark evaluates only weight-sharing subnets extracted from AutoFormer supernets; the paper justifies this by citing [20, 6] for the claim that subnet performance is comparable to independently trained architectures. Those citations support ID accuracy only. No experiment in the paper validates that the OoD accuracy ranking of subnets matches the ranking of the same architectures trained from scratch. This is load-bearing because every downstream analysis (the 11.85% OoD spread, the ID-OoD decoupling in Sec. 4.2, the training-free NAS comparison in Sec. 4.3, and the embedding-dimension claim in Sec. 4.4) is computed from these subnet evaluations. If OoD behavior is more sensitive to weight-sharing artifacts than ID behavior, the conclusions could be artifacts. The authors should either (a) train a representative subset of the 3,000 architectures independently and show that the OoD rankings correlate with the subnet rankings, or (b) explicitly reframe all claims as being about one-shot supernet subnets rather than ViT architectures in general. The current wording in the Abstract and Sec. 1 overstates the scope.","section":"Sec. 3, Protocol"},{"comment":"The claim that simple proxies like #Param and #Flops 'surprisingly outperform' complex training-free NAS is not justified by the reported statistics. The Kendall tau values are presented as means with standard deviations (e.g., #Param 0.3600±0.2321 vs AutoProx-A 0.3303±0.2384; #Flops 0.3537±0.2327 vs DSS 0.3421±0.2365). These differences are far smaller than the quoted standard deviations and are not tested for significance. Given the large uncertainty, the paper cannot support the superlative conclusion in the Abstract and Sec. 4.3. The authors should run a paired significance test (e.g., bootstrap over search spaces/datasets) and report confidence intervals, or temper the claim to a descriptive observation.","section":"Sec. 4.3, Table 2"},{"comment":"The comparison of the OoD accuracy range (e.g., 3.80%/4.86%/2.74%) with a 1.9% improvement from the SOTA training method [30] is not apples-to-apples: the former is a spread between the best and worst architecture in the search space, not an improvement over a baseline achieved by a method. Additionally, Table 3 compares an architecture trained on IN-100 (labelled 'Ours') with ViT-B-32 and ViT-L-32 without stating that the baselines were evaluated under the same training protocol; the table likely mixes IN-100-trained results with ImageNet-scale reference numbers. This makes the claim that the embedding-dimension insight yields architectures that 'outperform' well-established human-designed ViTs unsupported. The authors need to clarify the training/evaluation protocol for all models in Table 3 and, if the baselines were not trained identically, remove the direct comparison.","section":"Sec. 4.1 and Table 3"},{"comment":"The ablation in Appx 17.1 claims to fix all other architectural attributes when varying embedding dimension, but it does so by restricting to architectures with mean #Head = 6±0.05 and mean MLP_Ratio = 3.5±0.05. This is not a controlled ablation because the per-layer values of heads and MLP ratio still vary within the small range, and the selection of architectures by these means may introduce confounding with other unobserved variables. The layer-wise analysis (Appx 17.2) is a useful complement, and the human-designed ViT experiment (Appx 8) provides partial independent evidence, but the main-benchmark correlation in Fig. 1-c is computed over all 3,000 architectures without such controls. The causal language 'increasing embedding dimensions generally enhances performance' should be softened to 'is positively associated with OoD accuracy in this benchmark' unless a rigorous controlled study is added.","section":"Sec. 4.4 and Appx 17.1"}],"minor_comments":[{"comment":"The phrase 'with up to 11.85% improvement for some OoD shifts' is ambiguous: is this a range between two architectures, an improvement over a baseline, or something else? Please clarify in the text and define the quantity precisely when it is first used.","section":"Abstract and Sec. 1"},{"comment":"There is a duplicated phrase: 'analyzing 1,000 architectures from the Autoformer-Small search space within our OoD-ViT-NAS benchmark for1, 000 architectures in Autoformer-Small search space within our benchmark on IN-C.' The sentence should be rewritten.","section":"Sec. 4.1"},{"comment":"The rows for Supernet-Tiny appear to have Max and Min reversed (Max 192 vs Min 240), and the stated ranges with step sizes are inconsistent with typical AutoFormer settings. Please check and correct the table.","section":"Appx 10.1, Table 10.2"},{"comment":"The claim that all training-free methods 'consistently fail to predict IN-D performance' is based on near-zero negative correlations (e.g., -0.03 to -0.09 in Appx Table 15.5). This is better described as a lack of predictive power rather than 'failure'; negative values may also indicate a peculiarity of the IN-D dataset that deserves discussion.","section":"Sec. 4.3, Fig. 1-b"},{"comment":"The statement 'Our comprehensive OoD-ViT-NAS benchmark sheds light on a previously unknown relationship' is too strong; several prior works (e.g., [13,15]) have noted dependencies between ViT capacity and robustness. Please soften the novelty claim.","section":"Sec. 4.4"},{"comment":"The manuscript contains many typos and awkward phrases (e.g., 'MLP_Ration', 'MLP_Tatio', 'ration' for 'ratio', inconsistent notation in Appx 11, and duplicated sentences in Sec. 4.1). A thorough proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core concern is the unvalidated one-shot supernet proxy. Even if the authors cannot perform a full validation, they could reframe the benchmark as specifically for one-shot NAS evaluation, which would already be a useful contribution. However, the current claims about ViT architecture design go beyond that scope. The training-free NAS comparison also needs significance testing to be credible. I would recommend major revision with an option for the authors to re-scope rather than reject outright, as the benchmark itself has potential value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. The paper builds a real resource: 3,000 ViT subnets sampled from AutoFormer supernets, evaluated on 8 OoD datasets, with per-architecture JSON releases, detailed GPU-hour accounting, and code. That part is solid and likely to be used. It also gives the first comparison of nine training-free NAS proxies under OoD shift for ViTs, and the first attempt to attribute OoD behavior to specific ViT attributes. Credit where due: the benchmark construction and evaluation protocol are careful, and the data release is concrete and reproducible.\n\nThe soft spot is exactly the one the stress-test flags. Every headline claim — the 11.85% OoD spread, low ID-OoD correlation, failure of training-free NAS, embedding-dimension benefit, and the HFC analysis — is measured on subnets that inherit weights from a shared supernet. The paper cites BigNAS and AutoFormer for the claim that subnet performance is comparable to standalone training, but that evidence is about ID accuracy. OoD performance is more sensitive to optimization details, and one-shot supernets are known to rank subnets imperfectly even in distribution. If subnet OoD rankings diverge from independently trained ViTs, the insights could be artifacts of shared weights, not architecture design. The Appendix's human-designed ViT experiment varies one attribute at a time on IN-100 and does not provide the missing validation for the 3,000-subnet benchmark. The paper's own limitation section acknowledges cost and resource dependence but not this transfer issue.\n\nThere is a second, related weakness in the proxy comparison. Table 2 shows #Param/#Flops ahead of DSS and AutoProx by about 0.02 Kendall tau with standard deviations above 0.23. That difference is within noise. Given that larger subnets may get better-fitting shared weights, capacity-correlated proxies winning is exactly what a weight-sharing artifact would predict. The claim that simple proxies “surprisingly outperform” more complex methods is overstated for this evidence.\n\nThe Table 3 comparison also omits the training protocol for the “Ours” architectures, which makes the efficiency and accuracy comparisons hard to evaluate.\n\nWho gets value: anyone working on ViT NAS or OoD robustness who wants a large, ready-made evaluation set. The benchmark itself deserves serious engagement. I would accept it for peer review with a required validation study: train a modest number of sampled subnets independently and check ranking and range transfer before accepting the scientific conclusions. Without that, the abstract and insights should be reframed as one-shot subnet findings.\n\nNet: resource yes, insights conditional. Recommend major revision, not desk reject.","headline":"A genuinely useful benchmark resource whose central scientific claims rest on an unvalidated weight-sharing assumption; worth serious refereeing, but with major revision required.","tokens_in":26171,"tokens_out":2270,"would_cite":true,"duration_ms":22838,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that for Vision Transformers, architecture design—especially embedding dimension—drives out-of-distribution generalization as much as training-time robustness methods, and that standard in-distribution…","keywords":["Vision Transformer","out-of-distribution generalization","neural architecture search","training-free NAS","zero-cost proxy","embedding dimension","AutoFormer","high-frequency components"],"falsifier":"Take a random sample of 30 architectures from the benchmark, train each one from scratch under the standard ImageNet protocol, and measure OoD accuracy on ImageNet-R and ImageNet-C; if their rankings correlate weakly with the weight-sharing subnet scores, or if the embedding-dimension advantage reverses, the paper's central insights fail.","tokens_in":25157,"feed_emoji":"📊","tokens_out":5253,"duration_ms":48961,"temperature":0.7,"pith_summary":"The paper builds OoD-ViT-NAS, a benchmark of 3,000 Vision Transformer architectures sampled from the AutoFormer search space and scored on eight out-of-distribution (OoD) image datasets. It uses this benchmark to argue that ViT architecture choice has a large effect on OoD generalization, larger in some cases than a state-of-the-art training-time OoD method, and that in-distribution (ID) accuracy is a misleading guide to which architectures will generalize. It further claims that existing training-free neural architecture search (NAS) proxies predict ID accuracy better than OoD accuracy, and that simple parameter or FLOP counts beat all of them at predicting OoD performance. The paper's positive design insight is that increasing the embedding dimension of a ViT generally improves OoD accuracy, and it offers evidence that this happens because larger embedding dimensions help the model learn high-frequency image components.","feed_headline":"Architecture choice swings ViT out-of-distribution accuracy by 11.85%","feed_subtitle":"3,000-model benchmark finds ID accuracy misleads, embedding size matters, and simple size proxies beat complex NAS scores.","key_machinery":"The machinery is one-shot NAS weight sharing: three AutoFormer supernets (Tiny, Small, and Base) are trained once, and 3,000 subnets are extracted by letting them inherit the supernet weights, which makes large-scale OoD evaluation computationally feasible. The analysis tools are Kendall's $\\tau$ rank correlation for ID-versus-OoD and proxy-versus-OoD comparisons, plus a frequency-filtering experiment that removes high-frequency components at radius $r$ and measures the accuracy change to test whether larger embedding dimensions improve OoD by learning more high-frequency components. This machinery is load-bearing because every benchmark insight is a range or ranking computed on weight-sharing subnets.","core_discovery":"On the paper's own terms, the discovery is that OoD generalization in ViTs is much more architecture-dependent than previous studies of a handful of human-designed models could show, and that the dependency has a clear direction: among AutoFormer's searchable attributes, embedding dimension is the strongest and most consistent lever, with an ablation correlation around Kendall's $\\tau = 0.65$ when other attributes are held fixed. ID accuracy correlates only weakly with OoD accuracy on most shifts, and the ID-Pareto architectures are usually not the OoD-Pareto architectures. Nine training-free NAS scores, including recent ViT-specific ones, rank architectures worse than #Params or #FLOPs for OoD (best $\\tau \\approx 0.36$ for #Params versus $0.33$ for AutoProx), and all proxies fail on ImageNet-D. The paper frames this as evidence that architecture design itself should be a first-class object of OoD research, not a proxy for training methods.","pith_inferences":["Because the benchmark uses weight-sharing subnets, its rankings may not fully transfer to standalone-trained ViTs; the paper's cited evidence for equivalence concerns ID accuracy only, so the embedding-dimension rule should be re-tested with independent training before being treated as a design law. This is an extension of the paper's own caveat, not a claim it tests.","The high-frequency-component explanation suggests a testable intervention: if larger embedding dimensions help by preserving high-frequency content, then high-frequency-emphasizing augmentation during training might reproduce part of the OoD gain at constant architecture size, something the paper does not try.","The uniform failure of all proxies on ImageNet-D hints that diffusion-generated OoD shifts may need a fundamentally different architectural response than corruption or natural shifts, an avenue the paper leaves open.","The finding that OoD accuracy spread widens with shift severity implies that architecture design will matter more as deployment shifts get harder, so the 11.85% spread figure is likely a lower bound for harder future benchmarks."],"forward_implications":["If the claims hold, NAS for ViTs should benchmark OoD accuracy directly rather than rely on ID accuracy, because the two rankings diverge.","New training-free NAS proxies for ViT OoD accuracy must beat #Params and #FLOPs; the paper sets those simple counts as the bar.","Architects can improve OoD generalization by widening the embedding dimension, potentially more cheaply than compound scaling of depth, heads, and MLP ratio.","The benchmark's 3,000 scored architectures across eight shift types enable future studies of which components drive robustness on specific corruption or natural shift types."],"supporting_citations":[{"why":"Defines the AutoFormer search space and supplies the three pre-trained supernets from which the 3,000 benchmark architectures are sampled.","marker":"[6]"},{"why":"Cited as the evidence that weight-sharing subnets perform comparably to architectures trained alone, the key premise enabling the benchmark protocol.","marker":"[20]"},{"why":"Supplies the Kendall tau rank-correlation methodology used to measure ID-versus-OoD agreement.","marker":"[71]"},{"why":"Provides the high-frequency-component filtering experiment adapted to test why increasing embedding dimension improves OoD generalization.","marker":"[77]"},{"why":"Sources the ImageNet-C and ImageNet-P OoD datasets used for corruption and perturbation shifts.","marker":"[24]"},{"why":"Sources the ImageNet-A and ImageNet-O natural adversarial datasets used for OoD accuracy and AUPR evaluation.","marker":"[25]"},{"why":"Provides the state-of-the-art domain-invariant training method whose 1.9% OoD improvement serves as the baseline that architecture-driven gains are compared against.","marker":"[30]"},{"why":"One of the ViT-specific training-free NAS proxies benchmarked and found to predict OoD accuracy weakly.","marker":"[10]"},{"why":"Another recent ViT-specific training-free NAS proxy benchmarked, with its AutoProx correlation of 0.3303 cited as evidence of limited OoD predictability.","marker":"[17]"}],"fun_headline_variants":["ViT architecture choice swings OoD accuracy by 11.85%","3,000 ViTs show ID accuracy misguides OoD design","Embedding dimension, not NAS score, drives ViT OoD","Simple #Params beats complex NAS for ViT OoD ranking","ViT OoD robustness: architecture, not NAS, is the lever"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a subnetwork that inherits weights from a shared supernet behaves, for out-of-distribution accuracy, like the same architecture trained on its own; the paper only cites evidence for this equivalence on in-distribution accuracy.","fun_headline_variants_meta":{"raw":{"variants":["ViT architecture choice swings OoD accuracy by 11.85%","3,000 ViTs show ID accuracy misguides OoD design","Embedding dimension, not NAS score, drives ViT OoD","Simple #Params beats complex NAS for ViT OoD ranking","ViT OoD robustness: architecture, not NAS, is the lever"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000935,"raw_usage":{"total_tokens":4062,"prompt_tokens":1067,"completion_tokens":2995,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":2898}},"tokens_in":683,"tokens_out":2995,"duration_ms":18784,"temperature":1.0,"reasoning_tokens":2898,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:46:59.337235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 30 architectures from the benchmark, train each one from scratch under the standard ImageNet protocol, and measure OoD accuracy on ImageNet-R and ImageNet-C; if their rankings correlate weakly with the weight-sharing subnet scores, or if the embedding-dimension advantage reverses, the paper's central insights fail.","supporting_citations":[{"cited_title":"HYPO: Hyperspherical out-of-distribution generalization","cited_arxiv_id":null,"evidence_quote":"Provides the state-of-the-art domain-invariant training method whose 1.9% OoD improvement serves as the baseline that architecture-driven gains are compared against."},{"cited_title":"Autoformer: Searching transformers for visual recognition","cited_arxiv_id":null,"evidence_quote":"Defines the AutoFormer search space and supplies the three pre-trained supernets from which the 3,000 benchmark architectures are sampled."},{"cited_title":"Bignas: Scaling up neural architecture search with big single-stage models","cited_arxiv_id":null,"evidence_quote":"Cited as the evidence that weight-sharing subnets perform comparably to architectures trained alone, the key premise enabling the benchmark protocol."},{"cited_title":"Id and ood performance are sometimes inversely correlated on real-world datasets","cited_arxiv_id":null,"evidence_quote":"Supplies the Kendall tau rank-correlation methodology used to measure ID-versus-OoD agreement."},{"cited_title":"Improving vision transform- ers by revisiting high-frequency components","cited_arxiv_id":null,"evidence_quote":"Provides the high-frequency-component filtering experiment adapted to test why increasing embedding dimension improves OoD generalization."},{"cited_title":"Training-free transformer architecture search","cited_arxiv_id":null,"evidence_quote":"One of the ViT-specific training-free NAS proxies benchmarked and found to predict OoD accuracy weakly."},{"cited_title":"Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery","cited_arxiv_id":null,"evidence_quote":"Another recent ViT-specific training-free NAS proxy benchmarked, with its AutoProx correlation of 0.3303 cited as evidence of limited OoD predictability."}],"review_version":1}