Pith. sign in

REVIEW 4 major objections 4 minor 62 references

NN-Former: Rethinking Graph Structure in Neural Architecture Representation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sibling nodes that share a parent or child are the overlooked signal that makes neural architecture predictors more accurate, the paper argues.

desk verdict Useful, well-benchmarked NAS predictor; sibling-aware attention is genuinely new, but the 'pivotal sibling nodes' claim needs an isolation ablation before it can be taken literally. read the letter →

arxiv 2507.00880 v1 pith:24X4CI5G submitted 2025-07-01 cs.LG cs.AI

Ruihan Xu , Haokui Zhang , Yaowei Wang , Wei Zeng , Shiliang Zhang This is my paper
classification cs.LGcs.AI
keywords neuralarchitecturesearchlatencypredictionpredictorsdirectedacyclicgraphssiblingnodesattentionmasksgraphisomorphismfeed-forwardnetworktransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that neural architecture predictors have been reading the wrong parts of the computation graph. Alongside forward and backward neighbors, the operations that share a common parent or child—sibling nodes—carry information that accuracy and latency prediction both need. The authors build NN-Former, a transformer in which each of four attention heads is masked to one topological relation (forward adjacency, backward adjacency, same-parent siblings, same-child siblings), and a bidirectional graph-isomorphism feed-forward network replaces the usual channel mixer. On NAS-Bench-101 and NAS-Bench-201 accuracy prediction and NNLQ latency prediction, the model reports state-of-the-art Kendall's Tau (0.877 at 1% training on NAS-Bench-101) and latency error (average MAPE 2.85% in-domain on NNLQ), with the largest gains on deep networks where global attention degrades.

What carries the argument

The central mechanism is the Adjacency-Sibling Multi-head Attention (ASMA) paired with the Bidirectional Graph Isomorphism Feed-Forward Network (BGIFFN). ASMA derives sibling structure directly from the adjacency matrix: $AA^\top$ marks pairs of nodes sharing a parent and $A^\top A$ marks pairs sharing a child, and each of four heads attends only inside one mask so that attention stays local instead of global. BGIFFN concatenates forward and backward adjacency aggregations in the channel-mixing stage as an injective-style update, which the paper argues is why topology can be learned without position encoding.

What would settle it

Run the latency ablation on the NAS-Bench-201 family with four heads restricted to adjacency masks only ($I+A$, $I+A^\top$, $I+A$, $I+A^\top$) and with four heads restricted to sibling masks only ($I+AA^\top$, $I+A^\top A$, $I+AA^\top$, $I+A^\top A$), holding every other setting fixed. If the adjacency-only variant reproduces the reported 7.93 MAPE and 69.90 Acc(10%) while the sibling-only variant does not, the paper's sibling-centered explanation of the latency gain is not supported. A second check is to feed BGIFFN two non-isomorphic DAGs whose node features collapse to the same values after the ReLU step; if the final embeddings coincide, the implemented network does not realize the claimed isomorphism separation.

Watch

Extended reading notes

Core claim

The central claim is that sibling relations are a distinct, load-bearing channel of information for representing neural architectures, and that a transformer can exploit them by replacing global attention with four masked heads over $I+A$, $I+A^\top$, $I+AA^\top$, and $I+A^\top A$. The paper further claims that combining this adjacency-sibling attention with a bidirectional graph-isomorphism feed-forward network yields consistently strong accuracy and latency prediction, that the model scales to architectures with hundreds of operations, and that the injected topology is sufficient that explicit position encoding is unnecessary. On the paper's evidence, this design outperforms prior GNN, transformer, and hybrid predictors on almost every training-fraction setting tested, and it changes the standard picture of what graph structure matters in a DAG: short-range sibling and adjacency ties matter more than long-range transitive closure.

Load-bearing premise

The load-bearing premise is that sibling relationships, not just local graph masking in general, cause the reported gains; the latency ablation switches global attention to adjacency and sibling masks at the same time, and the appendix's isomorphism proof assumes a one-to-one combine step while the activation actually used can merge distinct inputs into the same output.

Editorial extensions

If this is right

  • Neural architecture search can rank candidates accurately from very small labeled sets: Kendall's Tau reaches 0.877 with 1% of NAS-Bench-101 and 0.804 with 1% of NAS-Bench-201, reducing the number of trained models needed to guide a search.
  • Latency predictors become reliable enough for deployment filtering across unseen network families, with average out-of-domain MAPE 8.39% on NNLQ and especially large improvements on EfficientNet, MnasNet, and MobileNetV2.
  • Because the position encodings tested in the paper did not improve the model, future architecture predictors can treat local topological masking as a substitute for hand-designed positional structure.
  • Sibling attention transfers to general DAG tasks selectively: it helps citation prediction on Cora but not AST-based code prediction, indicating that task-dependent DAG semantics decide whether sibling cues matter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the four fixed masks could be replaced by a learned or per-layer routing over adjacency and sibling relations, which would test whether the two relation types deserve different weights rather than one head each.
  • Editorial inference: because ASMA runs in $O(NK)$ time on sparse graphs where global attention is $O(N^2)$, the same masking scheme should extend to computation graphs much larger than the 200-node networks tested here.
  • Editorial inference: iterating the sibling construction, such as siblings of siblings via powers of $AA^\top$ and $A^\top A$, could capture wider parallel structure, an extension the paper does not test.
  • Editorial inference: the selective transfer to citation graphs but not code graphs suggests a per-task gating of sibling attention could be learned, letting the model decide when sibling structure carries signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes NN-Former, a transformer-based neural architecture predictor that augments self-attention with four topology masks (forward/backward adjacency and two sibling relations) and replaces the standard feed-forward network with a bidirectional graph-isomorphism feed-forward network (BGIFFN). The method is evaluated on accuracy prediction (NAS-Bench-101, NAS-Bench-201) and latency prediction (NNLQ), reporting state-of-the-art or competitive results, including Kendall's Tau of 0.877 at 1% training on NAS-Bench-101 and a 2.85% average in-domain MAPE on NNLQ. The paper also includes ablation studies and a supplementary proof that BGIFFN is graph-isomorphism-capable.

Significance. If the central claim is fully established, the paper is a useful contribution to neural architecture representation: it offers a concrete hybrid of GNN-style local topology and transformer attention, with strong empirical results on standard benchmarks and an open-source implementation. The paper also correctly identifies a gap in existing neural predictors, namely the lack of explicit sibling modeling, and tests this idea on both accuracy and latency prediction. The empirical comparisons are extensive and the ablation structure is thoughtful. However, the paper's main novelty attribution, that sibling masks drive the gains, is not isolated in the latency experiment, and the theoretical appendix does not prove the claimed isomorphism property for the implemented network. These issues are fixable but currently prevent the central claims from being accepted as stated.

major comments (4)
  1. [Section 4.3, Table 5] The claim that sibling modeling drives the latency improvement is not supported by the presented ablation. ASMA replaces global attention with four heads whose masks are I+A, I+A^T, I+AA^T, and I+A^T A, so the comparison bundles the adjacency masks (I+A, I+A^T) with the sibling masks (I+AA^T, I+A^T A). The reported gain from 10.83% to 7.93% MAPE could be produced by the adjacency heads alone. I request an ablation on NNLQ that also includes an adjacency-only mask configuration (e.g., four heads with I+A and I+A^T only) and a sibling-only configuration, so the effect of sibling masks is isolated. Without this, the paper's central novelty claim for latency prediction is not established.
  2. [Appendix 1.3, Eq. (19)] The graph-isomorphism proof does not cover the implemented BGIFFN. The proof invokes Theorem 3 of Xu et al. (2018), which requires injective aggregation and combine functions. The implemented combine is h_v = ReLU(h_v W1 + a_v) W2, and ReLU is not injective on R^d; the linear aggregation over neighbor sums is also not shown to be injective on multisets. The statement that there exist injective choices of f and phi establishes only existence, not that the network used in Tables 1-12 satisfies the condition. Please either revise the theoretical claim to describe a bound or intuition, or prove injectivity for the actual parameterized module, for example by using an injective activation and restricting the domain appropriately. This does not invalidate the empirical results but removes the theoretical justification currently claimed.
  3. [Section 4.3, Table 6] On accuracy, the full ASMA+BGIFFN model is not better than Global+BGIFFN (0.7654 versus 0.7656 on NAS-Bench-101), so the only accuracy evidence for sibling masks is the internal comparison in Table 7. The abstract's statement that the approach 'consistently achieves promising performance in both accuracy and latency prediction' and the paper's emphasis on sibling nodes as pivotal should be tempered, or an accuracy setting should be provided where ASMA beats global attention with BGIFFN held fixed.
  4. [Section 3.2 and Appendix 1.2] The sibling-relation definitions are inconsistent with the matrix algebra. With A_ij=1 for an edge from i to j, the matrix AA^T has a non-zero (i,j) entry when i and j share a common child/successor, while A^T A has a non-zero entry when i and j share a common parent/predecessor. Section 3.2 labels AA^T as 'siblings with the same parents' and A^T A as 'siblings with the same children', and Appendix 1.2 says A^T A denotes nodes sharing 'a same successor'; both are reversed. The actual masks in Listing 1 (A^T A and A A^T) should be labeled consistently with these definitions so the reader can verify which sibling relation each head encodes.
minor comments (4)
  1. [Throughout] There are several typos: 'refered' (Section 3.1), 'latecny' (Appendix 2.2.1), 'taht' (Appendix 2.1.2), 'predecossor' (Appendix 1.2), and 'BIGFFN' (Appendix 1.3).
  2. [Table 7, row 4] The masks listed as 'AA' and 'A^T A^T' are not defined in the paper; these are likely intended to be A^T A and AA^T, or another previously defined mask, and should be corrected.
  3. [Section 4.3] The phrase 'a large performance enhancement of 11.45% Acc(10%)' should read 'an improvement of 11.45 percentage points in Acc(10%)'.
  4. [Table 5] The caption says 'Results on the NAS-Bench-201 family are reported', but the experiment is on the NAS-Bench-201 test domain within NNLQ; clarifying this would avoid confusion with the accuracy-prediction experiments on NAS-Bench-201.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy and latency predictions are trained on external benchmark labels with graph-structure inputs, and no target quantity is reused as a predictor input.

full rationale

The paper's derivation chain is empirical rather than definitional. NN-Former takes an architecture graph (Z, A) as input and predicts accuracy or latency via f_theta(Z, A), with the target labels coming from external datasets (NAS-Bench-101, NAS-Bench-201, NNLQ). The proposed ASMA masks are computed directly from the adjacency matrix A (I+A, I+A^T, I+AA^T, I+A^T A), and BGIFFN aggregates A and A^T; neither module uses the prediction target y to define its structure. The reported gains are measured on held-out test sets after training on standard training subsets, not on the same data used to fit the model. The ablation tables compare model variants on fixed benchmarks, which is standard empirical validation, not a fitted-input-called-prediction loop. Self-citations to NAR-Former and ParcNetV2 are used for training settings, baselines, and a gating comparison, but none is load-bearing for the central claim in a way that reduces the result to the citation itself. A separate concern is that the graph-isomorphism proof assumes injective combine functions while the implementation uses ReLU (non-injective), and that the latency ablation changes adjacency and sibling masks simultaneously; these affect the validity and support of specific claims, but they are not circularity. No equation or procedure in the paper defines a predicted quantity in terms of the target it is supposed to predict.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central performance claim rests only on standard supervised learning assumptions and the benchmark protocols. The theoretical graph-isomorphism claim rests on the GIN theorem plus an unjustified assumption that ReLU can be part of an injective combine function. The sibling-importance premise is a domain assumption tested empirically but not fully isolated.

free parameters (2)
  • attention mask configuration = four heads with masks I+A, I+A^T, I+A^T A, I+A A^T
    Selected via ablation on NAS-Bench-101 (Table 7); the best configuration includes sibling masks. It is not derived from first principles and contributes to the results.
  • BGIFFN branch split count = 2 (forward and backward)
    Chosen via ablation (Table 9); the 4-split version performed worse, so this count is tuned to the benchmark.
assumptions (3)
  • standard math Theorem 3 of Xu et al. (GIN): a message-passing GNN with injective aggregation and readout is as powerful as the WL test
    Invoked in Appendix 1.3 to claim BGIFFN graph isomorphism.
  • ad hoc to paper ReLU nonlinearity can be treated as injective in the isomorphism proof
    The proof requires injective phi but the implementation uses ReLU, which is not injective. No argument is given that the specific linear layers make the composition injective.
  • domain assumption Sibling nodes carry predictive information for architecture accuracy and latency
    Motivates ASMA; empirically tested but not isolated from adjacency-only on latency (only Global vs ASMA is compared).

how reviews work

0 comments
Cite this review

Pith. "Pith review of NN-Former: Rethinking Graph Structure in Neural Architecture Representation." pith.science (2026). https://pith.science/paper/24X4CI5G

@misc{pith2026250700880,
  author       = {Pith},
  title        = {Pith review of: NN-Former: Rethinking Graph Structure in Neural Architecture Representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24X4CI5G}},
  note         = {Machine review of arXiv:2507.00880}
}
read the original abstract

The growing use of deep learning necessitates efficient network design and deployment, making neural predictors vital for estimating attributes such as accuracy and latency. Recently, Graph Neural Networks (GNNs) and transformers have shown promising performance in representing neural architectures. However, each of both methods has its disadvantages. GNNs lack the capabilities to represent complicated features, while transformers face poor generalization when the depth of architecture grows. To mitigate the above issues, we rethink neural architecture topology and show that sibling nodes are pivotal while overlooked in previous research. We thus propose a novel predictor leveraging the strengths of GNNs and transformers to learn the enhanced topology. We introduce a novel token mixer that considers siblings, and a new channel mixer named bidirectional graph isomorphism feed-forward network. Our approach consistently achieves promising performance in both accuracy and latency prediction, providing valuable insights for learning Directed Acyclic Graph (DAG) topology. The code is available at https://github.com/XuRuihan/NNFormer.

Figures

Figures reproduced from arXiv: 2507.00880 by the authors.

Figure 1
Figure 1. Comparison of different methods on DAG representation of neural architectures. (a) GCNs [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The proposed NN-Former framework. We introduce adjacency and sibling attention masks in the Adjacency-Sibling Multihead Attention (ASMA) to learn graph topology information. We also introduce adjacency aggregation in the Bidirectional Graph Isomorphism Feed-Forward Network (BGIFFN) to enhance the topology structure. capability of Directed Acyclic Graph (DAG) representa￾tion learning. In the channel-mixing part, we i… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 49 canonical work pages

  1. [1]

    Zero-cost proxies for lightweight nas

    Mohamed S Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. arXiv preprint arXiv:2101.08134, 2021. 4

    arXiv 2021

  2. [2]

    Once-for-all: Train one network and specialize it for efficient deployment

    Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791,

    arXiv 1908

  3. [3]

    Contrastive neural archi- tecture search with neural architecture comparators

    Yaofo Chen, Yong Guo, Qi Chen, Minli Li, Wei Zeng, Yaowei Wang, and Mingkui Tan. Contrastive neural archi- tecture search with neural architecture comparators. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9502–9511, 2021. 2, 3

    work page 2021

  4. [4]

    Peephole: Pre- dicting network performance before training

    Boyang Deng, Junjie Yan, and Dahua Lin. Peephole: Pre- dicting network performance before training. arXiv preprint arXiv:1712.03351, 2017. 2

    arXiv 2017

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 3

    2009

  6. [6]

    Repvgg: Making vgg-style convnets great again

    Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 13733–13742, 2021. 1

    work page 2021

  7. [7]

    ParZC: Parametric Zero-Cost Proxies for Efficient NAS

    Peijie Dong, Lujun Li, Xinglin Pan, Zimian Wei, Xiang Liu, Qiang Wang, and Xiaowen Chu. Parzc: Parametric zero-cost proxies for efficient nas. arXiv preprint arXiv:2402.02105,

    work page Pith review arXiv

  8. [8]

    Nas-bench-201: Extending the scope of reproducible neural architecture search

    Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326, 2020. 5, 3, 4

    arXiv 2001

Show all 62 references
  1. [9]

    Pace: A parallelizable computation encoder for directed acyclic graphs

    Zehao Dong, Muhan Zhang, Fuhai Li, and Yixin Chen. Pace: A parallelizable computation encoder for directed acyclic graphs. In International Conference on Machine Learning , pages 5360–5377. PMLR, 2022. 1, 2, 3, 7

    2022

  2. [10]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  3. [11]

    Brp-nas: Prediction-based nas using gcns

    Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: Prediction-based nas using gcns. Advances in Neural Infor- mation Processing Systems, 33:10480–10490, 2020. 1, 2

    2020

  4. [12]

    A general- ization of transformer networks to graphs

    Vijay Prakash Dwivedi and Xavier Bresson. A general- ization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020. 3

    2012 arXiv

  5. [13]

    Neural topological ordering for computation graphs

    Mukul Gagrani, Corrado Rainone, Yang Yang, Harris Teague, Wonseok Jeon, Roberto Bondesan, Herke van Hoof, Christopher Lott, Weiliang Zeng, and Piero Zappi. Neural topological ordering for computation graphs. Advances in Neural Information Processing Systems , 35:17327–17339,

  6. [14]

    Neural message passing for quantum chemistry

    Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263–1272. PMLR, 2017. 2

    2017

  7. [15]

    Accurate, large mini- batch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Doll ´ar, Ross Girshick, Pieter Noord- huis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large mini- batch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. 3, 4

    2017 arXiv

  8. [16]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12175–12185, 2022. 3

    2022

  9. [17]

    Inductive representation learning on large graphs

    Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017. 2

    2017

  10. [18]

    Flowerformer: Empowering neural architecture encod- ing using a flow-aware graph transformer

    Dongyeong Hwang, Hyunju Kim, Sunwoo Kim, and Kijung Shin. Flowerformer: Empowering neural architecture encod- ing using a flow-aware graph transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6128–6137, 2024. 3

    2024

  11. [19]

    Cap: a context-aware neural predictor for nas

    Han Ji, Yuqi Feng, and Yanan Sun. Cap: a context-aware neural predictor for nas. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelli- gence, pages 4219–4227, 2024. 3

    2024

  12. [20]

    Graph masked au- toencoder enhanced predictor for neural architecture search

    Kun Jing, Jungang Xu, and Pengfei Li. Graph masked au- toencoder enhanced predictor for neural architecture search. In IJCAI, pages 3114–3120, 2022. 5

    2022

  13. [21]

    A learned performance model for tensor processing units

    Sam Kaufman, Phitchaya Phothilimthana, Yanqi Zhou, Charith Mendis, Sudip Roy, Amit Sabne, and Mike Burrows. A learned performance model for tensor processing units. Proceedings of Machine Learning and Systems , 3:387–400,

  14. [22]

    Semi-supervised classi- fication with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classi- fication with graph convolutional networks. In International Conference on Learning Representations, 2016. 1, 2, 3, 5, 8

    2016

  15. [23]

    Answering complex queries in knowledge graphs with bidi- rectional sequence encoders

    Bhushan Kotnis, Carolin Lawrence, and Mathias Niepert. Answering complex queries in knowledge graphs with bidi- rectional sequence encoders. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 4968–4977,

  16. [24]

    Learning multiple layers of features from tiny images, 2009

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. 3

    2009

  17. [25]

    Neural graph em- bedding for neural architecture search

    Wei Li, Shaogang Gong, and Xiatian Zhu. Neural graph em- bedding for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4707– 4714, 2020. 1, 2, 3

    2020

  18. [26]

    Progressive neural architecture search

    Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference on com- puter vision (ECCV), pages 19–34, 2018. 2

    2018

  19. [27]

    Nnlqp: A multi-platform neural network la- tency query and prediction system with an evolving database

    Liang Liu, Mingzhu Shen, Ruihao Gong, Fengwei Yu, and Hailong Yang. Nnlqp: A multi-platform neural network la- tency query and prediction system with an evolving database. In Proceedings of the 51st International Conference on Par- allel Processing, pages 1–14, 2022. 1, 2, 3, ...

    2022

  20. [28]

    Sgdr: Stochas- tic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochas- tic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 3

    2016 arXiv

  21. [29]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 3, 4

    2017 arXiv

  22. [30]

    Tnasp: A transformer-based nas predictor with a self- evolution framework

    Shun Lu, Jixiang Li, Jianchao Tan, Sen Yang, and Ji Liu. Tnasp: A transformer-based nas predictor with a self- evolution framework. Advances in Neural Information Pro- cessing Systems, 34:15125–15137, 2021. 1, 2, 3, 5, 7

    2021

  23. [31]

    Pinat: A permutation invari- ance augmented transformer for nas predictor

    Shun Lu, Yu Hu, Peihao Wang, Yan Han, Jianchao Tan, Jix- iang Li, Sen Yang, and Ji Liu. Pinat: A permutation invari- ance augmented transformer for nas predictor. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pages 8957–8965, 2023. 5, 3

    2023

  24. [32]

    Neural architecture optimization

    Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. Advances in neural information processing systems, 31, 2018. 1, 2, 3, 5

    2018

  25. [33]

    Semi-supervised neural architecture search

    Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tie-Yan Liu. Semi-supervised neural architecture search. Advances in Neural Information Processing Systems, 33:10547–10557, 2020. 2

    2020

  26. [34]

    Transformers over directed acyclic graphs

    Yuankai Luo, Veronika Thost, and Lei Shi. Transformers over directed acyclic graphs. Advances in Neural Informa- tion Processing Systems, 36, 2023. 1, 2, 3, 7, 8

    2023

  27. [35]

    A generic graph-based neural architecture encoding scheme for predictor-based nas

    Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor-based nas. In European Con- ference on Computer Vision, pages 189–204. Springer, 2020. 3, 5, 8

    2020

  28. [36]

    Ta-gates: An encoding scheme for neu- ral network architectures

    Xuefei Ning, Zixuan Zhou, Junbo Zhao, Tianchen Zhao, Yiping Deng, Changcheng Tang, Shuang Liang, Huazhong Yang, and Yu Wang. Ta-gates: An encoding scheme for neu- ral network architectures. Advances in Neural Information Processing Systems, 35:32325–32339, 2022. 3

    2022

  29. [37]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992. 3

    1992

  30. [38]

    Estimates of the regression coefficient based on kendall’s tau

    Pranab Kumar Sen. Estimates of the regression coefficient based on kendall’s tau. Journal of the American statistical association, 63(324):1379–1389, 1968. 3

    1968

  31. [39]

    Bridging the gap between sample-based and one-shot neural architecture search with bonas

    Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Bridging the gap between sample-based and one-shot neural architecture search with bonas. Advances in Neural Information Processing Systems , 33:1808–1819,

  32. [40]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1–9, 2015. 1

    2015

  33. [41]

    Directed acyclic graph neural networks

    Veronika Thost and Jie Chen. Directed acyclic graph neural networks. arXiv preprint arXiv:2101.07965, 2021. 3

    2021 arXiv

  34. [42]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1, 2

    2017

  35. [43]

    Graph at- tention networks

    Petar Veli ˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li `o, and Yoshua Bengio. Graph at- tention networks. In International Conference on Learning Representations, 2018. 1, 2, 3, 8

    2018

  36. [44]

    Neural predictor for neural architecture search

    Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Ben- der, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX, pages 660–676. Springer, 202...

    2020

  37. [45]

    Bananas: Bayesian optimization with neural architectures for neural architecture search

    Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10293–10301, 2021. 2

    2021

  38. [46]

    Pay less attention with lightweight and dy- namic convolutions

    Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dy- namic convolutions. arXiv preprint arXiv:1901.10430, 2019. 3

    1901 arXiv

  39. [47]

    Representing long- range context for graph neural networks with global atten- tion

    Zhanghao Wu, Paras Jain, Matthew Wright, Azalia Mirho- seini, Joseph E Gonzalez, and Ion Stoica. Representing long- range context for graph neural networks with global atten- tion. Advances in Neural Information Processing Systems , 34:13266–13279, 2021. 3, 5

    2021

  40. [48]

    How powerful are graph neural networks? In International Conference on Learning Representations, 2018

    Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2018. 2, 3, 1

    2018

  41. [49]

    Parcnetv2: Oversized kernel with enhanced attention

    Ruihan Xu, Haokui Zhang, Wenze Hu, Shiliang Zhang, and Xiaoyu Wang. Parcnetv2: Oversized kernel with enhanced attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752–5762, 2023. 8

    2023

  42. [50]

    Renas: Relativistic eval- uation of neural architecture search

    Yixing Xu, Yunhe Wang, Kai Han, Yehui Tang, Shangling Jui, Chunjing Xu, and Chang Xu. Renas: Relativistic eval- uation of neural architecture search. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4411–4420, 2021. 1, 3, 5

    2021

  43. [51]

    Does unsupervised architecture representation learning help neural architecture search? Advances in Neural Information Processing Systems, 33:12486–12498, 2020

    Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? Advances in Neural Information Processing Systems, 33:12486–12498, 2020. 2, 3

    2020

  44. [52]

    Nar-former: Neural architecture representation learning towards holistic attributes prediction

    Yun Yi, Haokui Zhang, Wenze Hu, Nannan Wang, and Xi- aoyu Wang. Nar-former: Neural architecture representation learning towards holistic attributes prediction. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7715–7724, 2023. 1, 2, ...

    2023

  45. [53]

    Nar-former v2: Rethinking transformer for uni- versal neural network representation learning

    Yun Yi, Haokui Zhang, Rong Xiao, Nannan Wang, and Xi- aoyu Wang. Nar-former v2: Rethinking transformer for uni- versal neural network representation learning. Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 3, 5, 6, 4

    2024

  46. [54]

    Nas-bench-101: Towards reproducible neural architecture search

    Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International conference on machine learning, pages 7105–7114. PMLR,

  47. [55]

    Do transformers really perform badly for graph representation? In Thirty-Fifth Conference on Neural Information Process- ing Systems, 2021

    Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Thirty-Fifth Conference on Neural Information Process- ing Systems, 2021. 3, 5

    2021

  48. [56]

    Graph structure of neural networks

    Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks. In International Con- ference on Machine Learning, pages 10881–10891. PMLR,

  49. [57]

    Graph hy- pernetworks for neural architecture search

    Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hy- pernetworks for neural architecture search. arXiv preprint arXiv:1810.05749, 2018. 2, 3

    2018 arXiv

  50. [58]

    Nn-meter: Towards accurate latency prediction of deep-learning model inference on diverse edge devices

    Li Lyna Zhang, Shihao Han, Jianyu Wei, Ningxin Zheng, Ting Cao, Yuqing Yang, and Yunxin Liu. Nn-meter: Towards accurate latency prediction of deep-learning model inference on diverse edge devices. In Proceedings of the 19th Annual International Conference on Mobile Systems, Ap...

    2021

  51. [59]

    directed WL test

    Methods Details 1.1. Implementation for ASMA We present Python-style code for calculating the attention matrix in the ASMA module in Listing 1. ASMA is moti- vated by the importance of sibling nodes. In the accuracy prediction, sibling nodes provide complementary features, suc...

  52. [60]

    For accuracy prediction, we show the experiment settings on NAS-Bench-101 in Section 2.1.1 and NAS- Bench-201 in Section 4.1

    Experiment Details We present implementation details of our proposed NN- Former. For accuracy prediction, we show the experiment settings on NAS-Bench-101 in Section 2.1.1 and NAS- Bench-201 in Section 4.1. For latency prediction, we show the experiment settings on NNLQ [27] i...

    2023

  53. [61]

    Ablation on hyperparameters This work adopts a Transformer as the backbone, and the hyperparameters of Transformers have been well-settled in previous research

    Extensive experiments 3.1. Ablation on hyperparameters This work adopts a Transformer as the backbone, and the hyperparameters of Transformers have been well-settled in previous research. This article follows the common training settings (from NAR-Former) and has achieved good...

  54. [62]

    Theoretical Analysis Our ASMA method has less or equal computational com- plexity than the vanilla attention

    Model Complexity 4.1. Theoretical Analysis Our ASMA method has less or equal computational com- plexity than the vanilla attention. On the dense graph, the vanilla self-attention has a complexity of O(N 2) where N denotes the number of nodes. With the sibling connection prepro...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.