Pith. sign in

REVIEW 3 major objections 7 minor 53 references

What makes a good feedforward computational graph?

T0 review · 3 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two complementary measures—averaged mixing time and minimax fidelity—determine whether a feedforward computational graph serves a neural network well, and a recursively built sparse graph achieves the best known trade-off.

desk verdict Interesting metrics and a promising sparse graph, but Theorem 6.1's constant-alpha assumption is unsatisfiable for the recursive construction, so the central mixing-time guarantee is unproven as written. read the letter →

arxiv 2502.06751 v2 pith:NUYDBGID submitted 2025-02-10 cs.LG cs.AIcs.SIstat.ML

Alex Vitvitskyi , João G. M. Araújo , Marc Lackenby , Petar Veličković This is my paper · ORCID
classification cs.LGcs.AIcs.SIstat.ML MSC 05C2005C8105C8068T07
keywords feedforwardcomputationalgraphscausalattentionmasksmixingtimeminimaxfidelitygraphnetworkslengthgeneralizationsparsetransformersrewiring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a feedforward computational graph—the directed, time-respecting pattern of connections allowed by a causal attention mask—is good when two complementary quantities are good: the averaged mixing time, which measures how quickly a random walk from any node concentrates on the final sink, and the minimax fidelity, which measures how sharply the weakest node's information can still be read at the sink under averaging diffusion. If this pair of measures is right, sparse causal graphs can be designed by optimizing two numbers rather than by defaulting to fully connected attention or local sliding windows. The authors analyze standard families, show that line graphs and fully connected graphs sit at opposite extremes, and use program search to discover a recursively constructed family, the FS graph, that provably mixes in polylogarithmic time while keeping normalized minimax fidelity above that of a fully connected graph. In trained graph-attention models on maximum, second-maximum, and parity tasks, the FS graph matches or beats other sparse graphs and matches full attention on parity while using substantially fewer edges.

What carries the argument

The central objects are two dual matrices built from the adjacency matrix of a feedforward graph: the row-normalized walk matrix $W$ and the column-normalized averaging-diffusion matrix $\Delta$. Averaged mixing time is the smallest $t$ such that the average over starting vertices of $\|W^t e_i - \mathbf{1}_\tau\|_1$ is below $1/4$; it measures how quickly a random walk concentrates on the sink. Minimax fidelity is $\min_i \max_t \Delta^t_{\tau i}$, the smallest over nodes of the largest coefficient that node ever contributes to the sink after averaging diffusion, and normalizing by $n$ sets the fully connected graph to $1$. The construction carrying the argument is the FS graph: $O(\log n)$ blocks per recursion level, sparse bipartite expanders between consecutive blocks, and recursive FS graphs inside each block, giving self-similarity and bounded in-degree. Theorem 6.1's proof tracks the expected time to cross $D = \lceil \log n / \log \log n \rceil$ recursive levels with a binomial tail bound, establishing $O(\log^2 n / \log \log n)$ mixing time.

What would settle it

Train the paper's graph-attention architecture on a held-out battery of, say, fifty feedforward graph generators with matched in-degree budgets, measure averaged mixing time and normalized minimax fidelity for each, and regress out-of-distribution accuracy on the second-maximum and parity tasks against the two metrics; if the metric-ordering does not reproduce the empirical ranking, the claims that these measures capture graph quality are refuted. A sharper version would compute the same two quantities using the trained models' attention matrices instead of the uniform $\Delta$, and check whether the ranking survives.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the two measures are complementary because they arise from two dual normalizations of the adjacency matrix: the walk matrix $W$, normalized by out-degree, controls how fast information travels; the diffusion matrix $\Delta$, normalized by in-degree, controls how sharply it arrives. It proves that the fully connected graph has logarithmic averaged mixing time but normalized minimax fidelity exactly $1$, line graphs have the reverse profile, and oriented Erdős–Rényi and oriented expander graphs perform poorly on both measures. The constructive discovery is the FS graph generator: split the nodes into $O(\log n)$ blocks, connect consecutive blocks with bipartite expanders built from random perfect matchings, and recursively fill each intra-block triangle with a smaller copy of the same graph. Theorem 6.1 shows that, whenever every node keeps at least a constant fraction of its outgoing edges crossing blocks, the averaged mixing time is $O(\mathrm{polylog}\,n)$, with the proof yielding $O(\log^2 n / \log \log n)$, while the normalized minimax fidelity remains above that of the fully connected graph. Empirically, graph attention networks trained on the FS graph generalize from length 256 to length 1,024 better than other sparse graphs and match the fully connected graph on the parity task.

Load-bearing premise

The load-bearing premise is that minimax fidelity, computed under a uniform averaging-diffusion model, faithfully predicts how trained attention-based networks propagate information; the paper supports this link only by visual comparison on three synthetic tasks, so if trained attention's effective diffusion diverges from $\Delta$, the FS graph's design rationale would lose its foundation.

Editorial extensions

If this is right

  • If the two measures are right, the fully connected causal mask is not the gold standard: it has logarithmic mixing time but the lowest normalized fidelity, so sparse alternatives can dominate it on both axes.
  • The recursive block-expander template gives a concrete recipe for building sparse causal masks with polylogarithmic mixing time and self-similar structure, replacing hand-tuned sliding windows.
  • Graphs with the same mixing-time and fidelity profile should behave similarly in practice, so causal-attention designs can be screened by these two numbers before any training.
  • Training on sequences up to length 256 and testing to length 1,024, the FS graph's out-of-distribution accuracy is competitive with full attention, indicating sparsification by these principles does not sacrifice length generalization.
  • Fine-tuning a 2-billion-parameter language model with the FS graph as attention mask reaches perplexity comparable to full attention on Wikipedia text, suggesting the design can cut computational cost in realistic language modeling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader could go further and treat the two metrics as a Pareto front: among random graph generators with the same in-degree budget, the ones dominating on both mixing time and fidelity should also dominate empirically, a claim that is testable outside the paper's graph families.
  • Because minimax fidelity tracks the sharpest possible extraction of a single node at the sink, the framework naturally extends to long-context retrieval benchmarks; one could test whether recursive FS masks improve needle-in-a-haystack tasks in large language models.
  • The paper fixes one graph for all layers and only mentions layer-varying graphs as future work; a natural extension is to let the mask change across depth, with the same two metrics evaluated on the product of per-layer graphs.
  • The fidelity model assumes uniform averaging diffusion, so replacing $\Delta$ with measured trained-attention transitions would show whether the metrics remain predictive when attention is learned rather than uniform.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper studies feedforward computational graphs (directed graphs with edges only from earlier to later nodes) through two proposed measures: averaged mixing time, which tracks how quickly a lazy random walk reaches the sink node, and minimax fidelity, which tracks how sharply a node's feature can be recovered at the sink under mean-aggregation diffusion. The authors analyze these measures for line graphs, fully connected graphs, Erdős–Rényi graphs, oriented expanders, Poisson(p) graphs, and a new recursive graph family discovered via FunSearch (FS graphs). They prove that the line graph has O(n) averaged mixing time and high normalized fidelity, while the fully connected graph has O(log n) mixing time but normalized fidelity equal to 1. For the FS graphs they state a theorem claiming O(polylog n) mixing time under a constant-α assumption on cross-block edge proportions. Empirically, they train graph attention networks on maximum, second-maximum, and parity tasks, and report that FS graphs match or exceed the performance of other sparse graphs and sometimes approach the fully connected graph.

Significance. If the theoretical claims were fully established, the paper would offer a useful, task-agnostic design principle for sparse feedforward graphs in attention-based models, complementing recent work on over-squashing and under-reaching. The proposed metrics are defined independently of experiments, and the asymptotic analyses for line and fully connected graphs are correct and clearly presented. The empirical evaluation is careful and covers length generalization up to 1024. However, the central theoretical guarantee for the discovered FS graphs (O(polylog n) mixing time) depends on a condition that appears impossible to satisfy for the recursive construction, so the main theoretical contribution is currently not supported. The paper would be strengthened by a corrected theorem or by honestly presenting the mixing time of FS graphs as an empirical observation.

major comments (3)
  1. [6.1, Theorem 6.1, Remark C.1] The constant-α hypothesis of Theorem 6.1 is not satisfiable by the recursive FS graph for growing n. Let D ≈ ⌈log n / log⌈log n⌉⌉ be the number of recursive levels and let e_i(v) be the number of cross-block outgoing edges of node v at level i, with total outdegree T(v) = Σ_i e_i(v). If e_i(v)/T(v) ≥ α > 0 for all i, then summing over i gives T(v) ≥ D α T(v), so D α ≤ 1. Since D → ∞, no positive constant α can exist in any family with bounded outdegree. Remark C.1's proposed geometric decay δ_0 r^i yields, at the deepest level, δ_0 r^D = δ_0 n^{log r / log log n} → 0; for large n this is below 1 and cannot be realized as an integer number of edges. Consequently the proof in Appendix C does not apply to the graphs generated by the FS generator, and the claimed O(polylog n) mixing time is unproven as stated.
  2. [Section 6 / Appendix C] The FS generator is described informally ('full bipartite graphs across successive chunks', 'concatenating κ random perfect matchings', 'fill intra-cluster edges recursively'), and the proof of Theorem 6.1 abstracts each level as a chain of ⌈log n⌉ blocks with a fixed crossing probability α. The paper never specifies the precise construction—e.g., how many edges each expander contributes to each node, how recursion terminates, and how the 'geometric decay' of Remark C.1 is implemented with integer degrees—so the reader cannot check whether the assumptions of the theorem hold for the graphs actually evaluated in Figure 1 and Section 7. A precise, implementable definition of the FS generator is needed before the mixing-time claim can be assessed.
  3. [Section 7, Figure 3] The claim that the two metrics 'correlate' with trained model performance is supported only by visual inspection of Figure 3. No correlation coefficients, error bars, or statistical tests are reported. Because the FS graph was selected by optimizing fidelity subject to a mixing-time budget, the practical significance of the framework depends on this correlation being meaningful. The authors should provide a quantitative comparison, e.g., rank correlation between metric-based rankings and test accuracy across graph families at matched in-degree budgets, or clearly label the empirical match as qualitative.
minor comments (7)
  1. [Section 4.3.1] The word 'probabilty' should be 'probability'.
  2. [Section 5.1, Eq. (3)] The summation in (Δx)_j = Σ_j Δ_{ij} x_j should run over i (the incoming nodes), not over j; as written it is dimensionally inconsistent.
  3. [Section 5.2.2, Eq. (6)] Equation (6) is mis-rendered; the intended expression is max_a binom(a, n-1)/2^a (see Appendix B), not the fraction 'a/(n-1)' shown in the text.
  4. [Figure 1] The figure has no legend identifying which curve corresponds to which graph generator; the caption should list the generators and their plotting styles.
  5. [Appendix E.1] The phrase 'casual attention mask' should be 'causal attention mask'.
  6. [Impact Statement] The word 'alowing' should be 'allowing'.
  7. [Section 7] The graph name 'Erd s-Rényi' should be 'Erdős–Rényi'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the metrics are defined independently, the FS mixing-time theorem is proved from explicit assumptions, and the empirical tasks are held out from the graph search.

full rationale

The paper's derivation chain is self-contained. Mixing time (Section 4) and minimax fidelity (Eq. 2, Section 5) are defined independently of the experimental outcomes; fidelity is a mean-aggregation diffusion quantity, not a fitted parameter. The FS graph is found by FunSearch optimizing fidelity subject to a mixing-time constraint, but Theorem 6.1 proves an O(polylog n) mixing-time bound from an explicit recurrence (Appendix C) rather than importing the desired conclusion, and the empirical evaluation uses three held-out tasks (max, second-max, parity) that were not used to fit the metrics. The choices p=0.2 and 4 log n are specific generator hyperparameters and do not enter the theorem. Training hyperparameters were tuned on the fully connected graph only and reused everywhere else (Appendix E), so benchmark performance is not fitted to favor the proposed graphs. Self-citations to Barbero et al. (2024) and Veličković et al. (2024) are motivational context about over-squashing and dispersion, not load-bearing steps in the proof of the paper's central claims. One caveat is a soundness concern rather than circularity: Theorem 6.1's assumption of a constant cross-block outgoing-edge ratio α is not directly guaranteed by the recursive construction, and Remark C.1's geometric-decay scheme supplies a constant 1-r only under an additional realizability condition; however, the proof does not reduce the theorem to its conclusion, so this does not raise the circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central theoretical claims rest on standard probability tools and on the domain assumptions of unique sink, self-edges, uniform random walk, and mean-aggregation diffusion. The mixing time theorem adds a constant cross-block edge proportion assumption that is not verified in the actual FS generator used in experiments. Two hand-chosen parameters (p=0.2, k=4) affect the evaluated generators but not the core theorems.

free parameters (2)
  • Poisson graph acceptance probability p = 0.2
    Hand-picked to balance fidelity and mixing time in Section 5.2.4; not central to the theoretical claims.
  • FS expander indegree multiplier k = 4 log n
    Chosen based on fidelity scaling laws in Section 7, Figure 4; affects empirical performance but not the mixing time theorem.
assumptions (4)
  • domain assumption All feedforward graphs have all self-edges and a unique sink vertex tau
    Stated in Section 3 and used in Lemma 4.1 and the fidelity definitions.
  • domain assumption Random walk transition W chooses uniformly among outgoing edges
    Defined in Section 4.1; required for mixing time analyses.
  • domain assumption Diffusion Delta averages information over incoming edges (mean aggregation)
    Defined in Section 5.1; this is the basis for fidelity.
  • ad hoc to paper In Theorem 6.1, each node has a constant lower bound alpha on the proportion of its outgoing edges that cross block boundaries
    Assumption in Theorem 6.1; Remark C.1 argues a geometric decay construction can satisfy it, but it is imposed to make the proof work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What makes a good feedforward computational graph?." pith.science (2026). https://pith.science/paper/NUYDBGID

@misc{pith2026250206751,
  author       = {Pith},
  title        = {Pith review of: What makes a good feedforward computational graph?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUYDBGID}},
  note         = {Machine review of arXiv:2502.06751}
}
read the original abstract

As implied by the plethora of literature on graph rewiring, the choice of computational graph employed by a neural network can make a significant impact on its downstream performance. Certain effects related to the computational graph, such as under-reaching and over-squashing, may even render the model incapable of learning certain functions. Most of these effects have only been thoroughly studied in the domain of undirected graphs; however, recent years have seen a significant rise in interest in feedforward computational graphs: directed graphs without any back edges. In this paper, we study the desirable properties of a feedforward computational graph, discovering two important complementary measures: fidelity and mixing time, and evaluating a few popular choices of graphs through the lens of these measures. Our study is backed by both theoretical analyses of the metrics' asymptotic behaviour for various graphs, as well as correlating these metrics to the performance of trained neural network models using the corresponding graphs.

Figures

Figures reproduced from arXiv: 2502.06751 by the authors.

Figure 1
Figure 1. In this paper, we propose two measures which can be used to assess suitability of feedforward computational graphs for neural networks – mixing time (lower is better; Sec. 4) and (normalised) minimax fidelity (higher is better; Sec. 5). Here we plot these metrics as attained by various graph generators—at each point in a sequence, the number of nodes doubles, starting from 16. Using FunSearch (Romera-Paredes et al.,… view at source ↗
Figure 2
Figure 2. The evolution of the adjacency matrices generated by the FS graph generator. (Left:) The original solution found by FunSearch, when asked to optimise minimax fidelity while keeping the mixing time under 2 log2 n. (Middle:) Adapting the initial solution to bound the in-degree within each cluster. Each pair of successive blocks are now connected by a random bipartite expander. (Right:) The final FS graph generator, le… view at source ↗
Figure 3
Figure 3. Performance profiles of GATs trained with various computational graphs, on the max retrieval task (left), second max retrieval task (middle), and parity task (right), when trained up to size 256. The x-axis corresponds to test accuracy on sizes from 16 to 1, 024. 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0 log(n) 8.5 8.0 7.5 7.0 6.5 6.0 5.5 5.0 log(Fidelity) Fitted log(Fidelity) vs to_take=k*log(n) for different k's k = 2 … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Fitted scaling laws of log-minimax fidelity for different indegrees (of the form k log n for different values of k over n nodes) of sampled expander graphs in the FS graph. O(log n) indegree, we find that 4 log n offers a favourable “fidelity scaling law” (see [PITH_F…
Figure 5
Figure 5. Figure 5: Generated samples of adjacency matrices for the graph generators of Oriented Expander (left), Erdos-R ˝ enyi with constant ´ indegree (middle) and Poisson(0.2) with constant indegree (right) [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Generated samples of adjacency matrices for the graph generators of Line Graphs (left), Poisson(0.2) with √ n indegree (middle) and Erdos-R ˝ enyi with ´ √ n indegree (right) [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Generated samples of adjacency matrices for the graph generators of Poisson(0.2) with log2 n indegree (left), Erdos-R ˝ enyi with ´ log2 n indegree (middle) and the Fully Connected graph (right). 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages

  1. [1]

    and Yahav, E

    Alon, U. and Yahav, E. On the bottleneck of graph neural networks and its practical implications. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=i80OPhOCVH2

    2021

  2. [2]

    Diffwire: Inductive graph rewiring via the lov\'asz bound

    Arnaiz-Rodr \' guez, A., Begga, A., Escolano, F., and Oliver, N. Diffwire: Inductive graph rewiring via the lov\'asz bound. arXiv preprint arXiv:2206.07369, 2022

    arXiv 2022

  3. [3]

    Azabou, M., Ganesh, V., Thakoor, S., Lin, C.-H., Sathidevi, L., Liu, R., Valko, M., Veli c kovi \'c , P., and Dyer, E. L. Half-hop: A graph upsampling approach for slowing down message passing. In International Conference on Machine Learning, pp.\ 1341--1360. PMLR, 2023

    work page 2023

  4. [4]

    Barbero, F., Banino, A., Kapturowski, S., Kumaran, D., Araújo, J. G. M., Vitvitskyi, A., Pascanu, R., and Veličković, P. Transformers need glasses! information over-squashing in language tasks, 2024. URL https://arxiv.org/abs/2406.04267

    arXiv 2024

  5. [5]

    V., Monet, M., Pérez, J., Reutter, J., and Silva, J

    Barceló, P., Kostylev, E. V., Monet, M., Pérez, J., Reutter, J., and Silva, J. P. The logical expressiveness of graph neural networks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=r1lZ7AEKvB

    work page 2020

  6. [6]

    and Wang, Y

    Cai, C. and Wang, Y. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318, 2020

    arXiv 2006

  7. [7]

    Higher-Order Expander Graph Propagation

    Christie, T. and He, Y. Higher-order expander graph propagation. arXiv preprint arXiv:2311.07966, 2023

    work page Pith review arXiv 2023

  8. [8]

    Chung, F. R. Spectral graph theory, volume 92. American Mathematical Soc., 1997

    1997

Show all 53 references
  1. [9]

    and Grabowski,

    Cs \'o ka, E. and Grabowski, . On directed analogues of expander and hyperfinite graph sequences. Combinatorics, Probability and Computing, 31 0 (2): 0 184--197, 2022

    2022

  2. [10]

    Expander graph propagation

    Deac, A., Lackenby, M., and Veli c kovi \'c , P. Expander graph propagation. In Learning on Graphs Conference, pp.\ 38--1. PMLR, 2022

    2022

  3. [11]

    P., Markovich, T., and Bronstein, M

    Di Giovanni, F., Rowbottom, J., Chamberlain, B. P., Markovich, T., and Bronstein, M. M. Understanding convolution on graphs via energies. arXiv preprint arXiv:2206.10991, 2022

    2022 arXiv

  4. [12]

    K., Bronstein, M., Deac, A., Lackenby, M., Mishra, S., and Veli c kovi \'c , P

    Di Giovanni, F., Rusch, T. K., Bronstein, M., Deac, A., Lackenby, M., Mishra, S., and Veli c kovi \'c , P. How does over-squashing affect the power of GNN s? Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=KJRoQvRWNs

    2024

  5. [13]

    On the evolution of random graphs

    Erd o s, P., R \'e nyi, A., et al. On the evolution of random graphs. Publ. math. inst. hung. acad. sci, 5 0 (1): 0 17--60, 1960

    1960

  6. [14]

    Speeding up random walk mixing by starting from a uniform vertex

    Espuny D \' az , A., Morris, P., Perarnau, G., and Serra, O. Speeding up random walk mixing by starting from a uniform vertex. Electronic Journal of Probability, 29: 0 1--25, 2024

    2024

  7. [15]

    and Weber, M

    Fesser, L. and Weber, M. Mitigating over-smoothing and over-squashing using augmentations of forman-ricci curvature. In Learning on Graphs Conference, pp.\ 19--1. PMLR, 2024

    2024

  8. [16]

    Diffusion improves graph learning

    Gasteiger, J., Wei enberger, S., and G \"u nnemann, S. Diffusion improves graph learning. Advances in neural information processing systems, 32, 2019

    2019

  9. [17]

    Theoretical limitations of self-attention in neural sequence models

    Hahn, M. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8: 0 156--171, 2020

    2020

  10. [18]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  11. [19]

    Temporal graph benchmark for machine learning on temporal graphs

    Huang, S., Poursafaei, F., Danovitch, J., Fey, M., Hu, W., Rossi, E., Leskovec, J., Bronstein, M., Rabusseau, G., and Rabbany, R. Temporal graph benchmark for machine learning on temporal graphs. Advances in Neural Information Processing Systems, 36, 2024

    2024

  12. [20]

    Not too little, not too much: a theoretical analysis of graph (over) smoothing

    Keriven, N. Not too little, not too much: a theoretical analysis of graph (over) smoothing. Advances in Neural Information Processing Systems, 35: 0 2268--2281, 2022

    2022

  13. [21]

    and Suzuki, T

    Kim, J. and Suzuki, T. Transformers provably solve parity efficiently with chain of thought, 2024. URL https://arxiv.org/abs/2410.08633

    2024 arXiv

  14. [22]

    An introduction to expander graphs

    Kowalski, E. An introduction to expander graphs. Soci \'e t \'e math \'e matique de France Paris, 2019

    2019

  15. [23]

    Deeper insights into graph convolutional networks for semi-supervised learning

    Li, Q., Han, Z., and Wu, X.-M. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  16. [24]

    and P \'o czos, B

    Liao, T. and P \'o czos, B. Graph attention with random rewiring. arXiv preprint arXiv:2407.05649, 2024

    2024 arXiv

  17. [25]

    T., Lorraine, J., and Lucas, J

    Lim, D., Maron, H., Law, M. T., Lorraine, J., and Lucas, J. Graph metanetworks for processing diverse neural architectures. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=ijK5hyxs0n

    2024

  18. [26]

    Discrete groups, expanding graphs and invariant measures, volume 125

    Lubotzky, A. Discrete groups, expanding graphs and invariant measures, volume 125. Springer Science & Business Media, 1994

    1994

  19. [27]

    A fractional graph laplacian approach to oversmoothing

    Maskey, S., Paolino, R., Bacho, A., and Kutyniok, G. A fractional graph laplacian approach to oversmoothing. Advances in Neural Information Processing Systems, 36: 0 13022--13063, 2023

    2023

  20. [28]

    and Veli c kovi \'c , P

    Ong, E. and Veli c kovi \'c , P. Learnable commutative monoids for graph neural networks. In Learning on Graphs Conference, pp.\ 43--1. PMLR, 2022

    2022

  21. [29]

    and Suzuki, T

    Oono, K. and Suzuki, T. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947, 2019

    1905 arXiv

  22. [30]

    Improving language understanding by generative pre-training, 2018

    Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training, 2018. URL https://www.mikecaptain.com/resources/pdf/GPT-1.pdf

    2018

  23. [31]

    P., Dupont, E., Ruiz, F

    Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J., Wang, P., Fawzi, O., Kohli, P., and Fawzi, A. Mathematical discoveries from program search with large language models. Nature, 2023. doi:10.1038/s41586-023-06924-6

    2023 doi

  24. [32]

    Temporal graph networks for deep learning on dynamic graphs

    Rossi, E., Chamberlain, B., Frasca, F., Eynard, D., Monti, F., and Bronstein, M. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637, 2020

    2006 arXiv

  25. [33]

    K., Chamberlain, B., Rowbottom, J., Mishra, S., and Bronstein, M

    Rusch, T. K., Chamberlain, B., Rowbottom, J., Mishra, S., and Bronstein, M. Graph-coupled oscillator networks. In International Conference on Machine Learning, pp.\ 18888--18909. PMLR, 2022

    2022

  26. [34]

    Some applications of modular forms, volume 99

    Sarnak, P. Some applications of modular forms, volume 99. Cambridge University Press, 1990

    1990

  27. [35]

    and Stern, M

    Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL https://arxiv.org/abs/1804.04235

    2018 arXiv

  28. [36]

    J., and Sinop, A

    Shirzad, H., Velingker, A., Venkatachalam, B., Sutherland, D. J., and Sinop, A. K. Exphormer: Sparse transformers for graphs. In International Conference on Machine Learning, pp.\ 31613--31632. PMLR, 2023

    2023

  29. [37]

    Shirzad, H., Lin, H., Venkatachalam, B., Velingker, A., Woodruff, D., and Sutherland, D. J. Even sparser graph transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=K3k4bWuNnk

    2024

  30. [38]

    Going beyond the causal mask in language modeling

    Srambical, F. Going beyond the causal mask in language modeling. p(doom) blog, 2024. https://pdoom.org/blog.html

    2024

  31. [39]

    Commute-time-optimised graphs for gnns

    Sterner, I., Su, S., and Veli c kovi \'c , P. Commute-time-optimised graphs for gnns. In Geometry-grounded Representation Learning and Generative Modeling Workshop (GRaM) at ICML 2024, pp.\ 103--112. PMLR, 2024

    2024

  32. [40]

    Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL h...

    2014

  33. [41]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024 a

    2024 arXiv

  34. [42]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., ...

    2024 arXiv

  35. [43]

    and Chen, J

    Thost, V. and Chen, J. Directed acyclic graph neural networks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=JbuYF437WB6

    2021

  36. [44]

    D., Chamberlain, B

    Topping, J., Giovanni, F. D., Chamberlain, B. P., Dong, X., and Bronstein, M. M. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=7UmjRGzp-A

    2022

  37. [45]

    Wavenet: A generative model for raw audio

    Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., Kavukcuoglu, K., et al. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 12, 2016

    2016 arXiv

  38. [46]

    P., Budden, D., Pascanu, R., Banino, A., Dashevskiy, M., Hadsell, R., and Blundell, C

    Veli c kovi \'c , P., Badia, A. P., Budden, D., Pascanu, R., Banino, A., Dashevskiy, M., Hadsell, R., and Blundell, C. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, pp.\ 22084--22102. PMLR, 2022

    2022

  39. [47]

    Graph attention networks

    Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. Graph attention networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJXMpikCZ

    2018

  40. [48]

    softmax is not enough (for sharp out-of-distribution), 2024

    Veličković, P., Perivolaropoulos, C., Barbero, F., and Pascanu, R. softmax is not enough (for sharp out-of-distribution), 2024. URL https://arxiv.org/abs/2410.01104

    2024 arXiv

  41. [49]

    Cayley graph propagation

    Wilson, J., Bechler-Speicher, M., and Veli c kovi \'c , P. Cayley graph propagation. In The Third Learning on Graphs Conference, 2024. URL https://openreview.net/forum?id=VaTfEDs6lE

    2024

  42. [50]

    A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al

    Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33: 0 17283--17297, 2020

    2020

  43. [51]

    Dirichlet energy constrained learning for deep graph neural networks

    Zhou, K., Huang, X., Zha, D., Chen, R., Li, L., Choi, S.-H., and Hu, X. Dirichlet energy constrained learning for deep graph neural networks. Advances in Neural Information Processing Systems, 34: 0 21834--21846, 2021

    2021

  44. [52]

    T., and Ueda, M

    Ziyin, L., Wang, Z. T., and Ueda, M. Laprop: Separating momentum and adaptivity in adam, 2021. URL https://arxiv.org/abs/2002.04839

    2021 arXiv

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.