Pith. sign in

REVIEW 5 major objections 5 minor 52 references

Change of Thought: Adaptive Test-Time Computation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The SELF-Transformer runs a fixed-point loop on the attention alignment matrix inside each encoder layer, buying input-adaptive compute and, the paper reports, up to 20% accuracy gains without added parameters.

desk verdict The fixed-point attention idea is coherent and the toy induction result is credible, but the main benchmark claims are unsupported by the evidence as submitted. read the letter →

arxiv 2507.13569 v1 pith:WQ5MIVOO submitted 2025-07-17 cs.LG cs.AI

Mrinal Mathur , Mike Doan , Barak Pearlmutter , Sergey Plis This is my paper
classification cs.LGcs.AI
keywords fixed-pointself-attentiontest-timeadaptivecomputationlatentreasoningencodertransformersattentionalignmentrefinementimplicitdifferentiationtransformerexpressivityinput-adaptivecompute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a single forward pass through an encoder Transformer is too rigid: the attention alignment matrix, which decides how tokens mix, is computed once and never reconsidered. SELF-Transformer instead treats that matrix as a dynamical object, iterating it to a fixed point inside each layer so that harder inputs get more iterations and simpler inputs stop early. The reported payoff is up to 20% accuracy gains on encoder-style language, vision, and vision-language benchmarks with no additional parameters, because the same weights are reused across iterations. The reason a reader should care is the claimed trade: encoder models could gain some of the expressive power of autoregressive and chain-of-thought reasoning while staying in a single feedforward architecture.

What carries the argument

The central object is Fixed-Point Self-Attention (FPSA): within each attention head, the alignment matrix $T_k$ is recomputed from the current iterate $Z_k$ and immediately applied to the static value matrix $XW_V$, and the loop runs until the relative residual falls below $\epsilon$ or a maximum iteration count is reached. Dynamic Parameter Reuse lets the same $(W_Q,W_K,W_V,W_O,\mathrm{FFN})$ serve every iteration, and implicit differentiation / phantom-gradient updates give backward pass stability without unrolling. Spectral normalization on the qkv projection and a selective per-token freeze help keep the iteration contractive and stable; the residual itself doubles as a difficulty signal.

What would settle it

Train SELF-Transformer and a from-scratch BERT-base under identical data, tokenizer, schedule, seeds, and checkpoint selection on GLUE and SQuAD v1.1, and log per-example fixed-point iteration counts; if the reported margins (for example 88.4 versus 78.3 GLUE average) do not reproduce, or if harder examples do not consume more iterations than easy ones, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that iterating self-attention on the alignment matrix, rather than on the layer output or on tokens, makes encoder Transformers input-adaptive and removes their fixed-depth expressive ceiling. Concretely, each head refines $Z_{k+1} = T_k \, XW_V$ where $T_k = \mathrm{softmax}(Z_k W_Q W_K^\top Z_k^\top / \sqrt{d/h})$, iterating until $\|Z_{k+1}-Z_k\|/\|Z_k\| < \epsilon$ or $k=K_{\max}$. Values are taken from the static input, parameters are shared across iterations, and gradients come from implicit differentiation of the fixed point, so memory does not grow with the number of iterations. The paper reports that this recovers much of the expressive power of iterative reasoning while preserving a pure encoder design, with convergence justified by Wasserstein-1 contraction of softmax attention.

Load-bearing premise

The empirical headline rests on the assumption that the Section 4 comparison is a controlled apples-to-apples evaluation: SELF-Transformer and every baseline are trained and evaluated under the same protocol, with the prior-model scores in Table 1 being matched results rather than 'representative of strong published results'.

Editorial extensions

If this is right

  • Encoder Transformers get input-adaptive test-time compute: easy inputs can converge in a few iterations while hard inputs run longer, so the computation budget tracks difficulty.
  • The fixed-depth expressive ceiling (the constant-depth circuit class $\mathsf{TC}^0$) is lifted without token-level autoregression, opening a route to latent iterative reasoning in pure encoders.
  • No parameters are added: weight sharing across fixed-point iterations keeps memory constant in $K_{\max}$, avoiding the overhead of dynamic-depth and memory-augmented alternatives.
  • On the reported benchmarks the mechanism claims large accuracy gains, including 91.1% versus 63.1% on an induction-head toy task and up to 20% on encoder-style benchmarks.
  • The convergence stopping rule gives a per-example difficulty signal, since the residual $\|Z_{k+1}-Z_k\|$ is larger for harder inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the loop is genuinely input-adaptive, per-token iteration counts should track difficulty; logging iteration counts across benchmark examples would make that testable and would separate adaptivity from raw depth.
  • Editorial inference: the same latent-refinement principle, applied to decoder-only models, could reduce chain-of-thought token costs by letting models 'think' in hidden state rather than in text; the paper gestures at this for future work but does not claim it.
  • Editorial inference: iterating the alignment matrix rather than the hidden state suggests a general recipe for implicit layers — iterate the parameters that shape the computation, not just the activations — which could transfer to convolutional or state-space layers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SELF-Transformer, a modification of encoder self-attention in which each head iteratively refines its own attention alignment matrix through fixed-point iteration before producing the layer output. The authors claim this yields up to 20% accuracy gains on encoder-style benchmarks without increasing parameter count, and that the alignment-matrix tuning is provably convergent via Wasserstein-1 contraction of attention. The evaluation includes a toy induction-head experiment, an MNIST ViT convergence study, language benchmarks (GLUE, SQuAD, WikiText-2), vision benchmarks (ImageNet, image restoration), and vision-language benchmarks (VQA, retrieval). The paper also includes appendices with gradient computation details, attention heatmaps, and inference-time analyses.

Significance. The core idea—replacing a single-pass alignment computation with an input-adaptive fixed-point refinement of the attention matrix—is a reasonable and potentially useful design that connects to deep equilibrium models and adaptive computation. The toy induction-head experiment (§3.4) is a concrete, small-scale demonstration, and the MNIST convergence statistics (§4.1) provide a useful sanity check. If the main empirical claims were supported by controlled experiments, the contribution would be of interest to the adaptive-compute and efficient-transformer communities. As submitted, however, the load-bearing benchmark evidence is not verifiable and the theoretical convergence claim is imported rather than derived, so the significance of the reported results cannot be assessed.

major comments (5)
  1. [§4.2, Table 1; §5] The central 'up to 20% accuracy gain' claim is not supported by controlled comparisons. Table 1 explicitly labels baseline scores as 'representative of strong published results,' meaning the baselines were not rerun under the same protocol as SELF-Transformer. No training details, random seeds, error bars, checkpoints, or code are provided for the proposed model. The problem is compounded by an internal contradiction: §5 reports SELF-Transformer GLUE Avg 85.7 and SQuAD F1 91.8, while Table 1 reports GLUE Avg 88.4 and SQuAD F1 95.2 for the same model. These two sets of numbers cannot both be the reported result. Without a controlled protocol and a resolution of this discrepancy, the main empirical claim has no verifiable support.
  2. [§3.3, Eqs. (1)–(3), Eq. (7)] The claim that the alignment-matrix tuning is 'provably convergent' is not established. The paper cites Vuckovic et al. for Wasserstein-1 contraction of attention, but it does not show that the specific iterated map Z_{k+1} = softmax(Z_k W_Q W_K^T Z_k^T / sqrt(d/h)) X W_V is contractive in a norm relevant to the stopping criterion in Eq. (7). The argument does not account for the value projection X W_V, the per-head learned temperature τ, spectral normalization applied only to the qkv projection, or the selective-update rule z = torch.where(~converged_ever, z_next, z). As written, the convergence guarantee is an imported external result rather than a derivation for this architecture.
  3. [§3.3, Eq. (8); §B.1.1, Algorithm 1] The training procedure is internally inconsistent. The paper states that gradients are computed by implicit differentiation via Eq. (8), but §B.1.1 describes a Phantom Gradients approximation based on the last iteration's Jacobian, which is not the exact implicit gradient. The selective-update rule also changes the forward map, so it is unclear whether the Jacobian in Eq. (8) is evaluated at the true fixed point of the actual iteration. The text further admits that the gradient is 'invalid' for non-converged tokens and that the corresponding adjoints are discarded, but the practical effect of this discarding on the trained models is never quantified or discussed.
  4. [§4.3–§4.5, Tables 2–5] The vision and multimodal results are reported without a controlled protocol or error bars, and several entries appear implausible. For example, in Table 2, MIMIR reports BSD68 σ=50 PSNR 40.0 and SwinIR reports Set14 ×4 SSIM 0.485, both far outside the range of typical published values for these benchmarks; the Uformer-S, SwinIR, and Restormer denoising values also look inconsistent with their known performance. These issues suggest that the tables were compiled from heterogeneous sources rather than from a single evaluation pipeline, so they cannot be used to support the claimed gains.
  5. [§D.1, Figure 9] The claimed inference-efficiency advantage is not demonstrated. Figure 9 is described as showing a 1.5× speedup for sequences longer than 512 tokens and 'sublinear scaling,' but no measurement conditions, hardware details, baseline implementations, or iteration-count distributions are provided. Since the forward pass can run up to K_max=100 attention iterations per layer, the actual test-time compute may be much larger than the plot suggests. The efficiency claim needs a concrete measurement protocol and a comparison of total multiply-accumulate or wall-clock time.
minor comments (5)
  1. [§3.2, Eqs. (1)–(3)] The notation is confusing: Z^{(i)}_0 is defined as X W_Q^{(i)}, but in Eq. (2) the queries and keys are computed from the iterating state Z^{(i)}_k while the values are taken from the static X W_V. The role of the iterated state versus the static input should be stated more explicitly.
  2. [§4.2] The paragraph beginning 'To evaluate the effectivness of SELF-Transformer...' contains duplicated and incomplete sentences, and there is a typo 'effectivness.' The text should be rewritten for clarity.
  3. [§B.2] The appendix contains the typo 'adjoin computationn loops' and 'adjoin solve'; these should be corrected to 'adjoint.'
  4. [§D.2] The description of Figure 10 is partially duplicated in the following paragraph, and the figure caption says 'Heatmap of iteration counts' while the text describes specific ranges for early, middle, and late layers without a clear link to the figure axes.
  5. [§3.1] The latent-attention formula in §3.1 is introduced but never used in the actual SELF-Transformer definition in §3.2, which is a standard softmax attention; the connection between the two should be clarified or the latent-attention discussion should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracy gains are empirical, the convergence argument imports an external contraction theorem, and no load-bearing step reduces to the paper's own assumptions.

full rationale

None of the paper's load-bearing steps reduces to its own inputs by construction or by self-citation. The central claims in Sections 4 and 5 are benchmark numbers reported from experiments rather than quantities derived from a fitted parameter, so the fitted-input-called-prediction pattern does not apply. The convergence argument in Section 3.3 explicitly imports an external result, Vuckovic et al. [37], on Wasserstein-1 contractivity of attention; even if the application of that result is debatable, the citation is independent prior work and not a self-referential chain. The gradient computation in Section B is likewise imported from external work on phantom gradients [22] and implicit differentiation [26]. No author self-citations are used to justify the architecture or the empirical claims. The adaptive iteration count is defined by the stopping criteria in Eqs. (4) and (7), but that is a design choice, not a derivation of accuracy. The paper's explicit limitations in Section F are consistent with an empirical report. The uncontrolled baselines described as 'representative of strong published results' and the numeric discrepancy between Section 5 and Table 1 are reproducibility and validity threats, but they do not make the reported accuracy gains logically equivalent to the paper's own premises. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four unverified inputs: transfer of a contraction theorem to a modified update, existence of the inverse Jacobian for implicit gradients, safety of the selective freeze in forward and backward passes, and fairness of the benchmark comparisons. The first three are needed for the 'provably convergent' and gradient claims; the fourth is needed for the accuracy gains. The paper does not provide code or data to discharge any of them.

free parameters (5)
  • Convergence threshold epsilon = 1e-4 language, 1e-5 vision
    Hand-chosen in Section 4; controls the number of iterations and the definition of convergence.
  • Maximum iterations Kmax = 100
    Hand-chosen cap for all tasks in Sections 3.3 and 4; bounds worst-case compute and limits the fixed point approximation.
  • Spectral normalization coefficient = 1.0
    Set to promote contractivity of the update in Section B.2 and Section 4, with no reported tuning or validation.
  • Per-head learned temperature tau = learned, values not reported
    Introduced in Algorithm 2 to scale attention logits; absent from the main text and the convergence analysis.
  • Gradient clipping threshold T = not specified
    Used for stability in Section 3.3; the threshold is not given, so training is not reproducible.
assumptions (4)
  • domain assumption Contraction of softmax attention in Wasserstein-1 distance (Vuckovic et al. 2020) applies to the implemented FPSA update with parameter sharing and selective freezing.
    Invoked in Section 3.3 to claim 'provably convergent'; Algorithm 2 differs from the theorem's setting, so this transfer is not established.
  • standard math The inverse (I - J_f)^-1 exists at the computed iterate, so implicit differentiation in Eq. (8) is valid.
    Used for gradient computation in Section 3.3 and Section B; requires invertibility and differentiability, which the paper does not verify, and it discards adjoints for non-converged tokens.
  • ad hoc to paper Freezing converged elements with torch.where does not change the fixed point or the validity of the implicit gradient.
    The selective update in Section B.2 and Algorithm 1 is not analyzed; no proof that the frozen state satisfies the fixed point or adjoint equation.
  • domain assumption Tables 1-5 come from controlled, apples-to-apples comparisons of SELF-Transformer against the listed baselines.
    Baseline scores are labeled 'representative of strong published results' with no protocol, code, or checkpoints; this is an unverified assumption on which the accuracy claims rest.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Change of Thought: Adaptive Test-Time Computation." pith.science (2026). https://pith.science/paper/WQ5MIVOO

@misc{pith2026250713569,
  author       = {Pith},
  title        = {Pith review of: Change of Thought: Adaptive Test-Time Computation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQ5MIVOO}},
  note         = {Machine review of arXiv:2507.13569}
}
read the original abstract

Transformers evaluated in a single, fixed-depth pass are provably limited in expressive power to the constant-depth circuit class TC0. Running a Transformer autoregressively removes that ceiling -- first in next-token prediction and, more recently, in chain-of-thought reasoning. Both regimes rely on feedback loops that decode internal states into tokens only to re-encode them in subsequent steps. While this "thinking aloud" mirrors human reasoning, biological brains iterate without externalising intermediate states as language. To boost the expressive power of encoder Transformers without resorting to token-level autoregression, we introduce the SELF-Transformer: an encoder layer that iteratively refines its own attention weights to a fixed point. Instead of producing -- in one pass -- the alignment matrix that remixes the input sequence, the SELF-Transformer iteratively updates that matrix internally, scaling test-time computation with input difficulty. This adaptivity yields up to 20\% accuracy gains on encoder-style benchmarks without increasing parameter count, demonstrating that input-adaptive alignment at test time offers substantial benefits for only a modest extra compute budget. Self-Transformers thus recover much of the expressive power of iterative reasoning while preserving the simplicity of pure encoder architectures.

Figures

Figures reproduced from arXiv: 2507.13569 by the authors.

Figure 1
Figure 1. SELF-Transformer modifies vanilla self￾attention (A) to iteratively update its alignment transform T , adapting to the input sequence with￾out introducing additional parameters (B). Com￾pare with dynamic computation (C), which applies the same transformer block sequentially multiple times, and existing fixed-point iteration (D), which iterates the block to a fixed point. In this work, we propose the Self-Enhancing L… view at source ↗
Figure 2
Figure 2. Induction head attention patterns from (a) Stan [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. SELF-Transformer convergence: toy ViT MNIST classifier without any convergence guardrails exhibits stable convergence in tokens and alignment matrices. Notably, alignment matri￾ces start permutation-like and end highly structured but head, not token specific. Before proceeding with large scale experiments we have constructed a small models to study the behavior of FPSA. This is a ViT taking 12 × 12 patches of 28 × 2… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comparative attention patterns between FPI-based and vanilla self-attention mechanisms [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Iterative refinement of attention weights for the token “fox” across fixed-point iterations. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Iterative refinement of attention weights for the token “fox” across vanilla attention [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Iterative refinement of attention weights for the token “fox” across vanilla attention [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Iterative refinement of attention weights for the token “fox” across fixed-point iterations. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Inference time comparison across sequence lengths. SELF-Transformer exhibits sublinear [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Heatmap of iteration counts across layers and heads. Darker shades indicate more [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 23 canonical work pages

  1. [1]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz L...

    work page 1901

  3. [3]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  4. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

  5. [5]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  6. [6]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  7. [7]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning (ICML), 2021

    work page 2021

  8. [8]

    Dynamic neural networks: A survey

    Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7436–7456, 2021

    work page 2021

Show all 52 references
  1. [9]

    Nearest neighbor machine translation

    Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Nearest neighbor machine translation. In International Conference on Learning Representations, 2021

    2021

  2. [10]

    Compressive transformers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019

    1911 arXiv

  3. [11]

    Adaptive computation time for recurrent neural networks

    Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016

    2016 arXiv

  4. [12]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 646–661. Springer, 2016. 10

    2016

  5. [13]

    Branchynet: Fast inference via early exiting from deep neural networks

    Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pages 2464–2469. IEEE, 2016

    2016

  6. [14]

    Pondernet: Learning to ponder

    Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. arXiv preprint arXiv:2107.05407, 2021

    2021 arXiv

  7. [15]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024

    2024 arXiv

  8. [16]

    Dynamic diffusion transformer

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. arXiv preprint arXiv:2410.03456, 2024

    2024 arXiv

  9. [17]

    Forces in molecules

    Richard Phillips Feynman. Forces in molecules. Physical Review, 56(4):340–3, August 1939

    1939

  10. [18]

    A Learning Rule for Asynchronous Perceptrons with Feedback in a Combinatorial Environment

    Luis Borges Almeida. A Learning Rule for Asynchronous Perceptrons with Feedback in a Combinatorial Environment. In Proceedings of the IEEE First International Conference on Neural Networks, volume 2 of ICNN ’87, pages 609–618, New York, NY , USA, 1987. IEEE Press

    1987

  11. [19]

    Generalization of back propagation to recurrent and higher order neural networks

    Fernando Pineda. Generalization of back propagation to recurrent and higher order neural networks. In Neural information processing systems, 1987

    1987

  12. [20]

    Reviving and improving recurrent back-propagation

    Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, and Richard Zemel. Reviving and improving recurrent back-propagation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ...

    2018

  13. [21]

    Deep equilibrium models

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019

    2019

  14. [22]

    On training implicit models

    Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. Advances in Neural Information Processing Systems, 34:24247–24260, 2021

    2021

  15. [23]

    Differentiable forward and backward fixed-point iteration layers

    Younghan Jeon, Minsik Lee, and Jin Young Choi. Differentiable forward and backward fixed-point iteration layers. IEEE Access, 9:18383–18392, 2021

    2021

  16. [24]

    Invariant rectification of non-smooth planar curves

    David E Barrett and Michael D Bolt. Invariant rectification of non-smooth planar curves. Beiträge zur Algebra und Geometrie/Contributions to Algebra and Geometry, 65(3):657–679, 2024

    2024

  17. [25]

    Advancing the understand- ing of fixed point iterations in deep neural networks: A detailed analytical study

    Yekun Ke, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, and Zhao Song. Advancing the understand- ing of fixed point iterations in deep neural networks: A detailed analytical study. arXiv preprint arXiv:2410.11279, 2024

    2024 arXiv

  18. [26]

    Automatic differentiation of nonsmooth iterative algorithms

    Jérôme Bolte, Edouard Pauwels, and Samuel Vaiter. Automatic differentiation of nonsmooth iterative algorithms. Advances in Neural Information Processing Systems, 35:26404–26417, 2022

    2022

  19. [27]

    Spatially adaptive computation time for residual networks

    Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1039– 1048, 2017

    2017

  20. [28]

    Depth-adaptive transformer.arXiv preprint arXiv:1910.10073, 2019

    Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer.arXiv preprint arXiv:1910.10073, 2019

    1910 arXiv

  21. [29]

    Augmenting language models with long-term memory

    Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. Advances in Neural Information Processing Systems, 36, 2024. 11

    2024

  22. [30]

    Enhancing efficiency in sparse models with sparser selection

    Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, and Zenglin Xu. Enhancing efficiency in sparse models with sparser selection. arXiv preprint arXiv:2403.18926, 2024

    2024 arXiv

  23. [31]

    Adaptive attention span in transformers

    Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019

    1905 arXiv

  24. [32]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  25. [33]

    Droidspeak: Enhancing cross-llm communication

    Yuhan Liu, Esha Choukse, Shan Lu, Junchen Jiang, and Madan Musuvathi. Droidspeak: Enhancing cross-llm communication. arXiv preprint arXiv:2411.02820, 2024

    2024 arXiv

  26. [34]

    A novel convolution transformer-based network for histopathology-image classification using adaptive convolution and dynamic attention

    Tahir Mahmood, Abdul Wahid, Jin Seong Hong, Seung Gu Kim, and Kang Ryoung Park. A novel convolution transformer-based network for histopathology-image classification using adaptive convolution and dynamic attention. Engineering Applications of Artificial Intelligence, 135:108824, 2024

    2024

  27. [35]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  28. [36]

    Latent attention for linear time transform- ers

    Rares Dolga, Marius Cobzarenco, and David Barber. Latent attention for linear time transform- ers. arXiv preprint arXiv:2402.17512, 2024

    2024

  29. [37]

    A mathematical theory of attention

    James Vuckovic, Aristide Baratin, and Remi Tachet des Combes. A mathematical theory of attention. arXiv preprint arXiv:2007.02876, 2020

    2007 arXiv

  30. [38]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  31. [39]

    Glue: A multi-task benchmark and analysis platform for natural language under- standing

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language under- standing. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  32. [40]

    Efficientnetv2: Smaller models and faster training

    Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. InInternational conference on machine learning, pages 10096–10106. PMLR, 2021

    2021

  33. [41]

    Towards vision transformer unrolling fixed-point algorithm: a case study on image restoration

    Peng Qiao, Sidun Liu, Tao Sun, Ke Yang, and Yong Dou. Towards vision transformer unrolling fixed-point algorithm: a case study on image restoration. arXiv preprint arXiv:2301.12332, 2023

    2023 arXiv

  34. [42]

    Pirm challenge on perceptual image enhancement on smartphones: report

    Andrey Ignatov, Radu Timofte, et al. Pirm challenge on perceptual image enhancement on smartphones: report. In European Conference on Computer Vision (ECCV) Workshops, January 2019

    2019

  35. [43]

    Mimir: Masked image modeling for mutual information-based adversarial robustness

    Xiaoyun Xu, Shujian Yu, Zhuoran Liu, and Stjepan Picek. Mimir: Masked image modeling for mutual information-based adversarial robustness. arXiv preprint arXiv:2312.04960, 2023

    2023

  36. [44]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398–414, 2017

    2017

  37. [45]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings...

    2014

  38. [46]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision...

    2015

  39. [47]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 12

    2017 arXiv

  40. [48]

    The hidden attention of mamba models

    Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590, 2024

    2024 arXiv

  41. [49]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  42. [50]

    Repvit: Revisiting mobile cnn from vit perspective

    A Wang, H Chen, Z Lin, H Pu, and G Ding. Repvit: Revisiting mobile cnn from vit perspective. arxiv 2023. arXiv preprint arXiv:2307.09283, 2023. 13 A Fixed Point Self-Attention A.1 Preliminaries A.1.1 Standard Self-Attention Given an input sequence X ∈ Rn×d with n tokens and em...

    2023 arXiv

  43. [51]

    The quick brown fox jumps over the lazy dog

    to address the challenges of backpropagating through iterative attention updates, avoiding the computational cost of unrolling or inverting large Jacobians. B.1 Gradient Computation for Self-Attention Let Tk denote the attention matrix at iteration k, refined through fixed-poi...

  44. [52]

    while maintaining faster inference times and significantly fewer parameters. F Limitations and Future Work While our proposed SELF-attention mechanism, has demonstrated promising results on several evaluative tasks, we acknowledge certain limitations that also highlight avenue...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.