Pith. sign in

REVIEW 4 major objections 4 minor 29 references

Contextually Guided Transformers via Low-Rank Adaptation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a Transformer can compress any prefix into a context vector and use that vector to generate low-rank weight updates, letting the model specialize to the prefix without keeping it in the input.

desk verdict A genuinely new architecture with solid synthetic results, but the headline 'eliminates prompts' claim is only weakly supported on text and should be scoped down. read the letter →

arxiv 2506.05672 v1 pith:EBBZXQZE submitted 2025-06-06 cs.LG cs.CL

Andrey Zhmoginov , Jihwan Lee , Max Vladymyrov , Mark Sandler This is my paper
classification cs.LGcs.CL
keywords contextuallyguidedtransformerlow-rankadaptationin-contextlearningprompt-freeinferenceweightgenerationcontextrepresentationsmoothnessregularizationvariationalautoencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Contextually Guided Transformer (CGT), a Transformer that learns to compress the preceding prefix into a context vector $y^{\ell}$ and uses that vector to generate low-rank weight updates for the layers that follow, so the model can specialize to the prefix without keeping the prefix in the input. The authors' central claim is that CGT maps any given prefix into a specialized model that performs well on the remainder of the sequence. On a synthetic arithmetic in-context-learning task, freezing the context after two demonstrations yields 77.5% accuracy, close to the 77.7% baseline that still sees all examples in context. On text the claim is weaker: the frozen summary helps for roughly 200 tokens and then falls behind, which the authors address by updating $y^{\ell}$ as a moving average.

What carries the argument

The load-bearing mechanism is the low-rank weight generator driven by the context summary. For each insertion point $\kappa$, the model produces $\delta W_\kappa(y^{\ell}) = \sum_{k=1}^{r} L_\kappa(y^{\ell})_k R_\kappa(y^{\ell})_k^T$, where $L_\kappa(y^{\ell}) = \sum_m L_{\kappa,m} \sigma_m(y^{\ell})$, $R_\kappa(y^{\ell}) = \sum_m R_{\kappa,m} \sigma_m(y^{\ell})$, and $\sigma(y^{\ell}) = h(S_\kappa y^{\ell})$ are template-mixing coefficients from a learned linear map followed by a tanh or softmax nonlinearity. The supporting pieces are the dual-pathway architecture ($x$ independent of $y$ before layer $\ell$, $y$ seeing both), the auxiliary loss that freezes $y^{\ell}_{s-1}$ at a random cut and trains it to predict the tokens after the cut, the continuity regularizer $R_C = \sum_s \zeta_C(s) \|n_s - n_{s-1}\|^2$ on normalized summaries, and the batch-decorrelation regularizer $R_D$ that pushes summaries from different samples toward orthogonality. Together they make a single frozen vector $y^{\ell}$ capable of generating the per-token linear modulations that specialize the rest of the network.

What would settle it

A concrete falsifying experiment: take a text mixture where the topic changes sharply at the split point, freeze $y^{\ell}$ just before the change, and measure the specialized model's cross-entropy as a function of distance from the split. If for every split position the frozen model's loss converges to the uninformed baseline and never stays below the dynamic model beyond roughly 200 tokens, the claim that any prefix can be summarized into a sufficient frozen vector is false for natural text.

Watch

Extended reading notes

Core claim

The paper's central claim is that a Transformer can be split into two roles—parsing the prefix and generating the weights that specialize the model to that prefix—and that both roles can live in one architecture. In the Contextually Guided Transformer (CGT), the first $\ell$ layers maintain separate $x$ and $y$ activation pathways; the $x$ pathway is computed from prior $x$ alone, while the $y$ pathway sees both, so the final summary $y^{\ell}$ aggregates all prefix information. Starting at layer $\ell+1$, each MLP and self-attention block is preceded by a linear operator $T_\kappa(x; y^{\ell}) = x + \delta W_\kappa(y^{\ell})x$, with $\delta W_\kappa$ generated as a low-rank combination of learned template matrices whose mixing coefficients are a nonlinear function of $y^{\ell}$. When $y^{\ell}$ is frozen, these operators become position-independent and fold into the surrounding linear layers, producing a specialized model that runs on the remainder without the prefix. The auxiliary loss—cross-entropy on tokens after a randomly chosen cut, with $y^{\ell}$ taken from the intact prefix—is what forces $y^{\ell}$ to be a sufficient summary, and element-wise regularizers make it slow and decorrelated. Table 1 reports 77.5% specialized accuracy versus 77.7% for the full-context baseline; on text, the frozen summary helps only for about 200 tokens, after which a moving-average update of $y^{\ell}$ is used to retain the benefit.

Load-bearing premise

The whole construction rests on the assumption that one frozen vector $y^{\ell}_s$, extracted at an arbitrary sequence position, contains all the prefix information needed to generate a good specialized model for everything after that position, even when the future text drifts to a different topic.

Editorial extensions

If this is right

  • Freezing the context after two examples yields 77.5% specialized accuracy on the synthetic in-context-learning task, close to the 77.7% baseline that still attends to all four demonstrations.
  • Because the generated transformations are linear, a frozen $y^{\ell}$ can be folded into fixed position-independent weights, so the remainder of the sequence can be processed without carrying the prefix tokens or their activations.
  • Element-wise regularization turns $y^{\ell}$ into a slowly changing topic vector: the dot-product matrix shows clear blocks at task boundaries, and averaged summaries cluster Wikipedia pages by category.
  • The auxiliary loss alone raises specialized accuracy from 15.3% to 77.0%, and adding element-wise regularization raises it to 77.5% while also slightly improving dynamic-context accuracy.
  • On text, the specialized model is better than the dynamic model immediately after freezing but is overtaken after roughly 200 tokens; updating $y^{\ell}$ with a moving average keeps the contextual benefit across the whole sequence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If a frozen context summary can be reused, the expensive part of prompting can be amortized: compute $y^{\ell}$ once for a fixed instruction set, fold the resulting weight updates, and serve many continuations without the prompt; the paper does not make this serving argument, but it follows directly from the foldability of frozen linear updates.
  • The text results suggest the representation the method learns is not a constant summary but a slowly drifting latent, pointing toward learned update rates or learned covariance kernels $K_{s,t}$ rather than a fixed moving-average rate; this connects CGT to recurrent state-space sequence models.
  • Because the paper shows task coefficients $a$ and $b$ are nearly linear functions of $y^{\ell}$, a natural untested prediction is that arithmetic on summaries (averaging or adding them) should steer behavior predictably, giving a direct intervention test of whether $y^{\ell}$ is a true task variable.
  • The batch orthogonality regularizer presumes different sequences have unrelated context summaries; for open-domain text this may be too rigid, and a learned per-dimension scaling of $R_D$ could preserve smoothness without forcing global orthogonality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Contextually Guided Transformers (CGT), a modification of the Transformer architecture in which a context summary vector y_l, computed at layer l, is used to generate low-rank weight updates for the layers above l. The model is trained with an auxiliary loss so that freezing y_l after a prefix yields a specialized model for the rest of the sequence, ostensibly removing the need to keep prompts in the input during inference. The authors also introduce a smoothness prior on y_l, motivated by a variational-autoencoder view with a Gaussian process prior, and derive a simpler element-wise regularizer. Experiments cover a synthetic in-context-learning task, linear regression, and text mixtures (c4/wikipedia). The synthetic ICL experiment shows that freezing y_l after two examples achieves 77.5% accuracy versus 77.7% for the dynamic baseline, and the linear-regression results show plausible specialization curves. In the text experiment, however, the frozen model is overtaken by the dynamic model after roughly 200 tokens, and the paper resorts to a moving-average update of y_l as a fallback.

Significance. If the central claim held for arbitrary prefixes, CGT would be a noteworthy step toward prompt-free inference: a single model that computes its own weight adaptation from context, with potential efficiency gains from folding the generated linear maps into the network. The synthetic ICL result is clean and well-controlled, with multiple runs and error bars, and the interpretability analysis (linear decoding of task parameters from y_l, t-SNE clustering of topics) is interesting. The derivation from a GP-VAE prior to the element-wise regularizer is a useful conceptual contribution. However, the text-domain evidence does not support the advertised generality: the frozen model fails over longer horizons, and the efficiency motivation is never measured. The paper's value is therefore real but narrower than the abstract and introduction claim.

major comments (4)
  1. [Section 4.4, Fig. 6(a)] The central claim of Section 1 ('CGT model maps any given prefix into a specialized model that performs well on the remainder of the sequence') is directly contradicted by the text experiment. After the context representation is frozen at roughly 100 tokens, the specialized model is better than the dynamic model only near the freezing point, and the dynamic model surpasses it after about 200 tokens. The paper attributes this to thematic shifts from training on two concatenated excerpts, but no experiment tests that explanation. Please add a control experiment on a single coherent document (or topically homogeneous text) and, if the crossover persists, revise the claim to be scoped to short or topically stable contexts. This is load-bearing because the text experiment is the only language-domain evidence for the frozen-prefix mechanism.
  2. [Section 4.4, Fig. 14] The moving-average variant of y_l (the 'informed' model) is not a frozen context representation; it updates y_l throughout the second part of the text. Consequently, Fig. 14 does not demonstrate the advertised mechanism of generating a specialized model from a fixed prefix. The only text-domain evidence for long-range benefit relies on this non-freezing mechanism. The paper should explicitly state that the frozen-y_l variant is effective only over a limited horizon and that the moving-average variant is a separate contribution (or a fallback), not evidence for the central claim.
  3. [Section 4.3 and Appendix D.2, Fig. 11] The linear-regression experiments show a similar horizon dependence: the specialized (frozen y_l) model's error grows as more samples arrive after the freezing point, and some models diverge beyond the training length of 64 samples. This suggests that the 'performs well on the remainder' claim holds only within a limited horizon in this setup as well. Please report the horizon over which the frozen model remains competitive (e.g., number of samples after freezing) and discuss the limits of the specialization mechanism.
  4. [Section 3.1 and Abstract] The paper motivates CGT by the computational overhead of prompts and states that folding the generated linear maps into subsequent layers reduces cost when y_l is frozen. However, no experiment measures wall-clock time, FLOPs, or latency, and the accuracy-only results do not validate the efficiency motivation. Please add a cost analysis (theoretical FLOPs with and without folding, or measured latency) or explicitly re-scope the contribution to accuracy and representation properties.
minor comments (4)
  1. [Section 1, first sentence] The phrase 'laying at the foundation' should be 'lying at the foundation'.
  2. [Section 4.1, paragraph on synthetic dataset] There is a typo: 'whcih' should be 'which' in the description of the signed integer.
  3. [Section 4.4, reference to figure] The text refers to 'Fig. 4.4' in the y_l representation analysis; this should be 'Fig. 6(c)' or the appropriate figure number.
  4. [Section 3.2, mutual information claim] The statement that optimizing the auxiliary cross-entropy 'maximizes the lower bound on the mutual information I(t>s; y_l_s)' should be clarified: minimizing the conditional cross-entropy with respect to a fixed data distribution maximizes a lower bound on mutual information only if the entropy of t>s is treated as constant. A brief clarifying sentence would help.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the specialized-model result is the trained auxiliary objective, and self-citations are minor and non-load-bearing.

full rationale

The paper's central mechanism is constructively defined: the architecture separates x and y pathways, generates low-rank weight updates from the context summary y^l, and folds these updates when y^l is frozen. The auxiliary loss in Section 3.2 explicitly trains the model so that a frozen y^l performs well on the remainder of the sequence; the specialized-accuracy results in Table 1 and Section 4.3 are therefore evaluations of this trained objective rather than independent predictions. This is a limitation of what the experiment demonstrates, but it is not a circular reduction: the model could fail to optimize the auxiliary loss, and the paper shows that without this loss freezing fails. The mutual-information argument is a stated identity and the VAE derivation is an explicit modeling assumption with a factorized encoder and Gaussian-process prior, not an input disguised as an output. The self-citations, including the von Oswald et al. 2023 works with overlapping authors, are used only as related work and are not load-bearing for the correctness of CGT. One explicit limitation appears in Section 4.4: the frozen specialized model is surpassed by the dynamic model after roughly 200 tokens, and the authors resort to a moving-average update of y^l (Figure 14). This undermines the scope of the 'any prefix' claim and is a genuine correctness/scope concern, but it is not circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central result depends on several hand-chosen hyperparameters and on modeling assumptions about the context summary. The paper reports ablations for some of these, but no formal justification or released artifacts. No new physical or external entities are introduced; the contextual summary y is an activation vector within the model.

free parameters (6)
  • Layer index ell (context readout layer) = 4 (ICL), 8 (text), 7 (linear regression)
    Layer at which y is read out and then frozen; ablations show it strongly affects specialized model accuracy (Appendix B.2.1, D.2).
  • Regularization weights w_C and w_D = w_C=0.08, w_D=0.04 for ICL; w_C=0.04 and w_D=0.02 in Section 4.4 for text, but w_C=0.08, w_D=0.04 in Appendix B.2.2
    Chosen by tuning; they control smoothness and orthogonality of y and affect specialized accuracy. The main text and appendix report different values for the text experiments.
  • Loss mixing coefficient eta = 0.5
    Interpolates between cross-entropy loss and auxiliary loss; set by hand.
  • Low-rank rank r and template count M = r=4, M=16 for ICL; r=4, M=8 for linear regression
    Capacity choices for the weight generator; ablations show rank has a modest effect on performance.
  • Context embedding dimensionality dim y = 64 (ICL), 128 or 32 (text), 64 (linear regression)
    Capacity of the context summary; the authors report that larger dim y improves specialized model accuracy.
  • Temporal weighting profile zeta_C(s) = Constant for ICL; empirical for other experiments
    Element-wise regularizer weighting; the paper notes it can be derived from the covariance kernel or chosen empirically.
assumptions (4)
  • domain assumption Optimizing the auxiliary cross-entropy loss maximizes a lower bound on mutual information I(t>s; y_s), and this makes y_s condense all prefix information useful for future prediction.
    Section 3.2 makes this the mechanism that lets a frozen y generate a good specialized model; the paper does not prove the bound is tight for long text.
  • ad hoc to paper The context representation y follows a Gaussian process prior with known kernel K, and the causal encoder factorizes into independent Gaussians.
    Appendix A.1 (equations 6 and 7) assumes this structure to derive the VAE loss and the simpler element-wise regularizer; it is a chosen modeling assumption, not derived from data.
  • ad hoc to paper Low-rank weight updates with rank r and M template matrices are expressive enough to represent the context-dependent specialization.
    Section 3.1 defines the low-rank generator T_k; ablations show rank has limited impact, but the form of the generator is assumed rather than justified.
  • domain assumption The x activation path can remain independent of y before layer ell, so y can be frozen and folded into later layers.
    Section 3.1 and Figure 1(a) build the architecture on this separation; if x were coupled to y earlier, the specialization-by-freezing procedure would not work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contextually Guided Transformers via Low-Rank Adaptation." pith.science (2026). https://pith.science/paper/EBBZXQZE

@misc{pith2026250605672,
  author       = {Pith},
  title        = {Pith review of: Contextually Guided Transformers via Low-Rank Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EBBZXQZE}},
  note         = {Machine review of arXiv:2506.05672}
}
read the original abstract

Large Language Models (LLMs) based on Transformers excel at text processing, but their reliance on prompts for specialized behavior introduces computational overhead. We propose a modification to a Transformer architecture that eliminates the need for explicit prompts by learning to encode context into the model's weights. Our Contextually Guided Transformer (CGT) model maintains a contextual summary at each sequence position, allowing it to update the weights on the fly based on the preceding context. This approach enables the model to self-specialize, effectively creating a tailored model for processing information following a given prefix. We demonstrate the effectiveness of our method on synthetic in-context learning tasks and language modeling benchmarks. Furthermore, we introduce techniques for enhancing the interpretability of the learned contextual representations, drawing connections to Variational Autoencoders and promoting smoother, more consistent context encoding. This work offers a novel direction for efficient and adaptable language modeling by integrating context directly into the model's architecture.

Figures

Figures reproduced from arXiv: 2506.05672 by the authors.

Figure 1
Figure 1. (a) Model architecture showing processing of a single token with the activation component y ν being a function of (x ν−1 , y ν−1 ) and x ν being a function of x ν−1 alone for any ν ∈ [2, ℓ]; activations y ℓ are parameterizing transformations Tκ (·; y ℓ ) mapping x ν to x˜ ν op = Tκx ν op for ν ≥ ℓ (see Sec. 3.1); (b) Auxiliary loss incentivizing the Transformer to encode long-range information in y: the auxiliary lo… view at source ↗
Figure 2
Figure 2. An example of synthetic sequences analyzed in Sec. 4.2 with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. The model continuously learns and improves its predictions as it receives more samples. Models: (i) “baseline” Transformer (dim x = 128); (ii) CGT (dynamic y ℓ , dim x = 128, dim y = 64); (iii) CGT (frozen y ℓ after 5, 10, or 20 samples). Horizontal lines: Baseline L2 errors after 5, 10 and 20 samples. Ridge re￾gression error is also shown. (on the last two examples) across 4 models (see [PITH_FULL_IMAGE:figures/fu… view at source ↗
Figures from the paper (16 more)
Figure 5
Figure 5. Figure 5: Cross-entropy loss on c4 text dataset. While the improvement achieved with dim x = 128 and dim y = 128 is less than that observed when simply increasing the main em￾bedding dimensionality proportionally to 224 (roughly equivalent to dim x+dim y in a conventional model)…
Figure 6
Figure 6. Figure 6: (a) Average difference between the cross-entropy losses of the specialized (fixed y ℓ , blue curve) and non-specialized (dynamic y ℓ , green curve) models (typical loss value is ∼ 3); the specialized model is better where this difference is below zero. (b) Evolution of…
Figure 7
Figure 7. Figure 7: Ablation study results: (a) dependence of the model accuracy with frozen y ℓ on the layer index ℓ (model trained with the auxiliary loss and the element-wise regularization); (b) varying the rank of the generated matrices with wC = 2wD = 0.08 [PITH_FULL_IMAGE:figures/…
Figure 8
Figure 8. Figure 8: Specialized model accuracy for different values of [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Average accuracies on the last two examples in different specialization runs for [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: (a) A typical dependence on y ℓ on the sequence index for a synthetic in-context learning task with 8 examples; (b) dot product ni ·nj for normalized y ℓ embeddings at two different locations for this synthetic dataset. show the evolution of a typical L2 error between…
Figure 11
Figure 11. Figure 11: Average L2 loss computed at a given sample index. We compare specialized models obtained after seeing 20 samples for 5 different values of ℓ ∈ [3, 7]. Model behavior generally deteriorates towards the end of the sequence (for a large number of examples). Some models d…
Figure 12
Figure 12. Figure 12: Comparison of L2 errors for CGT model on the linear regression dataset with dim y = 64 and: (a) dim x = 64 (dashed), (b) dim x = 128 (solid). The plot shows 3 separate runs in both cases. One experiment with dim x = 128 shows degradation of performance for longer sequ…
Figure 13
Figure 13. Figure 13: (a) average difference (on the second part of the document) between cross-entropies of a specialized model with y ℓ pre-computed on the first part and a baseline language model; (b) dot￾product plot ni · nj for a combination of 3 different text excerpts described in A…
Figure 14
Figure 14. Figure 14: Average difference between the cross-entropy losses of the “informed” and “uninformed” [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: t-SNE plot for pages from 8 wikipedia categories using a model trained on individual c4 articles instead of pairs of randomly joined samples. This plot shows a much better separation between different categories, which is probably due to this test distribution being c…
Figure 16
Figure 16. Figure 16: t-SNE plot for pages from 8 wikipedia categories using a model trained on 2 merged c4 excerpts: (a) model without regularization; (b) model with element-wise regularization. The embeddings are obtained by averaging 16 sequential values of y ℓ at the end of the text. C…
Figure 17
Figure 17. Figure 17: (a) Normalized averaged intensity ⟨|fk| 2 ⟩ of discrete Fourier transform spectra fk of all y ℓ components for VAEs with β equal to 0.01, 0.1, 0.3 and 1.0. The averaging is performed over all components of y ℓ and over 256 samples. The averaged intensity is then norma…
Figure 18
Figure 18. Figure 18: Dot product ni ·nj plot computed for 3 different VAE models trained on c4 and evaluated on a mixture of 3 distinct texts (see Appendix B.4): (a) β = 1, (b) β = 3, (c) β = 10. 20 0 20 40 40 20 0 20 40 60 0 1 2 3 4 5 6 7 (a) 40 20 0 20 40 40 20 0 20 0 1 2 3 4 5 6 7 (b) …
Figure 19
Figure 19. Figure 19: t-SNE plots for 3 different VAE models trained on c4 and evaluated on wikipedia pages from 8 distinct categories: (a) β = 1, (b) β = 3, (c) β = 10. for different values of β ( [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Context representation y ℓ evolution along the sequence for 3 different VAE models trained on c4 and evaluated on a mixture of 3 distinct texts (see Appendix B.4): (a) β = 1, (b) β = 3, (c) β = 10. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 12 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    1901

  3. [3]

    Gaussian process prior variational autoencoders

    Francesco Paolo Casale, Adrian V Dalca, Luca Saglietti, Jennifer Listgarten, and Nicolo Fusi. Gaussian process prior variational autoencoders. 32nd Conference on Neural Information Processing Systems, 2018

    work page 2018

  4. [4]

    Cf-vit: A general coarse-to-fine method for vision transformer

    Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: A general coarse-to-fine method for vision transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 37 0 (6): 0 7042--7052, Jun. 2023. doi:10.1609/aaai.v37i6.25860. URL https://ojs.aaai.org/index.php/AAAI/article/view/25860

    work page doi:10.1609/aaai.v37i6.25860 2023

  5. [5]

    Transformer-based conditional variational autoencoder for controllable story generation

    Le Fang, Tao Zeng, Chaochun Liu, Liefeng Bo, Wen Dong, and Changyou Chen. Transformer-based conditional variational autoencoder for controllable story generation. arXiv preprint arXiv:2101.00828, 2021

    arXiv 2021

  6. [6]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 9318--9333, Singapore, December 2023. Association for Computational Linguistics. doi:10.18653/v1/2023.findings-emnlp.624. URL https://aclanthol...

    doi:10.18653/v1/2023.findings-emnlp.624 2023

  7. [7]

    A VAE for Transformers with Nonparametric Variational Information Bottleneck

    James Henderson and Fabio Fehr. A VAE for Transformers with Nonparametric Variational Information Bottleneck . In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6QkjC_cs03X

    work page 2023

  8. [8]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes . In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014

    2014

Show all 29 references
  1. [9]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT -3? In Eneko Agirre, Marianna Apidianaki, and Ivan Vuli \'c (eds.), Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Know...

    2022 doi

  2. [10]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference o...

    2022 doi

  3. [11]

    Fast-slow recurrent neural networks

    Asier Mujika, Florian Meier, and Angelika Steger. Fast-slow recurrent neural networks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., ...

    2017

  4. [12]

    Hypertuning: Toward adapting large language models without back-propagation

    Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. Hypertuning: Toward adapting large language models without back-propagation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machin...

    2023

  5. [13]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  6. [14]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020. URL...

    2020

  7. [15]

    Orthogonal projection loss

    Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, and Fahad Shahbaz Khan. Orthogonal projection loss. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 12333--12343, 2021

    2021

  8. [16]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Pro...

    2021

  9. [17]

    Learning to control fast-weight memories: An alternative to dynamic recurrent networks

    Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4 0 (1): 0 131--139, 1992. doi:10.1162/neco.1992.4.1.131

    1992 doi

  10. [18]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  11. [19]

    Patch slimming for efficient vision transformers

    Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 12155--12164, 2022. doi:10.1109/CVPR52688.2022.01185

    2022

  12. [20]

    Visualizing data using t-sne

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9 0 (86): 0 2579--2605, 2008. URL http://jmlr.org/papers/v9/vandermaaten08a.html

    2008

  13. [21]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jona...

    2023

  14. [22]

    Uncovering mesa-optimization algorithms in transformers, 2023

    Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers, 2023

    2023

  15. [23]

    T-cvae: Transformer-based conditioned variational autoencoder for story completion

    Tianming Wang and Xiaojun Wan. T-cvae: Transformer-based conditioned variational autoencoder for story completion. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pp.\ 5233--5239. International Joint Conferences on Arti...

    2019 doi

  16. [24]

    Wikimedia downloads

    Wikimedia Foundation . Wikimedia downloads. URL https://dumps.wikimedia.org

  17. [25]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer

    Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 2964--2972, 2022

    2022

  18. [26]

    Chi, Jason Wei, Jeff Dean, Liam B

    Barret Zoph, Colin Raffel, Dale Schuurmans, Dani Yogatama, Denny Zhou, Don Metzler, Ed H. Chi, Jason Wei, Jeff Dean, Liam B. Fedus, Maarten Paul Bosma, Oriol Vinyals, Percy Liang, Sebastian Borgeaud, Tatsunori B. Hashimoto, and Yi Tay. Emergent abilities of large language mode...

    2022

  19. [27]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  20. [28]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  21. [29]

    Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2009

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.