Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Transformers trained to predict next tokens deviate most from the Bayes-optimal predictor exactly in high-ambiguity contexts, and scaling does not remove the gap.

desk verdict The paper has a clean synthetic result showing transformers misallocate accuracy in high-ambiguity contexts, but the title is more confident than the evidence. read the letter →

arxiv 2506.16288 v1 pith:M2YT3EQ6 submitted 2025-06-19 cs.LG cs.AI

Leo Gagnon , Eric Elmoznino , Sarthak Mittal , Tom Marty , Tejas Kasetty , Dhanya Sridhar , Guillaume Lajoie This is my paper
classification cs.LGcs.AI
keywords next-tokenpredictionambiguityBayes-optimalmeta-learninghiddenMarkovmodelsin-contextlearningtest-timescalingMonteCarloinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard next-token prediction carries a detrimental inductive bias: it spends the same computation on every token even though the difficulty of implicit Bayesian inference over latent tasks varies wildly across a sequence. Using MetaHMM, a synthetic family of hidden Markov models with an exactly computable Bayes-optimal predictor, the paper shows that Transformers deviate most from that oracle at short context lengths, where the posterior over latent tasks is most ambiguous, producing a transient 'KL bump' that persists across model sizes from 1.1M to 25M parameters. The paper then converts a pre-trained autoregressive model into a Monte Carlo predictor that decouples task inference from token prediction, achieving substantial gains in ambiguous contexts for small models and a principled form of test-time scaling. If the finding transfers to natural language, it implies that foundation-model training should be ambiguity-sensitive, allocating more inference effort where the latent-task posterior is broad.

What carries the argument

Two mechanisms carry the argument. MetaHMM is a family of hidden Markov models assembled from shared building blocks — base cycles, cycle families, emission groups, and shift variables — in which the Bayes-optimal posterior predictive can be computed exactly with the forward algorithm, so model behavior can be measured against a true oracle rather than an approximation. The paper measures divergence with the symmetrized KL, $\mathrm{Div}_x(t) = \tfrac{1}{2} D_{\mathrm{KL}}[p^* \parallel p_\phi] + \tfrac{1}{2} D_{\mathrm{KL}}[p_\phi \parallel p^*]$, at each position. The proposed Monte Carlo predictor replaces the Bayesian integral over latent tasks with a sample average: a frozen base model provides a latent proxy $z = E(x_{\mathrm{ctx}})$ from average-pooled hidden states over an unambiguous context, a fine-tuned conditional model predicts $p_\phi(x_t \mid x_{<t}, z)$, and a diffusion transformer draws $z$ from the conditional embedding posterior given $x_{<t}$; increasing the number of samples $S$ is a direct test-time scaling knob.

What would settle it

Train a sequence model on a MetaHMM variant where posterior entropy is uniform across positions and check whether the KL bump disappears; separately, estimate the same symmetrized KL on natural text with a learned approximate oracle and group tokens by estimated ambiguity, because if high-ambiguity tokens show no KL bump the claim fails outside the synthetic setting.

Watch

Extended reading notes

Core claim

The paper's central discovery is a systematic failure mode rather than a new architecture win: evaluated against the exact Bayes-optimal posterior predictive on MetaHMM, Transformers show a bump in symmetrized KL divergence at short context lengths, exactly where the posterior over the latent HMM is most entropic. The bump shrinks with scale in the low-ambiguity regime but persists at all tested sizes, so additional parameters do not fix the ambiguity problem. Interpreting this as poor capacity allocation under a fixed per-token compute budget, the paper introduces a Monte Carlo predictor that averages conditional predictions over task embeddings sampled from a diffusion model; on small base models this improves divergence in ambiguous contexts and improves monotonically as the number of samples grows, though the advantage diminishes and eventually reverses for the largest model.

Load-bearing premise

The load-bearing premise is that MetaHMM's ambiguity structure — predictions are hardest early in a sequence, when the latent task is most uncertain — matches the ambiguity structure of the real sequences foundation models are trained on; if real corpora do not show that profile, the KL bump is an artifact of the synthetic family.

Editorial extensions

If this is right

  • Under MetaHMM, a Transformer's divergence from the Bayes-optimal predictor rises at short context lengths and decays later; the KL bump persists as model size grows from 1.1M to 25M parameters.
  • Fixed computation per token is therefore a poor inductive bias for sequence meta-learning: high-ambiguity predictions receive too little capacity and low-ambiguity ones receive too much.
  • A Monte Carlo predictor that separates task inference from token prediction improves divergence in ambiguous contexts for small base models and scales monotonically with the number of samples.
  • The Monte Carlo gains diminish with model size and reverse for the largest model, so the method is most useful where the base model underfits the oracle.
  • Training on a skewed distribution that emphasizes short sequences does not remove the KL bump, suggesting the failure is structural rather than a training-length artifact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If natural language exhibits a similar ambiguity profile, the same KL bump should be measurable on real corpora using an approximate Bayesian oracle, and adaptive-compute mechanisms such as early exiting, retrieval, or clarification-seeking become natural remedies.
  • The paper's decoupling suggests a taxonomy of failure: task inference (posterior over latent context) and token prediction (likelihood given the task) are separate bottlenecks, and future architectures could allocate capacity to each independently.
  • Because the Monte Carlo gains shrink with scale, a testable prediction follows: very large foundation models should still show ambiguity-sensitive errors, just smaller in absolute terms, and posterior-entropy-conditioned calibration on LLM text would reveal it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MetaHMM, a synthetic sequence meta-learning benchmark built from compositionally structured HMMs with an exactly computable Bayesian posterior predictive oracle. Causal Transformers of four sizes (1.1M to 25M parameters) are trained on this benchmark and evaluated by a symmetrized KL divergence from the Bayes-optimal predictor at each sequence position. The authors report a transient 'KL bump' in short, high-ambiguity contexts that persists across model sizes, and they propose a Monte Carlo predictor that separates task inference (a diffusion model over contextual embeddings) from token prediction (a fine-tuned conditional Transformer), showing gains for small models with test-time scaling and diminishing returns at larger scales.

Significance. If the empirical claims hold, the paper makes a useful diagnostic contribution: it provides a controlled environment in which the Bayes-optimal next-token predictor is exactly computable, allowing a non-circular quantitative comparison of learned predictors against a normative ideal. The MetaHMM framework with its 12,288 HMMs and the symmetrized KL metric are well suited for isolating ambiguity-driven failure modes. The Monte Carlo predictor is an interesting proof of concept that gives a principled interpretation of test-time scaling as approximate Bayesian inference. The main limitations are that the central persistence claim currently lacks uncertainty quantification and the proposed capacity-allocation mechanism is not directly tested; the paper also clearly acknowledges that adapting the method to naturalistic settings remains future work.

major comments (3)
  1. [Effect of increased model size; Figure 1c; Appendix B.2] The central claim that the KL bump persists across model sizes from 1.1M to 25M parameters is presented without any measure of seed-level dispersion, despite Appendix B.2 stating that all experiments use three seeds. Please report per-seed curves or error bars for Div(t) in Figure 1c, for the average Div values in Figure 2c, and for the skewed-length experiments in Figure 4, and provide a quantitative test (e.g., bump amplitude at 25M relative to run-to-run variance) to support the persistence claim.
  2. [Effect of increased model size; Section 2.1; Figure 4] The explanatory mechanism of 'poor capacity allocation'—under-parameterization in high-ambiguity regions and over-parameterization in low-ambiguity regions—is asserted but never directly measured. A direct test could vary per-token compute (e.g., adaptive computation time) or measure the complexity of the Bayes-optimal predictive as a function of position. The skewed-length training experiment in Figure 4 changes the training distribution, not the per-token compute budget, so it does not test this mechanism.
  3. [Evaluation of Transformers; Appendix B.2] The Div metric compares trained models to p* defined with the full uniform prior over 12,288 HMMs, while the models are trained on a held-out subset of roughly 11,288 HMMs (with about 1,000 HMMs held out for validation). If the empirical task distribution during training differs from the oracle's uniform prior, early-context divergence could partly reflect prior mismatch rather than ambiguity-driven difficulty. Please either match the training and oracle task priors or quantify the sensitivity of Div(t) to the train/oracle task distribution.
minor comments (6)
  1. [Appendix A.4] The sentence 'theses methods enhance can be seen as addressing cases where the conditional prediction ... is too difficult' is ungrammatical and should be rewritten.
  2. [Figure 5 caption] The caption for Figure 5 states only that xctx has length 10; it should clarify whether the conditional predictor and diffusion model are retrained for this setting or reused from the length-100 experiments, since that affects the interpretation.
  3. [Section 1] The statement that 'the model initially perfectly fits the Bayesian oracle' is too strong given the absence of error bars and the use of a symmetrized KL measure; 'closely matches' would be more precise.
  4. [Section 2.1] The comparison between the 1.1M MC predictor with S=5 and the 6.3M model in Figure 1c is made across separate figures; a joint panel or an explicit numerical comparison would make this claim easier to verify.
  5. [Abstract / Code availability] The text says 'Code is available here' but no repository URL appears in the visible text; please include the link.
  6. [Equation (2), Figure 2] The notation Divx(t) in Equation (2) is abbreviated as Div(t) or 'Div' in Figures and text; unify the notation throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core claims are evaluated against an exact, external Bayesian oracle on a synthetic benchmark, not against fitted parameters or self-referential definitions.

full rationale

The paper's central claim is that Transformers deviate most from the Bayes-optimal posterior predictive in high-ambiguity contexts, and that this divergence persists across model sizes. This is established by comparing model predictions p_phi(xt | x<t) to the exactly computed oracle p*(xt | x<t) defined by the MetaHMM environment prior in Equation (1). The divergence measure in Equation (2) is a symmetrized KL between the model and the oracle; no parameter of the oracle is fitted from the model's outputs. The MC predictor in Equation (3) is an explicit Monte Carlo approximation of Equation (1), and its gains are also measured against the same external oracle, so the improvement claim is not circular. The only overlapping-author citation, Mittal et al. (2024), appears in related work on task vectors and is not used to justify the paper's main hypothesis or to forbid alternative explanations. The paper's limitations—lack of error bars, the synthetic nature of MetaHMM, and the asserted rather than directly tested capacity-allocation mechanism—are threats to empirical robustness and external validity, not instances of circularity. No derivation step reduces a predicted quantity to a fitted input or to a self-citation, so the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central evaluation is against a known generative distribution, so the paper adds no fitted physical constants. The hand-chosen hyperparameters and the domain assumptions about transfer and the latent proxy are the main unverified inputs.

free parameters (2)
  • x_ctx length = 100
    Hand-chosen context length for the latent proxy z; used to define z = E(xctx) as unambiguous, and Figure 5 shows performance degrades with x_ctx = 10, so the method's gains depend on this choice.
  • DiT capacity = 8 layers, 8 heads, dimension 512
    Deliberately oversized so task inference is not the bottleneck (Sec. 2.1); a manually chosen capacity allocation rather than a tuned-to-task parameter.
assumptions (4)
  • domain assumption Next-token prediction in auto-regressive models can be written as Bayesian integration over a latent task theta (Eq. 1).
    The entire meta-learning framing relies on this decomposition, introduced in the Introduction and Eq. (1).
  • domain assumption The MetaHMM family of HMMs faithfully represents the ambiguity structure of sequence meta-learning relevant to foundation models.
    Used to justify generalizing synthetic results to next-token prediction; Section 1 and Discussion acknowledge naturalistic adaptation remains future work.
  • ad hoc to paper z = E(xctx) with unambiguous xctx is a good proxy for the latent task theta, and sampling z conditioned on x<t approximates sampling theta from the posterior.
    Introduced in the three-step method (Section 2); only validated internally via ablations and proxy behavior.
  • domain assumption Increasing the number of Monte Carlo samples S decreases divergence to the Bayesian oracle.
    Empirically observed on MetaHMM (Fig. 2b) and treated as a general property of the estimator rather than proven.
invented entities (1)
  • Contextual embedding z = E(xctx) used as latent task proxy
    purpose: Replaces the unknown latent theta so a conditional predictor and a diffusion sampler can be trained separately.
    The embedding is a constructed representation whose validity is only shown on MetaHMM via ablations (Fig. 5); it makes no falsifiable prediction outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective." pith.science (2026). https://pith.science/paper/M2YT3EQ6

@misc{pith2026250616288,
  author       = {Pith},
  title        = {Pith review of: Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M2YT3EQ6}},
  note         = {Machine review of arXiv:2506.16288}
}
read the original abstract

The rapid adaptation ability of auto-regressive foundation models is often attributed to the diversity of their pre-training data. This is because, from a Bayesian standpoint, minimizing prediction error in such settings requires integrating over all plausible latent hypotheses consistent with observations. While this behavior is desirable in principle, it often proves too ambitious in practice: under high ambiguity, the number of plausible latent alternatives makes Bayes-optimal prediction computationally intractable. Cognitive science has long recognized this limitation, suggesting that under such conditions, heuristics or information-seeking strategies are preferable to exhaustive inference. Translating this insight to next-token prediction, we hypothesize that low- and high-ambiguity predictions pose different computational demands, making ambiguity-agnostic next-token prediction a detrimental inductive bias. To test this, we introduce MetaHMM, a synthetic sequence meta-learning benchmark with rich compositional structure and a tractable Bayesian oracle. We show that Transformers indeed struggle with high-ambiguity predictions across model sizes. Motivated by cognitive theories, we propose a method to convert pre-trained models into Monte Carlo predictors that decouple task inference from token prediction. Preliminary results show substantial gains in ambiguous contexts through improved capacity allocation and test-time scalable inference, though challenges remain.

Figures

Figures reproduced from arXiv: 2506.16288 by the authors.

Figure 1
Figure 1. The MetaHMM benchmark. a) Latent structure of a MetaHMM environment. White rectangles contain mutually exclusive discrete choices θi which together define an HMM θ. Yellow circles represent hidden states and blue rectangles represent observable symbols. b) Example of an HMM sampled from the given building blocks. c) Divx(t) for different model sizes. ground this problem in meta-learning theory, make links to resourc… view at source ↗
Figure 2
Figure 2. The Monte-Carlo Predictor. a) Computational and training structure of the MC predictor. The parts in dotted lines are only used to train pϕ(xt | x<t, z), while the content of the green rectangle corresponds to the MC predictor. b) Performance of the MC predictor (green) with various number of samples compared with the original sequence model (gray, 1.1M). Y-axis is Div. c) Average Div(t) (across sequence length) of … view at source ↗
Figure 3
Figure 3. Hyperparameters 50 timesteps. Other hyperparameters are the same as in (Lovelace et al., 2023). B.4. Figures [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Transformer trained on random sequence lengths Variation of the next-token training setup of Figure 1c) where sequences have random lengths uniformly sampled between 1 and 200. This puts more pressure on the predictors to perform well at low context lengths. Batch size…
Figure 5
Figure 5. Figure 5: Short xctx variation of Figure 2b) where xctx has length 10. This means that z should be a poor proxy for θ and the MC predictor should do poorly. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 14 canonical work pages

  1. [1]

    What learning algorithm is in-context learn- ing? investigations with linear models

    Aky¨urek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learn- ing? investigations with linear models. arXiv preprint arXiv:2211.15661,

    arXiv

  2. [3]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258,

    arXiv

  3. [5]

    Analog bits: Gen- erating discrete data using diffusion models with self- conditioning

    Chen, T., Zhang, R., and Hinton, G. Analog bits: Gen- erating discrete data using diffusion models with self- conditioning. arXiv preprint arXiv:2208.04202,

    arXiv

  4. [6]

    Beyond Bayes-optimality: meta-learning what you know you don't know

    Grau-Moya, J., Del´etang, G., Kunesch, M., Genewein, T., Catt, E., Li, K., Ruoss, A., Cundy, C., Veness, J., Wang, J., et al. Beyond bayes-optimality: meta-learning what you know you don’t know. arXiv preprint arXiv:2209.15618,

    work page Pith review arXiv

  5. [7]

    K., Mat- tern, C., Aitchison, M., et al

    Grau-Moya, J., Genewein, T., Hutter, M., Orseau, L., Del´etang, G., Catt, E., Ruoss, A., Wenliang, L. K., Mat- tern, C., Aitchison, M., et al. Learning universal predic- tors. arXiv preprint arXiv:2401.14953,

    arXiv

  6. [11]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

    arXiv

  7. [12]

    and Grave, E

    Izacard, G. and Grave, E. Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:2012.04584,

    arXiv 2012

  8. [14]

    Transformer Language Models Handle Word Frequency in Prediction Head

    Kobayashi, G., Kuribayashi, T., Yokoi, S., and Inui, K. Transformer language models handle word frequency in prediction head. arXiv preprint arXiv:2305.18294,

    work page Pith review arXiv

Show all 41 references
  1. [15]

    K., Chan, S

    Lampinen, A. K., Chan, S. C., Singh, A. K., and Shanahan, M. The broader spectrum of in-context learning. arXiv preprint arXiv:2412.03782,

    arXiv

  2. [16]

    Towards an information theoretic frame- work of context-based offline meta-reinforcement learn- ing

    5 Next-Token Prediction Should be Ambiguity-Sensitive Li, L., Zhang, H., Zhang, X., Zhu, S., Yu, Y ., Zhao, J., and Heng, P.-A. Towards an information theoretic frame- work of context-based offline meta-reinforcement learn- ing. arXiv preprint arXiv:2402.02429,

    arXiv

  3. [18]

    Lovelace, J., Kishore, V ., Chen, Y ., and Weinberger, K. Q. Diffusion guided language modeling. arXiv preprint arXiv:2408.04220,

    arXiv

  4. [19]

    Understanding the origin of information-seeking exploration in probabilistic objectives for control

    Millidge, B., Seth, A., and Buckley, C. Understanding the origin of information-seeking exploration in probabilistic objectives for control. arXiv preprint arXiv:2103.06859,

    arXiv

  5. [21]

    Webgpt: Browser-assisted question-answering with hu- man feedback

    Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V ., Saunders, W., et al. Webgpt: Browser-assisted question-answering with hu- man feedback. arXiv preprint arXiv:2112.09332,

    arXiv

  6. [22]

    and Iso, H

    Niwa, A. and Iso, H. Ambignlg: Addressing task ambiguity in instruction for nlg. arXiv preprint arXiv:2402.17717,

    arXiv

  7. [23]

    Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv:1807.03748,

    arXiv

  8. [24]

    Ortega, P

    URL https: //arxiv.org/abs/2412.16720. Ortega, P. A., Wang, J. X., Rowland, M., Genewein, T., Kurth-Nelson, Z., Pascanu, R., Heess, N., Veness, J., Pritzel, A., Sprechmann, P., et al. Meta-learning of se- quential strategies. arXiv preprint arXiv:1905.03030 ,

    1905 arXiv

  9. [25]

    Peebles, W

    URL https: //arxiv.org/abs/2306.04891. Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision , pp. 4195–4205,

    arXiv

  10. [28]

    Scaling llm test- time compute optimally can be more effective than scal- ing model parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.03314,

    arXiv

  11. [30]

    M., Raghunathan, A., Liang, P., and Ma, T

    Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080,

    arXiv

  12. [31]

    Task vectors in in-context learning: Emergence, for- mation, and benefit

    Yang, L., Lin, Z., Lee, K., Papailiopoulos, D., and Nowak, R. Task vectors in in-context learning: Emergence, for- mation, and benefit. arXiv preprint arXiv:2501.09240,

    arXiv

  13. [32]

    Zhang, M. J. and Choi, E. Clarify when necessary: Re- solving ambiguity through interaction with lms. arXiv preprint arXiv:2311.09469,

    arXiv

  14. [33]

    Vector- icl: In-context learning with continuous vector represen- tations

    Zhuang, Y ., Singh, C., Liu, L., Shang, J., and Gao, J. Vector- icl: In-context learning with continuous vector represen- tations. arXiv preprint arXiv:2410.05629,

    arXiv

  15. [34]

    Xie et al

    and has become central to understanding foundation model behavior. Xie et al. (2021) were among the first to formalize this connection by framing foundation model pre-training as meta-learning over a distribution of tasks. This perspective has since inspired a wide range of st...

    2021

  16. [35]

    or Turing machines (Grau-Moya et al., 2024), but these lack a tractable Bayesian oracle, limiting their utility for quantitative evaluation. A.2. Latent variables in Transformers Multiple previous works have explored to what extent Transformers explicitly represent the latent ...

    2024

  17. [36]

    have trained Transformers with continuous task vectors in order to increase the performance on some ICL tasks. A.3. Ambiguity and the Limits of Bayesian Inference The challenge of inference under ambiguity has long been studied in cognitive science and decision theory. While B...

    1996

  18. [37]

    Further, system prompts, such as those used in chatbots, serve to disambiguate the model’s role and task (Niwa & Iso, 2024)

    can interact with APIs or calculators, actively reducing uncertainty—akin to epistemic actions in humans. Further, system prompts, such as those used in chatbots, serve to disambiguate the model’s role and task (Niwa & Iso, 2024). As a side note, chain-of-thought prompting (We...

    2024

  19. [38]

    However, theses methods enhance can be seen as addressing cases where the conditional prediction p(xt | x<t, θ), not the task inference, is too difficult for the model

    methods improve reasoning by encouraging intermediate steps. However, theses methods enhance can be seen as addressing cases where the conditional prediction p(xt | x<t, θ), not the task inference, is too difficult for the model. Finally, reinforcement learning from human feed...

    2024

  20. [39]

    In transformers, this has evolved into early exiting mechanisms (Zhu, 2021), which conditionally terminate processing

    allowed models to learn how many steps to take. In transformers, this has evolved into early exiting mechanisms (Zhu, 2021), which conditionally terminate processing. See Snell et al. (2024) for a modern discussion of test-time scaling. Our approach offers a specific, and prin...

    2024

  21. [40]

    Additionally, the embedding z may encode high- frequency details from xctx that are hard to sample accurately

    or self-conditioning (Chen et al., 2022)—could help bridge this gap. Additionally, the embedding z may encode high- frequency details from xctx that are hard to sample accurately. This issue could be exacerbated by increasing the dimension of x. A promising direction would be ...

    2022

  22. [41]

    Other hyperparameters are the same as in (Lovelace et al., 2023)

    Hyperparameters 50 timesteps. Other hyperparameters are the same as in (Lovelace et al., 2023). B.4. Figures Figure

    2023

  23. [1999]

    and Ho, J

    Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512,

    arXiv

  24. [2010]

    Schick, T., Dwivedi-Yu, J., Dess`ı, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T

    doi: 10.1037/a0020511. Schick, T., Dwivedi-Yu, J., Dess`ı, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551,

    doi

  25. [2016]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  26. [2018]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  27. [2019]

    L., Sharma, A

    Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. arXiv preprint arXiv:2310.15213,

    arXiv

  28. [2020]

    Do llms un- derstand ambiguity in text? a case study in open-world question answering

    Keluskar, A., Bhattacharjee, A., and Liu, H. Do llms un- derstand ambiguity in text? a case study in open-world question answering. In 2024 IEEE International Con- ference on Big Data (BigData) , pp. 7485–7490. IEEE,

    2024

  29. [2021]

    Does learning the right latent variables necessarily improve in-context learning? arXiv preprint arXiv:2405.19162,

    Mittal, S., Elmoznino, E., Gagnon, L., Bhardwaj, S., Sridhar, D., and Lajoie, G. Does learning the right latent variables necessarily improve in-context learning? arXiv preprint arXiv:2405.19162,

    arXiv

  30. [2022]

    In-context language learning: Architectures and algorithms

    Aky¨urek, E., Wang, B., Kim, Y ., and Andreas, J. In-context language learning: Architectures and algorithms. arXiv preprint arXiv:2401.12973,

    arXiv

  31. [2023]

    Hendel, R., Geva, M., and Globerson, A

    URL https://arxiv.org/abs/2303.07971. Hendel, R., Geva, M., and Globerson, A. In-context learning creates task vectors. arXiv preprint arXiv:2310.15916,

    arXiv

  32. [2024]

    Adaptive computation time for recurrent neural networks

    Graves, A. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983,

    arXiv

  33. [2025]

    A., and Choi, Y

    Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., and Choi, Y . We’re afraid language models aren’t modeling ambiguity. arXiv preprint arXiv:2304.14399,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.