Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that sequence perplexity is theoretically invariant under any token-ordering factorization, and shows that trained LLMs nonetheless deviate from this invariance in ways traceable to positional biases in self-attention.

desk verdict Solid empirical protocols and an interesting attention analysis attached to an elementary theorem that the paper over-sells as a contradiction of theory. read the letter →

arxiv 2505.08739 v1 pith:TTCHVMR4 submitted 2025-05-13 cs.CL

Xiaoliang Luo , Xinyi Xu , Michael Ramscar , Bradley C. Love This is my paper · ORCID
classification cs.CL
keywords perplexityinvariancechainruleofprobabilitytokenorderfactorizationautoregressivelanguagemodelsself-attentionpositionalbiasconsistencyLLMevaluationprotocolattentionsinks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a theorem: for any well-defined probability distribution over token sequences, perplexity is invariant under every factorization order, including forward, backward, or any fixed permutation, because the chain rule collapses the averaged conditional log-probabilities into the joint probability of the whole sequence. It then argues that earlier comparisons of forward- and backward-trained models violated the conditions of that proof by omitting begin-of-sequence tokens, retraining tokenizers on reversed text, or confusing logical reversal with token-level reversal, and therefore were comparing different sequences. Retraining GPT-2 models at three scales with corrected protocols, the paper finds small but systematic forward-over-backward perplexity advantages and much larger deviations for permuted order, which it traces to positional and locality biases in causal self-attention. The central claim is that when a trained model's perplexity depends on factorization order, that dependence is a measurable sign that the model's probability distribution is internally inconsistent, not a refutation of the theoretical invariance. A sympathetic reading of the paper is that it turns perplexity into a principled diagnostic: order-invariance is the benchmark, and deviations from it expose where a model's learned conditionals fail to form a single coherent joint distribution.

What carries the argument

The load-bearing object is the chain-rule telescoping identity for sequence perplexity, stated as Equation (1): $PP_\sigma = \exp\left(-\frac{1}{n}\ln P(X_0,X_1,\ldots,X_n)\right)$ for every permutation $\sigma$, proved by writing each conditional as a ratio of joint probabilities and observing that the log terms cancel in pairs. That identity turns perplexity into a pure function of the joint probability, and therefore into a concrete benchmark: any two factorizations of the same sequence must give the same number if the model is a faithful probability model. The experimental machinery is the protocol that makes the comparison legitimate, namely a BOS token with $P(X_0)=1$, one tokenizer trained only on forward text, strict token-level reversal or fixed permutation inside the context window, and full 1,024-token sequences, together with normalized attention entropy and normalized attention rank as diagnostics that localize the deviations within self-attention.

What would settle it

Train sibling models with the same protocol but with position information removed or made relative only, and check whether the forward/backward perplexity gaps vanish; if they persist, the cause is not simply the positional bias isolated here. Alternatively, evaluate a single fixed-permutation model on all factorizations of the same sequences via importance sampling, since a perfectly consistent model must yield a reweighted joint-probability estimate equal to its forward estimate to numerical precision.

Watch

Extended reading notes

Core claim

The paper establishes, by a telescoping-sum argument, the identity $PP_\sigma = \exp\left(-\frac{1}{n}\ln P(X_0,X_1,\ldots,X_n)\right)$ for any permutation $\sigma$ of the token indices, where $X_0$ is a begin-of-sequence token with $P(X_0)=1$. Because every chain-rule factorization of the same joint probability yields the same product, sequence perplexity is a property of the full sequence alone and not of the order in which its conditionals are read. The empirical core is a controlled comparison of 27 GPT-2 models trained from scratch at 124M, 355M, and 774M parameters on the same 1.3-billion-token neuroscience corpus in forward, backward, and fixed-permuted token orders, using identical tokenization and a BOS token on every sequence. Forward- and backward-trained models achieve near-identical sequence-level perplexities with correlations above 0.99, yet forward models are consistently lower, and the gap widens with model size; permuted training produces substantially higher perplexities and attention patterns that favor neither local nor long-range structure. The paper attributes these gaps to causal self-attention's positional biases rather than to the data, and presents them as evidence that practical LLMs do not maintain the probability consistency the theorem guarantees.

Load-bearing premise

The empirical comparison assumes that each trained model approximates the true conditional distribution of its own factorization well enough that perplexity gaps between separately trained models can be read as deviations from the theorem rather than as ordinary approximation error.

Editorial extensions

If this is right

  • Earlier forward/backward comparisons that omitted BOS tokens, retrained tokenizers, or reversed characters instead of tokens were comparing different sequences, so their ordering-effect claims cannot be interpreted against the theorem.
  • Because the invariance holds for any well-defined distribution, factorization-order dependence in a trained model is diagnostic: it marks where the model's conditional estimates fail to cohere into one joint distribution.
  • Forward and backward training are nearly interchangeable in downstream performance: on the BrainBench benchmark both directions match human experts, and backward-trained models are not worse aligned with human judgments than forward-trained ones.
  • Attention biases toward adjacent tokens and toward tokens at maximum distance are general across pretrained transformers and plausibly explain the lost-in-the-middle retrieval failure pattern.
  • The empirical deviations do not refute the invariance result; they locate the inconsistency in the learned approximation, motivating attention-based and architecture-level explanations of LLM behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If perplexity gaps are driven by positional priors, then a model that suppresses positional bias, for example through relative position encodings or sink-token removal, should show smaller forward/backward gaps under the same protocol; that is testable.
  • The same protocol could serve as a calibration check before deployment: a model whose perplexity depends strongly on factorization order is assigning meaning to token position beyond content, which may predict hallucination or out-of-distribution brittleness.
  • Because the theorem applies to any permutation, the fixed-permutation setting is the sharpest probe: it deliberately breaks the locality statistics of natural language, so the large deviations observed there suggest that much of an LLM's language understanding is a learned local-structure prior rather than a general joint model.
  • A direct extension would be to test invariance without retraining, by estimating the conditionals of one factorization from a single model through importance sampling; perfect consistency would require the reweighted joint-probability estimate to match the forward estimate, so any discrepancy would quantify the model's internal inconsistency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper makes a formal claim that sequence perplexity is invariant under any factorization order: for a fixed joint distribution over a token sequence, PP_σ = exp(−1/n ln P(X0,...,Xn)) regardless of the permutation σ used to order the conditional probabilities. It then trains GPT-2 models at three scales (124M, 355M, 774M) on forward, backward, and arbitrarily permuted versions of a neuroscience corpus, applying protocols designed to satisfy the theorem's conditions (BOS tokens, a single forward-trained tokenizer, and token-level permutations), and reports systematic perplexity gaps, attention-pattern differences, representational divergence, and BrainBench results. The paper also criticizes prior studies for methodological deviations such as missing BOS tokens, retraining tokenizers on reversed text, and conflating logical reversal with token reversal.

Significance. If the empirical interpretation were supported, the paper would provide a useful benchmark and diagnostic for order consistency in autoregressive LLMs. The chain-rule identity in Eq. (1) is elementary but worth stating as a conceptual baseline, and the protocol corrections for prior work are valuable and concrete. The empirical study is substantial: 27 models, multiple sizes, multiple initializations, public code and weights, and a clear description of training and evaluation setups. The descriptive findings about attention biases and the relationship between forward and backward models are interesting. However, the central interpretive claim that the measured gaps constitute deviations from the theoretical invariance is not established, because the theorem constrains a single distribution whereas the experiments compare several separately trained estimators. The contribution should be reframed as a descriptive study of training-order-induced estimation bias, with the theoretical benchmark used as a reference point rather than as a claim of theorem violation.

major comments (3)
  1. [§3.1, Table 2, Eq. (1)] The central empirical inference is not supported by the stated theorem. Equation (1) establishes an identity for a single fixed joint distribution P: for any permutation σ, the perplexity computed from P's conditionals on a given event equals exp(−1/n ln P(X0,...,Xn)). It imposes no constraint on two separately trained models, each of which realizes a different estimated distribution P̂_fwd, P̂_bwd, or P̂_perm. The reported gaps in Table 2 and Figure 1 are therefore entirely compatible with the theorem if the models' approximation errors differ, and the paper never measures or bounds the per-model approximation error relative to the true distribution or to the pushforward target distributions. Consequently, the abstract's claims of 'systematic deviations from theoretical invariance' and of probability distributions being 'inconsistent and therefore untrustworthy' go beyond what the data show; the empirical contribution should be reframed as a descriptive study of order-induced estimation bias.
  2. [§3.1, Appendix D.2] The paper does not specify, for the comparisons in Table 2 and Figure 1, whether the backward- and permuted-trained models are evaluated on the reversed/permuted versions of each validation sequence or on the original forward-order text. Under the theorem, only the former comparison is even notionally covered by Eq. (1); evaluating a backward model on forward-order text measures an off-factorization conditional that Eq. (1) never addresses. In addition, Appendix D.2 states that 'the starting token's probability is masked out in the softmax operation to align closely with the proof' without saying whether the corresponding term is removed from both the numerator and the denominator; if it is excluded from both, the empirical perplexity uses a different normalization than Eq. (1), which divides by n. Both points must be resolved before the quantitative comparisons can be interpreted.
  3. [§2, Implications] The theorem as stated applies to a fixed distribution P, but the experiment with permuted orders actually concerns pushforward distributions: if the training data for order σ are generated by applying σ to each natural string, the target distribution is Q_σ with Q_σ(σ(w)) = P(w). The paper does not state or prove the lemma that a perfect model of Q_σ has perplexity exp(−1/n ln P(w)) on σ(w); without this lemma, Eq. (1) does not by itself justify comparing forward, backward, and permuted models. Adding this step would also make explicit why the theoretical benchmark concerns the pair (model, its own factorization), which is exactly the point at which the approximation-error concern in Major Comment 1 enters.
minor comments (5)
  1. [§2, first paragraph] The phrase 'the same joint probability' should specify that the chain-rule factorizations refer to the same event, or to the same observed assignment to indexed positions, because for a non-exchangeable distribution the probability of the permuted string differs from the probability of the original string.
  2. [Appendix E, Eq. (2)] The index ranges and the meaning of t should be stated explicitly (number of real tokens after the BOS token) and matched to the denominator in Eq. (1), so that readers can verify the empirical perplexity uses the same normalization as the theorem.
  3. [Figure 1 caption] The caption contains a duplicated 'across across' that should be corrected.
  4. [Table 2] The t-tests treat the 9,413 validation sequences as independent; given the small effect sizes for the forward-backward comparisons (Cohen's d around 0.2-0.7), a discussion of practical significance and of possible dependence between sequences would strengthen the interpretation.
  5. [§3.4] The sentence 'We exclude permuted-text models, which converged during training' is unclear about whether the intended meaning is that the models converged to a poor solution or did not converge; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the invariance proof is self-contained; the empirical gap is an inferential limitation, not a circular derivation.

full rationale

The central derivation (Section 2, Eq. 1) is a direct application of the chain rule to a single fixed joint distribution: PPσ is defined as exp(-1/n Σ ln P(Xσ(i)|X0,Xσ(1),...,Xσ(i−1))), and by the chain rule the sum telescopes to exp(-1/n ln P(X0,X1,...,Xn)). This involves no fitted parameters, no benchmark values, and no self-citation; it is a self-contained mathematical identity. The empirical section trains separate GPT-2 models on forward, backward, and permuted factorizations and compares their achieved perplexities. The theorem itself places no constraint on the relative perplexities of different finite-capacity estimators, and the paper itself states the condition: "yields the same theoretical perplexity, provided the model accurately captures the conditional probabilities." Thus the observed gaps (Fig. 1, Table 2) do not reduce to the theorem's inputs by construction; they are differences between imperfect estimators, which is an interpretational gap rather than a circular step. Self-citations (BrainBench and the neuroscience corpus from Luo et al. 2024a,b) are used as external benchmarks and not to justify the proof. No circular step satisfying the required quote-and-reduction standard was found.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The proof is self-contained: it uses the chain rule, log/exp algebra, and the BOS convention. The empirical protocol adds domain assumptions about how training losses map to the proof and about model approximation quality. No new entities are introduced.

assumptions (5)
  • standard math Chain rule of probability
    Used in Section 2 to decompose joint probability into conditionals for arbitrary permutation.
  • domain assumption BOS token X0 with P(X0)=1 and ln P(X0)=0
    Section 2; ensures the first conditional is well-defined and the telescoping sum closes to ln P(X0,X1,...,Xn).
  • domain assumption The empirical loss over n real tokens matches the proof's product over i=1..n
    Appendix D.2 masks 'the starting token's probability'; the exact alignment with the proof is ambiguous.
  • domain assumption The trained model on a factorization approximates the true conditional distribution well enough for cross-model perplexity comparison to test the theorem
    Section 2 Implications and Section 3; without this, the empirical deviations are not 'contrary to theory'.
  • domain assumption The tokenizer trained on forward text preserves sequence identity when applied to reversed or permuted token IDs
    Appendix D.1; ensures the same sequences are compared across factorizations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies." pith.science (2026). https://pith.science/paper/TTCHVMR4

@misc{pith2026250508739,
  author       = {Pith},
  title        = {Pith review of: Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTCHVMR4}},
  note         = {Machine review of arXiv:2505.08739}
}
read the original abstract

Can autoregressive large language models (LLMs) learn consistent probability distributions when trained on sequences in different token orders? We prove formally that for any well-defined probability distribution, sequence perplexity is invariant under any factorization, including forward, backward, or arbitrary permutations. This result establishes a rigorous theoretical foundation for studying how LLMs learn from data and defines principled protocols for empirical evaluation. Applying these protocols, we show that prior studies examining ordering effects suffer from critical methodological flaws. We retrain GPT-2 models across forward, backward, and arbitrary permuted orders on scientific text. We find systematic deviations from theoretical invariance across all orderings with arbitrary permutations strongly deviating from both forward and backward models, which largely (but not completely) agreed with one another. Deviations were traceable to differences in self-attention, reflecting positional and locality biases in processing. Our theoretical and empirical results provide novel avenues for understanding positional biases in LLMs and suggest methods for detecting when LLMs' probability distributions are inconsistent and therefore untrustworthy.

Figures

Figures reproduced from arXiv: 2505.08739 by the authors.

Figure 1
Figure 1. Average validation perplexity differences across across model sizes and training directions. Forward and backward text training yields similar perplexities, though forward models consistently achieve lower values (difference below zero). This gap widens slightly with model size. Permuted text training yields much higher perplexity than both forward and backward models, with similar differences to each, causing the c… view at source ↗
Figure 2
Figure 2. Attention entropy across three data orders (GPT-2 124M). Normalized attention entropy (min = 0, max = 1) is measured across layers averaged over heads and sampled text sequences for varying context sizes. Models trained on forward, backward, and permuted token orders show distinct patterns despite using the same data. Forward and backward models exhibit similar trends, with larger differences at early layers. The mo… view at source ↗
Figure 3
Figure 3. Positional bias in self-attention varies with training directions and layers (GPT-2 124M). Normalized attention rank (min = 0, max = 1) is plotted as a function of token distance within the context, averaged across heads, sampled sequences, and layers. Compared to models at initialization (Init), forward (Fwd) and backward (Bwd) trained models show strong positional biases toward both nearby tokens and tokens at max… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Representational similarities across training directions. Forward- and backward-trained models show higher representational similarity to each other than to the model trained on permuted text. Across all comparisons, similarity declines in deeper layers, with the permu…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do Tabular Foundation Models Agree with Themselves?

    cs.LG 2026-08 accept novelty 6.0 of 10

    All evaluated tabular foundation models violate both marginalization and factorization consistency, so their autoregressive joint distributions are order-dependent and cannot arise from any single joint distribution.

Reference graph

Works this paper leans on

38 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Barbero, A

    F. Barbero, A. Banino, S. Kapturowski, D. Kumaran, J. G. M. Araújo, A. Vitvitskyi, R. Pascanu, and P. Veličković. Transformers need glasses! Information over-squashing in language tasks, Oct. 2024. URL http://arxiv.org/abs/2406.04267. arXiv:2406.04267 [cs]

    arXiv 2024

  2. [2]

    Bondarenko, M

    Y. Bondarenko, M. Nagel, and T. Blankevoort. Quantizable Transformers : Removing Outliers by Helping Attention Heads Do Nothing , Nov. 2023. URL http://arxiv.org/abs/2306.12929. arXiv:2306.12929 [cs]

    arXiv 2023

  3. [3]

    Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Somani, P. Hase, M. Wagner, F. Roger, V. Mikulik, S. Bowman, J. Leike, J. Kaplan, and E. Perez. Reasoning Models Don ’t Always Say What They Think . Anthropic, 2025

    work page 2025

  4. [4]

    Clark, U

    K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What Does BERT Look At ? An Analysis of BERT 's Attention , June 2019. URL http://arxiv.org/abs/1906.04341. arXiv:1906.04341 [cs]

    arXiv 2019

  5. [5]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding , May 2019. URL http://arxiv.org/abs/1810.04805. arXiv:1810.04805 [cs]

    arXiv 2019

  6. [6]

    P. Gage. A new algorithm for data compression. The C Users Journal archive, 12: 0 23--38, 1994. URL https://api.semanticscholar.org/CorpusID:59804030

    work page 1994

  7. [7]

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile : An 800GB Dataset of Diverse Text for Language Modeling , 2020. URL https://arxiv.org/abs/2101.00027. \_eprint: 2101.00027

    arXiv 2020

  8. [8]

    X. Gu, T. Pang, C. Du, Q. Liu, F. Zhang, C. Du, Y. Wang, and M. Lin. When Attention Sink Emerges in Language Models : An Empirical View , Mar. 2025. URL http://arxiv.org/abs/2410.10781. arXiv:2410.10781 [cs]

    arXiv 2025

Show all 38 references
  1. [9]

    Huang, W

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu. A Survey on Hallucination in Large Language Models : Principles , Taxonomy , Challenges , and Open Questions . ACM Transactions on Information Systems, 43 0 (2): 0 1--55, Mar. 20...

    2025 arXiv

  2. [10]

    Kallini, I

    J. Kallini, I. Papadimitriou, R. Futrell, K. Mahowald, and C. Potts. Mission: Impossible Language Models , Jan. 2024. URL http://arxiv.org/abs/2401.06416. arXiv:2401.06416 [cs]

    2024 arXiv

  3. [11]

    Kornblith, M

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of Neural Network Representations Revisited , 2019. URL https://arxiv.org/abs/1905.00414. \_eprint: 1905.00414

    2019 arXiv

  4. [12]

    Kovaleva, A

    O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. Revealing the Dark Secrets of BERT , Sept. 2019. URL http://arxiv.org/abs/1908.08593. arXiv:1908.08593 [cs]

    2019 arXiv

  5. [13]

    A. K. Lampinen, S. C. Y. Chan, and K. Hermann. Learned feature representations are biased by complexity, learning order, position, and more, Sept. 2024. URL http://arxiv.org/abs/2405.05847. arXiv:2405.05847 [cs]

    2024 arXiv

  6. [14]

    Lanham, A

    T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion, K. Lukosiute, K. Nguyen, N. Cheng, N. Joseph, N. Schiefer, O. Rausch, R. Larson, S. McCandlish, S. Kundu, S. Kadavath, S. Yang, T. Henighan, T. Maxwell, T. Te...

    2023

  7. [15]

    Latuske, O

    P. Latuske, O. Toader, and K. Allen. Interspike Intervals Reveal Functionally Distinct Cell Populations in the Medial Entorhinal Cortex . Journal of Neuroscience, 35 0 (31): 0 10963--10976, Aug. 2015. ISSN 0270-6474, 1529-2401. doi:10.1523/JNEUROSCI.0276-15.2015. URL https://w...

    2015 doi

  8. [16]

    Lindsey, W

    J. Lindsey, W. Gurnee, E. Ameisen, B. Chen, A. Pearce, N. L. Turner, C. Citro, D. Abrahams, S. Carter, B. Hosmer, J. Marcus, M. Sklar, A. Templeton, T. Bricken, C. McDougall, H. Cunningham, T. Henighan, A. Jermyn, A. Jones, A. Persic, Z. Qi, T. B. Thompson, S. Zimmerman, K. Ri...

    2025

  9. [17]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the Middle : How Language Models Use Long Contexts , Nov. 2023. URL http://arxiv.org/abs/2307.03172. arXiv:2307.03172 [cs]

    2023 arXiv

  10. [18]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization , Jan. 2019. URL http://arxiv.org/abs/1711.05101. arXiv:1711.05101 [cs, math]

    2019 arXiv

  11. [19]

    X. Luo, M. Ramscar, and B. C. Love. Beyond Human - Like Processing : Large Language Models Perform Equivalently on Forward and Backward Scientific Text , Nov. 2024 a . URL http://arxiv.org/abs/2411.11061. arXiv:2411.11061 [cs]

    2024 arXiv

  12. [20]

    X. Luo, A. Rechardt, G. Sun, K. K. Nejad, F. Yáñez, B. Yilmaz, K. Lee, A. O. Cohen, V. Borghesani, A. Pashkov, D. Marinazzo, J. Nicholas, A. Salatiello, I. Sucholutsky, P. Minervini, S. Razavi, R. Rocca, E. Yusifov, T. Okalova, N. Gu, M. Ferianc, M. Khona, K. R. Patil, P.-S. L...

    2024

  13. [21]

    X. Luo, G. Sun, and B. C. Love. Matching domain experts by training from scratch on domain knowledge, May 2024 c . URL http://arxiv.org/abs/2405.09395. arXiv:2405.09395 [cs, q-bio]

    2024 arXiv

  14. [22]

    Nguyen, N

    A. Nguyen, N. Karampatziakis, and W. Chen. Meet in the Middle : A New Pre -training Paradigm , Mar. 2023. URL http://arxiv.org/abs/2303.07295. arXiv:2303.07295 [cs]

    2023 arXiv

  15. [23]

    Papadopoulos, J

    V. Papadopoulos, J. Wenger, and C. Hongler. Arrows of Time for Large Language Models , July 2024. URL http://arxiv.org/abs/2401.17505. arXiv:2401.17505 [cs]

    2024 arXiv

  16. [24]

    J. Pfau, A. Infanger, A. Sheshadri, A. Panda, J. Michael, and C. Huebner. Eliciting Language Model Behaviors using Reverse Language Models . In Socially Responsible Language Modelling Research , 2023. URL https://openreview.net/forum?id=m6xyTie61H

    2023

  17. [25]

    Z. Qin, W. Sun, H. Deng, D. Li, Y. Wei, B. Lv, J. Yan, L. Kong, and Y. Zhong. cosFormer : Rethinking Softmax in Attention , Feb. 2022. URL http://arxiv.org/abs/2202.08791. arXiv:2202.08791 [cs]

    2022 arXiv

  18. [26]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsupervised Multitask Learners . OpenAI, 2019

    2019

  19. [27]

    Sennrich, B

    R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units , June 2016. URL http://arxiv.org/abs/1508.07909. arXiv:1508.07909 [cs]

    2016 arXiv

  20. [28]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need , Aug. 2023. URL http://arxiv.org/abs/1706.03762. arXiv:1706.03762 [cs]

    2023 arXiv

  21. [29]

    Warner, A

    B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, N. Cooper, G. Adams, J. Howard, and I. Poli. Smarter, Better , Faster , Longer : A Modern Bidirectional Encoder for Fast , Memory Efficient , and Long Cont...

    2024 arXiv

  22. [30]

    X. Wu, Y. Wang, S. Jegelka, and A. Jadbabaie. On the Emergence of Position Bias in Transformers , Feb. 2025. URL http://arxiv.org/abs/2502.01951. arXiv:2502.01951 [cs]

    2025 arXiv

  23. [31]

    G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient Streaming Language Models with Attention Sinks , Apr. 2024. URL http://arxiv.org/abs/2309.17453. arXiv:2309.17453 [cs]

    2024 arXiv

  24. [32]

    Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. XLNet : Generalized Autoregressive Pretraining for Language Understanding , Jan. 2020. URL http://arxiv.org/abs/1906.08237. arXiv:1906.08237 [cs]

    2020 arXiv

  25. [33]

    S. Yu, Y. Xu, C. Du, Y. Zhou, M. Qiu, Q. Sun, H. Zhang, and J. Wu. Reverse Modeling in Large Language Models , Feb. 2025. URL http://arxiv.org/abs/2410.09817. arXiv:2410.09817 [cs]

    2025 arXiv

  26. [34]

    Z. Yu, Z. Wang, Y. Fu, H. Shi, K. Shaikh, and Y. C. Lin. Unveiling and Harnessing Hidden Attention Sinks : Enhancing Large Language Models without Training through Attention Calibration , June 2024. URL http://arxiv.org/abs/2406.15765. arXiv:2406.15765 [cs]

    2024 arXiv

  27. [35]

    Yáñez, X

    F. Yáñez, X. Luo, O. V. Minero, and B. C. Love. Confidence-weighted integration of human and machine judgments for superior decision-making, 2025. URL https://arxiv.org/abs/2408.08083. \_eprint: 2408.08083

    2025

  28. [36]

    Zhang, R

    Y. Zhang, R. Bai, Z. Gu, R. Zhang, J. Gu, E. Abbe, S. Bengio, and N. Jaitly. Reversal Blessing : Thinking Backward May Outpace Thinking Forward in Multi -choice Questions , Mar. 2025. URL http://arxiv.org/abs/2502.18435. arXiv:2502.18435 [cs]

    2025 arXiv

  29. [37]

    Zhang, S

    Z. Zhang, S. Wu, S. Liu, M. Li, M. Zhou, and T. Xu. Regularizing Neural Machine Translation by Target -bidirectional Agreement , Nov. 2018. URL http://arxiv.org/abs/1808.04064. arXiv:1808.04064 [cs]

    2018 arXiv

  30. [38]

    Zucchet, J

    N. Zucchet, J. Bornschein, S. Chan, A. Lampinen, R. Pascanu, and S. De. How do language models learn facts? Dynamics , curricula and hallucinations, Mar. 2025. URL http://arxiv.org/abs/2503.21676. arXiv:2503.21676 [cs]

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.