Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

A Statistical Physics of Language Model Reasoning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sentence-level LLM reasoning can be modeled as a low-rank stochastic dynamical system whose latent regimes switch between distinct reasoning phases.

desk verdict Interesting empirical measurements undermined by an unverified projection-closure assumption and a partly circular misalignment case study, but worth a serious referee. read the letter →

arxiv 2506.04374 v1 pith:X5TLISQW submitted 2025-06-04 cs.AI cs.CL

Jack David Carson , Amir Reisizadeh This is my paper
classification cs.AIcs.CL MSC 60H1062M0562H25
keywords languagemodelreasoningstochasticdifferentialequationsswitchinglineardynamicalsystemslatentregimechain-of-thoughtprincipalcomponentanalysisfailure-modepredictionstatisticalphysicsofcognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the reasoning a transformer language model performs, read at sentence boundaries, is not an unanalyzable high-dimensional process but the trace of a low-dimensional stochastic dynamical system with a small number of switching regimes. It argues that a rank-40 projection of the sentence-level hidden states (capturing roughly half the variance) is a workable domain for fitting a Switching Linear Dynamical System (SLDS) whose drift, noise, and transition structure reproduce observed reasoning trajectories. On held-out data the fitted SLDS predicts the next sentence-level state with $R^2 \approx 0.68$, well above the $R^2 \approx 0.51$ of a single global linear model, and ablated variants confirm that regime switching, low-rank projection, and state-dependent drift each contribute. The paper applies the same machinery to adversarial belief-shift trajectories and reports that it can predict when a model slips into a misaligned, misinformation-affirming state at inference time. If true, reasoning dynamics are compressible and simulable at low cost, opening a path to cheaper red-teaming and to predicting LLM failure modes before they occur.

What carries the argument

The load-bearing object is the Switching Linear Dynamical System (SLDS), a discrete-time surrogate for the continuous-time regime-switching SDE $dh(t)=\mu_{Z(t)}(h(t))dt+B_{Z(t)}(h(t))dW(t)$. The SLDS updates projected hidden states as $h_{t+1}=h_t+V_k(M_{Z_t}(V_k^\top h_t)+b_{Z_t})+\varepsilon_t$, with $\varepsilon_t\sim\mathcal{N}(0,\Sigma_{Z_t})$, where $V_k$ is the rank-40 PCA basis, $M_i,b_i,\Sigma_i$ are regime-specific drift and noise parameters, and $Z_t$ follows a Markov chain with transition matrix $T$. Regimes are initialized by fitting a Gaussian mixture to the residuals of a global linear fit, with $K=4$ chosen by BIC/AIC; parameters are estimated by expectation-maximization with forward-backward inference. The mechanism that carries the argument is projection-plus-switching: the rank-40 subspace makes estimation feasible, and the latent regimes absorb the multimodal residual structure that a single linear model cannot explain.

What would settle it

Estimate the projection leakage $L_k$ empirically: on held-out sentence transitions, compute the norm of the drift component orthogonal to the rank-40 PCA subspace relative to the total drift, e.g. by comparing $\|\mu(h)-(V_k V_k^\top)\mu(h)\|$ with $\|\mu(h)\|$ where $\mu$ is estimated nonparametrically; if this ratio is of order one rather than $\ll 1$ on a substantial fraction of transitions, Assumption 2.1 collapses, and a rank-40 SLDS would be fitting a distorted process.

Watch

Extended reading notes

Core claim

The paper's central claim is that the sentence-level hidden-state trajectory of a transformer LM is well approximated by a continuous-time drift-diffusion process confined, for modeling purposes, to a rank-40 linear subspace, with a small number of latent regimes governing different phases of reasoning. Concretely, projecting final-layer sentence embeddings with PCA to $k=40$ dimensions explains about 50% of the variance; fitting a four-regime Switching Linear Dynamical System (SLDS) to the projected increments gives held-out one-step-ahead prediction $R^2 \approx 0.68$, versus $0.51$ for a single-regime global linear model, and simulated trajectories match empirical jump norms, autocorrelations, and regime occupancy statistics. The paper further claims that the learned transition matrix and regime-specific drifts capture the onset of failure: in an adversarial belief-manipulation setup, the SLDS predicts one-step-ahead projected states with $R^2 \approx 0.72$ and $R^2 \approx 0.69$ for two LLMs and predicts the final binary belief outcome with about 88% and 85% accuracy, outperforming linear and recurrent-network baselines.

Load-bearing premise

The paper assumes, without directly measuring it, that the true reasoning drift stays almost entirely inside the chosen 40-dimensional subspace (the projection leakage $L_k$ is much smaller than 1); if the drift leaks substantially out of that subspace, the fitted SLDS describes a distorted projection of the real process, and the regime and misalignment findings may not transfer to the full system.

Editorial extensions

If this is right

  • A four-regime, rank-40 SLDS is a workable surrogate for sentence-level reasoning dynamics, beating the single-regime global linear model on held-out prediction ($R^2 \approx 0.68$ vs $0.51$).
  • Reasoning trajectories can be simulated at low cost: the fitted SLDS reproduces jump norms, autocorrelations, and regime occupancy statistics, so surrogate rollouts can stand in for expensive model runs.
  • The learned transition matrix and regime drifts can flag an impending slip into a misaligned state; in the adversarial belief-shift study the SLDS predicts the final belief outcome at roughly 88% accuracy for one model and 85% for another.
  • Ablations imply that regime switching, low-rank projection, and state-dependent drift are each necessary; removing state dependence degrades $R^2$ to $0.35$, so the model is not merely classifying residuals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The posterior regime probabilities could be used as an online monitor: a system running an LLM could watch $\gamma_{t,j}$ and intervene when the model enters the regime associated with failure, before the final answer is produced.
  • The same surrogate could accelerate red-teaming: instead of many full forward passes, search over small perturbations of the projected state or of the transition matrix to find trajectories that land in the misaligned regime.
  • If the 40-dimensional drift manifold is shared across models and tasks, it could become a common coordinate system for comparing reasoning styles across architectures; the paper's transfer results suggest same-family transfer works well but cross-family transfer is only partial.
  • A direct test of the paper's interpretation: annotate the sentences in each inferred regime and check whether the four regimes correspond to distinguishable reasoning phases such as planning, computation, conclusion, or error loops; the paper associates textual behaviors with regimes but does not formally evaluate this mapping.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes to model sentence-level transformer hidden-state trajectories as realizations of a continuous-time switching stochastic differential equation, approximated in practice by a discrete-time switching linear dynamical system (SLDS) on a rank-40 PCA subspace. After reporting that a rank-40 projection explains roughly 50% of variance and that residual distributions are multimodal, the authors fit K=4 regimes by a Gaussian mixture model and estimate SLDS parameters by EM. Validation includes one-step-ahead R2 comparisons, transfer across 8 models and 7 benchmarks, ablations, and a case study on adversarial belief shifts with K=3 regimes. The paper concludes that the SLDS is an efficient surrogate that enables low-cost reasoning simulation and prediction of misaligned states or other failure modes.

Significance. If the conclusions were fully supported, a rank-40, four-regime SLDS would be an important, inexpensive surrogate for sentence-level LLM reasoning dynamics, and the transfer and failure-mode results would be of broad interest. The paper's strengths include the breadth of the empirical corpus, a clearly stated formal framework, a detailed EM derivation, a three-way ablation, and a safety-conscious decision to release only aggregate statistics. However, the current evidence supports a weaker statement: the SLDS is a predictive regression on projected coordinates. The physical interpretation as 'reasoning dynamics' and the failure-mode predictions require additional validation, specifically a direct test of Assumption 2.1, non-circular out-of-sample evaluation in Section 5.3, uncertainty quantification, and correction of the EM formula in Appendix B.

major comments (5)
  1. [Assumption 2.1 / Appendix A] The approximate projection closure assumption Lk << 1 is never tested. Appendix A's justification conflates the variance fraction with the sup-norm ratio rho_k: the statement that the first 40 PCs capture about 50% of drift variance gives a root-mean-square residual fraction of about 0.71, not a supremum ratio of 0.50, and the supremum over the empirical support can be larger still. The additive term L_mu * epsilon / mu_min is also never estimated, and for deep-network hidden states the Lipschitz constant is unknown. Please measure the empirical leakage ratio directly (for example, the high quantile of ||(I - V_k V_k^T) mu(h)|| / ||mu(h)|| over the trajectory data), report how it varies with k, and test whether downstream regime assignments are stable when the drift is not closed. Without this, the four regimes and the belief-shift results describe the projected process but cannot be claimed to describe the original reasoning dynamics.
  2. [Section 5.3 / Appendix C.5] The adversarial 'prediction' of belief shifts is partly circular. Appendix C.5 states that poisoned simulations use 'small fixed displacements estimated from empirical poisoned data' at preselected intervals, and the MLP mapping PCA states to belief scores is trained on the same empirical data that is then predicted. The final-belief accuracies in Table 3 therefore measure a fitted reconstruction rather than an independent prediction of failure. Please re-run the evaluation with perturbations estimated only from clean data, or train on the first part of each trajectory and test on later steps, or hold out entire misinformation themes; report the drop in accuracy under these non-circular protocols.
  3. [Section 5 / Tables 1 and 2] The central quantitative claims lack uncertainty estimates. The text reports held-out R2 approximately 0.68 in Section 5, but Table 2 reports full SLDS R2 = 0.74 and Table 1 reports same-model R2 = 0.73; no standard errors, confidence intervals, or number of random seeds are given, and it is unclear whether the same held-out split is used. Also, Section 3.1 filters out transitions with ||Delta h_t|| <= 10 before all fitting, but the paper never states what fraction of transitions is removed or whether R2 and the regime structure are robust to this threshold. Please add error bars or bootstrap intervals and a sensitivity analysis over the filter threshold and projection rank k.
  4. [Appendix B] The E-step formula for xi_t(i,j) is mathematically incorrect as written: the emission density N(Delta h'_t | M_j x_t + b_j, Sigma_j) is added to the normalized forward-backward product rather than multiplied into the numerator alpha_t(i) T_ij beta_{t+1}(j). In the displayed equation, the first term is a dimensionless probability and the second term is a density, so the right-hand side is not a probability and can exceed one. Please correct the formula and state whether the implementation follows the corrected expression; if it follows the displayed one, the fitted parameters are not EM estimates.
  5. [Section 4.2 / Regime interpretation] The paper assigns behavioral labels such as 'systematic decomposition' and 'failure loops' to the four GMM clusters, but no external validation is provided that these clusters correspond to distinct reasoning phases. The regime posterior probabilities may be useful for prediction, but the claim that regimes represent semantically meaningful reasoning states (or misalignment) requires, for example, showing that regime occupancy predicts task success or error, or that sentences with high posterior probability in a regime have distinct interpretable content. As written, 'reasoning regimes' are just fitted components of residual statistics.
minor comments (5)
  1. [Section 5.3 vs Table 3] The text states that baseline belief accuracies ranged from 0.62 to 0.78 and that GRU networks achieved R2 approximately 0.57 to 0.58, but Table 3 reports linear belief accuracies of 0.55 and 0.52, GRU accuracies of 0.68 and 0.65, and GRU R2 values of 0.48 and 0.46; please harmonize these numbers.
  2. [Figure 5 caption] The caption contains the typo 'posioned' and should read 'poisoned'.
  3. [Section 5.3] The same projection rank k=40 is reported to capture about 87% of variance in the belief dataset but roughly 50% in the main corpus; please explain why this discrepancy is expected or state whether a separate rank selection was performed for the case study.
  4. [Section 3.1] The filtering criterion '||Delta h_t|| <= 10 in normalized units' is not defined in the main text; please specify what normalization was applied to the hidden states before this threshold was applied.
  5. [Equations 8 and Appendix B] The SLDS indexing is inconsistent: in Eq. (8) the noise at step t is indexed by Z_t, while the EM appendix uses epsilon_{t+1} ~ N(0, Sigma_{Z_{t+1}}); please align the notation.

Circularity Check

2 steps flagged · score 6.0 of 10

Adversarial failure-mode 'predictions' are partly fitted inputs: poison perturbations are estimated from empirical poisoned data and injected at preselected intervals, and the belief-score readout is not shown to be out-of-sample; the main held-out SLDS R² comparison is not circular.

  1. fitted input called prediction [Appendix C.5 (Simulation Protocol and Validation), supporting Section 5.3 and Figure 5]
    "Poisoned simulations introduced adversarial perturbations (small fixed displacements estimated from empirical poisoned data) at random preselected intervals. Simulated trajectories matched empirical ones closely in timing/magnitude of belief shifts (slips into misaligned states), variance, and distributional characteristics (Kolmogorov-Smirnov test p >0.3 for final belief scores)."

    The claimed ability to reproduce and 'predict' belief shifts is not an out-of-sample prediction: the displacements that cause the simulated shift are estimated from the empirical poisoned trajectories, and the times at which they act are supplied by the experiment ('random preselected intervals'). Matching timing and magnitude under those injected inputs is a consistency check of the fitted simulator, not evidence that the SLDS predicts critical transitions. The Section 5.3 claim that simulated belief trajectories match empirical ones in timing and magnitude therefore holds largely by construction.

  2. fitted input called prediction [Appendix C.4 (Belief-Score Prediction) and Table 3]
    "Since SLDS models latent PCA dynamics, a small two-layer MLP regressor (32 ReLU units/layer, Adam, early stopping) mapped PCA-projected states to belief scores for validation and for assessing the prediction of the misaligned (high belief score) state."

    The final-belief accuracies in Table 3 (0.88 for Llama-2-70B, 0.85 for Gemma-7B) are produced by an MLP readout that is fit to belief scores, and the paper does not describe a separate held-out split for that readout. A readout fit on the scored trajectories can memorize the PCA-state-to-belief-score mapping, so 'Belief Acc.' partly measures the fit of the readout rather than a prediction made by the SLDS dynamics from clean inputs; at minimum, the attribution of these numbers to the SLDS is unsupported.

full rationale

The central SLDS evaluation (Section 5, R2 ≈ 0.68 vs 0.51) is a legitimate parameter-fit/held-out comparison: EM-fitted regime parameters are applied to held-out trajectories, and the comparison to a global linear model is not forced by construction. The GMM-based K=4 choice is descriptive rather than circular. However, the paper's flagship failure-mode 'prediction' (Section 5.3, Appendix C) is partially circular. Appendix C.5 injects adversarial perturbations estimated from empirical poisoned data at preselected intervals, so the simulated timing and magnitude of belief shifts are inputs rather than predictions, making Figure 5's and Section 5.3's 'captures timing' claims hold largely by construction. Appendix C.4 fits the MLP belief-score readout without describing a separate test split, so Table 3's Belief Acc. cannot be cleanly attributed to the SLDS dynamics. Separately, Assumption 2.1 (Lk ≪ 1) is load-bearing but never measured; Appendix A substitutes the variance-fraction ρ40 ≈ 0.50 for the sup-norm leakage ratio ρk and never estimates Lμ ε / μmin, which is a correctness risk for the low-rank closure, though not itself a circular step.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. The latent regimes are empirically derived clusters in residual space, not postulated objects, so the invented-entity ledger is empty. The main burden is carried by unverified modeling assumptions, primarily the projection-closure assumption, and by several hand-chosen or fitted constants.

free parameters (6)
  • Projection rank k = 40
    Chosen as a variance-versus-feasibility trade-off in Appendix A; no empirical verification that the Assumption 2.1 leakage bound is small.
  • Number of regimes K (main experiments) = 4
    Selected by BIC/AIC on the same data used to fit the model; regime labels are interpretive.
  • Number of regimes K (belief case study) = 3
    Chosen by BIC on validation data, motivated by the bimodal belief score distribution.
  • Threshold for filtering transitions = 10 (normalized units)
    Transitions with norm at or below 10 are excluded in Section 3.1; sensitivity not reported.
  • Ridge regularization lambda = 1.0
    Set for the global linear baseline in Equation 6; no grid search or sensitivity analysis is shown.
  • Adversarial perturbation displacement = Small fixed displacements estimated from empirical poisoned data
    Used in Appendix C.5 to simulate poisoned trajectories; because these are estimated from the same data, the simulated belief shift is partly a fit.
assumptions (4)
  • ad hoc to paper Assumption 2.1: projection leakage Lk is much less than 1 for k=40
    The paper bounds Lk by rho_k + L_mu epsilon / mu_min in Appendix A but never estimates Lk, L_mu, or mu_min. The validity of the low-dimensional drift approximation is asserted, not demonstrated.
  • domain assumption Sentence-stride embeddings are observations of an underlying continuous-time process with latent Markov regime switching
    Section 2 defines the sentence-stride process and the regime-switching SDE; continuity of semantic trajectories and the Markov property of regime transitions are assumed, not established.
  • ad hoc to paper The GMM clusters correspond to distinct reasoning phases with behavioral meaning
    Section 4.2 labels regimes as systematic decomposition, answer synthesis, exploratory variance, or failure loops, but no evidence links clusters to observed text outputs.
  • domain assumption The double-well Langevin analogy transfers to high-dimensional LLM hidden states
    Appendix E interprets multimodal residuals as metastable basins, but no barrier heights, escape rates, or Fokker-Planck fits are measured for the LLM data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Statistical Physics of Language Model Reasoning." pith.science (2026). https://pith.science/paper/X5TLISQW

@misc{pith2026250604374,
  author       = {Pith},
  title        = {Pith review of: A Statistical Physics of Language Model Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X5TLISQW}},
  note         = {Machine review of arXiv:2506.04374}
}
read the original abstract

Transformer LMs show emergent reasoning that resists mechanistic understanding. We offer a statistical physics framework for continuous-time chain-of-thought reasoning dynamics. We model sentence-level hidden state trajectories as a stochastic dynamical system on a lower-dimensional manifold. This drift-diffusion system uses latent regime switching to capture diverse reasoning phases, including misaligned states or failures. Empirical trajectories (8 models, 7 benchmarks) show a rank-40 projection (balancing variance capture and feasibility) explains ~50% variance. We find four latent reasoning regimes. An SLDS model is formulated and validated to capture these features. The framework enables low-cost reasoning simulation, offering tools to study and predict critical transitions like misaligned states or other LM failures.

Figures

Figures reproduced from arXiv: 2506.04374 by the authors.

Figure 2
Figure 2. Latent regimes (K = 4) uncovered by GMM fitting on low-rank residuals ζt. (a) Residuals projected onto their first two principal components, colored by GMM assignment, showing distinct clusters. (b) Histogram of residual norms ∥ζt∥, colored by GMM regime assignment, further illustrating regime separation. These regimes may capture different reasoning qualities, including potential misalignments [PITH_FULL_IMAGE:fig… view at source ↗
Figure 1
Figure 1. (a) CDF comparison of token and sentence jump norms, illustrating that sentence-level increments capture more substantial semantic shifts. (b) Histograms of residual norms from a global linear fit, showing raw residuals ∥ξt∥ (left) and residuals projected onto a low-rank PCA space ∥ζt∥ (right). Both reveal significant multimodality, motivating regime switching to capture distinct reasoning phases or potential misali… view at source ↗
Figure 4
Figure 4. Failure of single-mode noise models for the full residuals ξt (before projection). This plot shows mismatches between the empirical distribution of residual norms and fits from both Gaus￾sian and Laplace distributions, highlighting the inadequacy of a single noise process and further motivating the regime-switching approach to capture diverse reasoning states, including potential misalignments. {1, . . . , K} be the… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: SLDS model validation via adversarial belief manipula￾tion. Each row shows a distinct topic. Empirical belief trajectories where blue and red follow the clean and posioned belief trajecto￾ries, respectively (left). SLDS simulations where green and orange follow the pro…
Figure 6
Figure 6. Figure 6: shows the distribution of residuals ∆ht projected onto each of these 40 principal component dimensions, re￾vealing rich multimodal structures that motivate the regime￾switching approach. These regimes can be interpreted as dif￾ferent reasoning pathways or potential "mi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysis

    cs.CL 2026-05 conditional novelty 7.0 of 10

    Apparent phase transitions during fine-tuning on near-synonym tasks are phantoms originating in the softmax readout; an order parameter isolates kinematic and structural failure modes and a few dimensionless quantitie...

  2. The wall confronting large language models

    cs.AI 2025-07 conditional novelty 4.0 of 10

    LLM scaling exponents near 0.1 imply that reducing loss tenfold would need 10^10 more compute, making scientific-grade reliability unreachable by brute-force scaling.

Reference graph

Works this paper leans on

39 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

    Abdin et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone . arXiv preprint arXiv:2404.14219, Apr 2024. URL https://arxiv.org/abs/2404.14219

    arXiv 2024

  3. [3]

    Physics of language models: Part 1, learning hierarchical language structures

    Allen-Zhu et al. Physics of language models: Part 1, learning hierarchical language structures. arXiv preprint arXiv:2305.13673, 2023

    arXiv 2023

  4. [4]

    Qwen technical report

    Bai et al. Qwen technical report. arXiv preprint arXiv:2309.16609, Sep 2023. URL https://arxiv.org/abs/2309.16609

    arXiv 2023

  5. [5]

    PIQA : Reasoning about physical commonsense in natural language

    Bisk et al. PIQA : Reasoning about physical commonsense in natural language. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020 , pp.\ 7432--7439. AAAI Press, Feb 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/6241. arXiv:1911.11641

    arXiv 2020

  6. [6]

    Language models are few-shot learners

    Brown et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, pp.\ 1877--1901, 2020

    work page 1901

  7. [7]

    Computational principles of memory

    Chaudhuri et al. Computational principles of memory. Nature Neuroscience, 19 0 (3): 0 394--403, 2016. doi:10.1038/nn.4237

    doi:10.1038/nn.4237 2016

  8. [8]

    B ool Q : Exploring the surprising difficulty of natural yes/no questions

    Clark et al. B ool Q : Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp.\ 2924--2936, Minneapolis, Minnesota, June 2019. Association for Computational Ling...

    doi:10.18653/v1/n19-1090 2019

Show all 39 references
  1. [9]

    Training verifiers to solve math word problems

    Cobbe et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, Oct 2021. URL https://arxiv.org/abs/2110.14168

    2021 arXiv

  2. [10]

    The rotation of eigenvectors by a perturbation

    Davis et al. The rotation of eigenvectors by a perturbation. III . SIAM Journal on Numerical Analysis, 7 0 (1): 0 1--46, 1970. doi:10.1137/0707001

    1970 doi

  3. [11]

    DeepSeek LLM : Scaling open-source language models with longtermism

    DeepSeek-AI et al. DeepSeek LLM : Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, Jan 2024. URL https://arxiv.org/abs/2401.02954

    2024 arXiv

  4. [12]

    Maximum likelihood from incomplete data via the EM algorithm

    Dempster et al. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39 0 (1): 0 1--38, 1977. doi:10.1111/j.2517-6161.1977.tb01600.x

    1977

  5. [13]

    A mathematical framework for transformer circuits

    Elhage et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021

    2021

  6. [14]

    Gemma: Open models based on gemini research and technology

    Gemma Team et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, Mar 2024. URL https://arxiv.org/abs/2403.08295

    2024 arXiv

  7. [15]

    Did A ristotle use a laptop? A question answering benchmark with implicit reasoning strategies

    Geva et al. Did A ristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics (TACL), 9: 0 346--361, 2021. doi:10.1162/tacl_a_00370. URL https://aclanthology.org/2021.tacl-1.21

    2021 doi

  8. [16]

    Variational learning for switching state-space models

    Ghahramani et al. Variational learning for switching state-space models. Neural Computation, 12 0 (4): 0 831--864, 2000. doi:10.1162/089976600300015619

    2000 doi

  9. [17]

    Note on the derivatives with respect to a parameter of the solutions of a system of differential equations

    Gr \"o nwall. Note on the derivatives with respect to a parameter of the solutions of a system of differential equations. Annals of Mathematics, 20 0 (4): 0 292--296, 1919. doi:10.2307/1967124

    1919 doi

  10. [18]

    A new approach to the economic analysis of nonstationary time series and the business cycle

    Hamilton. A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica, 57 0 (2): 0 357--384, 1989

    1989

  11. [19]

    Ridge regression: Biased estimation for nonorthogonal problems

    Hoerl et al. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12 0 (1): 0 55--67, 1970. doi:10.1080/00401706.1970.10488634

    1970

  12. [20]

    Mistral 7b

    Jiang et al. Mistral 7b. arXiv preprint arXiv:2310.06825, Oct 2023. URL https://arxiv.org/abs/2310.06825

    2023 arXiv

  13. [21]

    Principal Component Analysis

    Jolliffe. Principal Component Analysis. Springer Series in Statistics. Springer-Verlag, New York, second edition, 2002. ISBN 0-387-95442-2. doi:10.1007/b98835

    2002 doi

  14. [22]

    Emergent world representations: Exploring a sequence model trained on a synthetic task

    Li et al. Emergent world representations: Exploring a sequence model trained on a synthetic task. In Proceedings of the International Conference on Learning Representations (ICLR), 2023

    2023

  15. [23]

    T ruthful QA : Measuring how models mimic human falsehoods

    Lin et al. T ruthful QA : Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3214--3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi...

    2022 doi

  16. [24]

    Linguistic interpretability of transformer-based language models: A systematic review

    L \'o pez-Otal et al. Linguistic interpretability of transformer-based language models: A systematic review. arXiv preprint arXiv:2404.08001, 2024

    2024 arXiv

  17. [25]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Mihaylov et al. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 2381--2391, Brussels, Belgium, October-November 2018. Association fo...

    2018 doi

  18. [26]

    Emergent linear representations in world models of self-supervised sequence models

    Nanda et al. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023

    2023 arXiv

  19. [27]

    Stochastic Differential Equations: An Introduction with Applications

    ksendal. Stochastic Differential Equations: An Introduction with Applications. Springer Science & Business Media, sixth edition, 2003. ISBN 978-3540047582

    2003

  20. [28]

    In-context learning and induction heads

    Olsson et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  21. [29]

    A tutorial on hidden markov models and selected applications in speech recognition

    Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77 0 (2): 0 257--286, 1989

    1989

  22. [30]

    Language models are unsupervised multitask learners

    Radford et al. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019

    2019

  23. [31]

    The Fokker-Planck Equation: Methods of Solution and Applications , volume 18 of Springer Series in Synergetics

    Risken et al. The Fokker-Planck Equation: Methods of Solution and Applications , volume 18 of Springer Series in Synergetics. Springer, Berlin, Heidelberg, 2nd ed. 1989, corrected 2nd printing edition, 1996. ISBN 978-3-540-61530-9. doi:10.1007/978-3-642-61530-9

    1989 doi

  24. [32]

    Optimal sequence memory in driven random networks

    Schuecker et al. Optimal sequence memory in driven random networks. Physical Review X, 8 0 (4): 0 041029, 2018. doi:10.1103/PhysRevX.8.041029

    2018 doi

  25. [33]

    C ommonsense QA : A question answering challenge targeting commonsense knowledge

    Talmor et al. C ommonsense QA : A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) ...

    2019 doi

  26. [34]

    C ommonsense QA 2.0: Exposing the limits of AI through gamification

    Talmor et al. C ommonsense QA 2.0: Exposing the limits of AI through gamification. In Scholkopf et al. (eds.), Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS 2021), December 2021. URL https://datasets-benchmarks-proceedi...

    2021 arXiv

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, Jul 2023. URL https://arxiv.org/abs/2307.09288

    2023 arXiv

  28. [36]

    Attention is all you need

    Vaswani et al. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp.\ 5998--6008, 2017

    2017

  29. [37]

    Towards understanding chain-of-thought prompting: An empirical study of what matters

    Wang et al. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2023

    2023 arXiv

  30. [38]

    Chain-of-thought prompting elicits reasoning in large language models

    Wei et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  31. [39]

    Zellers et al. H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp.\ 4799--4809, Florence, Italy, July 2019. Association for Computational Linguistics. doi:10.18653/v1/P19...

    2019 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.