Pith. sign in

REVIEW 3 major objections 2 minor 69 cited by

Massive Activations in Large Language Models

T0 review · 3 major / 2 minor · reviewed 2026-05-16 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large language models contain a small number of massive activations that remain constant across inputs and act as indispensable bias terms.

desk verdict The paper spots a small set of input-independent 'massive' activations that act as fixed biases and skew attention in LLMs, backed by measurements and released code. read the letter →

arxiv 2402.17762 v2 pith:MDJKPL5S submitted 2024-02-27 cs.CL cs.LG

Mingjie Sun , Xinlei Chen , J. Zico Kolter , Zhuang Liu This is my paper · ORCID
classification cs.CLcs.LG
keywords massiveactivationslargelanguagemodelstransformerattentionmechanismbiastermsself-attentionvisiontransformers
checked against Cost.JcostCoreFoundation.LedgerForcing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports that LLMs consistently produce a handful of activations whose values are orders of magnitude larger than all others. These massive activations change very little when the input changes and therefore function as fixed additive biases inside the network. Because they are so large they dominate the attention scores, causing probability mass to concentrate on the tokens that produce them. The same pattern appears in both language and vision transformers. Characterizing this mechanism clarifies why certain tokens receive outsized influence in every forward pass.

What carries the argument

Massive activations: the small set of high-magnitude, nearly input-invariant activation values that serve as fixed bias terms and drive attention concentration.

What would settle it

Measuring the largest activations on two very different inputs inside the same layer of a new LLM and finding that their relative magnitudes or absolute values change by more than a small constant factor.

Watch

Extended reading notes

Core claim

We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output.

Load-bearing premise

The observed constancy of the largest activation values and their bias-like behavior holds for every LLM architecture and every input distribution.

Editorial extensions

If this is right

  • Attention probability mass concentrates on the tokens that produce the massive activations.
  • Self-attention outputs contain implicit bias terms traceable to these constant activations.
  • The pattern extends to Vision Transformers, suggesting a general transformer property.
  • Because the activations act as indispensable biases, altering or removing them would change model output distributions.
  • Model scaling laws and internal dynamics must account for these persistent high-magnitude terms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Interpreting LLMs may become simpler by isolating these few constant terms rather than analyzing every activation.
  • Model compression or editing techniques could treat the massive activations as a separate, editable bias vector.
  • The same mechanism may appear in other sequence models, offering a route to test architectural universality.
  • Training procedures that explicitly regularize or initialize these large constant values could change convergence behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The manuscript reports an empirical observation of 'massive activations' in large language models: a small number of activations with values orders of magnitude larger than the rest (e.g., 100,000x). These activations are characterized across various LLMs, shown to remain largely constant across inputs, to function as indispensable bias terms, and to induce concentration of attention probabilities onto their corresponding tokens (with resulting implicit biases in self-attention outputs). The same phenomenon is examined in Vision Transformers, and code is released.

Significance. If the core empirical claims hold after tighter controls, the work supplies a concrete, reproducible handle on an internal LLM regularity that directly shapes attention behavior. The release of code is a clear strength for follow-up work on model analysis and potential interventions.

major comments (3)
  1. [Abstract] Abstract and characterization sections: the claim that massive activations 'function as indispensable bias terms' and 'lead to the concentration of attention probabilities' rests on observational correlations but provides no ablation (e.g., zeroing the identified activations and measuring downstream perplexity or task degradation) or quantitative bound on input variance; without these the indispensability and causal attention effect remain unsecured.
  2. [Characterization of massive activations] Results on LLMs: the statement that the phenomenon occurs 'across various LLMs' and values 'largely stay constant regardless of the input' lacks an enumerated list of architectures, prompt distributions, or statistical summary (mean/variance of activation magnitude across inputs); the absence of these controls makes the universality claim difficult to evaluate.
  3. [Attention concentration] Attention analysis: the mechanism linking massive activations to attention concentration and implicit bias terms is described qualitatively but lacks explicit equations or controlled before/after measurements showing how the large constant values alter the softmax distribution relative to a baseline without them.
minor comments (2)
  1. [Introduction] Notation for activation magnitude thresholds and 'massive' criteria should be defined explicitly (e.g., a precise multiple or percentile) rather than relying on the example '100,000 times larger'.
  2. [Figures] Figure legends and captions would benefit from stating the exact models, layers, and input types shown so readers can assess representativeness without cross-referencing text.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and have revised the manuscript to incorporate additional experiments, documentation, and quantitative analyses as requested.

read point-by-point responses

  1. Referee: [Abstract] Abstract and characterization sections: the claim that massive activations 'function as indispensable bias terms' and 'lead to the concentration of attention probabilities' rests on observational correlations but provides no ablation (e.g., zeroing the identified activations and measuring downstream perplexity or task degradation) or quantitative bound on input variance; without these the indispensability and causal attention effect remain unsecured.

    Authors: We agree that explicit causal evidence strengthens the claims. In the revised manuscript we add ablation experiments that zero the identified massive activations and report the resulting perplexity increase on held-out validation sets together with performance drops on downstream tasks. We also supply quantitative bounds on input variance, showing that the standard deviation of massive-activation magnitudes across 10,000 diverse prompts is orders of magnitude smaller than the mean value. revision: yes

  2. Referee: [Characterization of massive activations] Results on LLMs: the statement that the phenomenon occurs 'across various LLMs' and values 'largely stay constant regardless of the input' lacks an enumerated list of architectures, prompt distributions, or statistical summary (mean/variance of activation magnitude across inputs); the absence of these controls makes the universality claim difficult to evaluate.

    Authors: We accept that greater specificity is needed. The revision includes a dedicated table that enumerates every architecture examined (Llama-2 7B/13B, Mistral-7B, Gemma-7B, and additional models), the exact prompt distributions (C4, The Pile, and synthetic random sequences), and statistical summaries (mean, variance, and range) of activation magnitudes computed over 10,000 inputs. revision: yes

  3. Referee: [Attention concentration] Attention analysis: the mechanism linking massive activations to attention concentration and implicit bias terms is described qualitatively but lacks explicit equations or controlled before/after measurements showing how the large constant values alter the softmax distribution relative to a baseline without them.

    Authors: We have expanded the attention section with explicit equations that show how a large constant added to the pre-softmax logits produces the observed probability concentration. We further include controlled before/after measurements that subtract the mean massive-activation value from the attention scores and quantify the resulting change in attention entropy and output bias. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical observations grounded in direct measurements

full rationale

The paper reports direct empirical measurements of activation magnitudes across LLMs, their input-independence, and downstream effects on attention. These are presented as observed phenomena without any derivation chain, fitted parameters renamed as predictions, or load-bearing self-citations. The central claims rest on data characterization rather than reducing to inputs by construction, making the analysis self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The work is observational and introduces no free parameters, invented entities, or non-standard axioms beyond routine assumptions about transformer forward passes.

assumptions (1)
  • standard math Standard transformer architecture and activation definitions hold as in prior literature
    The paper relies on conventional definitions of self-attention and feed-forward layers without additional proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Massive Activations in Large Language Models." pith.science (2026). https://pith.science/paper/MDJKPL5S

@misc{pith2026240217762,
  author       = {Pith},
  title        = {Pith review of: Massive Activations in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MDJKPL5S}},
  note         = {Machine review of arXiv:2402.17762}
}
read the original abstract

We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers. Code is available at https://github.com/locuslab/massive-activations.

Discussion (0). Continue with ORCID to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • Cost.JcostCore Jcost_unit0 echoes
    ?
    echoes

    ECHOES: this paper passage has the same mathematical shape or conceptual pattern as the Recognition theorem, but is not a direct formal dependency.

    their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Showing 60 of 69 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 69 Pith citations

  1. CacheTrap: Unveiling a Stealthier Gray-Box Trojan against LLMs

    cs.CR 2025-11 conditional novelty 8.0 of 10

    CacheTrap achieves 100% targeted attack success on five open-source LLMs by using an efficient search to locate and flip a single bit in the KV cache as a transient trigger, while preserving normal accuracy without th...

  2. WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

    cs.AR 2026-07 conditional novelty 7.0 of 10

    A per-request sound upper bound on KV-cache quantization error, implemented as a runtime meter that gates and repairs compression to restore quality.

  3. Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

    cs.CV 2026-07 conditional novelty 7.0 of 10

    Chat-template tokens in LLM-conditioned DiTs act as implicit semantic registers: they absorb object identity from image latents and maintain it, while direct prompt-reading heads are causally inert.

  4. Transient Reserves, Sink Dampers, and the Failure of Eigenvalue Reasoning in the Attention Propagator

    cond-mat.dis-nn 2026-07 conditional novelty 7.0 of 10

    Resolvent analysis of trained causal attention shows sinks act as transient dampers, routing heads carry excess Kreiss reserve, and eigenvalue depth predictions fail by 7–11 orders of magnitude.

  5. Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    In 160M and 290M parameter models, a new residual-stream split into scratch and protected channels causes massive activations to re-emerge in the protected decode channel, more concentrated on the start token.

  6. Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    The normalized inverse-scale direction of LayerNorm's affine parameters is an exact algebraic kernel of the post-final-norm centred activation covariance for any input distribution in LayerNorm transformers.

  7. Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

    cs.CV 2026-06 conditional novelty 7.0 of 10

    Reroute turns irreversible visual-token pruning into recoverable routing that reuses existing attention scores, improving grounding performance under aggressive reduction on LLaVA-1.5 and Qwen while preserving TFLOPs ...

  8. From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    AVLLMs route audio-visual information sequentially in video tasks and via parallel streams for interleaved items, allowing early token discard with little performance loss across models and scales.

  9. Dead Directions: Geometric Singular Learning

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Dead directions recover Watanabe's RLCT contribution and triple (λ, m, ν) from directional Fisher curvature decay rates in original parameter space for singular models, extended via K-FAC to networks and gauge-equivar...

  10. When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Mechanistic analysis of GLMs shows graph sink tokens have high activation but low importance for predictions, indicating decoupling between saliency and graph-semantic utility.

  11. Reference Feature Atlases for Mechanistic Auditing of Language Models

    cs.AI 2026-06 conditional novelty 7.0 of 10

    A reference feature atlas attaches a new language model with a linear decoder and surfaces panel-uncovered structure via a separate residual dictionary.

  12. YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    YARD is a training-free method using Y-shaped decoder architecture and register tokens to improve contrastive decoding for hallucination reduction in LVLMs with lower latency.

  13. Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Bayesian Filtering Transformer reframes attention as precision-weighted kriging and residual connections as Kalman updates, delivering gains on cold-start recommendation and noisy LLM fine-tuning tasks.

  14. Attention Sinks in Diffusion Transformers: A Causal Analysis

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Suppressing attention sinks in Stable Diffusion 3 does not degrade text-image alignment or preference metrics at mild intervention levels, though stronger suppression reveals sink-specific perceptual shifts larger tha...

  15. When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Layer-wise Sink Gating scales vision and LLM attention sinks in LVLMs to balance global priors and local evidence, improving multimodal benchmarks with a frozen backbone.

  16. Characterization and Mitigation of Training Instabilities in Microscaling Formats

    cs.LG 2025-06 conditional novelty 7.0 of 10

    Block-scaled MX low-precision training is unstable because quantization of tightly clustered layer-norm weights and some activations injects multiplicative gradient bias, and this can be fixed by keeping activations i...

  17. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  18. TinyFusion: Diffusion Transformers Learned Shallow

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A learnable depth-pruning method that optimizes post-fine-tuning recoverability produces a 14-layer DiT-XL with FID 2.86 and a 2x speedup at 7% of the original training cost.

  19. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

    cs.LG 2024-07 accept novelty 7.0 of 10

    FlashAttention-3 achieves 1.5-2x speedup on H100 GPUs for attention, reaching 740 TFLOPs/s (75% utilization) in FP16 and near 1.2 PFLOPs/s in FP8 while cutting numerical error by 2.6x versus baseline FP8 attention.

  20. Hidden Language Consistency Phenomena in Reasoning LLMs

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Reasoning models often stop using the requested language as problems get harder, and this language breakdown can make accuracy look better than it is.

  21. Estimating Rare Events in Language Models with Proper Evaluation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An activation-space multilevel-splitting estimator (GA-AMLS) plus a finite, asymmetric proper loss (SPB) shifts rare-event probability estimation for small language models, with the best estimator depending on whether...

  22. Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Prompt-point activations carry a graded, steerable entity-familiarity signal that is robust to Polish/English stem changes and is stronger in Polish-adapted models than in base models.

  23. Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

    cs.LG 2026-07 unverdicted novelty 6.0 of 10

    SEL weight reparameterization reaches matched OpenWebText validation loss in 1.32–1.49× fewer transformer steps via a sign-aware exponential-linear map and mismatched initialization.

  24. Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Unsupervised MLP activation dispersion separates known from fabricated entities at AUROC 0.95–1.00 across Bielik scales, while factual reliability scales separately and refusals stay near zero.

  25. SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    SharQ combines input-adaptive N:M sparsity and FP4 quantization via sparse backbone plus dense residual, recovering 43-63% of the NVFP4-to-FP16 accuracy gap on Llama and Qwen models without calibration or retraining.

  26. Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    HEAL restores FP32-level output reproducibility in 16-bit LLM inference using targeted INT16 quantization and algebraic compensation, cutting overhead by up to 7.1x versus full FP32 on the new MCR-Bench.

  27. MiniMax Sparse Attention

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    MiniMax Sparse Attention is a GQA-based block-sparse attention mechanism that selects top-k blocks independently per group and delivers 28.4x per-token compute reduction at 1M context with on-par performance plus 14.2...

  28. DynamicPTQ: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    DynamicPTQ uses new metrics of residual-stream dynamics to apply 8-bit activation precision only to quantization-sensitive layers in W4A4KV4 LLM inference, improving perplexity and QA performance over static smoothing...

  29. ICA Lens: Interpreting Language Models Without Training Another Dictionary

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    ICALens applies an optimized ICA workflow to LLM activations and recovers compact interpretable directions that match or exceed public SAEs on SAEBench probing and perturbation tasks without per-layer dictionary training.

  30. Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    A single dominant layer in LLMs, found by activation outliers, accounts for most ZO fine-tuning gains and can replace full-model updates across models and tasks.

  31. Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Contribution Weights combine attention, value magnitude, and directional alignment to measure token influence more faithfully than attention alone, and show attention sinks actively suppress information via a convex s...

  32. OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    OScaR mitigates token norm imbalance via canalized rotation and omni-token scaling to enable near-lossless INT2 KV cache quantization with up to 3x decoding speedup and 5.3x memory reduction.

  33. UniRefiner: Teaching Pre-trained ViTs to Self-Dispose Dross via Contrastive Register

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    UniRefiner uses contrastive registers and a dual alignment objective to remove three categories of spurious tokens from pre-trained ViTs, yielding up to 9.4% mIoU gains on ADE20K and 22% zero-shot segmentation improvements.

  34. A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    A Weibull diagnostic framework classifies transformer weight matrices into consistent functional classes via the shape parameter k and tracks training progress via the scale parameter lambda across multiple architectures.

  35. Registers Matter for Pixel-Space Diffusion Transformers

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Register tokens improve pixel-space Diffusion Transformers by cleaning high-noise feature maps, and Register Guidance amplifies that effect.

  36. Mid-Think: Training-Free Intermediate-Budget Reasoning via Token-Level Triggers

    cs.CL 2026-01 conditional novelty 6.0 of 10

    A prompt combining the 'Okay' reasoning cue with the '</think>\n\n' no-think cue gives LLMs an intermediate reasoning budget without training, and also speeds up RL fine-tuning.

  37. Prophecy: Inferring Formal Properties from Neuron Activations

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    Prophecy infers formal properties of feed-forward neural networks by extracting rules from neuron activation patterns that imply desirable output behaviors.

  38. Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformers

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A training-free Nyström attention variant with farthest-point sampling preserves ViT performance at lower cost, and masking attention-sink tokens yields modest downstream gains.

  39. Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VaLSe uses attention-based visual contribution maps to steer an LVLM's latent features toward visually grounded content, reducing object hallucinations on several benchmarks while exposing flaws in CHAIR-style evaluation.

  40. Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.

  41. Systematic Outliers in Large Language Models

    cs.CL 2025-02 conditional novelty 6.0 of 10

    The paper identifies activation, weight, and attention outliers as a single phenomenon caused by softmax attention and demonstrates that explicit context-aware scaling eliminates them.

  42. Peri-LN: Revisiting Normalization Layer in the Transformer Architecture

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Peri-LN, which normalizes both the input and output of each sublayer, reduces activation-variance growth and gradient spikes during LLM pretraining, outperforming Pre-LN and Post-LN at scales up to 3.2B parameters.

  43. An Inquiry into Datacenter TCO for LLM Inference with FP8

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Measured FP8 thin-GEMM throughput shows Intel Gaudi 2/3 outperform NVIDIA H100/H200 on LLM decode-style workloads, making workload-level benchmarks more decisive than peak TFLOPS for datacenter TCO.

  44. AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AKVQ-VL quantizes VLM KV caches to mostly 2 bits with attention-aware token protection and Walsh-Hadamard outlier removal, staying near FP16 accuracy on MileBench.

  45. RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations

    cs.LG 2025-01 conditional novelty 6.0 of 10

    RotateKV combines outlier-aware, pre-RoPE grouped-head Hadamard rotation with attention-sink-aware retention to make 2-bit KV cache quantization accurate on LLaMA-2, LLaMA-3, and Mistral models.

  46. Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Aligning cross-attention map similarity to text self-attention maps at test time improves semantic alignment in Stable Diffusion for prompts with multiple objects and attributes.

  47. When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

    cs.CL 2024-11 conditional novelty 6.0 of 10

    BFloat16 rounding breaks RoPE's positional shift invariance, and AnchorAttention, which fixes the first token as a shared anchor, improves long-context training accuracy and speed.

  48. When Attention Sink Emerges in Language Models: An Empirical View

    cs.CL 2024-10 accept novelty 6.0 of 10

    Attention sinks emerge in language models from softmax-induced token dependence on attention scores and do not appear when using sigmoid attention without normalization in models up to 1B parameters.

  49. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  50. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A 2-bit base plus three 2-bit residual stages gives one checkpoint that runs at 2, 4, 6, or 8 bits, matching a prior multi-precision baseline at 6-8 bits in most tested models.

  51. Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Massive activations in DiTs are timestep-driven detail channels; suppressing them guides finer sampling and AdaLN-modulating them yields more discriminative dense features.

  52. Understanding and Mitigating Prompt Leaking Attacks in Real-World LLM-Based Applications

    cs.CR 2026-06 unverdicted novelty 5.0 of 10

    Large-scale empirical study finds widespread prompt leaking in commercial LLM apps and introduces AREA defense that improves usability while resisting leaks.

  53. P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8

    cs.AR 2026-06 unverdicted novelty 5.0 of 10

    Forward KV iteration in FP8 attention produces P-collapse under attention sink; reverse iteration with S=256 removes it and is optimal among bit-exact scales.

  54. Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Tensor decompositions face practical limits in large-scale LLM compression due to mismatch between assumed shared subspaces and heterogeneous model representations.

  55. Steered Generation via Gradient-Based Optimization on Sparse Query Features

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Prototype-Based Sparse Steering decomposes query activations with SAEs and optimizes sparse features via gradients to steer LLM outputs toward specific behaviors.

  56. Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models

    cs.CL 2026-01 unverdicted novelty 5.0 of 10

    The survey organizes mechanistic interpretability techniques into a Locate-Steer-Improve framework to enable actionable improvements in LLM alignment, capability, and efficiency.

  57. CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    A training method that injects token-level causal labels into attention improves out-of-distribution accuracy on a synthetic benchmark and slightly on math/reasoning tasks.

  58. Discriminating Distal Ischemic Stroke from Seizure-Induced Stroke Mimics Using Dynamic Susceptibility Contrast MRI

    eess.IV 2025-08 unverdicted novelty 5.0 of 10

    Perfusion MRI descriptors from brain regions are reported to separate distal ischemic stroke from seizure mimics with an AUROC of 0.90 in a 162-patient cohort.

  59. Mitigating Spurious Correlations in Weakly Supervised Semantic Segmentation via Cross-architecture Consistency Regularization

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A teacher-student CNN/ViT framework with feature-level consistency raises weakly supervised smoke-segmentation seed mIoU from 33.56 to 47.37 and to 52.93 with post-processing on a custom IJmond dataset.

  60. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

See all 69 Pith citations

Reference graph

Works this paper leans on

159 extracted references · 159 canonical work pages · cited by 69 Pith papers (see all)

  1. [1]

    Wrong-finished

    Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. arXiv:2207.04901, 2022

    work page arXiv 2022

  2. [2]

    Computational complexity: a modern approach

    Sanjeev Arora and Boaz Barak. Computational complexity: a modern approach. Cambridge University Press, 2009

    work page 2009

  3. [3]

    End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking, October 2022

    Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Logical extrapolation without overthinking. arXiv:-2202.05826, 2022

    work page arXiv 2022

  4. [4]

    Barak, B

    Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. arXiv:2207.08799, 2022

    work page arXiv 2022

  5. [5]

    Mix Barrington

    David A. Mix Barrington. Bounded-width polynomial-size branching programs recognize exactly those languages in NC ^1 . In Symposium on the Theory of Computing, 1986

    work page 1986

  6. [6]

    Mix Barrington and Denis Thérien

    David A. Mix Barrington and Denis Thérien. Finite monoids and the fine structure of NC ^1 . Journal of the ACM, 1988

    work page 1988

  7. [7]

    On the ability and limitations of transformers to recognize formal languages

    Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. In Conference on Empirical Methods in Natural Language Processing, 2020

    work page 2020

  8. [8]

    Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

    Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veli c kovi \'c . Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv:2104.13478, 2021

    work page Pith review arXiv 2021

Show all 159 references
  1. [9]

    Unbounded fan-in circuits and associative functions

    Ashok K Chandra, Steven Fortune, and Richard Lipton. Unbounded fan-in circuits and associative functions. In Symposium on Theory of Computing, 1983

    1983

  2. [10]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, 2021 a

    2021

  3. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  4. [12]

    Finite-automaton aperiodicity is PSPACE -complete

    Sang Cho and Dung T Huynh. Finite-automaton aperiodicity is PSPACE -complete. Theoretical Computer Science, 1991

    1991

  5. [13]

    The algebraic theory of context-free languages

    Noam Chomsky and Marcel P Sch \"u tzenberger. The algebraic theory of context-free languages. In Studies in Logic and the Foundations of Mathematics. 1959

    1959

  6. [14]

    Conditional positional encodings for vision transformers

    Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882, 2021

    2021

  7. [15]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? A n analysis of BERT 's attention. In ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , 2019

    2019

  8. [16]

    Approximation by superpositions of a sigmoidal function

    George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 1989

    1989

  9. [17]

    Depth separation for neural networks

    Amit Daniely. Depth separation for neural networks. In Conference on Learning Theory, pages 690--696. PMLR, 2017

    2017

  10. [18]

    Learning parities with neural networks

    Amit Daniely and Eran Malach. Learning parities with neural networks. Advances in Neural Information Processing Systems, 2020

    2020

  11. [19]

    Universal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019

    2019

  12. [20]

    Neural networks and the chomsky hierarchy

    Gr \'e goire Del \'e tang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Marcus Hutter, Shane Legg, and Pedro A Ortega. Neural networks and the chomsky hierarchy. arXiv preprint arXiv:2207.02098, 2022

    2022

  13. [22]

    Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, and Gilbert Strang

    Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, Roman Wang, Nikhil Singh, Taylor L. Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, and Gilbert Strang. A neural network solves, expla...

    2022

  14. [23]

    How can self-attention networks recognize D yck-n languages? In Findings of the Association for Computational Linguistics: EMNLP , 2020

    Javid Ebrahimi, Dhruv Gelda, and Wei Zhang. How can self-attention networks recognize D yck-n languages? In Findings of the Association for Computational Linguistics: EMNLP , 2020

    2020

  15. [24]

    Inductive biases and variable creation in self-attention mechanisms

    Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning, 2022

    2022

  16. [25]

    Computational holonomy decomposition of transformation semigroups

    Attila Egri-Nagy and Chrystopher L Nehaniv. Computational holonomy decomposition of transformation semigroups. arXiv:1508.06345, 2015

    2015 arXiv

  17. [26]

    Automata, languages, and machines

    Samuel Eilenberg. Automata, languages, and machines. Academic Press, 1974

    1974

  18. [27]

    The power of depth for feedforward neural networks

    Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference on learning theory, pages 907--940. PMLR, 2016

    2016

  19. [28]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

    2021

  20. [29]

    Saxe, and Michael Sipser

    Merrick Furst, James B. Saxe, and Michael Sipser. Parity, circuits, and the polynomial-time hierarchy. Mathematical Systems Theory, 1984

    1984

  21. [30]

    Shortcut learning in deep neural networks

    Robert Geirhos, J \"o rn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020

    2020

  22. [31]

    Looped transformers as programmable computers

    Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. arXiv preprint arXiv:2301.13196, 2023

    2023

  23. [32]

    Reliably learning the R e LU in polynomial time

    Surbhi Goel, Varun Kanade, Adam Klivans, and Justin Thaler. Reliably learning the R e LU in polynomial time. In Conference on Learning Theory, 2017

    2017

  24. [33]

    Adaptive computation time for recurrent neural networks

    Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016

    2016 arXiv

  25. [34]

    Neural turing machines

    Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014

    2014 arXiv

  26. [35]

    Li, and Richard Socher

    Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. arXiv:1711.02281, 2017

    2017 arXiv

  27. [36]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv:1912.01603, 2019

    1912 arXiv

  28. [37]

    Theoretical limitations of self-attention in neural sequence models

    Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 2020

    2020

  29. [38]

    Transformer language models without positional encodings still learn positional information

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv:2203.16634, 2022

    2022

  30. [39]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , 2016

    2016

  31. [40]

    Towards lower bounds on the depth of R e LU neural networks

    Christoph Hertrich, Amitabh Basu, Marco Di Summa, and Martin Skutella. Towards lower bounds on the depth of R e LU neural networks. In Advances in Neural Information Processing Systems, 2021

    2021

  32. [41]

    Steele Jr

    W Daniel Hillis and Guy L. Steele Jr. Data parallel algorithms. Communications of the ACM, 1986

    1986

  33. [42]

    Multilayer feedforward networks are universal approximators

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 1989

    1989

  34. [43]

    Universal language model fine-tuning for text classification

    Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv:1801.06146, 2018

    2018 arXiv

  35. [44]

    Block-recurrent transformers

    DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-recurrent transformers. arXiv:2203.07852, 2022

    2022

  36. [45]

    Offline reinforcement learning as one big sequence modeling problem

    Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021

    2021

  37. [46]

    Finetuning pretrained transformers into rnns

    Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A Smith. Finetuning pretrained transformers into rnns. arXiv:2103.13076, 2021

    2021

  38. [47]

    Rethinking positional encoding in language pre-training

    Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595, 2020

    2006

  39. [48]

    The number of semigroups of order n

    Daniel J Kleitman, Bruce R Rothschild, and Joel H Spencer. The number of semigroups of order n. Proceedings of the American Mathematical Society, 1976

    1976

  40. [49]

    Finite permutation groups with large abelian quotients

    L \'a szl \'o Kov \'a cs and Cheryl Praeger. Finite permutation groups with large abelian quotients. Pacific Journal of Mathematics, 1989

    1989

  41. [50]

    Produit complet des groupes de permutations et probleme d’extension de groupes II

    Marc Krasner and L \'e o Kaloujnine. Produit complet des groupes de permutations et probleme d’extension de groupes II . Acta Scientiarum Mathematicarum, 1951

    1951

  42. [51]

    Algebraic theory of machines, I : P rime decomposition theorem for finite semigroups and machines

    Kenneth Krohn and John Rhodes. Algebraic theory of machines, I : P rime decomposition theorem for finite semigroups and machines. Transactions of the American Mathematical Society, 1965

    1965

  43. [52]

    Deep learning for symbolic mathematics

    Guillaume Lample and Fran c ois Charton. Deep learning for symbolic mathematics. arXiv:1912.01412, 2019

    1912

  44. [53]

    Fractal N et: U ltra-deep neural networks without residuals

    Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractal N et: U ltra-deep neural networks without residuals. arXiv:1605.07648, 2016

    2016 arXiv

  45. [54]

    On the ability of neural nets to express distributions

    Holden Lee, Rong Ge, Tengyu Ma, Andrej Risteski, and Sanjeev Arora. On the ability of neural nets to express distributions. In Conference on Learning Theory, pages 1271--1296. PMLR, 2017

    2017

  46. [55]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal,...

    2022

  47. [56]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017

    2017 arXiv

  48. [57]

    On the K rohn- R hodes cascaded decomposition theorem

    Oded Maler. On the K rohn- R hodes cascaded decomposition theorem. In Time for Verification. 2010

    2010

  49. [58]

    On the cascaded decomposition of automata, its complexity and its application to logic ( D raft)

    Oded Maler and Amir Pnueli. On the cascaded decomposition of automata, its complexity and its application to logic ( D raft). 1994

    1994

  50. [59]

    Threshold circuits for iterated matrix product and powering

    Carlo Mereghetti and Beatrice Palano. Threshold circuits for iterated matrix product and powering. RAIRO-Theoretical Informatics and Applications, 2000

    2000

  51. [60]

    William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. On the power of saturated T ransformers: A view from circuit complexity. arXiv:2106.16213, 2021

    2021

  52. [61]

    Transformers are sample efficient world models

    Vincent Micheli, Eloi Alonso, and Fran c ois Fleuret. Transformers are sample efficient world models. arXiv:2209.00588, 2022

    2022

  53. [62]

    Lower bounds over boolean inputs for deep neural networks with R e LU gates

    Anirbit Mukherjee and Amitabh Basu. Lower bounds over boolean inputs for deep neural networks with R e LU gates. arXiv:1711.03073, 2017

    2017 arXiv

  54. [63]

    A mechanistic interpretability analysis of grokking

    Neel Nanda and Tom Lieberum. A mechanistic interpretability analysis of grokking. Alignment Forum, 2022. URL https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking

    2022

  55. [64]

    Benjamin Newman, John Hewitt, Percy Liang, and Christopher D. Manning. The EOS decision and length extrapolation. In BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020

    2020

  56. [65]

    Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials

    Eshaan Nichani, Yu Bai, and Jason D Lee. Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials. arXiv:2206.03688, 2022

    2022

  57. [66]

    Investigating the limitations of transformers with simple arithmetic tasks

    Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of transformers with simple arithmetic tasks. arXiv:2102.13019, 2021

    2021

  58. [67]

    Show your work: Scratchpads for intermediate computation with language models

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. arXiv:...

    2021 arXiv

  59. [68]

    The complexity of M arkov decision processes

    Christos H Papadimitriou and John N Tsitsiklis. The complexity of M arkov decision processes. Mathematics of Operations Research, 1987

    1987

  60. [69]

    Py T orch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K" o pf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  61. [70]

    Attention is turing complete

    Jorge P \'e rez, Pablo Barcel \'o , and Javier Marinkovic. Attention is turing complete. The Journal of Machine Learning Research, 22 0 (1): 0 3463--3497, 2021

    2021

  62. [71]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv:1802.05365, 2018

    2018 arXiv

  63. [72]

    Generative language modeling for automated theorem proving

    Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv:2009.03393, 2020

    2009

  64. [73]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022

    2022

  65. [74]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 2019

    2019

  66. [75]

    Reif and Stephen R

    John H. Reif and Stephen R. Tate. On threshold circuits and polynomial computation. SIAM Journal on Computing, 1992

    1992

  67. [76]

    Applications of automata theory and algebra: via the mathematical theory of complexity to biology, physics, psychology, philosophy, and games

    John Rhodes, Chrystopher L Nehaniv, and Morris W Hirsch. Applications of automata theory and algebra: via the mathematical theory of complexity to biology, physics, psychology, philosophy, and games. World Scientific, 2010

    2010

  68. [77]

    Can contrastive learning avoid shortcut solutions? Advances in Neural Information Processing Systems, 2021

    Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in Neural Information Processing Systems, 2021

    2021

  69. [78]

    Depth separations in neural networks: what is actually being separated? In Conference on Learning Theory, pages 2664--2666

    Itay Safran, Ronen Eldan, and Ohad Shamir. Depth separations in neural networks: what is actually being separated? In Conference on Learning Theory, pages 2664--2666. PMLR, 2019

    2019

  70. [79]

    Programming puzzles

    Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Kalai. Programming puzzles. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks, 2021

    2021

  71. [80]

    On finite monoids having only trivial subgroups

    Marcel Paul Sch \"u tzenberger. On finite monoids having only trivial subgroups. Information and Control, 1965

    1965

  72. [81]

    Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks

    Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. In Advances in Neural Information Processing Systems, 2021

    2021

  73. [82]

    On the computational power of neural nets

    Hava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. In Conference on Learning Theory, 1992

    1992

  74. [83]

    Benefits of depth in neural networks

    Matus Telgarsky. Benefits of depth in neural networks. In Conference on learning theory, pages 1517--1539. PMLR, 2016

    2016

  75. [84]

    BERT rediscovers the classical NLP pipeline

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. arXiv:1905.05950, 2019

    1905

  76. [85]

    Wave N et: A generative model for raw audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wave N et: A generative model for raw audio. arXiv:1609.03499, 2016

    2016 arXiv

  77. [86]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  78. [87]

    Hechtman, and Jonathon Shlens

    Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake A. Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In IEEE Conference on Computer Vision and Pattern Recognition , 2021

    2021

  79. [88]

    Visualizing attention in transformer-based language representation models

    Jesse Vig. Visualizing attention in transformer-based language representation models. arXiv:1904.02679, 2019

    1904 arXiv

  80. [89]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  81. [90]

    Chain of thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv:2201.11903, 2022

    2022 arXiv

  82. [91]

    Thinking like T ransformers

    Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like T ransformers. In International Conference on Machine Learning, 2021

    2021

  83. [92]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, ...

    1910 arXiv

  84. [93]

    Google's neural machine translation system: Bridging the gap between human and machine translation

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Ste...

    2016 arXiv

  85. [94]

    A survey on non-autoregressive generation for neural machine translation and beyond

    Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie-yan Liu. A survey on non-autoregressive generation for neural machine translation and beyond. arXiv:2204.09269, 2022

    2022

  86. [95]

    How neural networks extrapolate: From feedforward to graph neural networks

    Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv:2009.11848, 2020

    2009

  87. [96]

    Papadimitriou, and Karthik Narasimhan

    Shunyu Yao, Binghui Peng, Christos H. Papadimitriou, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. In Association for Computational Linguistics, 2021

    2021

  88. [97]

    Mastering atari games with limited data

    Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 2021

    2021

  89. [98]

    Cascade synthesis of finite-state machines

    H Paul Zeiger. Cascade synthesis of finite-state machines. Information and Control, 1967

    1967

  90. [99]

    Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization

    Chiyuan Zhang, Maithra Raghu, Jon Kleinberg, and Samy Bengio. Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization. arXiv:2107.12580, 2021 a

    2021

  91. [100]

    How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2021 b

    Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2021 b

    2021

  92. [101]

    Unveiling T ransformers with LEGO : a synthetic reasoning task

    Yi Zhang, Arturs Backurs, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling T ransformers with LEGO : a synthetic reasoning task. arXiv:2206.04301, 2022

    2022

  93. [102]

    On K rohn- R hodes theory for semiautomata

    Karl-Heinz Zimmermann. On K rohn- R hodes theory for semiautomata. arXiv:2010.16235, 2020

    2010

  94. [103]

    Intriguing properties of quantization at scale

    Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization at scale. In NeurIPS, 2023

    2023

  95. [104]

    The falcon series of open language models

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, and et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867, 2023

    2023

  96. [105]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    2016 arXiv

  97. [106]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. arXiv preprint arXiv:1911.11641, 2019

    1911

  98. [107]

    Understanding and overcoming the challenges of efficient transformer quantization

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. arXiv:2109.12948, 2021

    2021

  99. [108]

    Quantizable transformers: Removing outliers by helping attention heads do nothing

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. arXiv preprint arXiv:2306.12929, 2023

    2023

  100. [109]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  101. [110]

    Sparks of artificial general intelligence: Early experiments with gpt-4

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with g...

    2023 arXiv

  102. [111]

    BoolQ : Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ : Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019 a

    1905 arXiv

  103. [112]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341, 2019 b

    1906 arXiv

  104. [113]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  105. [114]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv:2309.16588, 2023

    2023 arXiv

  106. [115]

    ImageNet : A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet : A large-scale hierarchical image database. In CVPR, 2009

    2009

  107. [116]

    LLM .int8(): 8-bit matrix multiplication for transformers at scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM .int8(): 8-bit matrix multiplication for transformers at scale. In NeurIPS, 2022

    2022

  108. [117]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanove. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  109. [118]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  110. [119]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  111. [120]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2021

    2021 arXiv

  112. [121]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

    2016

  113. [122]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377, 2021

    2021 arXiv

  114. [123]

    Residual stream norms grow exponentially over the forward pass, 2023

    Stefan Heimersheim and Alex Turner. Residual stream norms grow exponentially over the forward pass, 2023. URL https://www.alignmentforum.org/posts/8mizBCm3dyc432nK8/residual-stream-norms-grow-exponentially-over-the-forward

    2023

  115. [124]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021

    2021

  116. [125]

    Outlier-efficient hopfield layers for large transformer-based models

    Jerry Yao-Chieh Hu, Pei-Hsuan Chang, Robin Luo, Hong-Yu Chen, Weijian Li, Wei-Po Wang, and Han Liu. Outlier-efficient hopfield layers for large transformer-based models. arXiv preprint arXiv:2404.03828, 2024

    2024

  117. [126]

    Phi-2: The surprising power of small language models, 2023

    Mojan Javaheripi, Sébastien Bubeck, and et al. Phi-2: The surprising power of small language models, 2023. URL https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/

    2023

  118. [127]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  119. [128]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  120. [129]

    Nanogpt, 2023

    Andrej Karpathy. Nanogpt, 2023. URL https://github.com/karpathy/nanoGPT

    2023

  121. [130]

    Gpt-4 passes the bar exam

    Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. SSRN, 2023

    2023

  122. [131]

    Revealing the dark secrets of bert

    Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXiv preprint arXiv:1908.08593, 2019

    1908

  123. [132]

    Bert busters: Outlier dimensions that disrupt transformers

    Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. In ACL Findings, 2021

    2021

  124. [133]

    Awq: Activation-aware weight quantization for llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023

    2023 arXiv

  125. [134]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  126. [135]

    Attention is off by one, 2023

    Evan Miller. Attention is off by one, 2023. URL https://www.evanmiller.org/attention-is-off-by-one.html

    2023

  127. [136]

    Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023

    MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b

    2023

  128. [137]

    Role of bias terms in dot-product attention

    Mahdi Namazifar, Devamanyu Hazarika, and Dilek Hakkani-Tur. Role of bias terms in dot-product attention. arXiv preprint arXiv:2302.08626, 2023

    2023

  129. [138]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  130. [139]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  131. [140]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, and Marc Szafraniec. Dinov2: Learning robust visual features without supervision. arXiv:2304.07193, 2024

    2024 arXiv

  132. [141]

    Wainwright, and et al

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, and et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022

    2022 arXiv

  133. [142]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021

    2021 arXiv

  134. [143]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical Report, 2019

    2019

  135. [144]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021

    2021 arXiv

  136. [145]

    Compressive transformers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint, 2019. URL https://arxiv.org/abs/1911.05507

    2019

  137. [146]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020

    2020

  138. [147]

    Robinson, Nathan Drenkow, Colin Conwell, and Michael F

    Brian S. Robinson, Nathan Drenkow, Colin Conwell, and Michael F. Bonner. A sparse null code emerges in deep neural networks. In NeurIPS UniReps Workshop, 2023

    2023

  139. [148]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019

    1907 arXiv

  140. [149]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  141. [150]

    All bark and no bite: Rogue dimensions in transformer language models obscure representational quality

    William Timkey and Marten van Schijndel. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. arXiv:2109.04404, 2021

    2021

  142. [151]

    Redpajama: an open dataset for training large language models, October 2023

    Together Computer . Redpajama: an open dataset for training large language models, October 2023. URL https://github.com/togethercomputer/RedPajama-Data

    2023

  143. [152]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  144. [153]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022

    2022

  145. [154]

    Liu, Lechao Xiao, Katie Everett, and et al

    Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, and et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023

    2023

  146. [155]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, 2023 a

    2023

  147. [156]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv:2309.17453, 2023 b

    2023 arXiv

  148. [157]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023

    2023 arXiv

  149. [158]

    Glm-130b: An open bilingual pre-trained model

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, and et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022

    2022 arXiv

  150. [159]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. In NeurIPS, 2019

    2019

  151. [160]

    Unveiling a core linguistic region in large language models

    Jun Zhao, Zhihao Zhang, Yide Ma, Qi Zhang, Tao Gui, Luhui Gao, and Xuanjing Huang. Unveiling a core linguistic region in large language models. arXiv:2310.14928, 2023

    2023

Pith tools

Reviewed May 16, 2026 · model on record in the stance chip above.