Pith. sign in

REVIEW 4 major objections 5 minor 21 references

RoPE scaling breaks short-text accuracy; LinearARD restores it using 4.25M tokens (60x fewer than baselines) by aligning Q/Q, K/K, and V/V attention-relation distributions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:32 UTC pith:2NWE4KU5

load-bearing objection Solid linear-memory KL kernel and plausible distillation recipe, but the 60x token-efficiency claim is not actually demonstrated by the experiments. the 4 major comments →

arxiv 2604.00004 v2 pith:2NWE4KU5 submitted 2026-03-09 cs.CL cs.AI

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

Ning Yang , Hengyu Zhong , Wentao Wang , Baoliang Tian , Haijun Zhang , Jun Wang This is my paper
classification cs.CL cs.AI
keywords RoPE scalingcontext window extensionattention distillationself-relation matriceslinear-memory kernelKL divergencecontinued pre-traininglong-context LLMs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Scaling a pretrained model's rotary position embeddings (RoPE) to extend its context window disrupts the attention geometry learned during pretraining, and standard fixes such as continued pre-training consume hundreds of millions of tokens. LinearARD proposes that this damage can be repaired far more cheaply by treating the original model as a frozen teacher and forcing the scaled student to reproduce the teacher's internal row-wise attention-relation distributions — the softmax-normalized Q/Q, K/K, and V/V similarity matrices inside every attention head. The paper introduces an exact linear-memory kernel that computes the KL divergence between these distributions and its gradients without materializing any n×n matrix, removing the quadratic-memory barrier that previously limited attention distillation to short sequences. On three 7B-8B backbones extended to 32K, LinearARD recovers about 95% of native short-text accuracy using 4.25M training tokens — roughly 60x fewer than prior restoration baselines — while matching or exceeding their long-context RULER scores, especially at 32K. The point of the paper, if right, is that restoring a scaled model is mostly a geometry-repair problem: align the internal relational structure and the output-level behavior comes back.

Core claim

The paper's central claim is that the short-context degradation caused by RoPE scaling is a fine-grained positional-geometry distortion inside attention, and that it can be corrected by distilling the row-wise distributions of three dense self-relation matrices — R_Q = softmax(QQ^T/√d + M), R_K = softmax(KK^T/√d + M), and R_V = softmax(VV^T/√d + M) — from the frozen native-RoPE teacher to the scaled student using forward KL. Because the gradient of this KL with respect to a student logit is simply R_s(i,j) − R_t(i,j), the loss can be accumulated tile by tile: precompute per-row log-sum-exp statistics, recompute logits in the backward pass, and reconstruct probabilities on-chip. The paper pro

What carries the argument

The central object is the masked row-wise relation distribution R_m = softmax(X_m X_m^T/√d + M) for X ∈ {Q,K,V}; distillation aligns teacher and student rows by forward KL, L = (1/n) Σ_i D_KL(R_t(i,:) ∥ R_s(i,:)). The load-bearing identity is ∂L/∂Z_s(i,j) = (R_s(i,j) − R_t(i,j))/n, which lets each gradient entry be formed locally. The kernel exploits this in a two-pass tiled scheme: Phase 1 stores only the per-row log-sum-exp statistics LSE_m(i) = log Σ_k exp(Z_m(i,k)) (O(n) memory); Phase 2 recomputes masked logits tile by tile in on-chip memory, reconstructs R_s and R_t from the precomputed LSEs, accumulates the KL loss, and streams gradients to the student's Q/K/V projections. This makes

Load-bearing premise

The load-bearing premise is that matching the row-wise softmax distributions of Q/Q, K/K, and V/V self-relations is a complete and sufficient supervision signal to restore the student's short-context behavior — in particular that the V/V constraint, which never appears in the attention logits, is necessary.

What would settle it

Take a new backbone/scaling pair not used in the paper (e.g., a 1B model with YaRN 4x) and run LinearARD with the V/V term ablated. If short-text recovery stays at the full-method level, the claimed holism of Q/Q-K/K-V/V alignment is wrong. Conversely, if a student whose Q/Q and K/K match the teacher but whose V/V is deliberately misaligned (e.g., value projections frozen at the scaled initialization) shows a clear accuracy drop, the claim survives.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Context extension becomes roughly 60x cheaper: restoring short-text behavior needs 4.25M tokens instead of 256M, so long-context deployment no longer requires a large high-quality long-context corpus.
  • Attention-structure consistency is sufficient supervision: because output-logit divergence falls in tandem with the relation KL during training, explicit logit-level distillation is unnecessary for restoration.
  • Dense, exact distribution matching is now practical at long contexts, retiring the need for sparse or selective attention-supervision objectives that sacrifice fidelity.
  • The restoration transfers across backbones and scaling factors: 4x and 8x RoPE scaling on three 7B-8B models all recover ~95% of native short-text accuracy, with the largest long-context gains at 32K.
  • A short continued-pretraining stage remains necessary for long-context activation: ARD alone restores short-text performance but leaves RULER low (36.8), while adding 2M tokens of continued pre-training raises it to 63.2.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the V/V constraint is the least theoretically motivated piece — V/V never enters the attention logits — so its contribution likely comes from regularizing what features the value pathway carries. A direct test would be to run the method on a backbone where V/V is omitted; the paper's own ablation shows a small but real drop, so the prediction is that V/V matters most when valu
  • Editorial inference: the linear-memory KL kernel is a general-purpose primitive, not tied to QKV relations. It should apply to any row-wise distribution-matching objective — attention-map distillation, logit KL, or even non-attention softmax layers — so the method's scope may be broader than context restoration.
  • Editorial inference: the 60x token reduction suggests the dominant cost of restoration is geometric recalibration, not knowledge acquisition. If that is right, the same budget should transfer to larger scaling factors (64K, 128K) and to newer positional schemes; the paper does not test those regimes.
  • Editorial inference: because the teacher is evaluated only within its native context length, the method cannot teach the student behaviors that live beyond the teacher's range; the optional continued-pretraining stage is doing that work, so the two stages are complementary rather than interchangeable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LinearARD, a self-distillation method that restores short-context performance of RoPE-scaled LLMs by aligning row-wise softmax distributions of Q/Q, K/K, and V/V self-relation matrices with those of a frozen native-RoPE teacher. To make this feasible at long sequence lengths, it introduces an IO-aware tiled kernel that computes the exact KL divergence and gradients with O(n) memory instead of O(n^2). The method is evaluated on LLaMA2-7B, LLaMA3-8B, and Mistral-7B-v0.1 extended to 32K context, with short-text benchmarks (MMLU, LAMBADA, BoolQ, etc.) and RULER. The paper claims that LinearARD achieves comparable or better restoration than CPT and LongReD using 4.25M training tokens versus 256M, roughly a 60x reduction, and provides theoretical proofs of linear memory and gradient exactness.

Significance. If the empirical claims hold, this is a practically valuable contribution: it makes dense structural attention distillation feasible at sequence lengths where materializing n x n relation maps is prohibitive, and it offers a concrete, theoretically grounded alternative to hidden-state distillation for RoPE restoration. The linear-memory KL kernel is elegant, with a clear proof of exactness and empirical numerical verification, and the code is released. The two-stage recipe (relation distillation followed by a light CPT stage) is well motivated and the ablations isolate the contribution of the kernel and loss terms. However, the central token-efficiency claim is currently supported only by cross-budget comparisons, and one ablation uses mismatched RoPE scaling settings, so the significance is conditional on additional experiments.

major comments (4)
  1. [§4.1, Table 1, Appendix A.1] The headline claim of a 60x token reduction is not supported by a same-token-budget baseline. Table 1 compares LinearARD at 4.25M tokens only against CPT and LongReD at 256M tokens. No experiment shows what CPT or LongReD achieve at 4.25M tokens, nor what LinearARD would achieve at larger budgets. The statement that LongReD and CPT 'require' 256M tokens is asserted, not demonstrated. A matched-budget comparison (or at least a token-budget learning curve) is needed to substantiate the efficiency claim.
  2. [§4.3.2, Tables 3 and 4] The CPT ablation is internally inconsistent with the main setting. Table 3, which shows that CPT has negligible short-text effect, is run on LLaMA2-7B with PI4x, while Table 4 and the main Table 1 use PI8x. Therefore the conclusion that ARD alone is sufficient for short-text restoration is not established in the exact configuration used for the headline result. Furthermore, Table 4 shows RULER dropping from 63.2 to 36.8 without the CPT stage, so the long-context improvement is substantially produced by the 2M-token CPT component. The attribution of long-context gains to relation distillation versus CPT needs a same-scaling-factor ablation.
  3. [§4.2, Table 1] All reported results are single runs without error bars or significance tests. Some differences that the text interprets are small: on Mistral-7B-v0.1, LinearARD's RULER is 60.8 vs. LongReD's 62.3 and short-text Avg is 57.91 vs. 59.38. Without variance estimates, the claims of 'competitive' or 'surpassing' performance are difficult to evaluate, especially at the 8K/16K lengths where differences are smaller. Reporting multiple seeds or at least bootstrap confidence intervals would substantially strengthen the comparative claims.
  4. [§3.2, Eq. (4c), Table 2] The inclusion of V/V relation distillation is a distinctive design choice, but the rationale is not fully developed. V/V relations do not enter the attention logits (which use QK^T), so aligning V/V is an extra assumption about value-feature geometry. The only evidence is the Table 2 ablation, which is on a different architecture (Qwen3-0.6B) and shows a small drop from 43.7 to 43.4 without V-rel. The paper should either provide a more direct analysis of why V/V alignment is necessary for RoPE restoration or acknowledge that this term is an empirical heuristic whose transferability across models is untested.
minor comments (5)
  1. [Abstract] The abstract states 'recovers 98.3% of the short-text performance of state-of-the-art baselines.' This figure corresponds only to LLaMA2-7B in Table 1 (51.87/52.75); for LLaMA3 and Mistral the ratios are 97.2% and 97.5%. The claim should be qualified.
  2. [Appendix A.1, Eq. (8)] The token-budget formula should specify whether L_s is the number of valid tokens per sample or the context length with padding, and how gradient accumulation interacts with the per-GPU batch size. The decomposition of 4.25M into the distillation and CPT stages is not shown, which makes it hard to verify the accounting.
  3. [§4.1] The training description says 'only the attention-module Q/K/V projection weights are updated' but also mentions optional LoRA/QLoRA. It is unclear which configuration produced Table 1. Please state explicitly for each experiment whether LoRA was used and what rank.
  4. [Table 2 caption] The main experiments use LLaMA2, LLaMA3, and Mistral, but the ARD component ablation is on Qwen3-0.6B. A sentence explaining why this smaller model was chosen, and whether the conclusions are expected to transfer, would be helpful.
  5. [§A.8.1, Figure 4] The correlational training-dynamics plot is interpreted as 'strong causal link.' Correlations alone do not establish causation; consider adding an intervention (e.g., early stopping, gradient norm analysis) or softening the causal language.

Circularity Check

0 steps flagged

No significant circularity: the distillation objective, kernel, and evaluation are self-contained; the token-efficiency and ablation concerns are experimental-evidence issues, not derivation-level circularity.

full rationale

The paper's central derivation is not circular. The distillation loss (Eq. 5) aligns teacher and student row-wise Q/Q, K/K, and V/V relation distributions, with all loss weights set to 1 and no parameter fitted to the reported benchmarks. The linear-memory kernel's exactness is established by a self-contained mathematical argument (Proposition 3.3 and Appendix A.6) comparing tiled recomputation against the analytic gradient, not by appealing to empirical success. The reported short-text recovery and RULER scores are measured on external benchmarks (MMLU, LAMBADA, RULER) and are not reconstructed from the training objective. The only overlapping-author citation (PoSE, Zhu et al. 2024) is contextual and not load-bearing. Concerns raised about the 60x token-efficiency claim lacking matched-budget baselines and the CPT ablation using different PI factors in Tables 3 and 4 are legitimate threats to attribution, but they concern experimental comparison design rather than equations reducing to their inputs. No 'prediction' is equivalent to a fitted value or to a self-citation, so the correct circularity score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central claim rests primarily on the sufficiency of relation-distribution matching (an unproven empirical premise), standard softmax/tiling math, and a small number of hand-set hyperparameters.

free parameters (2)
  • Loss weights λ_q, λ_k, λ_v = 1, 1, 1
    Set to equal weights without sensitivity analysis; the central result (Eq. 5) depends on this choice, and the paper does not explore its impact.
  • CPT stage token budget = 2M tokens (part of the 4.25M total)
    Chosen as 'lightweight' without systematic tuning; token-efficiency claims depend on this amount being sufficient.
axioms (3)
  • ad hoc to paper Row-wise softmax distributions of Q/Q, K/K, and V/V self-relations are a sufficient and appropriate distillation target for restoring RoPE-scaled students.
    The entire method is built on this premise (Eqs. 3-5); only correlational evidence (Fig. 4) and ablations are provided, no proof.
  • domain assumption The causal mask M should be applied to self-relation matrices even though self-similarity does not require causality.
    The paper reuses the attention mask in Eq. 1 for all three relation types; this discards future-token similarity information and is a modeling choice that is not justified.
  • domain assumption RoPE scaling modifies only Q and K, so the teacher's frozen Q/Q and K/K distributions are stable reference targets while V/V provides an auxiliary stabilizing signal.
    Section 3.2 states V/V stabilizes features; this is an empirical assumption not derived from the RoPE mechanism.

pith-pipeline@v1.3.0-alltime-deepseek · 18545 in / 14735 out tokens · 132110 ms · 2026-08-02T18:32:55.478578+00:00 · methodology

0 comments
read the original abstract

The extension of context windows in Large Language Models is typically facilitated by scaling positional encodings followed by lightweight Continual Pre-Training (CPT). While effective for processing long sequences, this paradigm often disrupts original model capabilities, leading to performance degradation on standard short-text benchmarks. We propose LinearARD, a self-distillation method that restores Rotary Position Embeddings (RoPE)-scaled students through attention-structure consistency with a frozen native-RoPE teacher. Rather than matching opaque hidden states, LinearARD aligns the row-wise distributions of dense $Q/Q$, $K/K$, and $V/V$ self-relation matrices to directly supervise attention dynamics. To overcome the quadratic memory bottleneck of $n \times n$ relation maps, we introduce a linear-memory kernel. This kernel leverages per-token log-sum-exp statistics and fuses logit recomputation into the backward pass to compute exact Kullback-Leibler divergence and gradients. On LLaMA2-7B extended from 4K to 32K, LinearARD recovers 98.3\% of the short-text performance of state-of-the-art baselines while surpassing them on long-context benchmarks. Notably, our method achieves these results using only \textbf{4.25M} training tokens compared to the \textbf{256M} tokens required by LongReD and CPT. Our code is available at https://github.com/gracefulning/LinearARD.

Figures

Figures reproduced from arXiv: 2604.00004 by Baoliang Tian, Haijun Zhang, Hengyu Zhong, Jun Wang, Ning Yang, Wentao Wang.

Figure 1
Figure 1. Figure 1: LinearARD pipeline. A frozen native-RoPE teacher provides dense row-wise relation distributions (Q/Q, K/K, and V /V ), and the RoPE-scaled student is restored by minimizing relation KL with an exact linear-memory kernel. RoPE. While effective for extension, this rescaling distorts the rotation-sensitive geometric relationships established during pretraining, leading to a collapse in short-context performan… view at source ↗
Figure 2
Figure 2. Figure 2: Standard attention vs. QKV self-relations. (a) Attention matrix A computed from QK⊤ (shown for reference), and (b– d) Q/Q, K/K, and V/V relation matrices computed from QQ⊤, KK⊤, and VV⊤ (Eq. 4). LinearARD distills the row-wise rela￾tion distributions in (b–d) by aligning teacher vs. student rows via forward KL (Eq. 3) under the same mask M. during both training and inference. The model employs masked causa… view at source ↗
Figure 3
Figure 3. Figure 3: Memory scaling of the linear-memory KL distillation Kernel in Sec. 3.3 as a function of sequence length. Proposition 3.3 (Mathematical Exactness). The gradients computed by the tiled formulation in Algorithm 2 are math￾ematically equivalent to the analytical gradients obtained from standard full-matrix backpropagation. The proofs for Theorem 3.2 and Proposition 3.3 are pro￾vided in Appendix A.5 and Appendi… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of distillation objectives. The relation/attention loss (the optimization target) and the logit distillation loss (monitored but not optimized) are plotted across training steps. The synchronous decline suggests a strong causal link between internal attention drift and output distribution shift. A.8.2. KERNEL NUMERICAL VERIFICATION Length Forward Error Backward Error (×10−7 ) Mean (×10−4 … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 16 linked inside Pith

  1. [1]

    Mathqa: Towards interpretable math word problem solving with operation-based for- malisms

    Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y ., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. InProceedings of the 2019 conference of the North American chapter of the association for computa- tional linguistics: Human language technologies, volume 1 (long and short papers)...

    2019

  2. [8]

    Minillm: Knowl- edge distillation of large language models.arXiv preprint arXiv:2306.08543,

    Gu, Y ., Dong, L., Wei, F., and Huang, M. Minillm: Knowl- edge distillation of large language models.arXiv preprint arXiv:2306.08543,

    Pith/arXiv arXiv

  3. [9]

    Seekr: Selective attention-guided knowledge retention for continual learning of large language models

    He, J., Guo, H., Zhu, K., Zhao, Z., Tang, M., and Wang, J. Seekr: Selective attention-guided knowledge retention for continual learning of large language models. InPro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3254–3266,

    2024

  4. [10]

    Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Pith/arXiv arXiv 2009

  5. [11]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Pith/arXiv arXiv

  6. [14]

    Continual pre-training of language models.arXiv preprint arXiv:2302.03241,

    Ke, Z., Shao, Y ., Lin, H., Konishi, T., Kim, G., and Liu, B. Continual pre-training of language models.arXiv preprint arXiv:2302.03241,

    Pith/arXiv arXiv

  7. [15]

    Ring attention with blockwise transformers for near-infinite context.arXiv preprint arXiv:2310.01889,

    Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context.arXiv preprint arXiv:2310.01889,

    Pith/arXiv arXiv

  8. [16]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    9 LinearARD: Linear-Memory Attention Distillation for RoPE Restoration Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391,

    2018

  9. [18]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    Sanh, V ., Debut, L., Chaumond, J., and Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    Pith/arXiv arXiv 1910

  10. [19]

    Socialiqa: Commonsense reasoning about social interac- tions.arXiv preprint arXiv:1904.09728,

    Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y . Socialiqa: Commonsense reasoning about social interac- tions.arXiv preprint arXiv:1904.09728,

    Pith/arXiv arXiv 1904

  11. [20]

    Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

    Pith/arXiv arXiv

  12. [21]

    Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers

    Wang, W., Bao, H., Huang, S., Dong, L., and Wei, F. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers. InFindings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pp. 2140–2151,

    2021

  13. [2015]

    Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654,

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y ., and Ginsburg, B. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654,

    Pith/arXiv arXiv

  14. [2016]

    Yarn: Efficient context window extension of large language models.arXiv preprint arXiv:2309.00071,

    Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models.arXiv preprint arXiv:2309.00071,

    Pith/arXiv arXiv

  15. [2018]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning.arXiv preprint arXiv:2307.08691,

    Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning.arXiv preprint arXiv:2307.08691,

    Pith/arXiv arXiv

  16. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  17. [2020]

    Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595,

    Chen, S., Wong, S., Chen, L., and Tian, Y . Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595,

    Pith/arXiv arXiv

  18. [2022]

    Tinybert: Distilling bert for natural language understanding

    Jiao, X., Yin, Y ., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. Tinybert: Distilling bert for natural language understanding. InFindings of the association for computational linguistics: EMNLP 2020, pp. 4163– 4174,

    2020

  19. [2023]

    Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    Pith/arXiv arXiv 1905

  20. [2024]

    X., Wang, B., and Chen, W

    Dong, Z., Li, J., Jiang, J., Xu, M., Zhao, W. X., Wang, B., and Chen, W. Longred: Mitigating short-text degradation of long-context large language models via restoration distillation.arXiv preprint arXiv:2502.07365,

    Pith/arXiv arXiv

  21. [2025]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv