Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that treating query, key, and value asymmetrically—compressing keys more than values and widening the query projection—cuts long-context attention cost by up to 33.36% over grouped-query attention while maintaining…

desk verdict Real architecture idea and careful ablations, but the 33.36% speedup is attention-only, not end-to-end inference; deserves review with major revision. read the letter →

arxiv 2501.13629 v2 pith:OUZZXSXZ submitted 2025-01-23 cs.CL

Zhenghao Lin , Zihao Tang , Xiao Liu , Yeyun Gong , Yi Cheng , Qi Chen , Hang Li , Ying Xin
show 26 more authors
Ziyue Yang Kailai Yang Yu Yan Xiao Liang Shuai Lu Yiming Huang Zheheng Luo Lei Qu Xuan Feng Yaoxiang Wang Yuqing Xia Feiyang Chen Yuting Jiang Yasen Hu Hao Ni Binyang Li Guoshuai Zhao Jui-Hao Chiang Zhongxin Guo Chen Lin Kun Kuang Wenjie Li Yelong Shen Jian Jiao Peng Cheng Mao Yang
This is my paper · ORCID
classification cs.CL
keywords DiffQKVattentiondifferentialKVcompressionaugmentedquerygrouped-querylong-contextinferencecachesystem-domainlanguagemodelAIMICIUSbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sigma sets out to show that the standard habit of giving Query, Key, and Value the same head count and head dimension is wasteful, and that differential rescaling—compressing keys aggressively, values lightly, and queries more amply—can make long-context attention faster without hurting quality. The paper demonstrates through controlled ablations that performance drops far less when K heads are cut than when V heads are cut, that halving K head dimension is nearly free, and that widening the Q projection recovers capacity more efficiently than widening the feed-forward network. On 64k-token outputs, the resulting DiffQKV attention reduces measured attention-module cost by up to 33.36% versus grouped-query attention, approaching the theoretical 37.5% saving from a smaller key cache. If these results hold at serving level, the design gives a direct, training-time-only recipe for cheaper long-context inference, and the same model family claims strong performance on system-infrastructure tasks through a new benchmark.

What carries the argument

The load-bearing object is DiffQKV attention, a generalization of MHA, MQA, and GQA in which Q, K, and V can each have different numbers of heads ($n_q^h$, $n_k^h$, $n_v^h$) and different per-head dimensions ($d_q^h$, $d_k^h$, $d_v^h$). Two mechanisms make it work. First, differential KV compression: Sigma uses $n_k^h = 4$, $n_v^h = 16$ (and optionally $d_k^h = d_v^h/2$), shrinking the cached key matrix while keeping values nearly intact, and a GroupSharing operation that lets multiple Q heads attend to the same K or V head. Second, augmented Q: the Q projection is widened to roughly 1.5 times the hidden dimension ($d_q^h = 3072$ at hidden size 2048), adding uncached parameters that improve representation capacity. Because keys and values have different head counts, standard fused attention kernels cannot run directly; the paper supplies FlexHeadFA, a split/combine attention kernel whose address calculation uses the query head index to look up independent key and value head indices, removing the equal-head-count constraint. The theoretical savings come from the key cache: GQA caches $16+16=32$ head-vectors per layer and Sigma caches $4+16=20$, a 37.5% reduction in cache elements, which dominates long-context memory traffic.

What would settle it

Run a long-context decode (for example, a 64k-token prefix generating 64k tokens) in a standard serving engine with a fixed batch, measuring tokens per second for the Sigma architecture versus a GQA model trained with the same total parameter budget; if the end-to-end gain is much smaller than the attention-stage 33.36% because sampling, feed-forward, or scheduler costs dominate, the inference-speed claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that query, key, and value in attention need not be treated symmetrically, and that breaking the symmetry is a cheap efficiency win. Based on ablations at the 1B scale, Sigma argues that model quality is far more sensitive to compressing Value heads than Key heads: cutting K heads from 32 to 16 can even slightly improve average benchmark score (52.40 to 52.83), while cutting V heads by the same amount costs 0.66 points. It therefore recommends a differentially compressed KV design in which the number of K heads (and optionally K head dimension) is aggressively reduced while V heads stay numerous, combined with an augmented Q projection whose head dimension is about 1.5 times the hidden dimension, because Q is never cached and extra parameters there boost performance more than the same parameters in the feed-forward network. On this architecture, the key cache shrinks from 16 to 4 heads per layer, giving a theoretical long-context reduction of 37.5% in KV-cache elements; measured attention-module cost drops by up to 33.36% relative to GQA at 64k-token outputs, with quality at least matching the GQA baseline. The same architecture, pre-trained on 6T tokens including 19.5B tokens of system-domain data, yields a 10B model that the paper reports beats GPT-4 by up to 52.5% on the new AIMICIUS system-domain benchmark.

Load-bearing premise

The headline 33.36% efficiency gain is measured on the attention modules alone (KV cache load/store, attention compute, and augmented Q), and the claim that this translates to real inference-speed improvement assumes those attention costs dominate end-to-end decoding in long-context serving, which the paper does not directly measure.

Editorial extensions

If this is right

  • If DiffQKV's measured attention-stage savings carry over, long-context decoding with 64k output tokens can be served at roughly a third lower attention cost than GQA, with no quality penalty, because the key cache is 37.5% smaller.
  • The optimal-configuration rule found at 1B scale—aggressive K compression, mild V compression, and 1.5x query width—becomes a direct design recipe for training new efficient LLMs.
  • The augmented-Q result implies that, at equal added parameters, widening Q beats widening the FFN for benchmark accuracy at this scale, giving a new allocation rule for model capacity.
  • Adoption requires attention kernels that support unbalanced Q/K/V head counts; the paper's FlexHeadFA address-probing approach removes that barrier, so existing serving stacks can implement DiffQKV once kernels support it.
  • For system-domain work, the 19.5B-token corpus plus AIMICIUS yields a specialized 10B model that outperforms large general-purpose models on command generation, infrastructure retrieval, topology optimization, and KQL generation, suggesting domain data matters more than raw scale for these tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the paper's efficiency result is attention-stage evidence, not end-to-end proof: the natural next test is serving-level throughput, where sampling, the feed-forward network, and scheduler overhead may dilute the headline 33.36% gain.
  • The asymmetry principle may generalize beyond head counts: selective V cache fetching (only loading the V vectors with highest attention scores) is explored briefly, and one could combine it with layer-wise K compression schedules that the paper lists as future work.
  • Because the ablation study fixes a 1B model and 100B tokens, the optimal ratios (4 K heads, 16 V heads, 1.5x Q width) may not be scale-invariant; testing at 10B and beyond is an open question.
  • The AIMICIUS benchmark is Azure-centric, so the claimed system-domain advantage may not transfer to non-Azure infrastructures without additional data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Sigma, an LLM specialized for the system domain, and proposes DiffQKV attention, in which the number of Key heads is reduced more aggressively than the number of Value heads and the Query projection is augmented with an intermediate dimension that is 1.5 times the hidden dimension. The authors present 1B-scale ablations to justify the differential treatment of K and V, a theoretical and empirical efficiency analysis based on KET and CEET measurements, a custom FlashAttention extension called FlexHeadFA, a 6T-token pre-training pipeline that includes 19.5B system-domain tokens, and a new system-domain benchmark called AIMICIUS on which Sigma is reported to outperform GPT-4 by up to 52.5%. The paper claims up to a 33.36% inference-speed improvement over grouped-query attention in long-context scenarios.

Significance. If the claims are supported, the paper makes two useful contributions. First, DiffQKV attention is a simple and plausible architecture-level change: reducing K heads while keeping V heads wider is directly aligned with the observation that the Key cache is a substantial part of the memory footprint, and the inclusion of an augmented-Q projection to recover quality is a reasonable design. The KET/CEET experiments, the nsys-based kernel measurements, and the FlexHeadFA code snippets provide concrete evidence about the attention-stage cost, which is more than many architecture papers offer. Second, the system-domain dataset collection effort and the AIMICIUS benchmark address a relatively underserved evaluation area. These strengths are substantial. The significance is tempered, however, by the fact that the headline efficiency number is an attention-module microbenchmark rather than an end-to-end serving measurement, and by the absence of error bars and statistical tests for several architecture decisions that are made on small performance differences. The benchmark comparison also rests on a proprietary, unreleased evaluation set.

major comments (4)
  1. [Abstract, §1, §3.3, Figure 2] The headline claim of "up to 33.36% improvement in inference speed" is measured as the CEET sum of KV cache load/store, attention computation, and the augmented-Q projection, and is explicitly called the "total cost of attention layers" in §3.3. This is not an end-to-end decoding measurement. No tokens/s, latency under a serving framework, or attention-fraction profile is reported, so the paper does not establish that attention-layer costs dominate total decoding time in long-context serving. Figure 2 also shows that Sigma is slower than the standard model for short outputs, so the real-world gain depends on the output-length mix. The efficiency claim should either be re-labeled as an attention-stage cost reduction throughout the paper, or supported by end-to-end serving benchmarks and a decomposition of where time is spent.
  2. [§3.1, Eq. (2)] Equation (2) presents the 37.5% reduction as a theoretical result, but it is simply the arithmetic of the chosen configuration: the standard model has 16 K heads plus 16 V heads, while Sigma has 4 K heads plus 16 V heads, giving (32 - 20)/32 = 37.5%. This does not predict the optimal head configuration; it restates the design choice that was already selected in Section 2. The empirical KET and CEET measurements are independent and valuable, but the manuscript should describe Eq. (2) as a definitional consequence of the cache-size reduction rather than as independent theoretical validation of the architecture.
  3. [§2, Tables 1–4] Several design conclusions are drawn from overall-score differences of 0.1–1.5 points across nine benchmarks, but no error bars, repeated runs, or significance tests are reported. For example, Table 2 shows that halving the K dimension changes the score by -0.08 and +0.26 in two settings, which is within the range one might expect from evaluation noise, and Table 3 selects d_h^q = 3072 even though the scores at 3072, 4096, and 5632 are non-monotonic (53.38, 52.93, 53.07). The statement in Observation 3 that augmented Q "consistently" improves performance would be much stronger with confidence intervals or paired significance tests. This is load-bearing because the final architecture is justified by these specific comparisons.
  4. [§5, Table 6, Appendix C] The AIMICIUS benchmark is introduced as the first comprehensive system-domain benchmark and is used to claim that Sigma "significantly outperforms" GPT-4 by up to 52.5%, but the benchmark is not released, several test sets are very small (e.g., 43 test cases for NL2KQL), and metrics such as Exact Match, Success Ratio, and Calibration Score depend on threshold choices that are not analyzed. There is also no contamination analysis between the 19.5B system-domain pre-training corpus and benchmark data that are drawn from overlapping sources such as Azure logs, documentation, and system-related websites. For the benchmark claims to be assessable, the authors should provide a release plan or detailed evaluation protocol, confidence intervals, human agreement for subjective components, and contamination checks.
minor comments (6)
  1. [§3.3] There is a typo: "soly" should be "solely" in the KET results paragraph.
  2. [Abstract] The abstract contains typos such as "varing" and "efficienct"; these should be corrected before publication.
  3. [Appendix D] The benchmark name is spelled inconsistently as AIMICIUS, AIM ICIUS, and AIM ICIOUS; please unify the spelling throughout.
  4. [§5 and Appendix H.1] The main text says the pre-training data total 6T tokens, while Appendix H.1 lists four phases of 3.5T + 1.0T + 1.0T + 1.0T = 6.5T tokens; this discrepancy should be reconciled.
  5. [Figure 1 and Figure 2] The subfigures would be easier to interpret with explicit axis labels, legends, and a legend entry for the grey crossover line; the current panels are dense and hard to read.
  6. [§3.2 and Appendix E] The shorturl links for FlexHeadFA and the KQL documentation are not permanent references; please use stable URLs or repository DOIs.

Circularity Check

1 steps flagged · score 2.0 of 10

The 37.5% theoretical bound restates the chosen head counts, but the headline 33.36% speedup is measured directly, so circularity is minor.

  1. self definitional [Section 3.1, Eq. (2), main text]
    "As the size of cache increases, the reduction rate r of the total cost of KV cache operations converges toward a theoretical value as shown in Eq. (2). r = lim_{s→+∞} (L_GQA − L_Sigma)/L_GQA = (32 − 20)/32 = 37.5%."

    The 37.5% figure is pure arithmetic on the chosen head counts: GQA has 16 K heads plus 16 V heads (32 units), while Sigma has 4 K heads plus 16 V heads (20 units). Since Eq. (1) defines KV-cache cost as proportional to b·s·(nh_k·dh_k + nh_v·dh_v), the ratio (32−20)/32 is already fixed by the architecture selected in Section 2. Thus the 'theoretical bound' does not independently predict the efficiency gain; it restates the design decision. This is a minor self-definitional overclaim, not the source of the headline number. The empirical 33.36% result is a separate CEET measurement, so the central efficiency claim is not circular.

full rationale

The central architecture claim is supported by direct ablations and by measured CEET/KET comparisons between Sigma and the GQA-based standard model; those numbers are observations, not quantities derived from the claim. No fitted parameter is relabeled as a prediction: the chosen configurations are selected from scratch-trained ablations and then evaluated independently. The AIMICIUS benchmark is constructed by the authors and used to compare against external baselines such as GPT-4, DeepSeek, Qwen, and Llama; a self-constructed benchmark is a validity concern, not a circularity. No load-bearing self-citation chain or imported uniqueness theorem was found. The only mildly circular element is the 37.5% theoretical bound in Eq. (2), which merely restates the selected head counts; because the empirical 33.36% speedup is measured directly, this does not undermine the main efficiency comparison. Overall, no significant circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

DiffQKV rests on small-scale ablations that are assumed to transfer to the 10B/6T model, on a linear cost model for KV cache, and on an author-built benchmark with no public data. The augmented-Q size and K/V head counts are selected from the same experiments used to justify them, so they are best treated as chosen hyperparameters rather than derived constants.

free parameters (3)
  • K-head count 4 (75% compression) = nh_k=4 vs nh_v=16
    Chosen from Tables 1 and 13; the claim that K can be compressed more aggressively than V depends on this specific ratio, and it is selected by observing the same benchmarks used to justify it.
  • Augmented Q intermediate dimension delta = 3072 for 1.5B, 6144 for 10B (1.5x hidden)
    Selected from Table 3/4 as 'optimal' after comparing 3072, 4096, and 5632; no error bars and no out-of-sample validation for the final choice.
  • V-head count 16 = 16
    The differential design keeps 16 Value heads while reducing Key heads to 4; the choice of which side to compress comes from Table 1 results.
assumptions (4)
  • domain assumption KV-cache and attention cost are proportional to cache element count, with linear cost model Eq. (1).
    Eq. (1) posits L = alpha * [b*s*(nh_k*dh_k + nh_v*dh_v)] + beta; the 37.5% theoretical bound in Eq. (2) follows from this linear model and memory-bound inference assumptions from Shazeer 2019.
  • domain assumption Compressibility ordering observed at 1B scale and 100B tokens transfers to the 1.5B/10B models trained on 6T tokens.
    Final configurations in Appendix B.5 are based on small-scale ablations in Section 2; no at-scale ablation is reported for the 10B model.
  • domain assumption Attention scores are more than 95% sparse in long-context inference, motivating differential K compression and selective V fetching.
    The paper cites Zhang et al. 2024 for the sparsity claim and uses it in Section 2 and Appendix B.2; selective V fetching (Table 8) is not used in the final model but is part of the argument that V is more important than K.
  • domain assumption The AIMICIUS benchmark measures the intended system-domain abilities without training/test overlap.
    The benchmark is constructed by the authors from Azure-related data and used for evaluation; no leakage analysis, external audit, or human evaluation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models." pith.science (2026). https://pith.science/paper/OUZZXSXZ

@misc{pith2026250113629,
  author       = {Pith},
  title        = {Pith review of: Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OUZZXSXZ}},
  note         = {Machine review of arXiv:2501.13629}
}
read the original abstract

We introduce Sigma, an efficient large language model specialized for the system domain, empowered by a novel architecture including DiffQKV attention, and pre-trained on our meticulously collected system domain data. DiffQKV attention significantly enhances the inference efficiency of Sigma by optimizing the Query (Q), Key (K), and Value (V) components in the attention mechanism differentially, based on their varying impacts on the model performance and efficiency indicators. Specifically, we (1) conduct extensive experiments that demonstrate the model's varying sensitivity to the compression of K and V components, leading to the development of differentially compressed KV, and (2) propose augmented Q to expand the Q head dimension, which enhances the model's representation capacity with minimal impacts on the inference speed. Rigorous theoretical and empirical analyses reveal that DiffQKV attention significantly enhances efficiency, achieving up to a 33.36% improvement in inference speed over the conventional grouped-query attention (GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various sources, including 19.5B system domain data that we carefully collect and 1T tokens of synthesized and rewritten data. In general domains, Sigma achieves comparable performance to other state-of-arts models. In the system domain, we introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates remarkable performance across all tasks, significantly outperforming GPT-4 with an absolute improvement up to 52.5%.

Figures

Figures reproduced from arXiv: 2501.13629 by the authors.

Figure 1
Figure 1. KET comparison of FlexHeadFA between Standard model(STD) and SIGMA. (a) Output Length = 2k. (b) Output Length = 4k. (c) Output Length = 8k. (d) Absolute CEET Improvment. (e) Output Length = 16k. (f) Output Length = 32k. (g) Output Length = 64k. (h) Relative CEET Improvment [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Comparison of total CEET cost between Standard model(STD) and SIGMA. The gray dashed line indicates where the inference costs of both models are equal. As the output length increases, this intersection point moves progressively earlier. n h v , and n h q is an integer multiple of n h k and n h v , which could not be directly applied to SIGMA. To address this issue, we introduce FlexHeadFA3 , which leverages address … view at source ↗
Figure 3
Figure 3. Overview of our proposed method for differential rescaling of QKV, compared alongside Multi-Head Attention (MHA), Multi-Query Attention (MQA), and Grouped Query Attention (GQA). Specifically, our method involves: (1) differentially compressed KV: applying more aggressive compression on the number of K heads and their dimensions than on the V components, which more significantly reduces the size of K cache. We can al… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: CEET comparison of augmented Q between Standard model(STD) and SIGMA. From (a) to (f), the output length increases progressively from 2k to 64k tokens. Here, we report detailed KET results and CEET results of three modules: KV Cache, Attention Computation, and Augmente…
Figure 5
Figure 5. Figure 5: CEET comparison of KV cache between Standard model(STD) and SIGMA. From (a) to (f), the output length increases progressively from 2k to 64k tokens. CEET Results - Augmented Q. The augmented Q module is not inherently included within the standard model(STD). In this ex…
Figure 6
Figure 6. Figure 6: CEET comparison of attention computation between Standard model(STD) and SIGMA. From (a) to (f), the output length increases progressively from 2k to 64k tokens. CEET Results - Attention Computation. CEET results of the attention computation are demonstrated in [PITH_…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hardware-Efficient Attention for Fast Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.

Reference graph

Works this paper leans on

53 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    A., Awan, A

    Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H. H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H. S., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V ., Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., H...

    arXiv

  2. [4]

    This phase encompassed a total training volume of 1 trillion tokens, during which the learning rate was reduced to 20% of its peak value, i.e., 3e-5. Finally, in the annealing phase, we utilized General Dataset III, which was selected for its highest quality, along with meticulously chosen synthesized and rewritten pre-training data and the system domain ...

    work page 2024

  3. [5]

    Stable lm 2 1.6 b technical report

    Bellagente, M., Tow, J., Mahan, D., Phung, D., Zhuravin- skyi, M., Adithyan, R., Baicoianu, J., Brooks, B., Cooper, N., Datta, A., et al. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834,

    arXiv

  4. [6]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150,

    arXiv 2004

  5. [8]

    V ., R´e, C., and Mirhoseini, A

    Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V ., R´e, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,

    arXiv

  6. [9]

    T., Li, Y ., Lundberg, S., et al

    Bubeck, S., Chandrasekaran, V ., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y . T., Li, Y ., Lundberg, S., et al. Sparks of artificial general intel- ligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712,

    arXiv

  7. [10]

    9 SIGMA : Differential Rescaling of Query, Key and Value for Efficient Language Models Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  8. [11]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

    arXiv 1905

Show all 53 references
  1. [14]

    DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts lan- guage models

    Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y ., et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts lan- guage models. arXiv preprint arXiv:2401.06066,

    arXiv

  2. [15]

    Cerebras-gpt: Open compute- optimal language models trained on the cerebras wafer- scale cluster

    Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al. Cerebras-gpt: Open compute- optimal language models trained on the cerebras wafer- scale cluster. arXiv preprint arXiv:2304.03208,

    arXiv

  3. [16]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  4. [17]

    ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools

    GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., et al. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools. arXiv preprint arXiv:2406.12793,

    arXiv

  5. [18]

    H., Ivison, H., Magnusson, I., Wang, Y ., et al

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y ., et al. Olmo: Accelerating the science of lan- guage models. arXiv preprint arXiv:2402.00838,

    arXiv

  6. [19]

    Gunasekar, S., Zhang, Y ., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644,

    arXiv

  7. [20]

    Memory-efficient transformers via top-k attention

    Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J. Memory-efficient transformers via top-k attention. arXiv preprint arXiv:2106.06899,

    arXiv

  8. [21]

    LM-infinite: Zero-shot extreme length generalization for large language models

    Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y ., Ji, H., and Wang, S. LM-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L...

    2024

  9. [22]

    and Zhai, J

    He, J. and Zhai, J. FastDecode: High-throughput GPU- efficient LLM serving using heterogeneous pipelines. arXiv preprint arXiv:2403.11421,

    arXiv

  10. [23]

    Measuring mas- sive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,

    2009 arXiv

  11. [24]

    MiniCPM: Unveiling the potential of small language models with scalable train- ing strategies

    Hu, S., Tu, Y ., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y ., Huang, Y ., Zhao, W., et al. MiniCPM: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint arXiv:2404.06395, 2024a. Hu, X., Xiong, T., Yi, B., Wei, Z., Xi...

    arXiv

  12. [25]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    10 SIGMA : Differential Rescaling of Query, Key and Value for Efficient Language Models Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  13. [26]

    Datacomp- lm: In search of the next generation of training sets for lan- guage models

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., et al. Datacomp- lm: In search of the next generation of training sets for lan- guage models. arXiv preprint arXiv:2406.11794,

    arXiv

  14. [27]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y ., and Zhang, Y . Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124,

    2007 arXiv

  15. [28]

    Scissorhands: Exploit- ing the persistence of importance hypothesis for LLM KV cache compression at test time

    Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V ., Xu, Z., Kyrillidis, A., and Shrivastava, A. Scissorhands: Exploit- ing the persistence of importance hypothesis for LLM KV cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024a. Liu, Z., Z...

    arXiv

  16. [29]

    H., Cao, Q., Horton, M., Jin, Y ., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al

    Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y ., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al. OpenELM: An efficient language model family with open-source training and inference framework.arXiv preprint arXiv:2404.14619,

    arXiv

  17. [30]

    L., Bari, M

    Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Bi- derman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786,

    arXiv

  18. [32]

    Fast transformer decoding: One write-head is all you need

    Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150,

    1911 arXiv

  19. [33]

    You only cache once: Decoder-decoder architectures for language models

    Sun, Y ., Dong, L., Zhu, Y ., Huang, S., Wang, W., Ma, S., Zhang, Q., Wang, J., and Wei, F. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254,

    arXiv

  20. [34]

    W., Chowdhery, A., Le, Q

    Suzgun, M., Scales, N., Sch ¨arli, N., Gehrmann, S., Tay, Y ., Chung, H. W., Chowdhery, A., Le, Q. V ., Chi, E. H., Zhou, D., , and Wei, J. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261,

    arXiv

  21. [37]

    F., and Gardner, M

    Welbl, J., Liu, N. F., and Gardner, M. Crowdsourc- ing multiple choice science questions. arXiv preprint arXiv:1707.06209,

    arXiv

  22. [38]

    Deepseek-prover-v1

    Xin, H., Ren, Z., Song, J., Shao, Z., Zhao, W., Wang, H., Liu, B., Zhang, L., Lu, X., Du, Q., et al. Deepseek-prover-v1. 5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search. arXiv preprint arXiv:2408.08152,

    arXiv

  23. [39]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671,

    arXiv

  24. [40]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  25. [41]

    TinyLlama: An open-source small language model

    Zhang, P., Zeng, G., Wang, T., and Lu, W. TinyLlama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024a. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V ., et al. Opt: Open pre-trained transform...

    arXiv

  26. [42]

    H2O: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y ., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y ., R´e, C., Barrett, C., et al. H2O: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024b. Zhou, Z., Ning, ...

    arXiv

  27. [43]

    12 SIGMA : Differential Rescaling of Query, Key and Value for Efficient Language Models A. Related Work Over the past few years, Large Language Models (LLMs) have exerted a considerable impact across various domains (Bubeck et al., 2023; Jiang et al., 2023a; GLM et al., 2024; ...

    2023

  28. [44]

    reduces the number of K and V heads in the attention layer to the same extent by organizing Query (Q) heads into groups, with each group sharing a single K/V head. The same applies to methods based on KV cache eviction, which carefully select a fixed number of tokens and cache...

    2020

  29. [45]

    MQA is a variant of the standard Multi-Head Attention (MHA) mechanism, where all query heads share a single key head and value head

    are the two most widely-used methods. MQA is a variant of the standard Multi-Head Attention (MHA) mechanism, where all query heads share a single key head and value head. GQA further generalizes MQA by using an intermediate number of shared key and value heads. Post-training s...

    2023

  30. [47]

    This operation significantly enhances inference efficiency by reducing memory usage

    The ablation studies of the model performance when only selectively loading the V vectors corresponding to the highest attention scores for approximate calculation. This operation significantly enhances inference efficiency by reducing memory usage. The number of Q heads is 32...

    2023

  31. [48]

    Sel.V-top100

    Model Overall Commonsense & Comprehension Continued LM Hella. ObQA Wino. ARC. PIQA SciQ Bool. Logi. LAMB. MHA 52.40 55.6 37.6 57.6 36.0 73.9 85.5 59.6 28.9 36.8 + AugQ (dh q =5632) 53.03 (↑0.63) 57.4 38.0 57.9 39.4 72.9 85.9 60.1 27.3 38.3 GQA (nh k=nh v=16) 52.14 55.1 39.6 56...

    2021

  32. [49]

    We utilize the same vocabulary as Llama3 (Dubey et al., 2024), with a vocabulary size of 128k

    For SIGMA -1.5B, we set dh q = 3072, and for SIGMA -10B, we set dh q = 6144, corresponding to 1.5 times the dimension of the hidden state, so as to extend the representational space of Q. We utilize the same vocabulary as Llama3 (Dubey et al., 2024), with a vocabulary size of ...

    2024

  33. [51]

    Both models exhibit notable performance improvements after fine-tuning, with absolute improvements of 1.5 and 6.4 on the Accuracy metric, respectively. H.3. Additional Results in General Domain Problem-Solving Tasks. We also evaluate on three general problem-solving benchmarks...

    2020

  34. [52]

    and MBPP (Austin et al., 2021), and two math problem datasets MATH (5-shot) (Hendrycks et al.,

    2021

  35. [53]

    The evaluation results on various general, coding and math problem-solving benchmarks are shown in Table

    and GSM8K (5-shot) (Cobbe et al., 2021). The evaluation results on various general, coding and math problem-solving benchmarks are shown in Table

    2021

  36. [2016]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Penedo, G., Kydl´ıˇcek, H., Lozhkov, A., Mitchell, M., Raffel, C., V on Werra, L., Wolf, T., et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557,

    arXiv

  37. [2017]

    Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark

    Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574,

    arXiv

  38. [2018]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  39. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  40. [2020]

    Brandon, W., Mishra, M., Nrusimha, A., Panda, R., and Kelly, J. R. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981,

    arXiv

  41. [2021]

    D., McAleer, S., Jiang, A

    Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for math- ematics. arXiv preprint arXiv:2310.10631,

    arXiv

  42. [2022]

    S., Love, J., et al

    11 SIGMA : Differential Rescaling of Query, Key and Value for Efficient Language Models Team, Gemma, Mesnard, T., Hardin, C., Dadashi, R., Bhu- patiraju, S., Pathak, S., Sifre, L., Rivi`ere, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini re- search and te...

    arXiv

  43. [2023]

    GQA: Training general- ized multi-query transformer models from multi-head checkpoints

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebron, F., and Sanghai, S. GQA: Training general- ized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901,

    2023

  44. [2024]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

    arXiv

  45. [2048]

    To evaluate the model’s performance on different experimental settings, we employed the following benchmarks: HellaSwag (Hella.) (Zellers et al., 2019), OpenBookQA (ObQA) (Mihaylov et al., 2018), WinoGrande (Wino.) (Sakaguchi et al., 2021), ARC Challenge (ARC.) (Clark et al., ...

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.