Pith. sign in

REVIEW 5 major objections 5 minor 43 references

Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Under a fixed KV-cache budget, routing each token to a weight-shared attention expert chosen by learned importance improves language-model quality over static grouped-query attention.

desk verdict A useful KV-grouping variant with large clean gains over static GQA, but the router's training signal is self-referential and key dynamic baselines are missing. read the letter →

arxiv 2506.13541 v1 pith:4VZEGWFN submitted 2025-06-16 cs.CL cs.LG

Guanghui Song , Dongping Liao , Yiren Zhao , Kejiang Ye , Cheng-zhong Xu , Xitong Gao This is my paper · ORCID
classification cs.CLcs.LG
keywords KVcachecompressionmixtureofexpertstoken-wiseroutinggroupedqueryattentioncausallanguagemodelingdynamicinferenceefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the KV cache of a causal language model can be compressed as aggressively as grouped-query attention, but with less quality loss, by making the grouping decision per token instead of per layer. mixSGA routes each token to one of several attention "experts" whose key and value projections are shared but whose KV groups differ in size, so important tokens keep many KV heads while unimportant tokens get few, and no token is dropped. The router is trained so that the same assignment is made during prefill and during decode, and the claim is that under the same average cache size the method improves instruction-following quality (Llama3.1-8B average ROUGE-L rises from 19.97 to 27.08) and lowers perplexity on continued pretraining (22.66 to 20.46 on Wikitext-2 with TinyLlama-1.1B). If right, the practical upshot is cheaper, longer-context inference that does not pay for memory savings by evicting information.

What carries the argument

The load-bearing mechanism is token-wise expert-choice routing with heterogeneous expert capacities, welded to weight-shared grouped-attention projections. A trainable linear layer maps each token embedding to $E$ sigmoid scores $S(x)=\sigma(x\phi+\beta)$; during prefill and training, expert $e$ receives the top $\lceil \rho_e L \rceil$ tokens not already claimed by earlier experts, producing a hard, exclusive one-hot mask $m_e(x)$; during decoding, each token simply picks $\arg\max_e S(x)$. To make those two assignments coincide, the auxiliary loss $\mathcal{L}_{\mathrm{aux}}$, a cross-entropy between $S(x)$ and the prefill assignment, is added to the language-modeling loss at every layer. The KV side uses one shared set of key and value projections for all experts and obtains expert $e$'s representation by averaging heads into groups of size $2^e$, so switching experts changes only memory and arithmetic, not parameters. That combination is what lets the paper claim a per-token allocation of KV size proportional to importance without adding parameters or discarding tokens.

What would settle it

Take a trained mixSGA model and, on held-out data, compare the router's assignments against an oracle importance measure such as the per-token perplexity change when a token's KV heads are averaged away, as in the paper's own Figure 1. If the router's chosen expert for a token does not track that oracle ordering, then the reported gains must come from something other than importance-aware allocation, and the claim that dynamic token importance drives the improvement fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that token importance in a sequence is not just a ranking, but a continuous spectrum that shifts with context, and the right response is not to evict low-importance tokens but to give them a coarser KV representation. mixSGA realizes this with a set of heterogeneous grouped-attention experts: expert e averages the shared key and value projections into groups of size 2^e, so its KV cache per token is H/2^e heads, and a learned router assigns each token to exactly one expert. Prefill assigns tokens greedily by capacity ratios, decode uses an argmax of the same scoring function, and an auxiliary cross-entropy loss is the only training signal for the router, because the hard one-hot masks block gradients from the language-modeling loss. With a 3:1:6 capacity split among group sizes 1, 2, and 4 (50% of the original cache), the paper reports consistent improvements over GQA across OPT, Llama3, and Gemma2 models on instruction-following tasks and over GQA and CLA in continued pretraining, and shows the method combines with H2O-style eviction.

Load-bearing premise

The routing assignments that decide which tokens get fine or coarse KV representations are learned only from the auxiliary consistency loss, since the hard one-hot masks block any gradient from the language-modeling loss; the whole method therefore rests on that single signal learning a meaningful, transferable ranking of token importance.

Editorial extensions

If this is right

  • Under the same average KV cache size as a GQA baseline, mixSGA raises average ROUGE-L across five instruction-following datasets, with the largest reported jump on Llama3.1-8B from 19.97 (GQA) to 27.08 (mixSGA 3:1:6).
  • In continued pretraining on TinyLlama-1.1B, mixSGA lowers Wikitext-2 perplexity from 22.66 (GQA) to 20.46 and matches or beats GQA and CLA on HellaSwag, PIQA, Winogrande, and ARC-Challenge.
  • The method composes with KV eviction: with H2O keep ratios from 80% down to 20%, mixSGA combined with H2O beats GQA with H2O on average accuracy and perplexity at every tested ratio.
  • The routing consistency loss is load-bearing: removing the auxiliary loss drops average ROUGE-L on Gemma2-2B from 21.20 to 7.35, and replacing the learned router with a random router costs about 3.5 points on the same average.
  • The optimal expert mix under a 50% KV budget is consistently the 3:1:6 ratio of group sizes 1, 2, and 4, which allocates 30% of tokens to fine, 10% to medium, and 60% to coarse representation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the learned importance scores are genuinely transferable, they could serve as a cheap token-importance signal for other memory-saving devices the paper does not test, such as prefix caching, prompt compression, or deciding where to place tokens in hierarchical caches; the paper only demonstrates routing learned jointly with fine-tuning or continued pretraining.
  • Because the routing weights are trained only by the auxiliary loss, the approach is easy to probe causally: freeze the model, supervise the router directly with an oracle importance ranking, and compare quality against the auxiliary-loss-trained router, a clean way to measure how much of the gain is routing accuracy versus the richer token-importance prior itself.
  • The paper fixes expert group sizes to powers of two (1, 2, 4); nothing in the weight-sharing design forces that, so one could interpolate to fractional or per-head granularities or condition capacity ratios on sequence length, which might extend the budget-linear scaling shown in Figure 3 to more extreme compression.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes mixSGA, an attention mechanism that assigns each token to one of several weight-shared KV-group experts with different group sizes (e.g., group sizes 1, 2, 4) via a learned router, while retaining all tokens instead of evicting low-importance ones. A progressive top-k routing is used during prefill/training (Eq. 2), and decode-time routing uses the argmax of the same score function (Eq. 3), with an auxiliary loss (Eq. 4) intended to align the two. Experiments on instruction-following (Dolly-15k fine-tuning) and continued pretraining (MiniPile on TinyLlama-1.1B) report consistent improvements in ROUGE-L and perplexity over GQA and CLA under matched KV budgets, plus compatibility with H2O eviction.

Significance. If the reported results are reproducible, mixSGA offers a simple and parameter-efficient way to allocate KV resources non-uniformly across tokens while preserving full context, with only a modest decoding-throughput overhead (~3-4%). The paper covers multiple model families (OPT, Llama3, Gemma2, TinyLlama), reports memory and FLOP overheads, and includes ablations over expert ratios, number of experts, and the auxiliary loss. However, the central claim that routing is guided by 'learned importance scores' is not supported by the training objective as written: the router receives no gradient from the language-modeling loss and its auxiliary target is derived from the router's own scores. The same-budget comparisons also depend on the decode-time argmax maintaining the predefined expert ratios, which is not verified on held-out data. Several dynamic KV-optimization baselines cited in the motivation are never compared, and no uncertainty estimates are provided.

major comments (5)
  1. [Section 3.3, Eq. (4)] The auxiliary loss L_sce is never defined, so the training signal for the router is not fully specified and the method is not reproducible as written. Moreover, the target argmax T(x) in Eq. (4) is itself the routing produced from S(x) by the progressive procedure in Eq. (2), making the objective self-referential: it only encourages the per-token argmax to agree with the sequential top-(rho_e L) assignment. No term in Eq. (5) rewards routing based on token importance, so the abstract's and Section 3.2's claim of 'learned importance scores' is not justified by the given training objective. Please define L_sce, and either add an external importance-supervision signal or empirically demonstrate, for example by correlating expert assignments with the per-token sensitivity measure used in Figure 1, that the learned routing is indeed importance-aligned.
  2. [Section 3.1-3.2; Tables 1-3] Because the masks in Eq. (2) are hard indicators and Eq. (3) uses argmax, no gradient from L_model in Eq. (5) reaches the router parameters phi and beta; the router is trained only by the auxiliary consistency loss. This is an assumption about training dynamics, and the paper never verifies that the trained router maintains the intended 3:1:6 expert proportions at decoding time on the evaluation sets. Table 6 shows that without the auxiliary loss the ratios become nearly uniform, but the paper does not report the measured ratios for the full model on held-out data. If the decode-time ratios drift, the 'same KV budget' comparisons in Tables 1-3 are not guaranteed to hold. Please report the actual expert-assignment distribution for the trained router on the test sets, or explicitly enforce the target ratios at decoding (e.g., via capacity-limited sampling) and describe how the KV budget is then computed.
  3. [Section 4.1-4.2; Section 2.1] The paper motivates mixSGA against DynamicKV [40] and PyramidKV [5] and in Section 1 claims 'superior efficiency and performance over static and dynamic baselines', yet none of the experiments compare against these or other token-level KV-compression methods. The only dynamic baseline is H2O, which is applied as an eviction post-processing on top of both GQA and mixSGA. Please either add comparisons to at least one of the cited dynamic methods (DynamicKV or PyramidKV) under matched budgets, or revise the contributions and abstract to claim superiority over static-grouping baselines only.
  4. [Tables 1-3] All reported results are from single runs, with no standard deviations, confidence intervals, or multiple seeds. Given that the fine-tuning uses a relatively small Dolly-15k subset (14,000 training samples) and the continued-pretraining run is one epoch, differences on the order of a few tenths of a percent (e.g., Table 2: HellaSwag 37.00 vs 36.70; ARC-E 54.84 vs 54.92) are within plausible random variation. Please report means and standard deviations over at least three independent seeds for the main comparisons, and indicate how many seeds were used for each reported number.
  5. [Section 4.1 vs Appendix C.2] The main text states that the grid search for the optimal expert density ratio was conducted on OPT-355M, while Appendix C.2 reports the grid search on Gemma2-2B. This inconsistency must be corrected. The paper should also explain how the same 3:1:6 ratio is justified across different architectures and whether the optimal ratio was re-tuned per model family.
minor comments (5)
  1. [Table 1] In the Llama3.2-3B row, the GQA Vicuna DeepSeek-v3 feedback score is listed as 35.4, which is an order of magnitude larger than all neighboring values (typically 2-5); this appears to be a typo for 3.54.
  2. [Section 3.4] The group-size notation is inconsistent: the text says 'group size of 2^e' and 'H/2^e groups', but the example with H=4 and E=3 gives G1 as four groups of size 1, G2 as two groups of size 2, and G3 as one group of size 4. This corresponds to group sizes 2^(e-1), not 2^e. Please fix the notation or the indexing.
  3. [Appendix C.2] The sentence 'while efficiently handling less critical tokens with the third [expert]' describes the third expert as having 'group size 3', but the method defines the third expert with group size 4; correct the typo.
  4. [Eq. (2)] The indicator notation 1[...] is used without defining its output; please state explicitly that it returns a zero-one tensor of the same shape as its argument, and clarify the behavior when the ceiling operation makes the sum of capacities exceed L.
  5. [References] Reference [33] is cited as a general KV-cache reference, but the cited work ('KV-cache: A scalable high-performance web-object cache for manycore') is about a web-object cache, not the transformer key-value cache; this citation is misplaced.

Circularity Check

1 steps flagged · score 3.0 of 10

Router 'importance' scores are trained only for self-consistency with a greedy partition of their own scores; the headline benchmark results remain externally evaluated.

  1. self definitional [Section 3.1 Eq. (2), Section 3.2 Eq. (3), Section 3.3 Eq. (4)]
    "me(x) =1[top⌈ρeL⌉( S(x)Qe−1 i=1 (1−m i(x)) )] ... Tprefill(x) =PE e=1me(x), Tdecode(x) =1[arg max(S(x)) =e]. ... To align arg maxS(x) with the expert assignment arg max Tprefill(x), we introduce the following consistency loss ... Laux(x) =Lsce(S(x), arg maxT(x))."

    The auxiliary loss target is the argmax of the prefill routing masks me(x), and those masks are themselves constructed by top-⌈ρeL⌉ thresholding of S(x). Thus Eq. (4) trains the router to match a piecewise-constant function of its own scores. Because me(x) is an indicator, gradients from Lmodel cannot pass through Eq. (9) into φ or β, so no external token-importance objective ever reaches the router. The 'learned importance scores' are therefore self-consistency scores by construction: S is trained to agree with the greedy partition that S itself defines. This does not make the empirical ROUGE-L/perplexity comparisons circular, since those use held-out external benchmarks, but it means the claimed importance mechanism is not grounded in an independent importance measure.

full rationale

The only load-bearing circularity is the router's training target. The token-expert scores S(x)=σ(xφ+β) define the hard masks in Eq. (2); the prefill assignment is the argmax of those masks; and Eq. (4) trains S to match that self-generated argmax. With hard indicators blocking Lmodel gradients to the router, the 'importance' signal is entirely self-referential. However, the paper's central claims are tested on held-out external benchmarks: Dolly/Self-Instruct/Vicuna/Super-Natural/Unnatural ROUGE-L and DeepSeek-V3 feedback, HellaSwag/PIQA/Winogrande/ARC/Wikitext-2, all with static GQA/CLA baselines. Those results are not fitted to the router's self-consistency target, so the overall conclusion retains independent empirical content. I found no load-bearing self-citation chain and no imported uniqueness theorem. Two non-circularity concerns are worth noting: Lsce in Eq. (4) is never defined, so the training loss is incompletely specified; and the paper does not report the actual decode-time expert distribution, so the 'same KV budget' claim depends on the unverified assumption that argmax routing follows the configured 3:1:6 ratios at inference. Both are correctness/robustness issues rather than circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities; its load-bearing assumptions are the expressiveness of the linear router, the efficacy of the self-consistency auxiliary loss, and the sufficiency of weight-sharing across grouping granularities.

free parameters (3)
  • Expert density ratios rho (3:1:6) = x=0.3 (group size 1), y=0.1 (group size 2), z=0.6 (group size 4)
    Selected by grid search on OPT-355M (Section 4.1) and confirmed on Gemma2-2B (Appendix C.2); determines the KV budget allocation across experts.
  • Auxiliary loss weight alpha = not reported
    Equation (5) introduces alpha but the text never gives its value; the router training depends on this hyperparameter.
  • Router linear layer weights phi and bias beta = learned, values not reported
    The scoring function S(x)=sigma(x phi + beta) is trained only by the auxiliary loss; initial values are He-initialized.
assumptions (3)
  • domain assumption Token importance is expressible as a linear function of token embeddings followed by sigmoid (eq. 1), and this captures the dynamic spectrum of token importance.
    Section 3.1 introduces the scoring function; no evidence is given that a linear score is sufficient.
  • ad hoc to paper The auxiliary consistency loss (eq. 4) can align decode-time argmax routing with the prefill-time top-k assignment.
    Section 3.3; the ablation shows removing the loss causes near-random assignments, so the method depends on this alignment being achievable.
  • domain assumption Weight-sharing across experts does not hurt: the same key/value projections can be averaged into groups of size 1, 2, and 4 without loss.
    Section 3.4 states weights are shared across all experts; the paper does not compare to separate projections.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization." pith.science (2026). https://pith.science/paper/4VZEGWFN

@misc{pith2026250613541,
  author       = {Pith},
  title        = {Pith review of: Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4VZEGWFN}},
  note         = {Machine review of arXiv:2506.13541}
}
read the original abstract

Transformer models face scalability challenges in causal language modeling (CLM) due to inefficient memory allocation for growing key-value (KV) caches, which strains compute and storage resources. Existing methods like Grouped Query Attention (GQA) and token-level KV optimization improve efficiency but rely on rigid resource allocation, often discarding "low-priority" tokens or statically grouping them, failing to address the dynamic spectrum of token importance. We propose mixSGA, a novel mixture-of-expert (MoE) approach that dynamically optimizes token-wise computation and memory allocation. Unlike prior approaches, mixSGA retains all tokens while adaptively routing them to specialized experts with varying KV group sizes, balancing granularity and efficiency. Our key novelties include: (1) a token-wise expert-choice routing mechanism guided by learned importance scores, enabling proportional resource allocation without token discard; (2) weight-sharing across grouped attention projections to minimize parameter overhead; and (3) an auxiliary loss to ensure one-hot routing decisions for training-inference consistency in CLMs. Extensive evaluations across Llama3, TinyLlama, OPT, and Gemma2 model families show mixSGA's superiority over static baselines. On instruction-following and continued pretraining tasks, mixSGA achieves higher ROUGE-L and lower perplexity under the same KV budgets.

Figures

Figures reproduced from arXiv: 2506.13541 by the authors.

Figure 1
Figure 1. Token importance is dynamic and has a wide spectrum. We replace each token’s forward pass with multi-query attention on Llama3.1-8b by averaging key and value states across heads, and see the sequence perplexity changes on a sample sequence of WikiText-2. of processed KV pairs. Nevertheless, GQA assumes static group sizes and allocates resources uniformly, disregarding variations in token importance. Some works have… view at source ↗
Figure 2
Figure 2. High-level overview of mixSGA. During training, the router learns to compute assignment scores for each token-expert pair. These scores are utilized to sequentially route ρe tokens to the e th expert, ordered by their computational and memory costs, while leaving the remaining tokens to be routed to other experts. The experts consist of a set of key and value projections that generate state representations at varyin… view at source ↗
Figure 4
Figure 4. System prompt template for DeepSeek-V3 feedback evaluation. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 20 canonical work pages

  1. [40]

    Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, and Liang Ding. 2024. DynamicKV: Task-aware adaptive kv cache compression for long context llms.Preprint, arXiv:2412.14838

    arXiv 2024

  2. [5]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. PyramidKV: Dynamic kv cache compression based on pyramidal information funneling.Preprint, arXiv:2406.02069

    arXiv 2024

  3. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai

    work page

  4. [2]

    Dosovitskiy Alexey. 2020. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv: 2010.11929

    arXiv 2020

  5. [3]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. InThirty-Fourth AAAI Conference on Artificial Intelligence

    work page 2020

  6. [4]

    William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan-Kelley. 2024. Reducing transformer key-value cache size with cross-layer attention. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  7. [6]

    Hailin Chen, Amrita Saha, Steven Hoi, and Shafiq Joty. 2023. Personalized distillation: Empowering open-sourced llms with adaptive learning for code generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6737–6749. 9 PREPRINT 30% 40% 50% 60% 70% KV Size 18 20 22 24 26 28Perplexity CLA mixSGA GQA Figure ...

    work page 2023

  8. [7]

    Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. 2024. Nacl: A general and effective kv cache eviction framework for llms at inference time.Preprint, arXiv:2408.03675

    arXiv 2024

Show all 43 references
  1. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    2023

  2. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord

  3. [10]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm

    2023

  4. [11]

    DeepSeek-AI et al. 2024. Deepseek-v3 technical report.Preprint, arXiv:2412.19437

    2024 arXiv

  5. [12]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Preprint, arXiv:2101.03961

    2022 arXiv

  6. [13]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2021

  7. [14]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive kv cache compression for llms.Preprint, arXiv:2310.01801

    2024 arXiv

  8. [15]

    Gemma Team et al. 2024. Gemma 2: Improving open language models at a practical size.Preprint, arXiv:2408.00118

    2024 arXiv

  9. [16]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. MiniLLM: Knowledge distillation of large language models. InThe Twelfth International Conference on Learning Representations

    2024

  10. [17]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.Preprint, arXiv:1502.01852

    2015 arXiv

  11. [18]

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. Unnatural instructions: Tuning language models with (almost) no human labor. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Tor...

    2023

  12. [19]

    Jean Kaddour. 2023. The minipile challenge for data-efficient language models.Preprint, arXiv:2304.08442

    2023 arXiv

  13. [20]

    Distillm: Towards streamlined distillation for large language models

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. InForty-first International Conference on Machine Learning

  14. [21]

    Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. InCOLING 2020 (long paper). 10 PREPRINT

    2020

  15. [22]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding.Preprint, arXiv:2006.16668

    2020 arXiv

  16. [23]

    R Li, LB Allal, Y Zi, N Muennighoff, D Kocetkov, C Mou, M Marone, C Akiki, J Li, J Chim, et al. 2023. Starcoder: May the source be with you!Transactions on machine learning research

    2023

  17. [24]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM knows what you are looking for before generation.Preprint, arXiv:2404.14469

    2024 arXiv

  18. [25]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843

    2016 arXiv

  19. [26]

    Emilio Parisotto and Russ Salakhutdinov. 2021. Efficient transformers in reinforcement learning using actor-learner distillation. InInternational Conference on Learning Representations

    2021

  20. [27]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641

    2019 arXiv

  21. [28]

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations

    2017

  22. [29]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://cerebras.ai/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama

    2023

  23. [30]

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. Efficient transformers: A survey.ACM Comput. Surv., 55(6)

    2022

  24. [31]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971

    2023 arXiv

  25. [32]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Preprint, arXiv:1706.03762

    2017 arXiv

  26. [33]

    Daniel Waddington, Juan Colmenares, Jilong Kuang, and Fengguang Song. 2013. Kv-cache: A scalable high- performance web-object cache for manycore. In2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing, pages 123–130. IEEE

    2013

  27. [34]

    Smith, Daniel Khashabi, and Han- naneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions.Preprint, arXiv:2212.10560

    2023 arXiv

  28. [35]

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzni...

    2022

  29. [36]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019

  30. [37]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. TinyLlama: An open-source small language model.Preprint, arXiv:2401.02385

    2024 arXiv

  31. [38]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. OPT: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068

    2022 arXiv

  32. [39]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2024. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing ...

    2024

  33. [41]

    Dai, Zhifeng Chen, Quoc V Le, and James Laudon

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y Zhao, Andrew M. Dai, Zhifeng Chen, Quoc V Le, and James Laudon. 2022. Mixture-of-experts with expert choice routing. InAdvances in Neural Information Processing Systems. 12 PREPRINT A Experimental Setup Our exp...

    2022

  34. [2018]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1

    arXiv

  35. [2023]

    InThe 2023 Conference on Empirical Methods in Natural Language Processing

    GQA: Training generalized multi-query transformer models from multi-head checkpoints. InThe 2023 Conference on Empirical Methods in Natural Language Processing

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.