Pith. sign in

REVIEW 5 major objections 5 minor 41 references

MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A router that picks per-chunk bit-widths for KV-cache quantization keeps long-context LLM accuracy near FP16 at about 4 bits.

desk verdict The chunk-routing idea is fresh, but the training loss in §3.3 is degenerate as written, so the headline results are not yet tied to the mechanism. read the letter →

arxiv 2506.07533 v1 pith:J5X5264S submitted 2025-06-09 cs.CV

Wei Tao , Haocheng Lu , Xiaoyang Qu , Bin Zhang , Kai Lu , Jiguang Wan , Jianzong Wang This is my paper
classification cs.CV
keywords mixed-precisionquantizationKVcachelong-contextLLMinferencemixtureofexpertsquantization-awarerouterfine-tuningmemoryreductionserving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MoQAE proposes to treat each KV-cache quantization bit-width (FP16, INT4, INT2, ...) as an "expert" in a mixture-of-experts router, and to let the router choose the bit-width for each chunk of tokens during long-context LLM inference. The paper claims this learns a mixed-precision quantization strategy that keeps model accuracy near the full-precision baseline at about 4 bits on average, while reducing KV cache memory and decoding latency compared with prior quantization methods. A lightweight router-only fine-tuning step with a composite accuracy-memory loss is said to make the trade-off tunable through one hyperparameter, and routing freezing plus routing sharing cut the router's own overhead. The authors report perplexity close to FP16 on WikiText2 across Llama, Llama-2, Llama-3, and Mistral models, and best scores on most LongBench tasks.

What carries the argument

The key object is the quantization-aware expert set: each expert is a distinct KV-cache bit-width configuration (e.g., FP16, INT4, INT2). A lightweight two-layer MLP router takes a chunk of tokens, outputs per-token expert probabilities, and the chunk is assigned the expert selected by the most tokens (Equation 2). The router is trained alone, with the LLM frozen, under loss L = lambda*L_model + (1-lambda)*L_mem, where L_model multiplies the negative log-likelihood by the router's selection probability (penalized by 1/bit-width) and L_mem rewards low bit-widths (penalized by 16/bit-width). Routing freezing pins the first chunk to FP16, and routing sharing groups blocks so later blocks reuse the first block's routing decision.

What would settle it

Run the router fine-tuning on a small model with a fixed calibration set, then compare the perplexity of the quantized model when quantizing according to the trained router versus quantizing the same chunks with randomly chosen bit-widths matched to the same average bit-width. If the random assignment performs equally well, the router training is not responsible for the reported accuracy; alternatively, check whether the router's chosen bit-widths correlate with an oracle importance score such as per-token attention sum.

Watch

Extended reading notes

Core claim

Treating quantization bit-width configurations as experts and selecting them with a chunk-level MoE router yields a mixed-precision KV cache whose performance tracks FP16 while the average bit-width falls to around 4 bits. On Llama-7B, MoQAE-lambda0.5 reaches 5.76 perplexity versus FP16's 5.68 at 4.13 average bits; on LongBench it scores highest on five of eight tasks. The routing-freezing mechanism keeps the first chunk in FP16 because initial tokens carry disproportionate attention, and routing sharing lets later blocks reuse the first block's strategy, reducing latency at a small accuracy cost. The paper concludes that, unlike prior mixed-precision methods that require expensive per-token search, the MoE router learns the bit-width configuration quickly and cheaply.

Load-bearing premise

The load-bearing premise is that the router-only fine-tuning loss genuinely teaches the router to assign high bit-widths to the chunks that matter for model accuracy; the loss contains a non-differentiable indicator and is multiplied by the router's own probability, and the paper does not state how gradients are obtained, so if that training signal is ineffective the accuracy gains would not transfer.

Editorial extensions

If this is right

  • Long-context serving can hold the KV cache of much longer prompts in GPU memory because average cache bit-width drops to about 4 bits while perplexity stays within roughly 0.1 of FP16.
  • The router-only fine-tuning requires only a small calibration set (5% of training data) and freezes the LLM, so the method can be applied to existing models without full retraining or per-token search.
  • The single hyperparameter lambda directly trades accuracy against memory, letting deployment choose operating points from near-lossless (lambda=0.5) to very low memory (lambda=0.1).
  • Routing sharing reduces router computation and memory proportionally to group size, at a slight accuracy cost, making the router's own overhead manageable.
  • Because the first chunk is always kept FP16, the method protects attention-sink tokens that prior work identifies as critical, which supports long-context accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The chunk-level routing design suggests a natural hardware co-design: since a whole chunk shares one bit-width, kernels can be dispatched at chunk granularity, making the approach compatible with tensor-core-friendly group quantization without per-token branching.
  • The same router-as-quantizer framing could be applied to weights or activations, not just the KV cache, reusing the frozen-LLM, router-only training recipe for other mixed-precision compression problems.
  • The dependence on the router's own probability in L_model is a testable weak point: if gradients cannot flow through the indicator term, the reported gains may come largely from the memory-loss term and the FP16 first chunk rather than from learned accuracy-aware routing.
  • One could benchmark MoQAE against per-token mixed-precision methods with matched average bit-width on tasks where attention is concentrated in later positions, since routing freezing biases toward initial tokens.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MoQAE, a mixed-precision KV-cache quantization method for long-context LLM inference. The method treats different quantization bit-width configurations as 'experts' in a mixture-of-experts (MoE) framework, uses a lightweight router to select per-chunk configurations, and introduces routing-freezing and routing-sharing mechanisms to reduce overhead. The router is trained on a small calibration set with a composite loss intended to trade off accuracy and memory. Experiments on WikiText2 perplexity and LongBench tasks claim that MoQAE achieves near-FP16 perplexity at an average of about 4 bits and reduces memory and latency compared with state-of-the-art quantization baselines. However, the router training objective in Section 3.3 is degenerate as written, the efficiency and effectiveness claims are supported by different hyperparameter settings, and several technical details, including the router architecture and reproducibility information, are incomplete.

Significance. If the method worked as described, it would be a practically useful contribution: it addresses the memory bottleneck of KV caches in long-context inference, and the idea of framing bit-width selection as a MoE routing problem is creative. The paper also ships a reasonable set of comparisons and an ablation study. However, the central mechanism—router training—is not credible as written, and the headline claim of simultaneous efficiency and effectiveness is not backed by a single operating point. Because the reported gains depend entirely on the router learning a useful policy, the weaknesses in the training loss and the unexplained gradient path undermine the validity of the experimental section. The absence of code and error bars further limits the ability to verify the results.

major comments (5)
  1. [§3.3, Eqs. (3)–(4)] The router training loss is not a coherent training signal as written. In Eq. (3), for a token i whose selected expert is j, the term is p_i^j * L_nll / B_j; both L_nll and 1/B_j are positive, so minimizing this term drives p_i^j toward zero, i.e., it decreases the probability of the expert that was actually selected and pushes the router toward a uniform distribution. Eq. (4) has the same structure with the positive coefficient 16/B_j, so it also penalizes the selected expert's probability rather than rewarding good bit-width choices. If L_nll is intended to depend on the selected bit-width, that dependence enters through the non-differentiable argmax and indicator in Eq. (3), and the paper does not specify any gradient estimator (straight-through estimator, Gumbel-softmax, REINFORCE, or similar). Either way, the stated objective cannot explain how the router learns to prefer bit-widths that improve accuracy. Since the reported perplexity and bit-width results in Tables 1–4 are attributed to this training process, this is a load-bearing flaw.
  2. [§4.2, Figs. 5–6 vs. Table 1] The central claim that MoQAE outperforms baselines in both efficiency and effectiveness is not supported by a single configuration. The perplexity results in Table 1 are reported for λ=0.5, while the memory and latency reductions in Figures 5–6 are reported for λ=0.1 with the statement 'MoQAE-λ0.1 achieves the least memory usage and decoding latency.' No single setting of MoQAE is shown to achieve both near-FP16 accuracy and the best efficiency simultaneously. Since λ is a free hyperparameter swept in Table 4, the headline claim conflates two different operating points. The authors should either report a single configuration that delivers both, or temper the claim accordingly.
  3. [§3.2, Eq. (1)] The router forward computation is dimensionally inconsistent. The paper states C ∈ R^{N×D}, W1, W2 ∈ R^{D×M}, and W3 ∈ R^{D×M}. The expression P = f(C W1 · C W2) W3 then has incompatible dimensions: C W1 and C W2 are both N×M, and even if '·' denotes element-wise multiplication, the result (N×M) cannot be multiplied by W3 (D×M) unless D = M. If a different contraction is intended, it needs to be written precisely; as printed, the router architecture is undefined, which affects the entire method description.
  4. [§4.3, Table 4] The text describing Table 4 contradicts the table. The paper states 'with the increase of λ, the model accuracy increases ... while average bits and memory usage decreases,' but the table shows average bits increasing from 3.45 at λ=0.1 to 12.12 at λ=0.9, and memory usage is non-monotonic (14.01, 14.04, 15.95, 15.33, 15.88). The text should be corrected to say that increasing λ increases accuracy at the cost of more bits and generally more memory, or the table should be revised. As written, this is an internal inconsistency in a key ablation.
  5. [§4.1–§4.2] Reproducibility information is insufficient for the empirical claims. No code is released, no error bars or standard deviations are reported for perplexity, F1 scores, latency, or memory, and the number of runs or seeds is not given. Several margins in Table 1 are small (e.g., MoQAE-λ0.5 perplexity 5.76 vs. FP16 5.68 on Llama-7B), and without variance estimates it is impossible to tell whether the differences are significant. Given that the central claim depends on the new training loss, the absence of these details is a load-bearing gap for the experimental section.
minor comments (5)
  1. [§1] There is a typo in the first paragraph: 'complicatrd' should be 'complicated.'
  2. [§4.2] The sentence 'the memory usage of FP16 model by 2.99GB on average and outperforms most of the baseline methods on decoding latency on decoding latency' contains a duplicated phrase 'on decoding latency.'
  3. [Table 1 caption] The abbreviation 'AvB' in the caption 'A vB means average bit-width' is unclear; consider using 'AvgBits' or spelling out 'average bit-width.'
  4. [§4.2] The sentence 'The quantization bit-width for each token is assigned based on their saliency' appears immediately after the KIVI description and is ambiguous; it likely refers to MiKV rather than KIVI. Please clarify the antecedent.
  5. [Fig. 3] The axes of the attention-weight plots are labeled 0–14 but the text refers to 'the first few tokens'; please clarify the exact token range shown.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: MoQAE's router predictions are validated against external benchmarks; the §3.3 loss is a self-referential training-design concern, not a circular step.

full rationale

MoQAE's derivation chain is: define bit-width configurations as MoE experts (§3.2), route chunks by arg-max probabilities (Eq. 2), fine-tune only the router on a calibration subset with a composite loss (§3.3), then evaluate PPL/F1/latency on WikiText2 and LongBench against external baselines (§4). None of the reported quantities is defined as its own input. The router is trained on calibration data and the test metrics are independently measured; the average-bit-width values are direct consequences of the routing decisions rather than fitted targets. λ is swept (§4.3), and different values support different efficiency/accuracy claims, which is hyperparameter selection rather than a circular fit. The only self-citation (Tao et al. 2025b, Section 2.2) is a related-work pointer and does not carry the argument. The one in-scope weakness is Section 3.3, Eqs. (3)–(4): as written, the loss multiplies the selected expert's probability by the positive scalar L_nll/B_j (respectively 16p_i^j/B_j), so gradient descent decreases the probability of the very expert selected, and no straight-through estimator, Gumbel-Softmax, or REINFORCE surrogate is specified for the discrete argmax. That is a degenerate/self-referential training signal and an omitted-proof/correctness issue, but it is not a case where a prediction reduces by construction to an input: the benchmark comparisons remain external measurements. Weighting this concern, I assign 2 rather than 0, but find no circular derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or ontological entities. Its core ingredients are free hyperparameters (lambda, chunk size, group size, expert set) and unverified modeling assumptions about the router's probabilistic interpretation and trainability.

free parameters (5)
  • lambda = 0.5 (accuracy), 0.1 (efficiency), tested at 0.1, 0.3, 0.5, 0.7, 0.9
    Trade-off between L_model and L_mem; selected based on benchmark evaluations rather than a validation set.
  • Chunk size = 32
    Balances training time, decoding latency, and accuracy; chosen after ablation testing 8, 16, 32, 64, 128.
  • Routing sharing group size = 3
    Controls accuracy vs. latency trade-off; ablation tests group sizes 2 and 4.
  • Expert set (bit-width configurations) = Not specified
    The number of experts M and exact bit-widths (e.g., FP16, INT4, INT2) are not stated in the experimental setup, leaving a key design choice unquantified.
  • Calibration data fraction = 5% of training set
    Router fine-tuning data; no analysis of sensitivity to this fraction.
assumptions (5)
  • domain assumption The router output P in Eq. (1) can be treated as selection probabilities.
    No softmax or normalization is described, and f is SiLU, so P does not necessarily sum to 1 over experts.
  • domain assumption L_nll can be treated as a constant scaling factor independent of the router when training the router.
    Section 3.3 states L_nll does not involve operators directly related to the router weights; this makes the router gradient only through p_i_j.
  • ad hoc to paper The non-differentiable indicator I(argmax) can be used in the loss and still yield useful gradients for the router.
    The paper does not describe a gradient estimator for the argmax/indicator, yet claims the loss trains the router.
  • domain assumption Quantizing all tokens in a chunk to the majority-vote bit-width preserves model accuracy.
    Section 3.2 adopts one bit-width per chunk via Eq. (2); the ablation on chunk size shows sensitivity to this assumption.
  • domain assumption The first chunk of tokens always deserves FP16 precision.
    Justified by prior attention-sink work (Xiao et al., 2024) and Figure 3, but applied universally across models and tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts." pith.science (2026). https://pith.science/paper/J5X5264S

@misc{pith2026250607533,
  author       = {Pith},
  title        = {Pith review of: MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5X5264S}},
  note         = {Machine review of arXiv:2506.07533}
}
read the original abstract

One of the primary challenges in optimizing large language models (LLMs) for long-context inference lies in the high memory consumption of the Key-Value (KV) cache. Existing approaches, such as quantization, have demonstrated promising results in reducing memory usage. However, current quantization methods cannot take both effectiveness and efficiency into account. In this paper, we propose MoQAE, a novel mixed-precision quantization method via mixture of quantization-aware experts. First, we view different quantization bit-width configurations as experts and use the traditional mixture of experts (MoE) method to select the optimal configuration. To avoid the inefficiency caused by inputting tokens one by one into the router in the traditional MoE method, we input the tokens into the router chunk by chunk. Second, we design a lightweight router-only fine-tuning process to train MoQAE with a comprehensive loss to learn the trade-off between model accuracy and memory usage. Finally, we introduce a routing freezing (RF) and a routing sharing (RS) mechanism to further reduce the inference overhead. Extensive experiments on multiple benchmark datasets demonstrate that our method outperforms state-of-the-art KV cache quantization approaches in both efficiency and effectiveness.

Figures

Figures reproduced from arXiv: 2506.07533 by the authors.

Figure 1
Figure 1. The composition of LLM inference memory under different context lengths on an NVIDIA A100 GPU with 80GB memory capacity. newest LLM can handle up to 2 million input to￾kens (Team et al., 2024), the long-context inference still presents substantial challenges in memory con￾sumption and computational efficiency. We have plotted the composition of the memory usage of the Llama2-13B model in relation to the context leng… view at source ↗
Figure 2
Figure 2. The overview of MoQAE. We use the router in MoE technology to learn the optimal quantization strategy. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. The routing sharing mechanism. Here we choose 16 Bj as the penalty term. This is because data with higher-bitwidth leads to more memory consumption. Finally, our loss is defined as follows: L = λLmodel + (1 − λ)Lmem (5) where λ is a pre-defined hyperparameter that con￾trols the trade-off between model accuracy and memory usage. We will discuss the impact of λ on model performance in Section 4.3. 3.4 Routing Freezing… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: The memory usage of MoQAE and baseline methods under different context lengths. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The decoding latency of MoQAE and baseline methods under different context lengths. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The impact of chunk size on model perfor [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 21 canonical work pages

  1. [1]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119--3137

    work page 2024

  2. [2]

    William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan-Kelley. 2024. Reducing transformer key-value cache size with cross-layer attention. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  3. [3]

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, pages 5209--5235

    work page 2024

  4. [4]

    Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flashattention: fast and memory-efficient exact attention with io-awareness. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 16344--16359

    work page 2022

  5. [5]

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547--5569. PMLR

    2022

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1--39

    2022

  8. [8]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. Gptq: Accurate post-training quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations

    work page 2023

Show all 41 references
  1. [9]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive kv cache compression for llms. In The Twelfth International Conference on Learning Representations

    2024

  2. [10]

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui...

    2024

  3. [11]

    Jiaao He and Jidong Zhai. 2024. Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines. arXiv preprint arXiv:2403.11421

    2024 arXiv

  4. [12]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270--1303

    2024

  5. [13]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  6. [14]

    Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. 2023. S ^ 3 : Increasing gpu utilization during generative inference for higher throughput. Advances in Neural Information Processing Systems, 36:18015--18027

    2023

  7. [15]

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2024. Squeezellm: dense-and-sparse quantization. In Proceedings of the 41st International Conference on Machine Learning, pages 23901--23923

    2024

  8. [16]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  9. [17]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024 a . Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971--5984

    2024

  10. [18]

    Hongzhan Lin, Ang Lv, Yang Song, Hengshu Zhu, Rui Yan, et al. 2024 b . Mixture of in-context experts enhance llms' long context awareness. Advances in Neural Information Processing Systems, 37:79573--79596

    2024

  11. [19]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024 c . Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, ...

    2024

  12. [20]

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2025. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. In Proceedings of Machine Learning and Systems

    2025

  13. [21]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2024 a . Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Informatio...

    2024

  14. [22]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 b . Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, pages 32332--32344

    2024

  15. [23]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference on Learning Representations

    2017

  16. [24]

    Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and Fran c ois Fleuret. 2023. Faster causal attention over large sequences through sparse flash attention. arXiv preprint arXiv:2306.01160

    2023 arXiv

  17. [25]

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 590--606

    2024

  18. [26]

    Wei Tao, Xiaoyang Qu, Kai Lu, Jiguang Wan, Guokuan Li, and Jianzong Wang. 2025 a . Madllm: Multivariate anomaly detection via pre-trained llms. arXiv preprint arXiv:2504.09504

    2025 arXiv

  19. [27]

    Wei Tao, Bin Zhang, Xiaoyang Qu, Jiguang Wan, and Jianzong Wang. 2025 b . Cocktail: Chunk-adaptive mixed-precision quantization for long-context llm inference. In 2025 Design, Automation & Test in Europe Conference (DATE), pages 1--7. IEEE

    2025

  20. [28]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  21. [29]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  22. [30]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  23. [31]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations

    2024

  24. [32]

    Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. 2024. Powerinfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282

    2024 arXiv

  25. [33]

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096

    2024 arXiv

  26. [34]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for \ Transformer-Based \ generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521--538

    2022

  27. [35]

    Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. 2024 a . Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems, 37:3304--3331

    2024

  28. [36]

    Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2022. Mixture of attention heads: Selecting attention heads per token. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4150--4162

    2022

  29. [37]

    Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Atlas Wang. 2024 b . Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6:381--394

    2024

  30. [38]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Proces...

    2023

  31. [39]

    Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems, 6:196--209

    2024

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.