Pith. sign in

REVIEW 5 major objections 6 minor 33 references

IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IAM replaces selected attention computations in a large LLM with attention matrices borrowed from a smaller same-series model, keeping quality near-lossless at 30% mapping and cutting KV cache by about 22% at 50%.

desk verdict Solid cross-scale attention-mapping idea with useful quality-preservation data, but the headline prefill speedup claim contradicts the paper's own Algorithm 1 and the fine-tuning ablation contradicts its own table. read the letter →

arxiv 2507.11953 v1 pith:7CWWBWY7 submitted 2025-07-16 cs.CL cs.LG

Yi Zhao , Zuchao Li , Hai Zhao This is my paper
classification cs.CLcs.LG
keywords attentionmatrixsimilaritycross-scalelanguagemodelsKVcachereductioninferenceaccelerationlong-contextLLMservingcosinemappinglayer-wiseefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large language model can skip computing its own attention on selected layers and instead reuse attention matrices from a smaller model of the same family, because same-series models attend to their context in nearly the same way. The proposed framework, called Inference through Attention Mapping (IAM), chooses the most similar small-model attention matrix by cosine similarity during prefill, then reuses that mapping throughout decoding. On Qwen2 models the method keeps performance close to lossless at a 30% mapping ratio and, at 50%, still retains high capability while cutting KV cache by about 22% and prefill time-to-first-token by 12–17%. The attraction of the idea is that it attacks both main bottlenecks of long-context inference at once, and it can be combined with token-level KV eviction methods such as H2O.

What carries the argument

The carrying object is the mapping function $f(i) = \arg\max_{j} S(A_i, A'_j)$, where $A_i$ is an LLM attention matrix, $A'_j$ is an SLM attention matrix, and $S$ is cosine similarity computed as $\mathrm{Tr}(A_i^T A'_j)/(\|A_i\|_F \|A'_j\|_F)$. The paper uses this function to pair each mapped LLM layer with its closest small-model counterpart during prefill, then substitutes $A_i \leftarrow A'_{f(i)}$ during decoding so the LLM avoids Q/K projection, QK multiplication, and softmax for those layers. Two supporting mechanisms keep the substitution reliable: a delayed establishment threshold that starts mapping only after enough prompt tokens have accumulated, and a truncation threshold that limits similarity computation on very long contexts.

What would settle it

Measure IAM's perplexity and downstream accuracy on long prompts with long generation outputs, for example 8k-token prompts with 2k generated tokens, while re-estimating the mapping every 128 tokens versus never; if the never-remapped version degrades noticeably as generation length grows, the consistency premise fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that attention matrices are transferable across models of different scale within the same series: Qwen2-7B and Qwen2-72B attain an average cosine similarity of 0.954 between best-matching attention matrices, and this transfer can be exploited at inference time. IAM replaces the LLM's attention output on a chosen set of layers with the most similar attention matrix from the SLM, using cosine similarity as the matching metric, after instruction-tuning the small model on Alpaca data. The authors show that mapping layers at the end of the network, plus a middle block, is much safer than mapping early layers, and that the prefill-established mapping stays stable during autoregressive generation with a mean consistency rate of 91.12%. With 30% of layers mapped, performance on language modeling, MMLU, HotpotQA, and GovReport stays close to the original LLM; at 50% mapping, KV cache use falls by about 22% and end-to-end throughput rises by about 10–11%.

Load-bearing premise

The load-bearing premise is that the matching between a large model's attention layers and a small model's attention layers, worked out from the prompt at the start of inference, stays stable enough during the rest of generation that reusing the same matching without recomputing attention does not degrade output quality.

Editorial extensions

If this is right

  • At a 30% mapping ratio, performance stays close to lossless across language modeling, MMLU, HotpotQA, and GovReport benchmarks.
  • At a 50% mapping ratio, the method reduces KV cache usage by about 21.7–22.5% and reduces time-to-first-token by 12–17%, with throughput gains of about 10–11%.
  • The layer-mapping strategy transfers across tasks: block-level performance on MMLU correlates strongly (Pearson up to 0.95) with perplexity, HotpotQA, and GovReport results.
  • IAM generalizes to a different model family: LLaMA 3.1-70B with LLaMA 3.2-1B preserves MMLU accuracy under mapping, with a single optimal mapping region at the end of the network.
  • IAM is orthogonal to token-level KV eviction such as H2O and can be stacked with it without harming language modeling performance beyond the original method's cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to move from layer-level to head-level mapping: the paper itself notes this limitation, and head-level substitution could recover some of the performance lost at high mapping ratios while saving more cache.
  • If the 91.12% consistency rate degrades on longer or out-of-distribution contexts, a practical remedy the paper does not explore is periodic re-estimation of the mapping during decoding, or a confidence trigger that recomputes attention when similarity drops.
  • Because cross-series attention similarity is much lower (0.568 between LLaMA and Qwen2 with token alignment), IAM's practical reach depends on the availability of a well-matched small model in the same family; combining IAM with token-alignment methods might widen its applicability.
  • The heavy-tailed reuse of certain SLM attention matrices suggests the LLM draws from a small vocabulary of attention patterns; if so, one could precompute a compact set of mapping matrices and skip running the SLM at serve time entirely.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes IAM, an inference-efficiency method that replaces attention matrices in selected layers of a large language model with attention matrices from a smaller same-family model. The mapping from LLM attention heads to SLM heads is established during prefill by computing pairwise cosine similarity over attention matrices, then reused during decode so that mapped layers skip Q/K projection and QK^T computation. The authors report near-lossless quality at 30% mapping ratio, a 22% reduction in KV-cache usage, and 12–17% TTFT reductions on Qwen2-72B, with additional experiments on LLaMA series models and compatibility with H2O.

Significance. The paper identifies a useful empirical phenomenon—same-series models of different scales exhibit high attention-matrix similarity—and converts it into an optimization that is orthogonal to token-level KV-cache compression. If the reported results survive scrutiny, IAM would be a novel and practical addition to LLM serving toolkits. Strengths include the systematic analysis of similarity metrics, layer selection, and mapping consistency; the evaluation across two model families; a compatibility study with H2O; and public code. However, the current evidence is weakened by selection of hyperparameters on the same benchmarks later reported, an ambiguous Algorithm 1 that does not clearly describe long-context handling, and an efficiency claim that does not account for the cost of mapping establishment.

major comments (5)
  1. [§3.4, Algorithm 1] Algorithm 1 is ambiguous about how mapping establishment interacts with the full prompt for long inputs. Lines 2–3 say that if len(x) > tau_t, the prompt is truncated to tau_t; lines 7–10 then establish the mapping from the truncated prompt, and line 13 generates using M(x, y) where x has been truncated. This would discard all tokens beyond tau_t, which cannot be what was done for the GovReport long-context experiments. The truncation must apply only to the similarity computation, not to the prompt actually processed. Please rewrite Algorithm 1 to make the full-prefill path explicit, including when and how the mapping is applied during the prefill forward pass.
  2. [§4.3, Table 3] The reported TTFT reductions of 12% and 17% are attributed to skipping Q/K projections and QK^T in mapped layers, but mapping establishment (Algorithm 1, lines 7–10) requires exactly those computations on at least a tau_t-token prefix, plus the SLM forward pass and the pairwise similarity search over L·D × l·d attention matrices. The paper does not report a cost breakdown, so it is unclear whether these overheads are included in the TTFT numbers. Please provide an ablation that isolates the mapping-establishment overhead and reports end-to-end prefill latency including the SLM forward and the similarity computation.
  3. [§3.1 and §3.2, Figure 5] The similarity metric is selected by evaluating WikiText-v2 perplexity (Table 1), and the layer-mapping schedule is selected using MMLU (Figure 2). These same two benchmarks are then reported as the headline results in Figure 5. This selection-on-test-set procedure inflates the reported quality; the authors should either reserve a held-out set for final evaluation or report results on benchmarks not used in any selection step.
  4. [Appendix A, Table 4] The table appears to contradict the surrounding text. The text states that fine-tuning decreases perplexity, but the row labeled 'w. tuning' has higher perplexity than 'w.o. tuning' at every mapping ratio (e.g., 8.53 vs. 8.47 at 10%). Please correct the labels or the numbers, or explain what 'w.o.' and 'w.' denote.
  5. [§3.3, Figure 4] The mean consistency rate of 91.12% implies that roughly 9% of layer-head mappings change during decoding. The paper asserts that the mapping remains 'stable and reliable' but does not test whether those changes degrade output quality, nor whether consistency holds for longer outputs or out-of-distribution contexts. Since the efficiency gains depend on reusing the prefill mapping, please add an analysis of generation quality as a function of mapping stability, or measure consistency over longer and more diverse prompts.
minor comments (6)
  1. [Abstract and §4.3] The abstract reports 'accelerate prefill by 15%', while §4.3 reports 12% and 17% (average 14.5%) and the introduction reports 'average acceleration of 11% in inference speed'; please align these numbers.
  2. [Figure 5] In Figure 5, the y-axis of the perplexity panel is labeled 'Perplexity', while Table 1 reports log perplexity; please indicate which quantity is plotted to avoid confusion.
  3. [§3.3 and §4.5] There are duplicated figure references: Section 3.3 refers to 'Figure 3' for the consistency plot, and Section 4.5 also refers to 'Figure 3' for the H2O compatibility plot, but that plot is actually Figure 8.
  4. [Algorithm 1] Algorithm 1 contains a typo: 'lenth' should be 'length'.
  5. [Table 3] Table 3's header 'Bsz Lenth' should be 'Bsz Length'.
  6. [§3.1, Eq. (2)] The definition of Minkowski distance has a broken summation formatting; please fix the equation.

Circularity Check

0 steps flagged · score 0.0 of 10

IAM's claims are empirically driven and self-contained; no circular step reduces a predicted result to its inputs.

full rationale

The paper does not derive any central result from its own assumptions in a circular way. Equation (4) defines cosine similarity between LLM and SLM attention matrices, and Equation (5) selects the most similar SLM matrix; Equation (6) then replaces the LLM attention matrix during decode. This is a direct, non-circular algorithmic construction: the mapping is computed from the actual attention matrices and then applied, with performance measured on external benchmarks. The choice of cosine similarity, the layer-mapping schedule, and the consistency claim are all empirically selected and then tested, which is tuning rather than circular reasoning. The layer schedule is chosen partly from MMLU measurements, but Section 3.2 and Table 2 explicitly validate that the schedule transfers to HotpotQA, GovReport, and perplexity via high Pearson correlations, so the schedule is not forced by the evaluation target alone. The consistency rate of 91.12% is an observed quantity, not an assumption used to derive itself. The prefill acceleration claim is potentially sensitive to how mapping-establishment costs are accounted, since Algorithm 1 requires a forward pass through both models to obtain attention matrices; however, the algorithm's truncation threshold (tau_t = 100) indicates mapping can be established on a truncated prefix, so the paper's TTFT numbers are not shown to be impossible or definitionally forced. That concern is an efficiency-accounting and implementation-detail issue, not a circularity of the claimed derivation chain. No load-bearing self-citation or imported uniqueness theorem appears; the only self-citation in the reference list is PyramidInfer, which is cited as related work and is not used to justify IAM's core mechanism. Therefore, no specific circular step can be identified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework rests on empirical similarity between same-series models, plus several hand-set thresholds and tuning choices. No new physical or architectural entities are introduced.

free parameters (3)
  • tau_e (mapping establishment threshold) = 20 tokens
    Chosen by hand; if the prompt is shorter than 20 tokens, IAM runs autoregressive generation until the threshold is met before mapping, which changes the efficiency profile for short prompts.
  • tau_t (similarity truncation threshold) = 100 tokens
    Chosen by hand; sequences longer than 100 tokens are truncated when computing pairwise attention similarity to avoid high-dimensional instability, which is a stated approximation.
  • Fine-tuning epochs for SLM = 2 of max 5
    Selected by observing that training loss stops decreasing; Appendix A's Table 4 gives contradictory evidence about whether tuning helps.
assumptions (4)
  • domain assumption Same-series models share tokenizer and compatible attention-matrix shapes (N x N), so direct substitution of attention matrices is possible.
    Section 3.4 relies on l*d < L*D and pairwise N x N similarity; Appendix B notes cross-series models with different tokenizers cannot be mapped directly.
  • domain assumption Attention matrices of same-series different-scale LLMs are similar enough that the argmax mapping is meaningful.
    Empirical premise established in Appendix B (average cosine similarity 0.954 for Qwen2-7B to Qwen2-72B); it is the foundation of Eq. 5.
  • domain assumption The mapping established in prefill remains consistent through decode.
    Section 3.3 measures mean consistency 91.12%, but IAM's decode stage assumes this without recomputation.
  • domain assumption Instruction tuning of the small model compensates for attention substitution error.
    Section 3.4 optimizes only the small model parameters; Appendix A's Table 4 contradicts the text about whether tuning helps, making this assumption questionable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IAM: Efficient Inference through Attention Mapping between Different-scale LLMs." pith.science (2026). https://pith.science/paper/7CWWBWY7

@misc{pith2026250711953,
  author       = {Pith},
  title        = {Pith review of: IAM: Efficient Inference through Attention Mapping between Different-scale LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CWWBWY7}},
  note         = {Machine review of arXiv:2507.11953}
}
read the original abstract

LLMs encounter significant challenges in resource consumption nowadays, especially with long contexts. Despite extensive efforts dedicate to enhancing inference efficiency, these methods primarily exploit internal sparsity within the models, without leveraging external information for optimization. We identify the high similarity of attention matrices across different-scale LLMs, which offers a novel perspective for optimization. We first conduct a comprehensive analysis of how to measure similarity, how to select mapping Layers and whether mapping is consistency. Based on these insights, we introduce the IAM framework, which achieves dual benefits of accelerated attention computation and reduced KV cache usage by performing attention mapping between small and large LLMs. Our experimental results demonstrate that IAM can accelerate prefill by 15% and reduce KV cache usage by 22.1% without appreciably sacrificing performance. Experiments on different series of models show the generalizability of IAM. Importantly, it is also orthogonal to many existing KV cache optimization methods, making it a versatile addition to the current toolkit for enhancing LLM efficiency.

Figures

Figures reproduced from arXiv: 2507.11953 by the authors.

Figure 1
Figure 1. IAM is orthogonal to the existing KV cache [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The performance on MMLU benchmark after mapping different attention layers of Qwen2-72B. 3.3 Consistency of Mapping IAM establishes a mapping relationship between the SLM and the LLM based on similarity during the prefill stage, which is then utilized in the sub￾sequent decode stage. Therefore, an essential guar￾antee is that the mappings established during the prefill stage remain consistent, which means map￾ping r… view at source ↗
Figure 3
Figure 3. The average consistency rate from the 16th [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Framework of IAM. the MMLU (Hendrycks et al., 2021). 3) QA task: we assess QA performance using the HotpotQA dataset (Yang et al., 2018) and adopt F1 score as evaluation metric. 4) Long context: we utilize Gov￾Report benchmark (Huang et al., 2021) to test the long text…
Figure 5
Figure 5. Figure 5: Benchmark results of IAM with mapping ratio varying from 0 to 50%. Default represents using the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The performance on MMLU benchmark after mapping different attention layers of LLaMA 3.1-70B. To evaluate the generalizability of the IAM, we conduct experiments on different series of models. In this experiment, the LLM is LLaMA 3.1-70B and the SLM is LLaMA 3.2-1B. The…
Figure 8
Figure 8. Figure 8: The perplexity of using IAM in combination [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: The average attention matrices across all layers and heads from four different-scale models of Qwen2 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 12
Figure 12. Figure 12: The statistical analysis of the most frequent [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 11
Figure 11. Figure 11: The log perplexity of LLM according to [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 3 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, et al. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . Preprint, arXiv:2404.14219

    arXiv 2024

  2. [2]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. https://arxiv.org/abs/2401.10774 Medusa: Simple llm inference acceleration framework with multiple decoding heads . Preprint, arXiv:2401.10774

    arXiv 2024

  3. [3]

    Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. 2021. https://arxiv.org/abs/2110.07143 bert2bert: Towards reusable pretrained language models . Preprint, arXiv:2110.07143

    work page Pith review arXiv 2021

  4. [4]

    Tri Dao. 2023. https://arxiv.org/abs/2307.08691 Flashattention-2: Faster attention with better parallelism and work partitioning . Preprint, arXiv:2307.08691

    arXiv 2023

  5. [5]

    DeepSeek-AI. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

    arXiv 2025

  6. [6]

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. https://arxiv.org/abs/2404.19737 Better & faster large language models via multi-token prediction . Preprint, arXiv:2404.19737

    arXiv 2024

  7. [7]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    arXiv 2021

  8. [8]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. https://arxiv.org/abs/2401.18079 Kvquant: Towards 10 million context length llm inference with kv cache quantization . Preprint, arXiv:2401.18079

    arXiv 2024

Show all 33 references
  1. [9]

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. https://doi.org/10.18653/v1/2021.naacl-main.112 Efficient attentions for long document summarization . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computati...

    2021 doi

  2. [10]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: efficient training of giant neural networks using pipeline parallelism. Curran Associates Inc., Red Hook, NY, USA

    2019

  3. [11]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.825 LLML ingua: Compressing prompts for accelerated inference of large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural...

    2023 doi

  4. [12]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. https://arxiv.org/abs/2309.06180 Efficient memory management for large language model serving with pagedattention . Preprint, arXiv:2309.06180

    2023 arXiv

  5. [13]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/a452a7c6c463e4ae8fbdc614c6e983e6-Paper-Conference.pdf Scissorhands: Exploiting the...

    2023

  6. [14]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. https://arxiv.org/abs/1609.07843 Pointer sentinel mixture models . Preprint, arXiv:1609.07843

    2016 arXiv

  7. [15]

    OpenAI. 2024 a . https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  8. [16]

    OpenAI. 2024 b . Introducing openai o1-preview. https://openai.com/index/introducing-openai-o1-preview/

    2024

  9. [17]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor R \"u hle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.57 LLML ingua-2: Data distillation for efficie...

    2024 doi

  10. [18]

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2024. https://arxiv.org/abs/2407.00079 Mooncake: A kvcache-centric disaggregated architecture for llm serving . Preprint, arXiv:2407.00079

    2024 arXiv

  11. [19]

    Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. https://arxiv.org/abs/2303.06865 Flexgen: High-throughput generative infer...

    2023 arXiv

  12. [20]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. https://arxiv.org/abs/1909.08053 Megatron-lm: Training multi-billion parameter language models using model parallelism . Preprint, arXiv:1909.08053

    2020 arXiv

  13. [21]

    Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. 2024. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254

    2024 arXiv

  14. [22]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  15. [23]

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. 2024. https://arxiv.org/abs/2407.08454 Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks . Preprint, arXiv:2407.08454

    2024 arXiv

  16. [24]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  17. [25]

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. https://arxiv.org/abs/2203.16487 Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation . Preprint, arXiv:2203.16487

    2023 arXiv

  18. [26]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. https://arxiv.org/abs/2309.17453 Efficient streaming language models with attention sinks . Preprint, arXiv:2309.17453

    2024 arXiv

  19. [27]

    An Yang, Baosong Yang, Binyuan Hui, et al. 2024 a . https://arxiv.org/abs/2407.10671 Qwen2 technical report . Preprint, arXiv:2407.10671

    2024 arXiv

  20. [28]

    Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.195 P yramid I nfer: Pyramid KV cache compression for high-throughput LLM inference . In Findings of the Association for Computational Linguistics: A...

    2024 doi

  21. [29]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://arxiv.org/abs/1809.09600 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . Preprint, arXiv:1809.09600

    2018 arXiv

  22. [30]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  23. [31]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\' e , Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb2827f...

    2023

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.