Pith. sign in

REVIEW 4 major objections 6 minor 98 references

SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SHARP keeps Llama2-7b near its original perplexity while sharing MLP layers and cutting mobile inference time by 42%.

desk verdict Useful layer-sharing recovery recipe with credible perplexity results, but the mobile savings headline is measured on a variant without the recovery parameters and needs re-measurement. read the letter →

arxiv 2502.07832 v1 pith:7OHHNKHJ submitted 2025-02-11 cs.LG cs.AI

Yiping Wang , Hanxian Huang , Yifang Chen , Jishen Zhao , Simon Shaolei Du , Yuandong Tian This is my paper · ORCID
classification cs.LGcs.AI
keywords layersharinglow-rankadaptationmodelcompressionLLMinferenceaccelerationmobiledeploymentperplexityrecoverystructuralpruningLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SHARP claims that a pretrained large language model can be compressed for inference by replacing many MLP layers with one reference layer plus small low-rank recovery parameters, without retraining from scratch. The authors show that adjacent MLP layers in Llama2-7b produce similar outputs even though their weights differ, and that a two-stage recovery -- a single-layer output-matching warmup followed by supervised fine-tuning -- brings perplexity back near the original using at most 50k examples. The payoff is concrete: 38--65% fewer stored MLP parameters and, on a mobile phone, 42.8% less storage and 42.2% less total inference time. The paper argues this makes layer-sharing practical for deploying existing LLMs in resource-constrained devices with only fine-tuning-scale data.

What carries the argument

The central object is the layer-sharing-with-recovery-parameters scheme: a stored reference layer $\Theta_j$ is loaded once, and each target layer $l$ is recreated on the fly as $g(\Theta_j, \Delta\Theta_l)$, where $g$ is a low-rank transformation such as LoRA addition $g_0(\Theta_j,(\alpha,A_l,B_l)) = \alpha\Theta_j + A_lB_l$; only reference layers plus small adapters are stored. The argument is carried by the compression identity $s = (32-X)/32 + (X/32)(4096r+11008r)/(4096\times 11008) \approx 1 - X/32 + X\cdot r\cdot 10^{-5}$ for Llama2-7b, and by the two-stage recovery procedure: SLW fits each adapter independently by minimizing $\|f(X;g(\Theta_j,\Delta\Theta_l))-f(X;\Theta_l)\|_2^2$ on sampled activations, giving a good initialization for the joint SFT stage that restores final model behavior.

What would settle it

Run direct adjacent-layer replacement on Llama2-7b and measure Arxiv-math perplexity: if replacing a middle MLP layer (say layer 15 with layer 14) raises perplexity from the 3.0 baseline by more than about 10%, the redundancy premise SHARP builds on is absent. A second decisive check is to run SHARP's Tnext schedule at rank 400 on a larger model with the 50k-example cap: the method's practical claim collapses if the required rank or data budget grows substantially with model scale.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that an LLM's MLP stack has a built-in redundancy that can be exploited after training: the current MLP layer can approximate a later MLP layer surprisingly well simply by being reused, and adding low-rank adapters (e.g., a LoRA-style update $g_0(\Theta_j, (\alpha, A, B)) = \alpha\Theta_j + AB$) nearly closes the residual gap. The paper establishes this by direct replacement experiments -- perplexity barely moves when a neighbor layer is substituted in most positions -- then shows that a two-stage recovery, SLW and SFT, recovers in-distribution perplexity with no more than 50k examples while storing only a fraction of the MLP weights. SLW fits each adapter to reproduce the original layer's output under $L_2$ loss on a 10% activation sample, and SFT jointly fine-tunes all adapters. Ablations indicate that later layers can be replaced more aggressively than early layers, and that different low-rank parameterizations behave similarly at matched parameter counts. The paper claims that Llama2-7b under the one-for-one adjacent replacement schedule called Tnext saves 42.8% model storage and 42.2% wall-clock inference time on a mobile benchmark.

Load-bearing premise

The load-bearing premise is that a later MLP layer's function can be captured by an earlier layer's frozen weights plus low-rank adapters of rank 400, and that a per-layer output-matching warmup on only 10% of the activation data is a sufficient starting point for the joint fine-tune to restore the model.

Editorial extensions

If this is right

  • With the Tnext schedule (stored ratio 56%), SHARP closes most of the perplexity gap across Arxiv-math, DialogSum, GPT4-Alpaca, Dolly, and OpenOrca using at most 50k fine-tuning examples, so a single post-training pass can cut stored MLP parameters by 38%.
  • More aggressive schedules such as Tmore (storing only 8 of 32 layers) still recover perplexity to a small gap, suggesting the sharing limit is well beyond one-for-one layer replacement.
  • Because the original weights stay frozen and only low-rank adapters are trained, SHARP composes with 4-bit quantization at roughly a 1% performance drop, so further compression can be stacked on top.
  • The ablation result that Tback beats Tfront at matched storage means replacement should be concentrated in later layers, giving deployers a positional rule for where to share.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not claimed by the paper but directly testable: if the redundancy is a general property of Transformer MLPs, the same SLW-plus-SFT recipe should transfer to larger pretrained models with rank-400 adapters and a 50k-example budget; a negative result on a much larger model would bound the method's scope to the 3B-7B class studied here.
  • The layer-sensitivity ablation suggests a task-aware sharing policy: replace later layers that the target benchmark barely uses, and keep the layers that knowledge-memorization tasks depend on, which could push compression beyond the uniform Tnext schedule.
  • A natural extension left implicit is input-dependent sharing, where the number of reused layers is decided per token or per prompt, potentially recovering even more latency on easy inputs without sacrificing rare reasoning cases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SHARP, a post-training method for compressing pretrained LLMs by sharing weights of adjacent layers and adding low-rank recovery parameters. A two-stage procedure is used: Single Layer Warmup (SLW) aligns the outputs of shared layers with L2 loss, followed by Supervised Fine-Tuning (SFT) of the recovery parameters. Experiments on Llama2-7b report that perplexity can be largely recovered on several in-distribution datasets with roughly 38%–65% reduction in stored MLP parameters, along with ablations on replacement patterns, candidate transformations, rank, and dataset size. On downstream tasks the method performs better on knowledge-memorization tasks than on reasoning tasks. The paper also reports mobile-device savings of 42.8% storage and 42.2% inference time.

Significance. If the perplexity-recovery claim holds, SHARP is a practically relevant compression method that requires only fine-tuning-scale data, not pretraining-scale resources. The observation that different low-rank parameterizations perform alike when parameter budgets are matched is a useful and somewhat surprising empirical finding. The cross-dataset results in Appendix D.1 and the LLaMA3.2-3B result in Appendix D.2 strengthen the generality of the core recovery claim beyond the in-distribution setting. The main caveat is that the headline mobile savings are measured on a simplified variant without the LoRA recovery parameters, so the paper's most visible practical claim is not yet supported for the method as defined.

major comments (4)
  1. [Section 3.5 and Appendix C.3] The headline savings in Table 7 are measured on a model that is not the SHARP model evaluated elsewhere. Appendix C.3 states that the deployed model is 'a simplified version of SHARP (Tnext) where we removed the LoRA parameters' and that only reference layers are stored, with target layers produced by calling the same layer multiple times. This omits the storage and computation of the rank-400 recovery parameters that define SHARP. In addition, the on-device model size (2.31GB vs 4.04GB, about 56% of the original) matches sharing 14 of 32 entire transformer layers, whereas the quality experiments in Tables 2 and 4 focus on sharing MLP layers while keeping attention weights intact. The reported 42.8%/42.2% savings are therefore an upper bound for a direct-sharing architecture, not for SHARP with recovery parameters. Please re-measure with the actual SHARP architecture, including LoRA adapters and consistent layer-scope, or explicitly relabel the numbers as a direct-sharing upper bound.
  2. [Section 3.4 and Table 6] The downstream evaluation shows substantial degradation on reasoning tasks even for the mild Tnext configuration: GSM8k drops from 14.2 to 3.6, ARC-Challenge from 43.3 to 34.7, and PIQA from 78.1 to 72.6. The text acknowledges this gap, but the abstract and conclusion present SHARP as generally 'recovering model performance' and accelerating inference without this scope qualifier. The central claim should be stated more precisely: recovery is demonstrated for perplexity and for several knowledge-memorization/commonsense tasks, while complex reasoning is not well recovered at the tested data budget. The paper should either add a prominent limitation statement or temper the general claim.
  3. [Section 3.1 and Tables 2, 4, 6] All quantitative results appear to come from single runs, with no standard deviations, confidence intervals, or seeds reported. This is particularly important in Table 2, where the claimed recovery gaps are small (e.g., 3.2 vs 3.0 for Tnext on Arxiv-math), and in Table 6, where several differences are within a few points. Without repeated runs, it is difficult to know whether the ranking of replacement types in Table 4 or the comparison with baselines in Table 2 is statistically meaningful. Please report mean and variance across at least three seeds for the main tables, or provide a justification for why run-to-run variance is negligible.
  4. [Section 3.1 and Appendix D.1] The in-distribution perplexity evaluation is performed on a held-out 1% slice of the same dataset used for fine-tuning, so the primary evidence is partly self-referential. The independent evidence in Appendix D.1 (SHARP trained on GPT4-Alpaca and evaluated on four other tasks) and in Table 10 (LLaMA3.2-3B) is important and should be promoted into the main text, since it addresses the concern that the recovery could be specific to the fine-tuning distribution. At minimum, the main text should state clearly that the headline 'in-distribution' numbers are same-distribution held-out perplexity, while cross-distribution results appear only in the appendix.
minor comments (6)
  1. [Abstract and title] The title as rendered contains spacing artifacts ('A CCELERATING', 'SH ARING'); these should be corrected in the camera-ready version.
  2. [Figure 2] The caption refers to a 'dotted line' for the original-model perplexity, but the text elsewhere refers to a 'dash line'; please make the line-style references consistent.
  3. [Section 3.5 and Appendix C.3] Appendix C.3 says the models use both 8-bit dynamic quantization and 4-bit weight quantization, while Table 7 labels them as 4-bit; please specify the final deployed precision unambiguously.
  4. [Table 5] The rank values for g1, g2, and g3 are chosen to match total parameter counts, but the parameter-count formulas for these transformations are not given; please add them so the reader can verify the matching.
  5. [Figure 4] The y-axis is defined as 'Baseline - Modified Model', so a larger value indicates a larger performance drop; the caption statement that 'the lower the value the better' appears to reverse this and should be corrected.
  6. [Section 2.2.2] The term 'warmup' is used both for the SLW stage and for the learning-rate warmup within SFT; consider using a different name for one of them to avoid confusion.

Circularity Check

1 steps flagged · score 6.0 of 10

Mobile storage/time savings are measured on a LoRA-free direct-sharing variant and are reported as SHARP's savings; the main recovery results are otherwise self-contained.

  1. self definitional [Section 3.5 (Table 7) and Appendix C.3; cf. Section 2.2.2 and Eq. (1)]
    "Overall, our SHARP saves 42.2% run time and 42.8% model storage compared to the original Llama2-7b. ... we tested: (1) the original Llama2-7b and (2) a simplified version of SHARP (Tnext) where we removed the LoRA parameters. We only store the reference layers, and call those layers multiple times for the target layers in model forwarding."

    SHARP is defined (Section 2.2.2) as sharing layers together with low-rank recovery parameters, and Eq. (1) counts those parameters in the compression ratio s. The mobile benchmark instead deletes the LoRA adapters and stores only reference layers, which is exactly the Direct Sharing baseline of Figure 1(c). The reported 42.8% storage saving is therefore the saving of the stripped architecture, not of SHARP; it is forced by omitting the fitted recovery parameters whose storage and compute Eq. (1) includes. For Tnext with r=400, Eq. (1) gives s about 62% (a 38% MLP-parameter saving), not a 42.8% total-model saving. The headline mobile numbers thus reduce by construction to the direct-sharing input variant.

full rationale

The paper's main algorithmic contribution is the two-stage recovery procedure (SLW followed by SFT) for restoring perplexity after layer sharing, and that part is not circular: the perplexity evaluations use a held-out 1% slice of each dataset, and Appendix D.1 explicitly tests transfer to other datasets, providing independent grounding for the recovery claim. The DejaVu and MobileLLM citations, though overlapping with an author, are not load-bearing because the paper supplies its own direct-replacement evidence in Figure 2. The one genuinely circular element is the mobile latency/storage headline: the reported savings are obtained from a 'simplified version of SHARP (Tnext) where we removed the LoRA parameters,' i.e., from the direct-sharing architecture that SHARP is defined to augment. Since the abstract and conclusion attribute the 42.8% storage and 42.2% time savings to SHARP itself, and since Eq. (1) explicitly includes LoRA parameters in SHARP's compression accounting, those headline numbers are equivalent by construction to the savings of the no-recovery baseline, not of the proposed method. This is a partial circularity affecting one prominent claim, while the core recovery results retain independent content.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger is moderate: the method's success rests on empirical assumptions (layer similarity, low-rank representability, warmup benefit) rather than on derived guarantees. There are no invented physical entities, and the free parameters are mostly hyperparameters of the training pipeline; the replacement-type choice is the most consequential hand-selected structural parameter.

free parameters (5)
  • LoRA rank r = 400 (also 5, 20 in ablations)
    Chosen by hand; directly sets the compression ratio s and the capacity of the recovery adapter.
  • SLW learning rate = 1e-3
    Fixed Adam LR for the single-layer warmup (Section 3.1), affects convergence of adapter initialization.
  • SFT learning rate = 2e-5
    Standard Open-Instruct LR for the joint fine-tuning stage (Section 3.1).
  • Recovery data fraction for SLW = 10% (5-30% per task)
    Fraction of training data used to compute activations and fit layer-wise output matching (Section 3.1).
  • Replacement type = Tnext/Tback/Tmore/Tmax selected per experiment
    Which layers are shared is a structural hyperparameter chosen after inspecting perplexity; results in Tables 2-4 depend on it.
assumptions (4)
  • domain assumption Consecutive layer output similarity in pretrained LLMs is high enough that one layer can approximate another (motivated by Deja Vu and verified in Figure 2).
    Used in Section 2.1 to justify layer sharing; the paper provides direct PPL evidence but the assumption is empirical and model-specific.
  • domain assumption Layer-wise L2 output matching (SLW) is an effective warmup for later joint SFT.
    Section 2.2.2 claims SLW improves convergence when many layers are shared; validated only empirically in Table 3, no proof of optimization benefit.
  • domain assumption The difference between adjacent MLP layers can be represented by low-rank adapters of rank 400.
    Section 2.2 defines recovery via g0 (LoRA) and Section 3.2 uses rank 400; if the inter-layer map is not low-rank, the compression claim fails.
  • ad hoc to paper First and last layers should be kept intact.
    Section 2.2.3 skips these layers based on Figure 2 observations; this is a design choice specific to the method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters." pith.science (2026). https://pith.science/paper/7OHHNKHJ

@misc{pith2026250207832,
  author       = {Pith},
  title        = {Pith review of: SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OHHNKHJ}},
  note         = {Machine review of arXiv:2502.07832}
}
read the original abstract

While Large language models (LLMs) have advanced natural language processing tasks, their growing computational and memory demands make deployment on resource-constrained devices like mobile phones increasingly challenging. In this paper, we propose SHARP (SHaring Adjacent Layers with Recovery Parameters), a novel approach to accelerate LLM inference by sharing parameters across adjacent layers, thus reducing memory load overhead, while introducing low-rank recovery parameters to maintain performance. Inspired by observations that consecutive layers have similar outputs, SHARP employs a two-stage recovery process: Single Layer Warmup (SLW), and Supervised Fine-Tuning (SFT). The SLW stage aligns the outputs of the shared layers using L_2 loss, providing a good initialization for the following SFT stage to further restore the model performance. Extensive experiments demonstrate that SHARP can recover the model's perplexity on various in-distribution tasks using no more than 50k fine-tuning data while reducing the number of stored MLP parameters by 38% to 65%. We also conduct several ablation studies of SHARP and show that replacing layers towards the later parts of the model yields better performance retention, and that different recovery parameterizations perform similarly when parameter counts are matched. Furthermore, SHARP saves 42.8% in model storage and reduces the total inference time by 42.2% compared to the original Llama2-7b model on mobile devices. Our results highlight SHARP as an efficient solution for reducing inference costs in deploying LLMs without the need for pretraining-scale resources.

Figures

Figures reproduced from arXiv: 2502.07832 by the authors.

Figure 1
Figure 1. (a) Regular pretrained baseline model without layer sharing. (b) Adjacent layer sharing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Language models are robust to the replacement of adjacent MLP layers. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average rela￾tive error between adjacent layers (mean of {∥Θi+1 − Θi∥/∥Θi∥}31 i=1). What’s more, it’s also worth noting that even though layers with recov￾ery parameters can be good approximations of each other, the parame￾ters themselves are quite different. This is supported by evaluating the average relative error between adjacent layers as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Impact of different layers on model capabilities. The x-axis denotes the index of the zero [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

98 extracted references · 16 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\...

    2022

  3. [3]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019

    2019

  4. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [5]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    2023

  6. [6]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  7. [7]

    Gpt-neox-20b: An open-source autoregressive language model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022

    arXiv 2022

  8. [8]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

Show all 98 references
  1. [9]

    Sparks of artificial general intelligence: Early experiments with gpt-4

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023

    2023 arXiv

  2. [10]

    Code alpaca: An instruction-following llama model for code generation

    Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023

    2023

  3. [11]

    Learning to maximize mutual information for chain-of-thought distillation

    Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding. Learning to maximize mutual information for chain-of-thought distillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp.\ ...

    2024 doi

  4. [12]

    Longlora: Efficient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023

    2023 arXiv

  5. [13]

    D ialog S um: A real-life scenario dialogue summarization dataset

    Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. D ialog S um: A real-life scenario dialogue summarization dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp.\ 5062--5074, ...

    2021 doi

  6. [14]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  7. [15]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019

    2019

  8. [16]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018

    2018 arXiv

  9. [17]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  10. [18]

    Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/12/dolly-first-ope...

    2023

  11. [19]

    Mutual: A dataset for multi-turn dialogue reasoning

    Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. Mutual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Conference of the Association for Computational Linguistics. Association for Computational Linguistics, 2020

    2020

  12. [20]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  13. [21]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  14. [22]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 0 30318--30332, 2022

    2022

  15. [23]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  16. [24]

    Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster

    Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv preprint arXiv:2304.03208, 2023

    2023 arXiv

  17. [25]

    Blockwise compression of transformer-based models without retraining

    Gaochen Dong and Wei Chen. Blockwise compression of transformer-based models without retraining. Neural Networks, 171: 0 423--428, 2024

    2024

  18. [26]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  19. [27]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  20. [28]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  21. [29]

    Learn-to-share: A hardware-friendly transfer learning framework exploiting computation and parameter sharing

    Cheng Fu, Hanxian Huang, Xinyun Chen, Yuandong Tian, and Jishen Zhao. Learn-to-share: A hardware-friendly transfer learning framework exploiting computation and parameter sharing. In International Conference on Machine Learning, pp.\ 3469--3479. PMLR, 2021

    2021

  22. [30]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  23. [31]

    The unreasonable ineffectiveness of the deeper layers

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024

    2024 arXiv

  24. [32]

    Textbooks are all you need

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023

    2023 arXiv

  25. [33]

    Compressing pre-trained language models using progressive low rank decomposition

    Habib Hajimolahoseini, Mehdi Rezagholizadeh, Vahid Partovinia, Marzieh Tahaei, Omar Mohamed Awad, and Yang Liu. Compressing pre-trained language models using progressive low rank decomposition. Advances in Neural Information Processing Systems, 2021

    2021

  26. [34]

    Distilling the knowledge in a neural network

    Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  27. [35]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  28. [36]

    Language model compression with weighted low-rank factorization

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022

    2022 arXiv

  29. [37]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  30. [38]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  31. [39]

    safetensors

    HuggingFace. safetensors. https://github.com/huggingface/safetensors, 2022. Accessed: 2024-04-27

    2022

  32. [40]

    Camels in a changing climate: Enhancing lm adaptation with tulu 2

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702, 2023

    2023 arXiv

  33. [41]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  34. [42]

    Multi-domain neural machine translation with word-level adaptive layer-wise domain mixing

    Haoming Jiang, Chen Liang, Chong Wang, and Tuo Zhao. Multi-domain neural machine translation with word-level adaptive layer-wise domain mixing. arXiv preprint arXiv:1911.02692, 2019

    1911 arXiv

  35. [43]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 2017. ...

    2017

  36. [44]

    Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm

    Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. arXiv preprint arXiv:2403.05527, 2024

    2024 arXiv

  37. [45]

    arxiv-math-instruct-50

    Matthew Kenny. arxiv-math-instruct-50. 2023

    2023

  38. [46]

    Squeezellm: Dense-and-sparse quantization

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023 a

    2023 arXiv

  39. [47]

    Full stack optimization of transformer inference: a survey

    Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W Mahoney, et al. Full stack optimization of transformer inference: a survey. arXiv preprint arXiv:2302.14017, 2023 b

    2023 arXiv

  40. [48]

    Speculative decoding with big little decoder

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    Reformer: The efficient transformer

    Nikita Kitaev, ukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  42. [50]

    o pf, Yannic Kilcher, Dimitri von R \

    Andreas K \"o pf, Yannic Kilcher, Dimitri von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Info...

    2024

  43. [51]

    Loftq: Lora-fine-tuning-aware quantization for large language models

    Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659, 2023 a

    2023 arXiv

  44. [52]

    Losparse: Structured compression of large language models based on low-rank and sparse approximation

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. In International Conference on Machine Learning, pp.\ 20336--20350. PMLR, 2023 b

    2023

  45. [53]

    The microsoft toolkit of multi-task deep neural networks for natural language understanding

    Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, et al. The microsoft toolkit of multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:2002.07972, 2020

    2002 arXiv

  46. [54]

    Llm-qat: Data-free quantization aware training for large language models

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888, 2023 a

    2023 arXiv

  47. [55]

    Mobilellm: Optimizing sub-billion parameter language models for on-device use cases

    Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402.14905, 2024

    2024 arXiv

  48. [56]

    Deja vu: Contextual sparsity for efficient llms at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pp.\ 22137--22176. P...

    2023

  49. [57]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp.\ 22631--22648. PMLR, 2023

    2023

  50. [58]

    Fineweb-edu, May 2024

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu, May 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

    2024

  51. [59]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023

    2023

  52. [60]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022

    2022

  53. [61]

    Relu strikes back: Exploiting activation sparsity in large language models

    Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564, 2023

    arXiv

  54. [62]

    Orca: Progressive learning from complex explanation traces of gpt-4

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707, 2023

    2023 arXiv

  55. [63]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (eds.), Proceedings of the Conference...

    2022

  56. [64]

    The lambada dataset, Aug 2016

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset, Aug 2016

    2016

  57. [65]

    Hovy, Pamela Forner, \'A lvaro Rodrigo, Richard F

    Anselmo Pe \ n as, Eduard H. Hovy, Pamela Forner, \'A lvaro Rodrigo, Richard F. E. Sutcliffe, and Roser Morante. Qa4mre 2011-2013: Overview of question answering for machine reading evaluation. In CLEF, 2013

    2011

  58. [66]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  59. [67]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  60. [68]

    Efficiently scaling transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5: 0 606--624, 2023

    2023

  61. [69]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019

    1907 arXiv

  62. [70]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608, 2024

    2024 arXiv

  63. [71]

    S-lora: Serving thousands of concurrent lora adapters

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285, 2023

    2023 arXiv

  64. [72]

    Turbo sparse: Achieving llm sota performance with minimal activated parameters

    Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, and Haibo Chen. Turbo sparse: Achieving llm sota performance with minimal activated parameters. arXiv preprint arXiv:2406.05955, 2024

    2024 arXiv

  65. [73]

    Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali...

    2022

  66. [74]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  67. [75]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  68. [76]

    Kroneckerbert: Learning kronecker decomposition for pre-trained language models via knowledge distillation

    Marzieh S Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. Kroneckerbert: Learning kronecker decomposition for pre-trained language models via knowledge distillation. arXiv preprint arXiv:2109.06243, 2021

    2021 arXiv

  69. [77]

    C ommonsense QA : A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. C ommonsense QA : A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human La...

    2019 doi

  70. [78]

    Multi-domain neural machine translation

    Sander Tars and Mark Fishel. Multi-domain neural machine translation. arXiv preprint arXiv:1805.02282, 2018

    2018 arXiv

  71. [79]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  72. [80]

    Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

    Inar Timiryasov and Jean-Loup Tastet. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty. arXiv preprint arXiv:2308.02019, 2023

    2023 arXiv

  73. [81]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  74. [82]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023

    2023

  75. [83]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In NUT@EMNLP, 2017

    2017

  76. [84]

    Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity

    Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. arXiv preprint arXiv:2309.10285, 2023 a

    2023 arXiv

  77. [85]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023 b

    2023 arXiv

  78. [86]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp.\ 38087--38099. PMLR, 2023

    2023

  79. [87]

    Wizard LM : Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizard LM : Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024. ...

    2024

  80. [88]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35: 0 27168--27183, 2022

    2022

  81. [89]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024 a

    2024 arXiv

  82. [90]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  83. [91]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  84. [92]

    Adaptive-precision framework for sgd using deep q-learning

    Wentai Zhang, Hanxian Huang, Jiaxi Zhang, Ming Jiang, and Guojie Luo. Adaptive-precision framework for sgd using deep q-learning. In 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp.\ 1--8. IEEE, 2018

    2018

  85. [93]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing S...

    2024

  86. [94]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

    2024

  87. [95]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  88. [96]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  89. [97]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  90. [98]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.