EfficientLLM: Efficiency in Large Language Models

Haolong Jia; Huichi Zhou; Jianfeng Gao; Keerthiram Murugesan; Lichao Sun; Lifang He; Rong Zhou; Wei Song; Weixiang Sun; Yanfang Ye

REVIEW 5 major objections 5 minor 2 cited by

EfficientLLM: Efficiency in Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that no LLM efficiency technique is universally optimal—across more than 100 model-technique pairs, every method improves at least one resource metric while regressing another, so the right choice depends on…

desk verdict A genuinely useful benchmark with a mislabeled headline claim: the 3.9x memory/energy result is really just model compression, and the accuracy drops are larger than stated. read the letter →

arxiv 2505.13840 v1 pith:DXTGS63X submitted 2025-05-20 cs.CL cs.AIcs.LG

Zhengqing Yuan , Weixiang Sun , Yixin Liu , Huichi Zhou , Rong Zhou , Yiyang Li , Zheyuan Zhang , Wei Song

show 8 more authors

Yue Huang Haolong Jia Keerthiram Murugesan Yu Wang Lifang He Jianfeng Gao Lichao Sun Yanfang Ye

This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords largelanguagemodelsefficiencybenchmarkquantizationmixture-of-expertsparameter-efficientfine-tuningattentionmechanismsenergyconsumptionmulti-objectivetrade-offs

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EfficientLLM is a benchmark and large-scale empirical study that evaluates LLM efficiency techniques along three axes—architecture pretraining, fine-tuning, and inference—across more than 100 model–technique pairs on a production GH200/H200 cluster. The paper's central claim is that no efficiency technique is universally optimal: every method improves at least one resource metric while regressing another, so efficiency must be treated as a multi-objective trade-off. It quantifies these trade-offs with six metrics, finding, for example, that int4 quantization cuts memory and energy by up to $3.9\times$ at a 3–5% accuracy drop, while MoE reduces FLOPs and boosts accuracy but inflates VRAM by about 40%. The paper also claims the optima are task- and scale-dependent, and that the same techniques transfer to vision and vision–language models.

What carries the argument

The load-bearing object is the EfficientLLM measurement stack: a modality-agnostic harness that logs six fine-grained metrics—Average Memory Utilization (time-averaged memory pressure), Peak Compute Utilization (real GPU occupancy vs. peak), Average Latency, Token/Sample/Inference Throughput, Average Energy Consumption (mean watts), and Model Compression Rate (size reduction adjusted for performance retention)—under a unified three-axis taxonomy. The same collector is applied to architecture pretraining, fine-tuning, and bit-width quantization experiments, so trade-offs across techniques are directly comparable in a single coordinate system.

What would settle it

Run the same inference benchmark with a fully specified int4 quantization pipeline—named algorithm, documented calibration data, and fixed kernels—on the same model families; if the average task-score drop exceeds $5\%$ or the memory/energy reduction falls well short of the reported $3.9\times$, the headline int4 trade-off is implementation-specific rather than a general property of int4 quantization.

Watch

Extended reading notes

Core claim

After pretraining, fine-tuning, and serving over one hundred model–technique combinations spanning 0.5B to 72B parameters, the authors report that every efficiency technique evaluated improves at least one of memory, compute utilization, latency, throughput, energy, or compression while worsening another; no method Pareto-dominates. Specific measured trade-offs include: int4 post-training quantization achieves up to a $3.9\times$ model compression ratio with a 3–5 percentage-point drop in average task scores; Mixture-of-Experts architectures cut training FLOPs by about $1.8\times$ and add up to 3.5 points of accuracy but increase VRAM by roughly 40%; MQA provides the best memory–latency frontier, MLA the lowest perplexity, and NSA the lowest energy consumption; bfloat16 beats float16 by about 6% in latency and 9% in energy on Hopper GPUs; and RSLoRA only surpasses LoRA in efficiency for models of 14B parameters and above. The authors interpret these results as concrete empirical validation that LLM efficiency is a multi-objective optimization problem with no free lunch.

Load-bearing premise

The int4 quantization results are reported without specifying the quantization algorithm, calibration data, or kernels, so the measured $3.9\times$ compression and $3$–$5\%$ accuracy drop may be properties of one unidentified implementation rather than of int4 quantization in general.

Editorial extensions

If this is right

Engineers deploying models under memory or energy limits can treat int4 quantization as a near-free lever when a 3–5% average score drop is acceptable.
Architecture choices should follow the bottleneck: MQA for memory-constrained devices, MLA when language quality dominates, NSA when energy is the constraint.
For fine-tuning models above roughly 14B parameters, RSLoRA outperforms LoRA on both loss and latency, so PEFT selection should be scale-aware.
BF16 is a safer default than FP16 on Hopper-generation GPUs.
Efficiency reporting should present multi-objective trade-offs rather than a single aggregate leaderboard score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A fair, implementation-specified int8 benchmark on stable kernels might well fill the Pareto frontier between int4 and bf16, potentially changing deployment advice once Hopper-class int8 support matures.
The RSLoRA crossover at 14B suggests other PEFT rankings could invert at even larger scales, so small-model comparisons should not be extrapolated to 70B+ checkpoints.
The no-free-lunch pattern is hardware-sensitive: if next-generation GPUs natively accelerate very low-precision arithmetic, the measured int4 advantage over bf16 could shrink or widen, and the rankings should be re-measured on new silicon.
The transfer result to vision and vision-language models invites a further test: applying the same metric suite to video or audio generation models, where KV-cache and memory profiles differ qualitatively, would show how far the trade-offs generalize.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper delivers a real asset—a hundred-scale benchmark of efficiency techniques across attention, PEFT, and quantization, with six metrics and an open pipeline. That is genuinely useful. The qualitative finding that no technique dominates is not surprising but is good to have measured. The LVM/VLM extension is a nice bonus.

The problem is the flagship claim. The abstract says int4 cuts memory/energy by up to 3.9x at a 3–5% accuracy drop. That is not what Table 9 shows. The 3.9x figure is the model compression ratio, not a measured reduction in memory utilization or energy. Actual AMU reductions in Table 9 range from about 1.1x to 1.5x, and AEC sometimes goes up—Phi-4 int4 consumes 319 W versus 217 W in bf16. Accuracy drops are often 6–9%, not 3–5%. So the central quantitative anchor is mislabeled and overstated.

The AMU metric is also defined as a ratio in the formula but reported in GB throughout, which is confusing. The MoE VRAM claim of 40% is not directly supported by the table either—the 1.5B MoE shows ~1.7x AMU versus the 1.5B dense. And the int4 results lack the quantization algorithm, calibration data, and kernel details, so they may not generalize even after the numbers are fixed.

That said, the benchmark artifact and the qualitative trade-off map are worth preserving. The flaws are serious but fixable. This paper should go to peer review, because a good referee can force the corrections and make the published version reliable. As it stands, I would not cite the quantitative claims.

Referee Report

5 major / 5 minor

Summary. The paper introduces EfficientLLM, a benchmark and empirical study that evaluates efficiency techniques across LLM pretraining (efficient attention variants, positional encodings, MoE, attention-free architectures), fine-tuning (LoRA-family methods, parameter freezing, full fine-tuning), and inference quantization (int4, float16, bfloat16) on a production cluster of GH200 and H200 GPUs. It defines six metrics (AMU, PCU, AL, TT/ST/IT, AEC, MCR), reports results for over 100 model-technique pairs, and extends the evaluation to LVMs and VLMs. The central claims are that no single technique is universally optimal, that efficiency optima depend on task and model scale, and that techniques validated on LLMs transfer to vision-language models. The authors state that datasets, evaluation pipelines, and leaderboards are open-sourced.

Significance. The study addresses a genuine gap: systematic, large-scale, end-to-end efficiency measurements with energy reporting on modern hardware are rare, and the paper's explicit multi-metric framing is a useful contribution. The multimodal extension and the finding that every tested technique improves some metric while regressing another are concrete and falsifiable. The paper also ships open-sourced pipelines and leaderboards, which supports reproducibility and follow-up work. However, the flagship quantitative claim about int4 quantization cutting memory and energy by up to 3.9x is not supported by the paper's own Table 9, and the associated accuracy-drop claim is also overstated. This qualification substantially weakens the headline contribution until the numbers and definitions are corrected.

major comments (5)

[Abstract, Section 2.1, Table 9] The claim that 'int4 quantization cuts memory/energy by up to 3.9x' is not supported by Table 9. The 3.9x values in the MCR column are raw model-size compression ratios, not measured reductions in AMU or AEC. For DeepSeek-R1-Distill-Qwen-1.5B, the AMU reduction from bf16 to int4 is 21.26 GB to 19.49 GB (about 1.09x), and the AEC reduction is 144.39 W to 134.89 W (about 1.07x). For Qwen2.5-32B, AMU drops from 71.33 GB to 48.30 GB (about 1.48x), while AEC drops from 279.23 W to 214.57 W (about 1.30x). For Phi-4, AEC increases from 217.16 W to 319.11 W, and for Yi-34B from 295.10 W to 334.46 W. The abstract and Section 2.1 must either report MCR as a separate compression metric or be revised to state the actual measured AMU and AEC reductions, which are much smaller and sometimes negative.
[Table 9, Avg Perf. column] The abstract's '3-5% average task score drop' for int4 is not supported by Table 9. Relative drops from bfloat16 to int4 include DeepSeek-R1-Distill-Llama-8B from 0.3421 to 0.3116 (about 8.9%), Qwen2.5-7B from 0.4448 to 0.4152 (about 6.7%), and Phi-3.5-mini from 0.3683 to 0.3355 (about 8.9%). If the intended reporting is absolute percentage-point differences, that must be stated explicitly and the values recomputed; otherwise the claim should be revised to reflect the actual per-model degradation, which exceeds 5% for several models.
[Section 5.5, 'Evaluation Results' and Table 9] The int4 quantization experiment is under-specified in a way that affects the central inference claim. The paper reports only 'int4' as the precision and does not state the quantization algorithm, calibration data, group size, back-end kernels, or evaluation harness. Because int4 implementations vary widely in both speed and accuracy, the measured trade-offs cannot be attributed to int4 quantization in general. The authors should document the exact quantization setup or restrict the claims to the specific implementation used.
[Section 5.1.1, Eq. (1)] The definition of AMU is internally inconsistent. The text states that AMU is 'the ratio of the memory used by the model throughout the entire training process to the total available memory on the device, averaged over time,' but Eq. (1) defines AMU = (1/T) ∫ Memory Used(t) dt with no normalization by total available memory. The values in Tables 3-9 are reported in GB, which further indicates that the ratio interpretation was not applied. The paper should either correct Eq. (1) to include the normalization or explain why the reported values are not ratios. The direction of the metric is also inconsistent: Section 5.1.1 says higher AMU indicates better utilization, while Table 9 lists AMU↓ as better.
[Section 5.1.3, Eq. (9) vs Table 9] The definition of MCR in Eq. (9) includes a performance-retention factor, MCR = (Size_original / Size_compressed) × (Performance_compressed / Performance_original), but the MCR values in Table 9 (e.g., 3.87 for DeepSeek-R1-Distill-Qwen-1.5B int4) appear to be pure size ratios without any performance adjustment. If the performance factor is applied, the reported MCR values would be lower. The paper must clarify which definition is used in Table 9 and in the associated '3.9x' claim, and reconcile the formula with the reported numbers.

minor comments (5)

[Section 5.3.3 and Table 5] The text reports AEC for the MoE 1.5Bx8 model as 405321.86 J, while Table 5 lists AEC in watts as 692.45 W; the units for AEC are inconsistent across the text and tables and should be unified.
[Table 4 and Section 5.3.2] The TT column is labeled 'Tokens/param/s' but the text describes the value as 'TFloats,' and the units in the table (e.g., 8.98x10^-2) are not clearly tokens per parameter per second. The metric definition and table units should be aligned.
[Table 7] Several rows appear to contain copy-paste errors: Qwen-2.5-14B LoRA and LoRA-plus loss values (0.4795 and 0.4621) are identical to the Qwen-2.5-7B rows, and the Llama-3.2-3B 'full*' row duplicates the LoRA row values. These entries should be checked and corrected.
[Section 5.3.2] The text states that RoPE achieved PPL = 8.04, but Table 4 reports PPL = 8.09 for the same configuration; the discrepancy should be resolved.
[Figures 2 and 5 and Section 5.1.4] Figure 2's caption contains a typo ('frist'), Figure 5's caption contains 'deilted,' and Section 5.1.4 contains an unresolved cross-reference ('like Section ??'). The bottom block of Figure 2 also appears to have missing or duplicated values for the int4 row.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: benchmark conclusions rest on direct measurements; the disputed 3.9x claim is a correctness issue, not a circular derivation.

full rationale

EfficientLLM is a measurement study rather than a derivation. The metrics in Section 5.1 (AMU, PCU, AL, TT, ST, IT, AEC, MCR) are explicitly defined measurement quantities, not fitted parameters, and the core insights in Section 2 are empirical summaries of the tabulated measurements (Tables 3-11). No prediction is obtained by fitting a parameter to a subset of data and then re-announcing it; no load-bearing self-citation chain forces a conclusion; and no uniqueness theorem is imported from the authors' own prior work. The only candidate for a definitional concern is the abstract's statement that int4 quantization 'cuts memory/energy by up to 3.9x'. That figure is the MCR column in Table 9, and MCR is defined in Eq. 9 as a model-size compression ratio adjusted by performance, not as measured memory utilization or energy consumption. The paper's own AMU and AEC columns show much smaller memory reductions and sometimes increased energy for int4. This is an internal-consistency and correctness problem in how a definition-based metric is described in prose, not a circular derivation: the paper never defines MCR to be AMU or AEC, so the headline claim is a mislabeling rather than a reduction of a prediction to its inputs. Similarly, the Efficiency Score in Figure 4 is said to be a weighted harmonic combination, but the weights are unspecified; this is a transparency limitation, not circularity. The benchmark's broader conclusions — that no single technique dominates on all measured axes, that relative efficiency is task- and scale-dependent, and that the technique rankings transfer to vision models — are grounded in the reported measurements and do not reduce by construction to the metric definitions. Accordingly, no significant circularity is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on measured quantities and metric definitions, not on fitted constants. The main hidden choices are the undisclosed Efficiency Score weights, the unspecified int4 quantization implementation, and the per-scale batch sizes, all of which can change the reported rankings.

free parameters (5)

Efficiency Score weights = not specified
The Efficiency Score shown in Figures 4 and 7 is a weighted harmonic combination of normalized resource metrics, but the weights are not reported, so rankings depend on undisclosed choices.
Micro-batch sizes = varies by scale (e.g., 4, 2, 1 in Table 3)
Batch size affects latency, memory, and throughput measurements; the paper sets different micro-batch sizes across model scales and techniques.
Pretraining context length = 8192 tokens
All pretraining runs use an 8K context (Tables 4 and 6), bounding the generality of the latency and memory results.
MoE top-k routing = 2
MoE experiments use top-2 expert routing (Table 5), so the reported FLOP and VRAM trade-offs are specific to this routing choice.
FineWeb-Edu score threshold = 3
The pretraining corpus uses documents with an educational value score of at least 3, which influences downstream perplexity.

assumptions (4)

standard math The No-Free-Lunch theorem is the appropriate organizing principle for LLM efficiency trade-offs.
The paper invokes NFL (Section 2.1) to frame the observation that no technique dominates, but this is an interpretive framework, not a proof that the specific measured trade-offs follow from NFL.
domain assumption Perplexity, FID, and the six proposed efficiency metrics are valid proxies for model quality and resource use.
All conclusions rest on these metrics being meaningful and consistently measured; the AMU metric is defined in a self-contradictory way (Section 5.1.1).
domain assumption The GH200/H200 cluster and the Megatron-Core/LlamaFactory frameworks are representative of real deployment conditions.
The paper generalizes its trade-off numbers to the broader LLM ecosystem without comparing across other hardware or software stacks.
domain assumption Energy measurements from the cluster are accurate and unaffected by concurrent workloads.
AEC is computed from instantaneous power readings, but no validation or isolation procedure is described.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EfficientLLM: Efficiency in Large Language Models." pith.science (2026). https://pith.science/paper/DXTGS63X

@misc{pith2026250513840,
  author       = {Pith},
  title        = {Pith review of: EfficientLLM: Efficiency in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXTGS63X}},
  note         = {Machine review of arXiv:2505.13840}
}

read the original abstract

Large Language Models (LLMs) have driven significant progress, yet their growing parameter counts and context windows incur prohibitive compute, energy, and monetary costs. We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale. Conducted on a production-class cluster (48xGH200, 8xH200 GPUs), our study systematically explores three key axes: (1) architecture pretraining (efficient attention variants: MQA, GQA, MLA, NSA; sparse Mixture-of-Experts (MoE)), (2) fine-tuning (parameter-efficient methods: LoRA, RSLoRA, DoRA), and (3) inference (quantization methods: int4, float16). We define six fine-grained metrics (Memory Utilization, Compute Utilization, Latency, Throughput, Energy Consumption, Compression Rate) to capture hardware saturation, latency-throughput balance, and carbon cost. Evaluating over 100 model-technique pairs (0.5B-72B parameters), we derive three core insights: (i) Efficiency involves quantifiable trade-offs: no single method is universally optimal; e.g., MoE reduces FLOPs and improves accuracy but increases VRAM by 40%, while int4 quantization cuts memory/energy by up to 3.9x at a 3-5% accuracy drop. (ii) Optima are task- and scale-dependent: MQA offers optimal memory-latency trade-offs for constrained devices, MLA achieves lowest perplexity for quality-critical tasks, and RSLoRA surpasses LoRA efficiency only beyond 14B parameters. (iii) Techniques generalize across modalities: we extend evaluations to Large Vision Models (Stable Diffusion 3.5, Wan 2.1) and Vision-Language Models (Qwen2.5-VL), confirming effective transferability. By open-sourcing datasets, evaluation pipelines, and leaderboards, EfficientLLM provides essential guidance for researchers and engineers navigating the efficiency-performance landscape of next-generation foundation models.

Figures

Figures reproduced from arXiv: 2505.13840 by the authors.

**Figure 1.** Overview of the EfficientLLM framework. In the Architecture Pretraining Efficiency dimension, we systematically assessed how different architectural optimizations during the pretraining stage impact model performance and efficiency. We tested multiple attention mechanisms (e.g., MQA, GQA, MLA, NSA) across varying model scales (0.5B to 3B parameters) and found that MQA excels in memory utilization and latency, making… view at source ↗

**Figure 2.** Ranking of LLM training and inference efficiency and performance across various techniques. The chart compares [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Efficiency LLM Results. This figure illustrates the performance and efficiency trade-offs of various architectural [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Assessment of training and fine-tuning efficiency across multiple LLMs. (a) Comparison of different fine-tuning methods (LoRA, LoRA-plus, RSLoRA, DoRA, PISSA, Freeze, and full fine-tuning using DeepSpeed) across seven model architectures (Llama-3.2-1B/3B, Llama-3.1-8B…

**Figure 5.** Figure 5: Assessment of quantization-based inference efficiency across model precisions. Radar plots compare normalized efficiency metrics across three quantization formats: bfloat16, float16, and int4. Each plot evaluates models from DeepSeek, Qwen, Phi, and Yi families using s…

**Figure 6.** Figure 6: The development trends of computational efficiency and memory capacity across NVIDIA GPU series. Note that different [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Scalability analysis of EfficientLLM for LVM and VLM optimization. (a) Normalized efficiency scores across five metrics (FID↑, AMU↑, AL↑, TT↑, AEC↑) for attention variants (MHA, MQA, GQA, MLA, NSA) in three DiT-based LVM architectures (DiT-XL/2, L/8, B/4). All metrics …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation
cs.AR 2026-03 conditional novelty 6.0 of 10

An FPGA accelerator for SkipGPT-style pruned LLMs that fuses router/normalization/attention computation to hide nonlinear latency, packs two FP16 products in each DSP, and serves reused KV entries from on-chip URAM to...
APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning
cs.CV 2026-08 conditional novelty 5.0 of 10

A profiling-guided, LLM-driven framework combines structured pruning and mixed-precision quantization-aware training, reporting 13-18x bit-operation reductions with modest accuracy loss on ImageNet and CIFAR-10.

Reference graph

Works this paper leans on

299 extracted references · 8 canonical work pages · cited by 2 Pith papers

[1]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877--1901, 2020

1901
[2]

Palm: Scaling language modeling with pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022

arXiv 2022
[3]

Scaling laws for neural language models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020

arXiv 2001
[4]

Training compute-optimal large language models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022

arXiv 2022
[5]

Keivalya Pandya and M. Holia. Automating customer service using langchain: Building custom open-source gpt chatbot for organizations. ArXiv , abs/2310.05421, 2023

arXiv 2023
[6]

Litllm: A toolkit for scientific literature review

Shubham Agarwal, Issam H Laradji, Laurent Charlin, and Christopher Pal. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788 , 2024

arXiv 2024
[7]

A general purpose device for interaction with llms

Jiajun Xu, Qun Wang, Yuhang Cao, Baitao Zeng, and Sicheng Liu. A general purpose device for interaction with llms. In Proceedings of the Future Technologies Conference , pages 613--626. Springer, 2024

2024
[8]

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurIPS , 35:22199--22213, 2022

2022

Show all 299 references

[9]

Trustworthy llms: a survey and guideline for evaluating large language models' alignment

Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.05374 , 2023

2023 arXiv
[10]

Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P....

2024
[11]

On the trustworthiness of generative foundation models: Guideline, assessment, and perspective

Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Yujun Zhou, Yanbo Wang, Jiayi Ye, Jiawen Shi, Qihui Zhang, et al. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective. arXiv preprint arXiv:2502.14296 , 2025

2025 arXiv
[12]

Energy and policy considerations for deep learning in NLP

Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP . In Proceedings of the 57th Annual Meeting of the ACL , pages 3645--3650, 2019

2019
[13]

Efficient large language models: A survey

Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863 , 2023

2023 arXiv
[14]

Beyond efficiency: A systematic survey of resource-efficient large language models

Guangji Bai, Zheng Chai, Chen Ling, et al. Beyond efficiency: A systematic survey of resource-efficient large language models. ArXiv preprint arXiv:2401.00625 , 2024

2024 arXiv
[15]

A survey on efficient inference for large language models

Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. A survey on efficient inference for large language models. ArXiv , abs/2404.14294, 2024

2024 arXiv
[16]

Compressing context to enhance inference efficiency of large language models

Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. pages 6342--6353, 2023

2023
[18]

Generating long sequences with sparse transformers

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019

1904 arXiv
[19]

Gshard: Scaling giant models with conditional computation and automatic sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020

2006 arXiv
[20]

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research , 23(1):5232--5270, 2022

2022
[21]

Fast transformer decoding: One write-head is all you need

Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150 , 2019

1911 arXiv
[22]

Gqa: Training generalized multi-query transformer models from multi-head checkpoints

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 , 2023

2023 arXiv
[23]

Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 , 2024

2024 arXiv
[24]

Native sparse attention: Hardware-aligned and natively trainable sparse attention

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089 , 2025

2025 arXiv
[25]

Mixed precision training

Paulius Micikevicius, Sharan Narang, Jonah Alben, Greg Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations (ICLR) , 2018

2018
[26]

Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In SIGKDD , pages 3505--3506, 2020

2020
[27]

Megatron-LM : Training multi-billion parameter language models using model parallelism

Mohammad Shoeybi, Mostofa Ali Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM : Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019

1909 arXiv
[28]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

2021
[29]

Fedpara: Low-rank hadamard product for communication-efficient federated learning

Hyeon-Woo Nam, Ye-Bin Moon, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficient federated learning. arXiv preprint arXiv:2108.06098 , 2021. Introduces the LoHa variant

2021 arXiv
[30]

Clark, and Mehdi Rezagholizadeh

Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J. Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650 , 2022. Basis for the LoKr variant

2022 arXiv
[31]

One-for-all: Generalized lora for parameter-efficient fine-tuning

Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter-efficient fine-tuning. arXiv preprint arXiv:2306.07967 , 2023

2023 arXiv
[32]

LLM.int8() : 8-bit matrix multiplication for transformers at scale

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8() : 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , pages 12508--12521, 2022

2022
[33]

Gptq: Accurate post-training quantization for generative pre-trained transformers

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 , 2022

2022 arXiv
[34]

Qlora: Efficient finetuning of quantized llms, 2023

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023

2023
[35]

Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023

Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023

2023
[36]

Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems 33 , 2020

2020
[37]

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015

2015 arXiv
[38]

Fast inference from transformers via speculative decoding

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML) , 2023

2023
[39]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th ACM Symposium on Operating Systems Princi...

2023
[40]

Mixtral of experts

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024

2024 arXiv
[41]

Understanding int4 quantization for language models: latency speedup, composability, and failure cases

Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quantization for language models: latency speedup, composability, and failure cases. In International Conference on Machine Learning , pages 37524--37539. PMLR, 2023

2023
[42]

No free lunch theorems for optimization

David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation , 1(1):67--82, 1997

1997
[43]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations (ICLR) , 2022

2022
[44]

Dora: Weight-decomposed low-rank adaptation

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML) , 2024

2024
[45]

A rank stabilization scaling factor for fine-tuning with lora

Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732 , 2023

2023 arXiv
[46]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025

2025 arXiv
[47]

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

2018
[48]

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019

2019
[49]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

2019
[50]

Roberta: A robustly optimized bert pretraining approach, 2019

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019

2019
[51]

Albert: A lite bert for self-supervised learning of language representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019

1909 arXiv
[52]

Leveraging large language models to enhance personalized recommendations in e-commerce

Wei Xu, Jue Xiao, and Jianlong Chen. Leveraging large language models to enhance personalized recommendations in e-commerce. 2024

2024
[53]

Text understanding and generation using transformer models for intelligent e-commerce recommendations

Yafei Xiang, Hanyi Yu, Yulu Gong, Shuning Huo, and Mengran Zhu. Text understanding and generation using transformer models for intelligent e-commerce recommendations. ArXiv , abs/2402.16035, 2024

2024 arXiv
[54]

Recommendation systems in the era of llms

Ipsita Mohanty. Recommendation systems in the era of llms. Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation , 2023

2023
[55]

Recommender systems in the era of large language models (llms)

Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. Recommender systems in the era of large language models (llms). IEEE Transactions on Knowledge and Data Engineering , 36:6889--6907, 2023

2023
[56]

Adapting large language models for education: Foundational capabilities, potentials, and challenges, 2023

Qingyao Li, Lingyue Fu, Weiming Zhang, Xianyu Chen, Jingwei Yu, Wei Xia, Weinan Zhang, Ruiming Tang, and Yong Yu. Adapting large language models for education: Foundational capabilities, potentials, and challenges, 2023

2023
[57]

Yu, and Qingsong Wen

Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. Large language models for education: A survey and outlook, 2024

2024
[58]

Simulating classroom education with llm-empowered agents

Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhanxin Hao, Jianxiao Jiang, Jie Cao, Huiqin Liu, Zhiyuan Liu, et al. Simulating classroom education with llm-empowered agents. arXiv preprint arXiv:2406.19226 , 2024

2024 arXiv
[59]

A survey on large language models for code generation

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515 , 2024

2024 arXiv
[60]

Make every move count: Llm-based high-quality rtl code generation using mcts

Matthew DeLorenzo, Animesh Basak Chowdhury, Vasudev Gohil, Shailja Thakur, Ramesh Karri, Siddharth Garg, and Jeyavijayan Rajendran. Make every move count: Llm-based high-quality rtl code generation using mcts. arXiv preprint arXiv:2402.03289 , 2024

2024 arXiv
[61]

Mapping the increasing use of llms in scientific papers

Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, et al. Mapping the increasing use of llms in scientific papers. arXiv preprint arXiv:2404.01268 , 2024

2024 arXiv
[62]

u bler, Jiaji Huang, Matth \

Youngsuk Park, Kailash Budhathoki, Liangfu Chen, Jonas M K \"u bler, Jiaji Huang, Matth \"a us Kleindessner, Jun Huan, Volkan Cevher, Yida Wang, and George Karypis. Inference optimization of foundation models on ai accelerators. In Proceedings of the 30th ACM SIGKDD Conference...

2024
[63]

Leveraging compute-in-memory for efficient generative model inference in tpus

Zhantong Zhu, Hongou Li, Wenjie Ren, Meng Wu, Le Ye, Ru Huang, and Tianyu Jia. Leveraging compute-in-memory for efficient generative model inference in tpus. arXiv preprint arXiv:2503.00461 , 2025

2025 arXiv
[64]

In-datacenter performance analysis of a tensor processing unit

Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on...

2017
[65]

A novel neuromorphic processors realization of spiking deep reinforcement learning for portfolio management

Seyyed Amirhossein Saeidi, Forouzan Fallah, Soroush Barmaki, and Hamed Farbeh. A novel neuromorphic processors realization of spiking deep reinforcement learning for portfolio management. In 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE) , pages 68--71...

2022
[66]

Machine learning with neuromorphic photonics

Thomas Ferreira De Lima, Hsuan-Tung Peng, Alexander N Tait, Mitchell A Nahmias, Heidi B Miller, Bhavin J Shastri, and Paul R Prucnal. Machine learning with neuromorphic photonics. Journal of Lightwave Technology , 37(5):1515--1534, 2019

2019
[67]

Efficient training of large language models on distributed infrastructures: a survey

Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, et al. Efficient training of large language models on distributed infrastructures: a survey. arXiv preprint arXiv:2407.20018 , 2024

2024 arXiv
[68]

A survey on distributed machine learning

Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. A survey on distributed machine learning. Acm computing surveys (csur) , 53(2):1--33, 2020

2020
[69]

A survey of model compression and acceleration for deep neural networks

Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 , 2017

2017 arXiv
[70]

Model compression via distillation and quantization

Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668 , 2018

2018 arXiv
[71]

A survey on model compression for large language models

Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics , 12:1556--1577, 2024

2024
[72]

Flashattention: Fast and memory-efficient exact attention with io-awareness

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurIPS , 35:16344--16359, 2022

2022
[73]

Flashattention-2: Faster attention with better parallelism and work partitioning

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023

2023 arXiv
[74]

Operator fusion in xla: analysis and evaluation

Daniel Snider and Ruofan Liang. Operator fusion in xla: analysis and evaluation. arXiv preprint arXiv:2301.13062 , 2023

2023 arXiv
[75]

u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing...

2020
[76]

Knowledge distillation: A survey

Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision , 129:1789--1819, 2021

2021
[77]

Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019

1910 arXiv
[78]

Improving language models by retrieving from trillions of tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference...

2022
[79]

Jurassic-1: Technical details and evaluation

Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs , 1(9):1--17, 2021

2021
[80]

Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases

Jiarui Li, Ye Yuan, and Zehua Zhang. Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases. arXiv preprint arXiv:2403.10446 , 2024

2024 arXiv
[81]

Benchmarking retrieval-augmented generation for medicine

Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024 , pages 6233--6251, 2024

2024
[82]

Longformer: The long-document transformer

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020

2004 arXiv
[83]

Big bird: Transformers for longer sequences

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. NeurIPS , pages 17283--17297, 2020

2020
[84]

Reformer: The efficient transformer

Nikita Kitaev, ukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ICLR , 2021

2021
[85]

Mixture of experts: a literature survey

Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelligence Review , 42:275--293, 2014

2014
[86]

Exploring the benefit of activation sparsity in pre-training

Zhengyan Zhang, Chaojun Xiao, Qiujieli Qin, Yankai Lin, Zhiyuan Zeng, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. Exploring the benefit of activation sparsity in pre-training. arXiv preprint arXiv:2410.03440 , 2024

arXiv
[87]

Moma: Efficient early-fusion pre-training with mixture of modality-aware experts

Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer, and Armen Aghajanyan. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts. arXiv preprint arXiv:2407.21770 , 2024

2024 arXiv
[88]

Adam: A method for stochastic optimization

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014

2014 arXiv
[89]

Adaptive subgradient methods for online learning and stochastic optimization

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7), 2011

2011
[90]

Large batch optimization for deep learning: Training bert in 76 minutes

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 , 2019

1904 arXiv
[91]

Hyp-rl: Hyperparameter optimization by reinforcement learning

Hadi S Jomaa, Josif Grabocka, and Lars Schmidt-Thieme. Hyp-rl: Hyperparameter optimization by reinforcement learning. arXiv preprint arXiv:1906.11527 , 2019

1906 arXiv
[92]

Zeus: Understanding and optimizing GPU energy consumption of DNN training

Jie You, Jae-Won Chung, and Mosharaf Chowdhury. Zeus: Understanding and optimizing GPU energy consumption of DNN training. In USENIX NSDI , 2023

2023
[93]

A self-tuning actor-critic algorithm

Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. Advances in neural information processing systems , 33:20913--20924, 2020

2020
[94]

A comprehensive survey of neural architecture search: Challenges and solutions

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR) , 54(4):1--34, 2021

2021
[95]

Efficientnet: Rethinking model scaling for convolutional neural networks

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105--6114. PMLR, 2019

2019
[96]

Teaching large language models to reason with reinforcement learning

Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642 , 2024

2024 arXiv
[97]

Dpo meets ppo: Reinforced token optimization for rlhf

Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922 , 2024

2024 arXiv
[98]

Fine-tuning language models with advantage-induced policy alignment

Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I Jordan, and Jiantao Jiao. Fine-tuning language models with advantage-induced policy alignment. arXiv preprint arXiv:2306.02231 , 2023

2023 arXiv
[99]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. 2023

2023
[100]

The efficiency spectrum of large language models: An algorithmic survey

Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, Ilya Zharkov, and Luming Liang. The efficiency spectrum of large language models: An algorithmic survey. 2023

2023
[101]

Energy and policy considerations for deep learning in nlp

Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. 2019

2019
[102]

Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. 2020

2020
[103]

Rae, Oriol Vinyals, and Laurent Sifre

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

2022
[104]

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

2023
[105]

Deepnet: Scaling transformers to 1,000 layers

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. 2022

2022
[106]

Demaine, David Eppstein, Adam Hesterberg, Kshitij Jain, Anna Lubiw, Ryuhei Uehara, and Yushi Uno

Erik D. Demaine, David Eppstein, Adam Hesterberg, Kshitij Jain, Anna Lubiw, Ryuhei Uehara, and Yushi Uno. Reconfiguring undirected paths. 2019

2019
[107]

Not all samples are created equal: Deep learning with importance sampling

Angelos Katharopoulos and Francois Fleuret. Not all samples are created equal: Deep learning with importance sampling. 2018

2018
[108]

On the generalized zakharov-kuznetsov equation at critical regularity

Axel Gruenrock. On the generalized zakharov-kuznetsov equation at critical regularity. 2015

2015
[109]

Learning and development in neural networks: The importance of starting small

Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition , 48(1):71--99, 1993

1993
[110]

Curriculum learning

Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML , pages 41--48, 2009

2009
[111]

Multi-organ segmentation via co-training weight-averaged models from few-organ datasets

Rui Huang, Yuanjie Zheng, Zhiqiang Hu, Shaoting Zhang, and Hongsheng Li. Multi-organ segmentation via co-training weight-averaged models from few-organ datasets. 2020

2020
[112]

Self-paced learning for latent variable models

M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. NeurIPS , 23, 2010

2010
[113]

Towards understanding what code language models learned

Toufique Ahmed, Dian Yu, Chengxuan Huang, Cathy Wang, Prem Devanbu, and Kenji Sagae. Towards understanding what code language models learned. arXiv preprint arXiv:2306.11943 , 2023

2023 arXiv
[114]

Long code arena: a set of benchmarks for long-context code models

Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, and Timofey Bryksin. Long code arena: a set of benchmarks for long-context code models. arXiv preprint a...

2024 arXiv
[115]

Deepcodeprobe: Towards understanding what models trained on code learn

Vahid Majdinasab, Amin Nikanjam, and Foutse Khomh. Deepcodeprobe: Towards understanding what models trained on code learn. arXiv preprint arXiv:2407.08890 , 2024

2024 arXiv
[116]

Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K

Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. When is multilinguality a curse? language modeling for 250 high- and low-resource languages. arXiv preprint arXiv:2311.09205 , 2023

2023 arXiv
[117]

An efficient approach for studying cross-lingual transfer in multilingual language models

Fahim Faisal and Antonios Anastasopoulos. An efficient approach for studying cross-lingual transfer in multilingual language models. arXiv preprint arXiv:2403.20088 , 2024

2024 arXiv
[118]

Since the scientific literature is multilingual, our models should be too

Abteen Ebrahimi and Kenneth Church. Since the scientific literature is multilingual, our models should be too. arXiv preprint arXiv:2403.18251 , 2024

2024 arXiv
[119]

The less the merrier? investigating language representation in multilingual models

Hellina Hailu Nigatu, Atnafu Lambebo Tonja, and Jugal Kalita. The less the merrier? investigating language representation in multilingual models. arXiv preprint arXiv:2310.13228 , 2023. Accepted to EMNLP 2023(Findings)

2023 arXiv
[120]

Towards a semantic search engine for scientific articles

Bastien Latard, Jonathan Weber, Germain Forestier, and Michel Hassenforder. Towards a semantic search engine for scientific articles. arXiv preprint arXiv:1709.09836 , 2017

2017 arXiv
[121]

Eren, Kim O

Nicholas Solovyev, Ryan Barron, Manish Bhattarai, Maksim E. Eren, Kim O. Rasmussen, and Boian S. Alexandrov. Interactive distillation of large single-topic corpora of scientific papers. arXiv preprint arXiv:2309.10772 , 2023. Accepted at 2023 IEEE ICMLA conference

2023 arXiv
[122]

From rag to riches: Retrieval interlaced with sequence generation

Palak Jain, Livio Baldini Soares, and Tom Kwiatkowski. From rag to riches: Retrieval interlaced with sequence generation. arXiv preprint arXiv:2407.00361 , 2024. 18 pages, 3 figures, Preprint

2024 arXiv
[123]

Author and B

A. Author and B. Author. Deepseek-r1: Emergent reasoning in reinforcement learning fine-tuned large language models, 2025. arXiv preprint. Available at https://arxiv.org/abs/XXXX.XXXX

2025
[124]

Author and D

C. Author and D. Author. Kimi k1.5: A curriculum-based approach for enhancing reasoning in multi-modal llms, 2025. Open-source project. Available at https://github.com/MoonshotAI/Kimi-k1.5

2025
[125]

Author and F

E. Author and F. Author. Wisdom: Progressive curriculum data synthesis for enhancing reasoning in large language models, 2024. arXiv preprint. Available at https://arxiv.org/abs/XXXX.XXXX

2024
[126]

Author and H

G. Author and H. Author. Lbs3: Curriculum-inspired prompting for automated reasoning in large language models, 2024. arXiv preprint. Available at https://arxiv.org/abs/XXXX.XXXX

2024
[127]

Author and J

I. Author and J. Author. Curllm-reasoner: A curriculum reasoning framework for visual and language models. In Proceedings of the 2024 ACM SIGKDD Conference on Knowledge Discovery and Data Mining . ACM, 2024. Available at https://doi.org/10.1145/XXXXXX

2024 doi
[128]

Author and L

K. Author and L. Author. Logic-rl: A curriculum learning approach for reinforcement learning on logic puzzles, 2025. arXiv preprint. Available at https://arxiv.org/abs/XXXX.XXXX

2025
[129]

Author and N

M. Author and N. Author. Alphallm-cpl: Curriculum preference learning for enhanced reasoning via mcts in llms, 2024. arXiv preprint. Available at https://arxiv.org/abs/XXXX.XXXX

2024
[130]

Generating long sequences with sparse transformers

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. 2019

2019
[131]

Peters, and Arman Cohan

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. 2020

2020
[132]

Big bird: Transformers for longer sequences

Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. Neural Information Processing Systems (NeurIPS) 2020 , 2020

2020
[133]

Li, Madian Khabsa, Han Fang, and Hao Ma

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. 2020

2020
[134]

Nystromformer: A nystrom-based algorithm for approximating self-attention

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystromformer: A nystrom-based algorithm for approximating self-attention. 2021

2021
[135]

Rethinking attention with performers

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. 2020

2020
[136]

Reformer: The efficient transformer

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. 2020

2020
[137]

Fu, Stefano Ermon, Atri Rudra, and Christopher Re

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficient exact attention with io-awareness. 2022

2022
[138]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

2017
[139]

Train short, test long: Attention with linear biases enables input length extrapolation

Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. ICLR , 2023

2023
[140]

Kerple: Kernelized relative positional embedding for length extrapolation

Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. NeurIPS , 35:8386--8399, 2022

2022
[141]

Dissecting transformer length extrapolation via the lens of receptive field analysis

Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In ACL , pages 13522--13537, 2023

2023
[142]

Functional interpolation for relative positions improves long context transformers

Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418 , 2023

arXiv
[143]

Roformer: Enhanced transformer with rotary position embedding

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 , 2021

2021 arXiv
[144]

Yarn: Efficient context window extension of large language models

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071 , 2023

2023 arXiv
[145]

Randomized positional encodings boost length generalization of transformers

Anian Ruoss, Gr \'e goire Del \'e tang, Tim Genewein, Jordi Grau-Moya, R \'o bert Csord \'a s, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843 , 2023

2023 arXiv
[146]

The impact of positional encoding on length generalization in transformers

Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems , 36:24892--24928, 2023

2023
[147]

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485--5551, 2020

2020
[148]

The case for translation-invariant self-attention in transformer-based language models

Ulme Wennberg and Gustav Eje Henter. The case for translation-invariant self-attention in transformer-based language models. arXiv preprint arXiv:2106.01950 , 2021

2021 arXiv
[149]

Hope: A novel positional encoding without long-term decay for enhanced context awareness and extrapolation

Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, and Wei Liu. Hope: A novel positional encoding without long-term decay for enhanced context awareness and extrapolation. arXiv preprint arXiv:2410.21216 , 2024

2024 arXiv
[150]

3d-rpe: Enhancing long-context modeling through 3d rotary position encoding

Xindian Ma, Wenyuan Liu, Peng Zhang, and Nan Xu. 3d-rpe: Enhancing long-context modeling through 3d rotary position encoding. arXiv preprint arXiv:2406.09897 , 2024

2024 arXiv
[151]

Extending context window of large language models via positional interpolation

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 , 2023

2023 arXiv
[152]

Pope: Legendre orthogonal polynomials based position encoding for large language models

Arpit Aggarwal. Pope: Legendre orthogonal polynomials based position encoding for large language models. arXiv preprint arXiv:2405.04585 , 2024

2024 arXiv
[153]

Algebraic positional encodings

Konstantinos Kogkalidis, Jean-Philippe Bernardy, and Vikas Garg. Algebraic positional encodings. arXiv preprint arXiv:2312.16045 , 2023

2023 arXiv
[154]

Wavelet-based positional representation for long context

Yui Oka, Taku Hasegawa, Kyosuke Nishida, and Kuniko Saito. Wavelet-based positional representation for long context. arXiv preprint arXiv:2502.02004 , 2025. Accepted to ICLR 2025

2025 arXiv
[155]

Dense training, sparse inference: Rethinking training of mixture-of-experts language models

Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. Dense training, sparse inference: Rethinking training of mixture-of-experts language models. arXiv preprint arXiv:2404.05567 , 2024

2024 arXiv
[156]

Merging experts into one: Improving computational efficiency of mixture of experts

Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Merging experts into one: Improving computational efficiency of mixture of experts. arXiv preprint arXiv:2310.09832 , 2023. EMNLP 2023 Main Conference (Oral)

2023 arXiv
[157]

Multi-head mixture-of-experts

Xun Wu, Shaohan Huang, Wenhui Wang, and Furu Wei. Multi-head mixture-of-experts. arXiv preprint arXiv:2404.15045 , 2024

2024 arXiv
[158]

Mh-moe: Multi-head mixture-of-experts

Shaohan Huang, Xun Wu, Shuming Ma, and Furu Wei. Mh-moe: Multi-head mixture-of-experts. arXiv preprint arXiv:2411.16205 , 2024

2024 arXiv
[159]

Adaptive gating in mixture-of-experts based language models

Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu. Adaptive gating in mixture-of-experts based language models. arXiv preprint arXiv:2310.07188 , 2023

2023 arXiv
[160]

Mixture-of-depths: Dynamically allocating compute in transformer-based language models

David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258 , 2024

2024 arXiv
[161]

Scattered mixture-of-experts implementation

Shawn Tan, Yikang Shen, Rameswar Panda, and Aaron Courville. Scattered mixture-of-experts implementation. arXiv preprint arXiv:2403.08245 , 2024

2024 arXiv
[162]

Towards an empirical understanding of moe design choices

Dongyang Fan, Bettina Messmer, and Martin Jaggi. Towards an empirical understanding of moe design choices. arXiv preprint arXiv:2402.13089 , 2024

2024 arXiv
[163]

Part-of-speech sensitivity of routers in mixture of experts models

Elie Antoine, Frédéric Béchet, and Philippe Langlais. Part-of-speech sensitivity of routers in mixture of experts models. arXiv preprint arXiv:2412.16971 , 2024. Accepted at COLING 2025

2024 arXiv
[164]

Layerwise recurrent router for mixture-of-experts

Zihan Qiu, Zeyu Huang, Shuang Cheng, Yizhi Zhou, Zili Wang, Ivan Titov, and Jie Fu. Layerwise recurrent router for mixture-of-experts. arXiv preprint arXiv:2408.06793 , 2024

2024 arXiv
[165]

Llm-based routing in mixture of experts: A novel framework for trading

Kuan-Ming Liu and Ming-Chih Lo. Llm-based routing in mixture of experts: A novel framework for trading. arXiv preprint arXiv:2501.09636 , 2025. Accepted by AAAI 2025 Workshop on AI for Social Impact

2025 arXiv
[166]

Routing in sparsely-gated language models responds to context

Stefan Arnold, Marian Fietta, and Dilara Yesilbas. Routing in sparsely-gated language models responds to context. arXiv preprint arXiv:2409.14107 , 2024

2024 arXiv
[167]

Mixture of a million experts

Xu Owen He. Mixture of a million experts. arXiv preprint arXiv:2407.04153 , 2024

2024 arXiv
[168]

Rwkv: Reinventing rnns for the transformer era

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048 , 2023

2023 arXiv
[169]

Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De

Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. Proceedings of Machine Learning Research , 202:26227--26253, 2023

2023
[170]

Efficiently modeling long sequences with structured state spaces

Albert Gu, Karan Goel, and Christopher R \'e . Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations , 2022

2022
[171]

Simplified state space layers for sequence modeling

Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. 2023

2023
[172]

Mamba: Linear-time sequence modeling with selective state spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023

2023 arXiv
[173]

Hyena hierarchy: Towards larger convolutional language models

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R \'e . Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866 , 2023

2023 arXiv
[174]

Retentive network: A successor to transformer for large language models

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621 , 2023

2023 arXiv
[175]

Mixed precision training

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. 2017

2017
[176]

Megatron-lm: Training multi-billion parameter language models using model parallelism

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. 2019

2019
[177]

Le, Yonghui Wu, and Zhifeng Chen

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. 2018

2018
[178]

Skew throttling

Emelie Curl, Jesse Geneson, and Leslie Hogben. Skew throttling. 2019

2019
[179]

Zero: Memory optimizations toward training trillion parameter models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. 2019

2019
[180]

Parameter-efficient fine-tuning for large models: A comprehensive survey

Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608 , 2024

2024 arXiv
[181]

Adapterfusion: Non-destructive task composition for transfer learning

Jonas Pfeiffer, Aishwarya Kamath, Andreas R \"u ckl \'e , Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (E...

2021
[182]

Prefix-tuning: Optimizing continuous prompts for generation

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL/IJCNLP) , pages 4582--4597. Association for Computational Linguistics, 2021

2021
[183]

Counter-interference adapter for multilingual machine translation

Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, and Lei Li. Counter-interference adapter for multilingual machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2812--2823. Association for Computational Linguistics, 2021

2021
[184]

Zhao, Yuexin Wu, Bo Li, Yu Zhang, and Ming-Wei Chang

Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, and Ming-Wei Chang. Conditional adapters: Parameter-efficient transfer learning with fast inference. In Advances in Neural Information Processin...

2023
[185]

P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ...

2022
[186]

Spot: Better frozen model adaptation through soft prompt transfer

Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers) , pages 5039--5059. Ass...

2022
[187]

Ptp: Boosting stability and performance of prompt tuning with perturbation-based regularizer

Lichang Chen, Jiuhai Chen, Heng Huang, and Minhao Cheng. Ptp: Boosting stability and performance of prompt tuning with perturbation-based regularizer. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 13512--13525. Associ...

2023
[188]

Smop: Towards efficient and effective prompt tuning with sparse mixture-of-prompts

Joon-Young Choi, Junho Kim, Jun-Hyung Park, Wing-Lam Mok, and SangKeun Lee. Smop: Towards efficient and effective prompt tuning with sparse mixture-of-prompts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 14306--1431...

2023
[189]

Rush, and Yoon Kim

Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL/IJCNLP) (Volume 1: Long Papers) , pages 4884--4896. Association for Computational...

2021
[190]

Training neural networks with fixed sparse masks

Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021) , 2021

2021
[191]

Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning

Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 69...

2023
[192]

On the effectiveness of parameter-efficient fine-tuning

Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 12799--12807, 2023

2023
[193]

Raise a child in large language model: Towards effective and generalizable fine-tuning

Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNL...

2021
[194]

Clark, Brett H

Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard, James J. Clark, Brett H. Meyer, and Warren J. Gross. Efficient fine-tuning of bert models on the edge. In Proceedings of the 2022 IEEE International Symposium on Circuits and Systems (ISCAS) , pages 1838--1842. IEEE, 2022

2022
[195]

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 2: Short Papers) , pages 1-...

2022
[196]

Cross-attention is all you need: Adapting pretrained transformers for machine translation

Mozhdeh Gheini, Xiang Ren, and Jonathan May. Cross-attention is all you need: Adapting pretrained transformers for machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1754--1765. Association for Computa...

2021
[197]

Sensitivity-aware visual parameter-efficient fine-tuning

Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023

2023
[198]

Intrinsic dimensionality explains the effectiveness of language model fine-tuning

Armen Aghajanyan, Akshat Gupta, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 , 2020

2012 arXiv
[199]

Lora+: Efficient low rank adaptation of large models

Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354 , 2024

2024 arXiv
[200]

Pissa: Principal singular values and singular vectors adaptation of large language models

Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948 , 2024

2024 arXiv
[201]

Unipelt: A unified framework for parameter-efficient language model tuning

Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, and Madian Khabsa. Unipelt: A unified framework for parameter-efficient language model tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1:...

2022
[202]

Towards a unified view of parameter-efficient transfer learning

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In Proceedings of the 10th International Conference on Learning Representations (ICLR) , 2022

2022
[203]

Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models, 2023

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models, 2023

2023
[204]

Neural prompt search

Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. In Proceedings of the 17th European Conference on Computer Vision (ECCV) , pages 594--611. Springer, 2022

2022
[205]

Autopeft: Automatic configuration search for parameter-efficient fine-tuning

Han Zhou, Xingchen Wan, Ivan Vuli \'c , and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efficient fine-tuning. Transactions of the Association for Computational Linguistics , 12:525--542, 2024

2024
[206]

Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. 2020

2020
[207]

Llm-pruner: On the structural pruning of large language models

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. 2023

2023
[208]

Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. 2019

2019
[209]

Gptq: Accurate post-training quantization for generative pre-trained transformers

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. 2022

2022
[210]

Qlora: Efficient finetuning of quantized llms

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. 2023

2023
[211]

Albert: A lite bert for self-supervised learning of language representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. 2019

2019
[212]

Fast inference from transformers via speculative decoding

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. 2022

2022
[213]

Llm-fp4: 4-bit floating-point quantized transformers

Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang-Ting Cheng. Llm-fp4: 4-bit floating-point quantized transformers. arXiv preprint arXiv:2310.16836 , 2023

2023 arXiv
[214]

Training and inference of large language models using 8-bit floating point

Sergio P Perez, Yan Zhang, James Briggs, Charlie Blake, Josh Levy-Kramer, Paul Balanca, Carlo Luschi, Stephen Barlow, and Andrew William Fitzgibbon. Training and inference of large language models using 8-bit floating point. arXiv preprint arXiv:2309.17224 , 2023

2023 arXiv
[215]

Tallrec: An effective and efficient tuning framework to align large language model with recommendation

Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems , pages 1007--1014, 2023

2023
[216]

Cobra: Extending mamba to multi-modal large language model for efficient inference

Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 10421--10429, 2025

2025
[217]

Llm4effi: Leveraging large language models to enhance code efficiency and correctness

Tong Ye, Weigang Huang, Xuhong Zhang, Tengfei Ma, Peiyu Liu, Jianwei Yin, and Wenhai Wang. Llm4effi: Leveraging large language models to enhance code efficiency and correctness. arXiv preprint arXiv:2502.18489 , 2025

2025 arXiv
[218]

Andersen, Jun Woo Park, Alexander J

Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation (OS...

2014
[219]

Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F

Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. The computational limits of deep learning, 2022

2022
[220]

Beyond human-level accuracy: Computational challenges in deep learning

Joel Hestness, Newsha Ardalani, and Gregory Diamos. Beyond human-level accuracy: Computational challenges in deep learning. In Proceedings of the 24th symposium on principles and practice of parallel programming , pages 1--14, 2019

2019
[221]

Parallel computing, graphics processing unit (gpu) and new hardware for deep learning in computational intelligence research

M Madiajagan and S Sridhar Raj. Parallel computing, graphics processing unit (gpu) and new hardware for deep learning in computational intelligence research. In Deep learning and parallel computing environment for bioengineering systems , pages 1--15. Elsevier, 2019

2019
[222]

A survey of techniques for optimizing deep learning on gpus

Sparsh Mittal and Shraiysh Vaishay. A survey of techniques for optimizing deep learning on gpus. Journal of Systems Architecture , 99:101635, 2019

2019
[223]

Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B

Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, It...

2020
[224]

Standard performance evaluation corporation - contact

Standard Performance Evaluation Corporation . Standard performance evaluation corporation - contact. https://www.spec.org/spec/contact.html, 2024. Accessed: 2024-09-13

2024
[225]

Deepbench: Benchmarking deep learning operations on different hardware

Baidu Research. Deepbench: Benchmarking deep learning operations on different hardware. https://github.com/baidu-research/DeepBench, 2024. Accessed: 2024-09-13

2024
[226]

Dawnbench: An end-to-end deep learning benchmark and competition

Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris R \'e , and Matei Zaharia. Dawnbench: An end-to-end deep learning benchmark and competition. Training , 100(101):102, 2017

2017
[227]

Projectionnet: Learning efficient on-device deep networks using neural projections, 2017

Sujith Ravi. Projectionnet: Learning efficient on-device deep networks using neural projections, 2017

2017
[228]

Horovod: fast and easy distributed deep learning in TensorFlow

Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in TensorFlow . arXiv preprint arXiv:1802.05799 , 2018

2018 arXiv
[229]

Intel mpi benchmarks

Intel Corporation. Intel mpi benchmarks. https://github.com/intel/mpi-benchmarks, 2024. Accessed: 2024-09-13

2024
[230]

Llmperf: A tool for evaluating the performance of llm apis

Ray Project. Llmperf: A tool for evaluating the performance of llm apis. https://github.com/ray-project/llmperf, 2024. Accessed: 2024-09-13

2024
[231]

Intel extension for pytorch

Intel Corporation. Intel extension for pytorch. https://github.com/intel/intel-extension-for-pytorch, 2024. Accessed: 2024-09-13

2024
[232]

Aim: Adapting image models for efficient video action recognition, 2023

Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition, 2023

2023
[233]

Parameter-efficient transfer learning for nlp, 2019

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019

2019
[234]

Hulk: Graph neural networks for optimizing regionally distributed computing systems, 2023

Zhengqing Yuan, Huiwen Xue, Chao Zhang, and Yongming Liu. Hulk: Graph neural networks for optimizing regionally distributed computing systems, 2023

2023
[235]

Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity, 2023

Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity, 2023

2023
[236]

GPTC ache: An open-source semantic cache for LLM applications enabling faster answers and cost savings

Fu Bang. GPTC ache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. In Liling Tan, Dmitrijs Milajevs, Geeticka Chauhan, Jeremy Gwinnup, and Elijah Rippeth, editors, Proceedings of the 3rd Workshop for Natural Language Processing Ope...

2023
[237]

Efficiently training 7b llm with 1 million sequence length on 8 gpus, 2024

Pinxue Zhao, Hailin Zhang, Fangcheng Fu, Xiaonan Nie, Qibin Liu, Fang Yang, Yuanbo Peng, Dian Jiao, Shuaipeng Li, Jinbao Xue, Yangyu Tao, and Bin Cui. Efficiently training 7b llm with 1 million sequence length on 8 gpus, 2024

2024
[238]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023

2023
[239]

Efficientvit: Memory efficient vision transformer with cascaded group attention

Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14420--14430, 2023

2023
[240]

A 3.0 tflops 0.62 v scalable processor core for high compute utilization ai training and inference

Jinwook Oh, Sae Kyu Lee, Mingu Kang, Matthew Ziegler, Joel Silberman, Ankur Agrawal, Swagath Venkataramani, Bruce Fleischer, Michael Guillorn, Jungwook Choi, et al. A 3.0 tflops 0.62 v scalable processor core for high compute utilization ai training and inference. In 2020 IEEE...

2020
[241]

Scalify: scale propagation for efficient low-precision llm training

Paul Balan c a, Sam Hosegood, Carlo Luschi, and Andrew Fitzgibbon. Scalify: scale propagation for efficient low-precision llm training. arXiv preprint arXiv:2407.17353 , 2024

2024 arXiv
[242]

Efficient and robust parallel dnn training through model parallelism on multi-gpu platform

Chi-Chung Chen, Chia-Lin Yang, and Hsiang-Yun Cheng. Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv preprint arXiv:1809.02839 , 2018

2018 arXiv
[243]

Lp-bnn: Ultra-low-latency bnn inference with layer parallelism

Tong Geng, Tianqi Wang, Chunshu Wu, Chen Yang, Shuaiwen Leon Song, Ang Li, and Martin Herbordt. Lp-bnn: Ultra-low-latency bnn inference with layer parallelism. In 2019 IEEE 30th International Conference on Application-specific Systems, Architectures and Processors (ASAP) , vol...

2019
[244]

Gonzalez, and Ion Stoica

Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. AlpaServe : Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating S...

2023
[245]

Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve . In 18th USENIX Symposium on Operating Systems Design and Implementation ...

2024
[246]

Ebird: Elastic batch for improving responsiveness and throughput of deep learning services

Weihao Cui, Mengze Wei, Quan Chen, Xiaoxin Tang, Jingwen Leng, Li Li, and Mingyi Guo. Ebird: Elastic batch for improving responsiveness and throughput of deep learning services. In 2019 IEEE 37th International Conference on Computer Design (ICCD) , pages 497--505. IEEE, 2019

2019
[247]

Towards greener llms: Bringing energy-efficiency to the forefront of llm inference

Jovan Stojkovic, Esha Choukse, Chaojie Zhang, Inigo Goiri, and Josep Torrellas. Towards greener llms: Bringing energy-efficiency to the forefront of llm inference. arXiv preprint arXiv:2403.20306 , 2024

2024 arXiv
[248]

Optimizing llm inference clusters for enhanced performance and energy efficiency

Soka Hisaharo, Yuki Nishimura, and Aoi Takahashi. Optimizing llm inference clusters for enhanced performance and energy efficiency. Authorea Preprints , 2024

2024
[249]

Model compression and efficient inference for large language models: A survey

Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748 , 2024

2024 arXiv
[250]

Model compression and hardware acceleration for neural networks: A comprehensive survey

Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE , 108(4):485--532, 2020

2020
[251]

The knowledge within: Methods for data-free model compression

Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8494--8502, 2020

2020
[252]

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

2024
[253]

Challenging big-bench tasks and whether chain-of-thought can solve them

Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022

arXiv
[254]

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprin...

2022 arXiv
[255]

Gpqa: A graduate-level google-proof q&a benchmark

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling , 2024

2024
[256]

Instruction-following evaluation for large language models

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911 , 2023

2023 arXiv
[257]

Evaluating large language models trained on code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021

2021 arXiv
[258]

Hardmath: A benchmark dataset for challenging problems in applied mathematics

Jingxuan Fan, Sarah Martinson, Erik Y Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, and Michael P Brenner. Hardmath: A benchmark dataset for challenging problems in applied mathematics. arXiv preprint arXiv:2410.09988 , 2024

arXiv
[259]

Musr: Testing the limits of chain-of-thought with multistep soft reasoning

Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310.16049 , 2023

2023 arXiv
[260]

The llama 3 herd of models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

2024 arXiv
[261]

A comprehensive overview of large language models

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435 , 2023

2023 arXiv
[262]

Deepseek llm: Scaling open-source language models with longtermism

Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 , 2024

2024 arXiv
[263]

Qwen technical report

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023

2023 arXiv
[264]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 , 2024

2024 arXiv
[265]

Phi-4 technical report

Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905 , 2024

2024 arXiv
[266]

Textbooks are all you need, 2023

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Y...

2023
[267]

Yi: Open foundation models by 01

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652 , 2024

2024 arXiv
[268]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023
[269]

Fineweb-edu: the finest collection of educational content, 2024

Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024

2024
[270]

Openo1-sft dataset, December 2024

OpenO1 Team. Openo1-sft dataset, December 2024

2024
[271]

Huatuogpt-o1, towards medical complex reasoning with llms, 2024

Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms, 2024

2024
[272]

Large language models encode clinical knowledge

Benjamin Chew, Yuan Zhang, Pranav Baheti, Russ B Altman, and Jason Poon. Large language models encode clinical knowledge. Nature Communications , 14(1):6188, 2023

2023
[273]

Llm performance predictors are good initializers for architecture search

Ganesh Jawahar, Muhammad Abdul-Mageed, Laks VS Lakshmanan, and Dujian Ding. Llm performance predictors are good initializers for architecture search. arXiv preprint arXiv:2310.16712 , 2023

2023 arXiv
[274]

Large-language-models (llm)-based ai chatbots: Architecture, in-depth analysis and their performance evaluation

Vimal Kumar, Priyam Srivastava, Ashay Dwivedi, Ishan Budhiraja, Debjani Ghosh, Vikas Goyal, and Ruchika Arora. Large-language-models (llm)-based ai chatbots: Architecture, in-depth analysis and their performance evaluation. In International Conference on Recent Trends in Image...

2023
[275]

The efficiency spectrum of large language models: An algorithmic survey

Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, Ilya Zharkov, and Luming Liang. The efficiency spectrum of large language models: An algorithmic survey. arXiv preprint arXiv:2312.00678 , 2023

2023 arXiv
[276]

Llm in a flash: Efficient large language model inference with limited memory

Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Associ...

2024
[277]

A survey of resource-efficient llm and multimodal foundation models

Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource-efficient llm and multimodal foundation models. arXiv preprint arXiv:2401.08092 , 2024

2024 arXiv
[278]

Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

2020
[279]

Attention mechanisms in computer vision: A survey

Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. Computational visual media , 8(3):331--368, 2022

2022
[280]

Attend first, consolidate later: On the importance of attention in different llm layers

Amit Ben-Artzy and Roy Schwartz. Attend first, consolidate later: On the importance of attention in different llm layers. arXiv preprint arXiv:2409.03621 , 2024

2024 arXiv
[281]

Round attention: A novel round-level attention mechanism to accelerate llm inference

Yaohua Tang, Zhicheng Hu, Kun Cheng, Fan Mo, Qiheng Lv, Hua Wang, and Zhi Chen. Round attention: A novel round-level attention mechanism to accelerate llm inference. arXiv preprint arXiv:2502.15294 , 2025

2025 arXiv
[282]

Lserve: Efficient long-sequence llm serving with unified sparse attention

Shang Yang, Junxian Guo, Haotian Tang, Qinghao Hu, Guangxuan Xiao, Jiaming Tang, Yujun Lin, Zhijian Liu, Yao Lu, and Song Han. Lserve: Efficient long-sequence llm serving with unified sparse attention. arXiv preprint arXiv:2502.14866 , 2025

2025 arXiv
[283]

The multi-modal fusion in visual question answering: a review of attention mechanisms

Siyu Lu, Mingzhe Liu, Lirong Yin, Zhengtong Yin, Xuan Liu, and Wenfeng Zheng. The multi-modal fusion in visual question answering: a review of attention mechanisms. PeerJ Computer Science , 9:e1400, 2023

2023
[284]

Attention mechanism in neural networks: where it comes and where it goes

Derya Soydaner. Attention mechanism in neural networks: where it comes and where it goes. Neural Computing and Applications , 34(16):13371--13385, 2022

2022
[285]

An introductory survey on attention mechanisms in nlp problems

Dichao Hu. An introductory survey on attention mechanisms in nlp problems. In Intelligent Systems and Applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) Volume 2 , pages 432--448. Springer, 2020

2019
[286]

A general survey on attention mechanisms in deep learning

Gianni Brauwers and Flavius Frasincar. A general survey on attention mechanisms in deep learning. IEEE Transactions on Knowledge and Data Engineering , 35(4):3279--3298, 2021

2021
[287]

Attention mechanism, transformers, bert, and gpt: tutorial and survey

Benyamin Ghojogh and Ali Ghodsi. Attention mechanism, transformers, bert, and gpt: tutorial and survey. 2020

2020
[288]

Research progress in attention mechanism in deep learning

Jian-wei LIU, Jun-wen LIU, and Xiong-lin LUO. Research progress in attention mechanism in deep learning. Chinese Journal of Engineering , 43(11):1499--1511, 2021

2021
[289]

Position-aware parameter efficient fine-tuning approach for reducing positional bias in llms

Zheng Zhang, Fan Yang, Ziyan Jiang, Zheng Chen, Zhengyang Zhao, Chengyuan Ma, Liang Zhao, and Yang Liu. Position-aware parameter efficient fine-tuning approach for reducing positional bias in llms. arXiv preprint arXiv:2404.01430 , 2024

2024 arXiv
[290]

Length extrapolation of transformers: A survey from the perspective of positional encoding

Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, and Ting Liu. Length extrapolation of transformers: A survey from the perspective of positional encoding. arXiv preprint arXiv:2312.17044 , 2023

2023 arXiv
[291]

Deepextract: Semantic-driven extractive text summarization framework using llms and hierarchical positional encoding

Aytu g Onan and Hesham A Alhumyani. Deepextract: Semantic-driven extractive text summarization framework using llms and hierarchical positional encoding. Journal of King Saud University-Computer and Information Sciences , 36(8):102178, 2024

2024
[292]

A simple and effective positional encoding for transformers

Pu-Chin Chen, Henry Tsai, Srinadh Bhojanapalli, Hyung Won Chung, Yin-Wen Chang, and Chun-Sung Ferng. A simple and effective positional encoding for transformers. arXiv preprint arXiv:2104.08698 , 2021

2021 arXiv
[293]

Rethinking positional encoding in language pre-training

Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595 , 2020

2006 arXiv
[294]

Length generalization of causal transformers without position encoding

Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding. arXiv preprint arXiv:2404.12224 , 2024

2024 arXiv
[295]

Promoe: Fast moe-based llm serving using proactive caching

Xiaoniu Song, Zihang Zhong, Rong Chen, and Haibo Chen. Promoe: Fast moe-based llm serving using proactive caching. arXiv preprint arXiv:2410.22134 , 2024

2024 arXiv
[296]

When moe meets llms: Parameter efficient fine-tuning for multi-task medical applications

Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets llms: Parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Infor...

2024
[297]

Revisiting moe and dense speed-accuracy comparisons for llm training

Xianzhi Du, Tom Gunter, Xiang Kong, Mark Lee, Zirui Wang, Aonan Zhang, Nan Du, and Ruoming Pang. Revisiting moe and dense speed-accuracy comparisons for llm training. arXiv preprint arXiv:2405.15052 , 2024

2024 arXiv
[298]

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems , 35:30318--30332, 2022

2022
[299]

Apt: Adaptive pruning and tuning pretrained language models for efficient training and inference

Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. Apt: Adaptive pruning and tuning pretrained language models for efficient training and inference. arXiv preprint arXiv:2401.12200 , 2024

2024 arXiv
[300]

Deja vu: Contextual sparsity for efficient llms at inference time

Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning , pages 22137--22176....

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877--1901, 2020

1901

[2] [2]

Palm: Scaling language modeling with pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022

arXiv 2022

[3] [3]

Scaling laws for neural language models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020

arXiv 2001

[4] [4]

Training compute-optimal large language models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022

arXiv 2022

[5] [5]

Keivalya Pandya and M. Holia. Automating customer service using langchain: Building custom open-source gpt chatbot for organizations. ArXiv , abs/2310.05421, 2023

arXiv 2023

[6] [6]

Litllm: A toolkit for scientific literature review

Shubham Agarwal, Issam H Laradji, Laurent Charlin, and Christopher Pal. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788 , 2024

arXiv 2024

[7] [7]

A general purpose device for interaction with llms

Jiajun Xu, Qun Wang, Yuhang Cao, Baitao Zeng, and Sicheng Liu. A general purpose device for interaction with llms. In Proceedings of the Future Technologies Conference , pages 613--626. Springer, 2024

2024

[8] [8]

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurIPS , 35:22199--22213, 2022

2022