Pith. sign in

REVIEW 2 major objections 6 minor 4 cited by

On Accelerating Edge AI: Optimizing Resource-Constrained Environments

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that combining model compression, neural architecture search, and hardware-aware compilation lets edge AI cut latency, memory, and energy while preserving accuracy.

desk verdict A solid, current survey whose integrated-pipeline promise outruns the evidence it cites; fix the abstract and it earns its place. read the letter →

arxiv 2501.15014 v2 pith:R67PUJJH submitted 2025-01-25 cs.LG cs.AIcs.NE

Jacob Sander , Achraf Cohen , Venkat R. Dasari , Brent Venable , Brian Jalaian This is my paper · ORCID
classification cs.LGcs.AIcs.NE
keywords edgeAImodelcompressionquantizationpruningknowledgedistillationneuralarchitecturesearchhardware-awarecompilationresource-constraineddeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey organizes the many scattered techniques for accelerating deep learning at the edge into three families: model compression, neural architecture search, and compiler and deployment frameworks. Its central thesis is that these families are complementary, not competing, and that chaining them into one pipeline can reduce latency, memory use, and energy consumption simultaneously while keeping accuracy competitive. A reader should care because this is the practical path to running capable AI on devices with tight compute, memory, and power budgets, where sending data to a cloud is not an option. The paper positions its own contribution as a map of the field and a blueprint for a scalable, platform-independent optimization framework.

What carries the argument

The load-bearing mechanism is the three-pillar pipeline with hardware feedback as the connective tissue. Model compression attacks model size through four named operations: quantization (fewer bits per weight and activation), pruning (removing unimportant weights, channels, or larger structures), tensor decomposition (factoring a high-rank weight tensor into lower-rank factors), and knowledge distillation (training a small student to mimic a large teacher). Neural architecture search supplies the search space, the search strategy, and cheap evaluators such as accuracy predictors and hardware lookup tables that estimate per-layer latency and energy. Compiler frameworks fuse operations and generate device-specific kernels, and specialized hardware can carry sparsity and low-precision natively. What carries the argument is the composability claim: because each pillar addresses a different bottleneck--model size, architecture suitability, and low-level execution--their gains are expected to stack, and the hardware lookup tables are what keep the search honest about real deployment costs.

What would settle it

Take one fixed model, dataset, and edge device, then run a single integrated pipeline that quantizes or prunes, searches an architecture, and compiles for that device; if the combined result is no faster or smaller than the best single pillar applied alone, or if its accuracy loss exceeds what quantization alone produces, the composability claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that edge AI acceleration has three reusable pillars and they belong in the same pipeline. Compression techniques--lowering numeric precision, pruning weights or channels, replacing weight tensors with low-rank factors, and distilling a large teacher into a small student--shrink a model that already exists. Neural architecture search automates the search for a network topology that natively fits a task and a hardware budget. Compiler and deployment frameworks translate the chosen model into low-level kernels that exploit the device's specific capabilities, so the theoretical gains from the first two pillars actually appear at inference time. The paper argues that combining all three yields multi-objective gains--lower latency, lower memory footprint, and lower energy--without a competitive-accuracy penalty, and it points to case studies in which these pieces have already been linked, such as joint search over architecture and quantization policies and Bayesian search that distills a teacher into a hardware-constrained student. This is a survey's synthesis claim: it organizes existing results and asserts their composability, rather than reporting a new end-to-end experiment of its own.

Load-bearing premise

The entire thesis rests on the assumption that model compression, neural architecture search, and hardware-aware compilation can be stacked in one pipeline without one step erasing another's benefit or pushing accuracy below an acceptable level, an integration the paper asserts but does not measure with an end-to-end experiment.

Editorial extensions

If this is right

  • A practitioner can deploy accurate models on constrained devices by running compression and architecture search first, then compiling for the target hardware rather than treating optimization as one technique.
  • Hardware lookup tables and accuracy predictors become first-class tools: they let a search evaluate latency and energy without running every candidate on the device.
  • Quantization is not just a fallback; the survey reports cases where quantizing weights can preserve performance better than pruning them, especially for difficult downstream tasks.
  • The same pipeline logic should extend to large language models, where post-training quantization, structured pruning, and distillation are already the main practical levers.
  • Choice of compiler or hardware backend determines whether the theoretical speedups from compression and search actually show up in measured inference time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The composability thesis is the part most worth testing; a single end-to-end benchmark on one fixed device, measuring accuracy, latency, memory, and energy after each pipeline stage, would give the field a concrete target.
  • Editorial inference: If combined pipelines work as claimed, the natural next step is an automated orchestrator that takes a task description and a hardware budget and outputs a compiled, compressed, searched model without human tuning.
  • Editorial inference: The paper's emphasis on pre-training pruning for huge models suggests the biggest untapped energy savings may come before deployment, during training, rather than only at inference time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper is a survey of techniques for accelerating deep learning inference on edge devices. It organizes the literature into three pillars: model compression (pruning, quantization, tensor decomposition, knowledge distillation), neural architecture search (including hyperparameter optimization and multi-objective search), and compiler/deployment frameworks (TVM, XLA, TensorRT, OpenVINO, and IBM NorthPole). It then discusses integrated case studies, publicly available NAS benchmarks, and open research challenges. The abstract claims that integrating these three pillars into unified pipelines allows practitioners to achieve latency, memory, and energy benefits while maintaining competitive accuracy.

Significance. If taken as a synthesis of the field, the survey is useful in scope: it brings together recent LLM-era compression literature, NAS benchmarks, and compiler tooling in one place, with sensible taxonomies and helpful pointers to open problems such as pre-training pruning and distillation loss topology. The case studies in Section 5 are well chosen to illustrate partial integrations. However, the paper's headline contribution—the assertion that the three pillars compose into a reliable pipeline—is not supported by the surveyed evidence or by any experiment in the paper. With the claim appropriately qualified, the survey could serve as a solid reference for practitioners and researchers; as written, the abstract overstates the maturity of the field.

major comments (2)
  1. [Abstract; §5.2; §6.1] The abstract's claim that 'By integrating these three pillars into unified pipelines, practitioners can achieve multi-objective goals, including latency reduction, memory savings, and energy efficiency—all while maintaining competitive accuracy' is not supported by the survey's own evidence. In §5.2, the integrated case studies combine at most two pillars: APQ combines pruning, quantization, and NAS but omits a compiler or deployment framework; AutoDistill combines NAS and distillation but no quantization or compiler; AWQ is quantization alone; and DARTS is NAS alone. The paper reports no end-to-end experiment or controlled comparison of a full three-pillar pipeline. Moreover, §6.1 states that 'A central challenge remains formalizing this pipeline as a tractable multi-objective optimization problem,' which frames the unified pipeline as future work rather than an established recipe. Because the composability of compression, NAS, and compiler passes is the load-bearing assumption of the abstract, the claim should be weakened to a research direction or supported by cited existing work that validates the full pipeline.
  2. [Table 1; §4.1] The quantitative performance claims are presented without measurement context, and some are used to support the survey's practical guidance. Table 1 reports latency and accuracy from a single secondary source ('AI [2021]') without specifying the hardware, batch size, compiler, or measurement methodology; the text relies on the MobileNet-v2 PTQ latency increase (89 ms to 98 ms) as evidence that post-training quantization can degrade inference speed. Similarly, §4.1 asserts 'speedups of 1.2–3.8×' for TVM on CPU/GPU/FPGA and 'at least a 50% reduction in latency and energy usage for LLM inference' for TensorRT, but gives no model, benchmark, or measurement conditions for these numbers. Since the survey's claim that compiler frameworks deliver tangible gains is part of the three-pillar argument, these figures should be reported with their source and with an explicit caveat that they are not independent measurements.
minor comments (6)
  1. [§2.4, Eq. (1)] In Eq. (1), the notation is incomplete: the subscript for the Kullback-Leibler divergence is missing, and the roles of L and M (teacher/student layer counts) versus the summed indices a and t are unclear; please define all symbols before use.
  2. [§2.4, Eq. (2)] The same symbol LAT is used for two different losses, the attention-based loss in Eq. (1) and the activation-based loss in Eq. (2), which is confusing; rename one of the two losses.
  3. [§4.1] The reference for OpenVINO is malformed: 'Ria Chruvu' appears to be a misattribution of a web page author; the citation should be corrected.
  4. [§5.2.3] The heading 'A WQ' and the inline text 'A WQfocuses' contain spacing errors; the acronym is AWQ elsewhere and should be used consistently.
  5. [§6] The sentence 'how the model packaged and and sent for computation' contains a duplicated 'and' and is ungrammatical; also, 'Pre-Training Pruning' appears with inconsistent capitalization across the paper.
  6. [§3.4] The statement that lexicographic optimization 'yields a single final solution guaranteed to lie on the Pareto frontier' is imprecise when ties exist; it should say 'yields a Pareto-optimal solution'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: this is a literature survey whose claims rest on external prior work; the single author-affiliated citation is a minor, non-load-bearing example.

full rationale

This manuscript is a literature survey, not a derivation or prediction exercise, so the circularity patterns based on fitting parameters, renamed predictions, or imported uniqueness theorems do not apply. The only author-affiliated citation is Vadera et al. (2020), used in Section 2.4 as an illustrative example of multi-teacher distillation with a Bayesian neural network ensemble; it does not support the survey's central organizing claim about integrating compression, NAS, and compilers. The abstract's claim that unified pipelines can achieve latency, memory, and energy gains while maintaining accuracy is a synthesis of external results, and the paper itself flags a limitation in Section 6.1: 'A central challenge remains formalizing this pipeline as a tractable multi-objective optimization problem.' That is an evidence gap or a strength-of-claim concern, not circularity: no equation is defined in terms of a target result, no fitted value is renamed a prediction, and no load-bearing argument reduces to a self-citation chain. The survey's technical content is drawn from citable external benchmarks, compilers, and case studies, so no circular steps are identified.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters, no new mathematical axioms, and no invented entities. It depends instead on three background assumptions: over-parameterization enables compression, integrated pipelines compose gains, and the cited numbers are trustworthy. These assumptions are domain assumptions typical of a survey.

assumptions (3)
  • domain assumption Deep neural networks are over-parameterized, which makes pruning, quantization, and decomposition feasible without immediate catastrophic accuracy loss.
    Invoked in the introduction of Section 2 and used to motivate the entire model compression family. The survey treats this as established consensus rather than proving it.
  • domain assumption The gains from compression, neural architecture search, and compiler optimization compose without harmful interference in a unified pipeline.
    The abstract and Section 4.3 assert integrated pipelines achieve latency, memory, and energy gains. No experiment in this survey measures the combined effect, so this is an unverified premise of the central claim.
  • domain assumption The quantitative results cited from other papers, including speedups and accuracy retention figures, are accurate as reported by the original sources.
    The survey makes no independent measurements. Its empirical support is entirely second-hand, so its correctness depends on the reliability of the cited literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Accelerating Edge AI: Optimizing Resource-Constrained Environments." pith.science (2026). https://pith.science/paper/R67PUJJH

@misc{pith2026250115014,
  author       = {Pith},
  title        = {Pith review of: On Accelerating Edge AI: Optimizing Resource-Constrained Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R67PUJJH}},
  note         = {Machine review of arXiv:2501.15014}
}
read the original abstract

Resource-constrained edge deployments demand AI solutions that balance high performance with stringent compute, memory, and energy limitations. In this survey, we present a comprehensive overview of the primary strategies for accelerating deep learning models under such constraints. First, we examine model compression techniques-pruning, quantization, tensor decomposition, and knowledge distillation-that streamline large models into smaller, faster, and more efficient variants. Next, we explore Neural Architecture Search (NAS), a class of automated methods that discover architectures inherently optimized for particular tasks and hardware budgets. We then discuss compiler and deployment frameworks, such as TVM, TensorRT, and OpenVINO, which provide hardware-tailored optimizations at inference time. By integrating these three pillars into unified pipelines, practitioners can achieve multi-objective goals, including latency reduction, memory savings, and energy efficiency-all while maintaining competitive accuracy. We also highlight emerging frontiers in hierarchical NAS, neurosymbolic approaches, and advanced distillation tailored to large language models, underscoring open challenges like pre-training pruning for massive networks. Our survey offers practical insights, identifies current research gaps, and outlines promising directions for building scalable, platform-independent frameworks to accelerate deep learning models at the edge.

Figures

Figures reproduced from arXiv: 2501.15014 by the authors.

Figure 1
Figure 1. Strategies and Tools for Enhancing AI Edge Computing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pruning Taxonomy as formulated by Cheng et al. [2023] [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Kernel approximation of low-rank decomposition matrix as depicted in Li et al. [2023b] [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Distillation Components Numerous knowledge distillation methods have been established, each with design considerations, techniques, and uses. Adversarial Distillation highlights that the compact model produced through distillation may be susceptible to adversarial thre…
Figure 5
Figure 5. Figure 5: Concepts and Tools in Neural Architecture Search [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: A Study on Encodings for NAS White et al. [2020]. (a) A CNN visualized as a DAG; (b) Adjacency matrix of [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Graph Hypernetwork framework Zhang et al. [2018]. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: NAS-Bench-201: Each candidate cell has four nodes, and the overall network consists of five stacked cells [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: APQ Framework Wang et al. [2020b]. The supernetwork (left) supports multiple sub-architectures via pruning, [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: DARTS Framework Liu et al. [2018]. A DAG with multiple candidate operations per edge is learned via [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: AutoDistill pipeline Zhang et al. [2022]. The system repeatedly samples architectures (driven by Bayesian Optimization), does a quick “flash” distillation, and measures hardware metrics. By combining Bayesian NAS and knowledge distillation in a single loop, AutoDistil…
Figure 12
Figure 12. Figure 12: Flash Distillation correlates strongly with Full Distillation Zhang et al. [2022]. Each point represents a [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions

    cs.HC 2025-05 conditional novelty 6.0 of 10

    EdgeWisePersona is a new synthetic dataset and benchmark for reconstructing structured smart-home user routines from multi-session dialogues, on which large LLMs clearly outperform small on-device models.

  2. Performance Isolation for Inference Processes in Edge GPU Systems

    cs.OS 2026-01 conditional novelty 5.0 of 10

    On NVIDIA GPUs, MIG partitions provide the strongest temporal isolation for concurrent inference, and Green Contexts approach that level on edge devices only when power throttling is removed.

  3. Constrained Edge AI Deployment: Fine-Tuning vs Distillation for LLM Compression

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Under identical MLP-only pruning schedules, KL self-distillation matches or exceeds cross-entropy fine-tuning for recovering test accuracy of a pruned 7B language model, with a 2.5 to 5.5 point edge at 50% retention.

  4. Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI

    cs.CV 2025-07 reject novelty 4.0 of 10

    A survey and one-model benchmark concluding that token compression methods hurt compact Vision Transformers when used off the shelf, a conclusion supported only by an unverified experiment on AutoFormer-S.

Reference graph

Works this paper leans on

140 extracted references · 23 canonical work pages · cited by 4 Pith papers

  1. [1]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704--2713, 2018

    2018

  2. [2]

    Efficientqat: Efficient quantization-aware training for large language models, 2024 a

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models, 2024 a . URL https://arxiv.org/abs/2407.11062

    arXiv 2024

  3. [3]

    Art and science of quantizing large-scale models: A comprehensive overview

    Yanshu Wang, Tong Yang, Xiyan Liang, Guoan Wang, Hanning Lu, Xu Zhe, Yaoming Li, and Li Weitao. Art and science of quantizing large-scale models: A comprehensive overview. arXiv preprint arXiv:2409.11650, 2024

    arXiv 2024

  4. [4]

    Foundations of large language model compression--part 1: Weight quantization

    Sean I Young. Foundations of large language model compression--part 1: Weight quantization. arXiv preprint arXiv:2409.02026, 2024

    arXiv 2024

  5. [5]

    AWQ : Activation -aware Weight Quantization for LLM Compression and Acceleration , July 2024 a

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ : Activation -aware Weight Quantization for LLM Compression and Acceleration , July 2024 a . URL http://arxiv.org/abs/2306.00978. arXiv:2306.00978 [cs]

    arXiv 2024

  6. [6]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314

    arXiv 2023

  7. [7]

    Integer or floating point? new outlooks for low-bit quantization on large language models

    Yijia Zhang, Lingran Zhao, Shijie Cao, Sicheng Zhang, Wenqiang Wang, Ting Cao, Fan Yang, Mao Yang, Shanghang Zhang, and Ningyi Xu. Integer or floating point? new outlooks for low-bit quantization on large language models. In 2024 IEEE International Conference on Multimedia and Expo (ICME), page 1–6. IEEE, July 2024. doi:10.1109/icme57554.2024.10688089. UR...

    arXiv 2024

  8. [8]

    Rptq: Reorder-based post-training quantization for large language models, 2023

    Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models, 2023. URL https://arxiv.org/abs/2304.01089

    arXiv 2023

Show all 140 references
  1. [9]

    Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023. URL https://arxiv.org/abs/2304.09145

    2023 arXiv

  2. [10]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Frederick Diamos, Erich Elsen, David Garc \'i a, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. ArXiv, abs/1710.03740, 2017. URL https://api.semanticscholar.or...

    arXiv

  3. [11]

    Layered mixed-precision training: A new training method for large-scale ai models

    Hao Li, Yuzhu Wang, Yan Hong, Fei Li, and Xiaohui Ji. Layered mixed-precision training: A new training method for large-scale ai models. Journal of King Saud University - Computer and Information Sciences, 35 0 (8): 0 101656, September 2023 a . ISSN 1319-1578. doi:10.1016/j.jk...

    2023

  4. [12]

    Channel-wise mixed-precision quantization for large language models, 2024 b

    Zihan Chen, Bike Xie, Jundong Li, and Cong Shen. Channel-wise mixed-precision quantization for large language models, 2024 b . URL https://arxiv.org/abs/2410.13056

    2024 arXiv

  5. [13]

    Mixed precision low-bit quantization of neural network language models for speech recognition

    Junhao Xu, Jianwei Yu, Shoukang Hu, Xunying Liu, and Helen Meng. Mixed precision low-bit quantization of neural network language models for speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 0 3679–3693, 2021. ISSN 2329-9304. doi:10.1109/t...

    2021

  6. [14]

    Efficient deep learning: A survey on making deep learning models smaller, faster, and better

    Gaurav Menghani. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Comput. Surv., 55 0 (12), March 2023. ISSN 0360-0300. doi:10.1145/3578938. URL https://doi.org/10.1145/3578938

    2023 doi

  7. [15]

    Model optimization - google ai

    Google AI. Model optimization - google ai. https://ai.google.dev/edge/litert/models/model_optimization, 2021

    2021

  8. [16]

    A comprehensive study on quantization techniques for large language models, 2024

    Jiedong Lang, Zhehao Guo, and Shuyu Huang. A comprehensive study on quantization techniques for large language models, 2024. URL https://arxiv.org/abs/2411.02530

    2024 arXiv

  9. [17]

    The loss surface of multilayer networks

    Anna Choromanska, Mikael Henaff, Micha \" e l Mathieu, G \' e rard Ben Arous, and Yann LeCun. The loss surface of multilayer networks. CoRR, abs/1412.0233, 2014. URL http://arxiv.org/abs/1412.0233

    2014 arXiv

  10. [18]

    A survey on deep neural network pruning-taxonomy, comparison

    Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning-taxonomy, comparison. Analysis, and Recommendations, 2023

    2023

  11. [19]

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: Single-shot network pruning based on connection sensitivity, 2019. URL https://arxiv.org/abs/1810.02340

    2019 arXiv

  12. [20]

    Picking winning tickets before training by preserving gradient flow, 2020 a

    Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow, 2020 a . URL https://arxiv.org/abs/2002.07376

    2020 arXiv

  13. [21]

    Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023. URL https://arxiv.org/abs/2301.00774

    2023 arXiv

  14. [22]

    Channel pruning for accelerating very deep neural networks, 2017

    Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks, 2017. URL https://arxiv.org/abs/1707.06168

    2017 arXiv

  15. [23]

    Gate Decorator : Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks

    Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate Decorator : Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks . In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedin...

    2019

  16. [24]

    Layer-wise pruning of transformer attention heads for efficient language modeling

    Kyuhong Shim, Iksoo Choi, Wonyong Sung, and Jungwook Choi. Layer-wise pruning of transformer attention heads for efficient language modeling. In 2021 18th International SoC Design Conference (ISOCC), pages 357--358, 2021. doi:10.1109/ISOCC53507.2021.9613933

    2021

  17. [25]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. URL https://arxiv.org/abs/2401.15024

    2024 arXiv

  18. [26]

    Shortgpt: Layers in large language models are more redundant than you expect, 2024

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL https://arxiv.org/abs/2403.03853

    2024 arXiv

  19. [27]

    Llm-pruner: On the structural pruning of large language models, 2023

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models, 2023. URL https://arxiv.org/abs/2305.11627

    2023 arXiv

  20. [28]

    Lpvit: Low-power semi-structured pruning for vision transformers, 2024 a

    Kaixin Xu, Zhe Wang, Chunyun Chen, Xue Geng, Jie Lin, Xulei Yang, Min Wu, Xiaoli Li, and Weisi Lin. Lpvit: Low-power semi-structured pruning for vision transformers, 2024 a . URL https://arxiv.org/abs/2407.02068

    2024 arXiv

  21. [29]

    An image enhancing pattern-based sparsity for real-time inference on mobile devices, 2020

    Xiaolong Ma, Wei Niu, Tianyun Zhang, Sijia Liu, Sheng Lin, Hongjia Li, Xiang Chen, Jian Tang, Kaisheng Ma, Bin Ren, and Yanzhi Wang. An image enhancing pattern-based sparsity for real-time inference on mobile devices, 2020. URL https://arxiv.org/abs/2001.07710

    2020 arXiv

  22. [30]

    Pruning filter in filter, 2020

    Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter, 2020. URL https://arxiv.org/abs/2009.14410

    2020 arXiv

  23. [31]

    Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow, 2020. URL https://arxiv.org/abs/2006.05467

    2020 arXiv

  24. [32]

    Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Torr. A signal propagation perspective for pruning neural networks at initialization, 2020. URL https://arxiv.org/abs/1906.06307

    2020 arXiv

  25. [33]

    Rigging the lottery: Making all tickets winners, 2021

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners, 2021. URL https://arxiv.org/abs/1911.11134

    2021 arXiv

  26. [34]

    Learning structured sparsity in deep neural networks, 2016

    Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks, 2016. URL https://arxiv.org/abs/1608.03665

    2016 arXiv

  27. [35]

    Learning efficient convolutional networks through network slimming, 2017

    Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming, 2017. URL https://arxiv.org/abs/1708.06519

    2017 arXiv

  28. [36]

    Dsa: More efficient budgeted pruning via differentiable sparsity allocation, 2020

    Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation, 2020. URL https://arxiv.org/abs/2004.02164

    2020 arXiv

  29. [37]

    APQ : Joint Search for Network Architecture , Pruning and Quantization Policy , June 2020 b

    Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, and Song Han. APQ : Joint Search for Network Architecture , Pruning and Quantization Policy , June 2020 b . URL http://arxiv.org/abs/2006.08509. arXiv:2006.08509 [cs, stat]

    2020 arXiv

  30. [38]

    Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami

    Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers, 2022. URL https://arxiv.org/abs/2204.09656

    2022 arXiv

  31. [39]

    Group fisher pruning for practical network compression, 2021

    Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression, 2021. URL https://arxiv.org/abs/2108.00708

    2021 arXiv

  32. [40]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019. URL https://arxiv.org/abs/1803.03635

    2019 arXiv

  33. [41]

    Baraniuk, Zhangyang Wang, and Yingyan Lin

    Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks, 2022. URL https://arxiv.org/abs/1909.11957

    2022 arXiv

  34. [42]

    Earlybert: Efficient bert training via early-bird lottery tickets, 2021

    Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets, 2021. URL https://arxiv.org/abs/2101.00063

    2021 arXiv

  35. [43]

    Runtime network routing for efficient image classification

    Yongming Rao, Jiwen Lu, Ji Lin, and Jie Zhou. Runtime network routing for efficient image classification. IEEE Trans. Pattern Anal. Mach. Intell., 41 0 (10): 0 2291–2304, oct 2019. ISSN 0162-8828. doi:10.1109/TPAMI.2018.2878258. URL https://doi.org/10.1109/TPAMI.2018.2878258

    2019

  36. [44]

    Manifold regularized dynamic network pruning, 2021

    Yehui Tang, Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning, 2021. URL https://arxiv.org/abs/2103.05861

    2021 arXiv

  37. [45]

    Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016. URL https://arxiv.org/abs/1510.00149

    2016 arXiv

  38. [46]

    Pruning filters for efficient convnets, 2017

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets, 2017. URL https://arxiv.org/abs/1608.08710

    2017 arXiv

  39. [47]

    Variational convolutional neural network pruning

    Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian. Variational convolutional neural network pruning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2775--2784, 2019. doi:10.1109/CVPR.2019.00289

    2019

  40. [48]

    What matters in the structured pruning of generative language models?, 2023

    Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li. What matters in the structured pruning of generative language models?, 2023. URL https://arxiv.org/abs/2302.03773

    2023 arXiv

  41. [49]

    Everybody prune now: Structured pruning of llms with only forward passes, 2024

    Lucio Dery, Steven Kolawole, Jean-François Kagy, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of llms with only forward passes, 2024. URL https://arxiv.org/abs/2402.05406

    2024

  42. [50]

    Graph pruning for model compression, 2021

    Mingyang Zhang, Xinyi Yu, Jingtao Rong, and Linlin Ou. Graph pruning for model compression, 2021. URL https://arxiv.org/abs/1911.09817

    2021 arXiv

  43. [51]

    AMC: AutoML for Model Compression and Acceleration on Mobile Devices, page 815–832

    Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for Model Compression and Acceleration on Mobile Devices, page 815–832. Springer International Publishing, 2018. ISBN 9783030012342. doi:10.1007/978-3-030-01234-2_48. URL http://dx.doi.org/10.1007...

    2018 doi

  44. [52]

    Efficient neural network pruning using model-based reinforcement learning

    Blanka Bencsik and Márton Szemenyei. Efficient neural network pruning using model-based reinforcement learning. In 2022 International Symposium on Measurement and Control in Robotics (ISMCR), pages 1--8, 2022. doi:10.1109/ISMCR56534.2022.9950598

    2022

  45. [53]

    Model compression for deep neural networks: A survey

    Zhuo Li, Hengyi Li, and Lin Meng. Model compression for deep neural networks: A survey. Computers, 12 0 (3), 2023 b . ISSN 2073-431X. doi:10.3390/computers12030060. URL https://www.mdpi.com/2073-431X/12/3/60

    2023 doi

  46. [54]

    Compressing pre-trained language models using progressive low rank decomposition

    Habib Hajimolahoseini, Mehdi Rezagholizadeh, Vahid Partovinia, Marzieh Tahaei, Omar Mohamed Awad, and Yang Liu. Compressing pre-trained language models using progressive low rank decomposition. Advances in Neural Information Processing Systems, 2021

    2021

  47. [55]

    Goldsmith, and Mert Pilanci

    Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J. Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition, 2024. URL https://arxiv.org/abs/2405.18886

    2024 arXiv

  48. [56]

    Mingxue Xu, Yao Lei Xu, and Danilo P. Mandic. Tensorgpt: Efficient compression of large language models based on tensor-train decomposition, 2023 a . URL https://arxiv.org/abs/2307.00526

    2023 arXiv

  49. [57]

    Modegpt: Modular decomposition for large language model compression, 2024 b

    Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression, 2024 b . URL https://arxiv.org/abs/2408.09632

    2024 arXiv

  50. [58]

    Distilling the knowledge in a neural network, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL https://arxiv.org/abs/1503.02531

    2015 arXiv

  51. [59]

    A Survey on Knowledge Distillation of Large Language Models , March 2024 b

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A Survey on Knowledge Distillation of Large Language Models , March 2024 b . URL http://arxiv.org/abs/2402.13116. arXiv:2402.13116 [cs]

    2024 arXiv

  52. [60]

    Adversarially robust distillation

    Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 34 0 (04): 0 3996–4003, April 2020. ISSN 2159-5399. doi:10.1609/aaai.v34i04.5816. URL http://dx.doi.org/10.1609/aaai.v3...

    2020 doi

  53. [61]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https://arxiv.org/abs/1406.2661

    2014 arXiv

  54. [62]

    Peeraid: Improving adversarial distillation from a specialized peer tutor, 2024

    Jaewon Jung, Hongsun Jang, Jaeyong Song, and Jinho Lee. Peeraid: Improving adversarial distillation from a specialized peer tutor, 2024. URL https://arxiv.org/abs/2403.06668

    2024 arXiv

  55. [63]

    Data-free learning of student networks, 2019 a

    Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks, 2019 a . URL https://arxiv.org/abs/1904.01186

    2019 arXiv

  56. [64]

    Hospedales, and Huchuan Lu

    Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning, 2017. URL https://arxiv.org/abs/1706.00384

    2017 arXiv

  57. [65]

    Vadera, Brian Jalaian, and Benjamin M

    Meet P. Vadera, Brian Jalaian, and Benjamin M. Marlin. Generalized bayesian posterior expectation distillation for deep neural networks. CoRR, abs/2005.08110, 2020. URL https://arxiv.org/abs/2005.08110

    2005 arXiv

  58. [66]

    Cross modal distillation for supervision transfer, 2015

    Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer, 2015. URL https://arxiv.org/abs/1507.00448

    2015 arXiv

  59. [67]

    Robust cross-modal representation learning with progressive self-distillation

    Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16430--16441, 2022

    2022

  60. [68]

    C2kd: Bridging the modality gap for cross-modal knowledge distillation

    Fushuo Huo, Wenchao Xu, Jingcai Guo, Haozhao Wang, and Song Guo. C2kd: Bridging the modality gap for cross-modal knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16006--16015, 2024

    2024

  61. [69]

    Graph-based knowledge distillation: A survey and experimental evaluation, 2023 a

    Jing Liu, Tongya Zheng, Guanzheng Zhang, and Qinfen Hao. Graph-based knowledge distillation: A survey and experimental evaluation, 2023 a . URL https://arxiv.org/abs/2302.14643

    2023 arXiv

  62. [70]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  63. [71]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information P...

    2020

  64. [72]

    Attention distillation: self-supervised vision transformer students need more guidance, 2022 a

    Kai Wang, Fei Yang, and Joost van de Weijer. Attention distillation: self-supervised vision transformer students need more guidance, 2022 a . URL https://arxiv.org/abs/2210.00944

    2022 arXiv

  65. [73]

    Crowley, Gavin Gray, and Amos Storkey

    Elliot J. Crowley, Gavin Gray, and Amos Storkey. Moonshine: Distilling with cheap convolutions, 2019. URL https://arxiv.org/abs/1711.02613

    2019 arXiv

  66. [74]

    Maybank, and Dacheng Tao

    Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129 0 (6): 0 1789–1819, March 2021. ISSN 1573-1405. doi:10.1007/s11263-021-01453-z. URL http://dx.doi.org/10.1007/s11263-021-01453-z

    2021 doi

  67. [75]

    Online knowledge distillation with diverse peers

    Defang Chen, Jian - Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. CoRR, abs/1912.00350, 2019 b . URL http://arxiv.org/abs/1912.00350

    1912 arXiv

  68. [76]

    Self-distillation amplifies regularization in hilbert space

    Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3351--3361. Curr...

    2020

  69. [77]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. CoRR, abs/1905.08094, 2019. URL http://arxiv.org/abs/1905.08094

    1905 arXiv

  70. [78]

    Self-instruct: Aligning language model with self generated instructions, 12 2022 b

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 12 2022 b

    2022

  71. [79]

    Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning

    Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. ArXiv, abs/2402.10110, 2024. URL https://api.semanticscholar.org/CorpusID:267682220

    2024 arXiv

  72. [80]

    Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023. URL https://arxiv.org/abs/2306.02707

    2023 arXiv

  73. [81]

    Baize: An open-source chat model with parameter-efficient tuning on self-chat data, 2023 b

    Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data, 2023 b . URL https://arxiv.org/abs/2304.01196

    2023 arXiv

  74. [82]

    Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks, 2023

    Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks, 2023. URL https://arxiv.org/abs/2305.18395

    2023 arXiv

  75. [83]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL https://arxiv.org/abs/2310.11511

    2023 arXiv

  76. [84]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685

    2023 arXiv

  77. [85]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024. URL https://arxiv.org/abs/2404.04475

    2024 arXiv

  78. [86]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment...

    2023 arXiv

  79. [87]

    Yuntao Bai et. al. Constitutional ai: Harmlessness from ai feedback, 2022. URL https://arxiv.org/abs/2212.08073

    2022 arXiv

  80. [88]

    Sub-goal distillation: A method to improve small language agents, 2024

    Maryam Hashemzadeh, Elias Stengel-Eskin, Sarath Chandar, and Marc-Alexandre Cote. Sub-goal distillation: A method to improve small language agents, 2024. URL https://arxiv.org/abs/2405.02749

    2024 arXiv

  81. [89]

    Agent lumos: Unified and modular training for open-source language agents, 2024 a

    Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents, 2024 a . URL https://arxiv.org/abs/2311.05657

    2024 arXiv

  82. [90]

    Fireact: Toward language agent fine-tuning, 2023

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning, 2023. URL https://arxiv.org/abs/2310.05915

    2023 arXiv

  83. [91]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023. URL https://arxiv.org/abs/2305.15334

    2023 arXiv

  84. [92]

    Minillm: Knowledge distillation of large language models, 2024

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models, 2024. URL https://arxiv.org/abs/2306.08543

    2024 arXiv

  85. [93]

    Distillm: Towards streamlined distillation for large language models, 2024

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models, 2024. URL https://arxiv.org/abs/2402.03898

    2024 arXiv

  86. [94]

    Impossible distillation: from low-quality model to high-quality dataset and model for summarization and paraphrasing, 05 2023

    Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Choi Yejin. Impossible distillation: from low-quality model to high-quality dataset and model for summarization and paraphrasing, 05 2023

    2023

  87. [95]

    Quill: Query intent with large language models using retrieval augmentation and multi-stage distillation, 2022

    Krishna Srinivasan, Karthik Raman, Anupam Samanta, Lingrui Liao, Luca Bertelli, and Mike Bendersky. Quill: Query intent with large language models using retrieval augmentation and multi-stage distillation, 2022. URL https://arxiv.org/abs/2210.15718

    2022 arXiv

  88. [96]

    Visual instruction tuning, 2023 b

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023 b . URL https://arxiv.org/abs/2304.08485

    2023 arXiv

  89. [97]

    Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration, 2023

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration, 2023. URL https://arxiv.org/abs/2306.09093

    2023 arXiv

  90. [98]

    Junk DNA hypothesis: Pruning small pre-trained weights \ textit\ Irreversibly\ \ and \ textit\ Monotonically\ \ impairs ``difficult'' downstream tasks in LLM s

    Lu Yin, AJAY KUMAR JAISWAL, Shiwei Liu, Souvik Kundu, and Zhangyang Wang. Junk DNA hypothesis: Pruning small pre-trained weights \ textit\ Irreversibly\ \ and \ textit\ Monotonically\ \ impairs ``difficult'' downstream tasks in LLM s. In Forty-first International Conference on...

    2024

  91. [99]

    AutoML : A survey of the state-of-the-art

    Xin He, Kaiyong Zhao, and Xiaowen Chu. AutoML : A survey of the state-of-the-art. Knowledge-Based Systems, 212: 0 106622, January 2021. ISSN 0950-7051. doi:10.1016/j.knosys.2020.106622. URL https://www.sciencedirect.com/science/article/pii/S0950705120307516

    2021

  92. [100]

    Barret Zoph and Quoc V. Le. Neural Architecture Search with Reinforcement Learning , February 2017. URL http://arxiv.org/abs/1611.01578. arXiv:1611.01578 [cs]

    2017 arXiv

  93. [101]

    Efficient Architecture Search by Network Transformation , November 2017

    Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient Architecture Search by Network Transformation , November 2017. URL http://arxiv.org/abs/1707.04873. arXiv:1707.04873 [cs]

    2017 arXiv

  94. [102]

    A study on encodings for neural architecture search

    Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study on encodings for neural architecture search. CoRR, abs/2007.04965, 2020. URL https://arxiv.org/abs/2007.04965

    2007 arXiv

  95. [103]

    Neural architecture search: Insights from 1000 papers, 2023

    Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers, 2023. URL https://arxiv.org/abs/2301.08727

    2023 arXiv

  96. [104]

    DARTS: differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. CoRR, abs/1806.09055, 2018. URL http://arxiv.org/abs/1806.09055

    2018 arXiv

  97. [105]

    Neural architecture search using property guided synthesis

    Charles Jin, Phitchaya Mangpo Phothilimthana, and Sudip Roy. Neural architecture search using property guided synthesis. Proceedings of the ACM on Programming Languages, 6 0 (OOPSLA2): 0 1150–1179, October 2022. ISSN 2475-1421. doi:10.1145/3563329. URL http://dx.doi.org/10.114...

    2022 doi

  98. [106]

    Esperan c a, and Fabio Maria Carlucci

    Xingchen Wan, Binxin Ru, Pedro M. Esperan c a, and Fabio Maria Carlucci. Approximate neural architecture search via operation distribution learning. CoRR, abs/2111.04670, 2021. URL https://arxiv.org/abs/2111.04670

    2021 arXiv

  99. [107]

    Once-for- All : Train One Network and Specialize it for Efficient Deployment , April 2020

    Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for- All : Train One Network and Specialize it for Efficient Deployment , April 2020. URL http://arxiv.org/abs/1908.09791. arXiv:1908.09791 [cs, stat]

    2020 arXiv

  100. [108]

    Weight-sharing neural architecture search: A battle to shrink the optimization gap

    Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Zhengsu Chen, Lanfei Wang, An Xiao, Jianlong Chang, Xiaopeng Zhang, and Qi Tian. Weight-sharing neural architecture search: A battle to shrink the optimization gap. CoRR, abs/2008.01475, 2020. URL https://arxiv.org/abs/2...

    2008 arXiv

  101. [109]

    Dhp: Differentiable meta pruning via hypernetworks, 2020

    Yawei Li, Shuhang Gu, Kai Zhang, Luc Van Gool, and Radu Timofte. Dhp: Differentiable meta pruning via hypernetworks, 2020. URL https://arxiv.org/abs/2003.13683

    2020 arXiv

  102. [110]

    Metapruning: Meta learning for automatic neural network channel pruning, 2019

    Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning, 2019. URL https://arxiv.org/abs/1903.10258

    2019 arXiv

  103. [111]

    Graph hypernetworks for neural architecture search

    Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. CoRR, abs/1810.05749, 2018. URL http://arxiv.org/abs/1810.05749

    2018 arXiv

  104. [112]

    Path- Level Network Transformation for Efficient Architecture Search , June 2018

    Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path- Level Network Transformation for Efficient Architecture Search , June 2018. URL http://arxiv.org/abs/1806.02639. arXiv:1806.02639 [cs, stat]

    2018 arXiv

  105. [113]

    Autodistill: an end-to-end framework to explore and distill hardware-efficient language models

    Xiaofan Zhang, Zongwei Zhou, Deming Chen, and Yu Emma Wang. Autodistill: an end-to-end framework to explore and distill hardware-efficient language models. CoRR, abs/2201.08539, 2022. URL https://arxiv.org/abs/2201.08539

    2022 arXiv

  106. [114]

    How powerful are performance predictors in neural architecture search? In M

    Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volu...

    2021

  107. [115]

    Le, and Norman P

    Sheng Li, Mingxing Tan, Ruoming Pang, Andrew Li, Liqun Cheng, Quoc V. Le, and Norman P. Jouppi. Searching for fast model families on datacenter accelerators. CoRR, abs/2102.05610, 2021. URL https://arxiv.org/abs/2102.05610

    2021 arXiv

  108. [116]

    Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. CoRR, abs/1807.11626, 2018. URL http://arxiv.org/abs/1807.11626

    2018 arXiv

  109. [117]

    Multi-accelerator neural network inference via tensorrt in heterogeneous embedded systems

    Yuxiao Zhou, Zhishan Guo, Zheng Dong, and Kecheng Yang. Multi-accelerator neural network inference via tensorrt in heterogeneous embedded systems. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), pages 463--472, 2024. doi:10.1109/COMPSAC6110...

    2024 arXiv

  110. [118]

    Robust nas under adversarial training: benchmark, theory, and beyond, 2024

    Yongtao Wu, Fanghui Liu, Carl-Johann Simon-Gabriel, Grigorios G Chrysos, and Volkan Cevher. Robust nas under adversarial training: benchmark, theory, and beyond, 2024. URL https://arxiv.org/abs/2403.13134

    2024 arXiv

  111. [119]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  112. [120]

    Explaining neural scaling laws

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121 0 (27): 0 e2311878121, 2024

    2024

  113. [121]

    Alex A. Freitas. The case for hybrid multi-objective optimisation in high-stakes machine learning applications. SIGKDD Explor. Newsl., 26 0 (1): 0 24–33, jul 2024. ISSN 1931-0145. doi:10.1145/3682112.3682116. URL https://doi.org/10.1145/3682112.3682116

    2024

  114. [122]

    Schapire, and Umar Syed

    Jacob Abernethy, Robert E. Schapire, and Umar Syed. Lexicographic optimization: Algorithms and stability, 2024. URL https://arxiv.org/abs/2405.01387

    2024 arXiv

  115. [123]

    Guan, Barret Zoph, Quoc V

    Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018. URL https://arxiv.org/abs/1802.03268

    2018 arXiv

  116. [124]

    Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. CoRR, abs/1802.01548, 2018. URL http://arxiv.org/abs/1802.01548

    2018 arXiv

  117. [125]

    Leon - Suematsu, Quoc V

    Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka I. Leon - Suematsu, Quoc V. Le, and Alex Kurakin. Large-scale evolution of image classifiers. CoRR, abs/1703.01041, 2017. URL http://arxiv.org/abs/1703.01041

    2017 arXiv

  118. [126]

    Multiple-gradient descent algorithm ( MGDA ) for multiobjective optimization

    Jean-Antoine Désidéri. Multiple-gradient descent algorithm ( MGDA ) for multiobjective optimization. Comptes Rendus Mathematique, 350 0 (5): 0 313--318, 2012. ISSN 1631-073X. doi:https://doi.org/10.1016/j.crma.2012.03.014. URL https://www.sciencedirect.com/science/article/pii/...

    2012 doi

  119. [127]

    Multi-objective differentiable neural architecture search, 2024

    Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Samuel Dooley, Josif Grabocka, and Frank Hutter. Multi-objective differentiable neural architecture search, 2024. URL https://arxiv.org/abs/2402.18213

    2024 arXiv

  120. [128]

    BANANAS: bayesian optimization with neural architectures for neural architecture search

    Colin White, Willie Neiswanger, and Yash Savani. BANANAS: bayesian optimization with neural architectures for neural architecture search. CoRR, abs/1910.11858, 2019. URL http://arxiv.org/abs/1910.11858

    1910 arXiv

  121. [129]

    Transfer NAS with meta-learned bayesian surrogates

    Gresa Shala, Thomas Elsken, Frank Hutter, and Josif Grabocka. Transfer NAS with meta-learned bayesian surrogates. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=paGvsrl4Ntr

    2023

  122. [130]

    Tvm: An automated end-to-end optimizing compiler for deep learning

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. arXiv preprint arXiv:1802.04799, 2018

    2018 arXiv

  123. [131]

    GROQ ROCKS NEURAL NETWORKS

    Linley Gwennap. GROQ ROCKS NEURAL NETWORKS . 2020

    2020

  124. [132]

    Optimizing large language models with the OpenVINO ™ toolkit

    Ryan Loney Ria Chruvu. Optimizing large language models with the OpenVINO ™ toolkit. URL https://www.intel.com/content/www/us/en/content-details/817010/optimizing-large-language-models-with-the-openvino-toolkit.html

  125. [133]

    Exploring tensorrt to improve real-time inference for deep learning

    Yuxiao Zhou and Kecheng Yang. Exploring tensorrt to improve real-time inference for deep learning. In 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in S...

    2022

  126. [134]

    Cassidy, John V

    Andrew S. Cassidy, John V. Arthur, Filipp Akopyan, Alexander Andreopoulos, Rathinakumar Appuswamy, Pallab Datta, Michael V. Debole, Steven K. Esser, Carlos Ortega Otero, Jun Sawada, Brian Taba, Arnon Amir, Deepika Bablani, Peter J. Carlson, Myron D. Flickner, Rajamohan Gandhas...

    2024

  127. [135]

    Nas-bench-101: Towards reproducible neural architecture search

    Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. CoRR, abs/1902.09635, 2019. URL http://arxiv.org/abs/1902.09635

    1902 arXiv

  128. [136]

    Nas-bench-201: Extending the scope of reproducible neural architecture search

    Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. CoRR, abs/2001.00326, 2020. URL http://arxiv.org/abs/2001.00326

    2001 arXiv

  129. [137]

    Nas-bench-301 and the case for surrogate benchmarks for neural architecture search

    Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09777, 4: 0 14, 2020

    2008 arXiv

  130. [138]

    Nas-bench-suite: NAS evaluation is (now) surprisingly easy

    Yash Mehta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, and Frank Hutter. Nas-bench-suite: NAS evaluation is (now) surprisingly easy. CoRR, abs/2201.13396, 2022. URL https://arxiv.org/abs/2201.13396

    2022 arXiv

  131. [139]

    Mobilebert: a compact task-agnostic bert for resource-limited devices, 2020

    Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices, 2020. URL https://arxiv.org/abs/2004.02984

    2020 arXiv

  132. [140]

    Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 1487–1495, N...

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.