MoE structure, not parameter count, governs INT4 robustness in sub-3B VLMs; SigLIP INT8 latency spikes on Jetson Ampere are a BitsAndBytes-Ampere interaction, and INT4 VRAM savings come with TPOT and energy penalties.
arXiv preprint arXiv:2401.08092 (2024)
9 Pith papers cite this work. Polarity classification is still indexing.
9
Pith papers citing it
abstract
Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment. However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources. To support the growth of these large models in a scalable and environmentally sustainable way, there has been a considerable focus on developing resource-efficient strategies. This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects. It offers a comprehensive analysis and valuable insights gleaned from existing literature, encompassing a broad array of topics from cutting-edge model architectures and training/serving algorithms to practical system designs and implementations. The goal of this survey is to provide an overarching understanding of how current approaches are tackling the resource challenges posed by large foundation models and to potentially inspire future breakthroughs in this field.
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
Deeper transformer layers produce intermediate representations that are harder to lossy-compress, and the paper links this to growing covariance and Rademacher complexity.
ShadowNPU presents shadowAttn, a co-designed sparse attention system that uses NPU pilot compute and techniques like graph bucketing and per-head sparsity to minimize CPU/GPU fallback during on-device LLM inference while maintaining accuracy.
Empirical tests show compressed code language models retain task performance but suffer markedly lower robustness under four standard adversarial attacks.
ELEVATE is a framework and prototype for deploying LLM-powered 3D avatar tutors locally on consumer hardware with a three-stratum design separating interaction, execution, and governance layers.
A cross-platform mobile application deploys an ensemble of quantized open-source LLMs for fully local, DSM-5-aligned psychiatric decision support with claimed accuracy comparable to prior cloud versions.
Reviews NAS methods through bilevel optimization lens, categorizing them into sampling-based and theory-based, and proposes an auxiliary math programming framework for more principled architecture and weight updates.
The paper surveys techniques to speed up and reduce the resource needs of LLM inference, organized by data-level, model-level, and system-level changes, with comparative experiments on representative methods.
The paper compiles hardware-software co-design techniques including mixed-precision quantization, structural pruning, speculative decoding, and transformer accelerators to speed up multimodal foundation models, with examples in medical and code tasks.
citing papers explorer
-
Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment
MoE structure, not parameter count, governs INT4 robustness in sub-3B VLMs; SigLIP INT8 latency spikes on Jetson Ampere are a BitsAndBytes-Ampere interaction, and INT4 VRAM savings come with TPOT and energy penalties.
-
Understanding Rate-Distortion Performance in Distributed Transformer Inference
Deeper transformer layers produce intermediate representations that are harder to lossy-compress, and the paper links this to growing covariance and Rademacher complexity.
-
ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference
ShadowNPU presents shadowAttn, a co-designed sparse attention system that uses NPU pilot compute and techniques like graph bucketing and per-head sparsity to minimize CPU/GPU fallback during on-device LLM inference while maintaining accuracy.
-
Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code
Empirical tests show compressed code language models retain task performance but suffer markedly lower robustness under four standard adversarial attacks.
-
ELEVATE: Designing Human-Centered GenAI Virtual Tutors for Scalable and Inclusive Education
ELEVATE is a framework and prototype for deploying LLM-powered 3D avatar tutors locally on consumer hardware with a three-stratum design separating interaction, execution, and governance layers.
-
Toward Zero-Egress Psychiatric AI: On-Device LLM Deployment for Privacy-Preserving Mental Health Decision Support
A cross-platform mobile application deploys an ensemble of quantized open-source LLMs for fully local, DSM-5-aligned psychiatric decision support with claimed accuracy comparable to prior cloud versions.
-
Bilevel Optimization for Neural Architecture Search
Reviews NAS methods through bilevel optimization lens, categorizing them into sampling-based and theory-based, and proposes an auxiliary math programming framework for more principled architecture and weight updates.
-
A Survey on Efficient Inference for Large Language Models
The paper surveys techniques to speed up and reduce the resource needs of LLM inference, organized by data-level, model-level, and system-level changes, with comparative experiments on representative methods.
-
Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models
The paper compiles hardware-software co-design techniques including mixed-precision quantization, structural pruning, speculative decoding, and transformer accelerators to speed up multimodal foundation models, with examples in medical and code tasks.