Pith. sign in

REVIEW 9 cited by

LLM in a flash: Efficient Large Language Model Inference with Limited Memory

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.11514 v3 pith:46JVBM74 submitted 2023-12-12 cs.CL cs.AIcs.LG

Keivan Alizadeh , Iman Mirzadeh , Dmitry Belenko , Karen Khatamifard , Minsik Cho , Carlo C Del Mundo , Mohammad Rastegari , Mehrdad Farajtabar This is my paper
classification cs.CLcs.AIcs.LG
keywords memoryflashdatadraminferencelanguagelimitedllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their substantial computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters in flash memory, but bringing them on demand to DRAM. Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this hardware-informed framework, we introduce two principal techniques. First, "windowing" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models

    cs.LG 2025-05 conditional novelty 7.0 of 10

    CoreMatching couples token pruning and neuron pruning in vision-language models by selecting tokens that activate the most core neurons, achieving large inference speedups with minor accuracy loss.

  2. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    cs.PF 2026-06 accept novelty 6.5 of 10

    On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.

  3. Architectural Implications of Agentic AI Workflows

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Agentic AI workloads fragment execution into bursts of LLM calls and tool runs, and role-aware harvesting and pooling can reclaim idle CPU and GPU capacity.

  4. A CXL Memory Rack for Multi-Turn LLM Serving

    cs.DC 2026-07 conditional novelty 6.0 of 10

    HyMCache makes SSD-backed CXL-hybrid memory behave like a DRAM-tier KV cache for multi-turn LLM serving by prefetching known prefix blocks and isolating writes, achieving ~3x over local caching and ~70% of distributed...

  5. Transition-Aware Backend Dispatch for Edge LLM Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Transition-aware backend dispatch—conditioning each operator's backend choice on the previous backend—reduces replayed latency, energy, and EDP by up to 17.4%, 14.4%, and 28.5% versus static assignment in seven edge-L...

  6. Waltz: Temperature-Aware Cooperative Compression for High-Performance Compression-Based CSDs

    cs.PF 2025-09 conditional novelty 6.0 of 10

    Waltz uses drive temperature to schedule each compression task to either the host CPU or the SSD's hardware engine, eliminating thermal shutdowns and raising write throughput up to 3.9x and read throughput up to 6.3x ...

  7. CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge

    cs.AR 2025-06 conditional novelty 5.0 of 10

    CLONE combines generative pruning, LoRA adapters, a parameter-free Mixture-of-Experts router, and learning-based DVFS to make LLM inference on edge devices faster and more energy-efficient, claiming up to 11.92x speed...

  8. AVEC: Bootstrapping Privacy for Local LLMs

    cs.CR 2025-09 conditional novelty 4.0 of 10

    AVEC is a proposed framework for per-query differential privacy budgeting, entity-level randomized response, and hash-based verification when delegating LLM queries to a remote model.

  9. MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices

    cs.LG 2025-06 conditional novelty 4.0 of 10

    MNN-LLM, a mobile LLM inference engine based on MNN, reports up to 8.6x faster prefill than llama.cpp on a smartphone CPU through quantization, hybrid DRAM-Flash storage, and hardware-tuned kernels.

Pith tools