Pith. sign in

REVIEW 8 cited by

CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.08763 v4 pith:RCOHOGZF submitted 2024-04-12 cs.LG cs.CL

Donghyun Lee , Je-Yong Lee , Genghan Zhang , Mo Tiwari , Azalia Mirhoseini This is my paper
classification cs.LGcs.CL
keywords catsmodelssparsityactivationperformancebasecostsdownstream
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have dramatically advanced AI applications, yet their deployment remains challenging due to their immense inference costs. Recent studies ameliorate the computational costs of LLMs by increasing their activation sparsity but suffer from significant performance degradation on downstream tasks. In this work, we introduce a new framework for sparsifying the activations of base LLMs and reducing inference costs, dubbed Contextually Aware Thresholding for Sparsity (CATS). CATS is relatively simple, easy to implement, and highly effective. At the heart of our framework is a new non-linear activation function. We demonstrate that CATS can be applied to various base models, including Mistral-7B and Llama2-7B, and outperforms existing sparsification techniques in downstream task performance. More precisely, CATS-based models often achieve downstream task performance within 1-2% of their base models without any fine-tuning and even at activation sparsity levels of 50%. Furthermore, CATS-based models converge faster and display better task performance than competing techniques when fine-tuning is applied. Finally, we develop a custom GPU kernel for efficient implementation of CATS that translates the activation of sparsity of CATS to real wall-clock time speedups. Our custom kernel implementation of CATS results in a ~15% improvement in wall-clock inference latency of token generation on both Llama-7B and Mistral-7B.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes

    cs.LG 2026-05 accept novelty 7.0 of 10

    Standard losses induce negative weight drift with positive-biased activations, producing up to 90% sparsity in GPT-nano and an accuracy cliff above ~70% sparsity; clipped ReLU² and GELU² improve the tradeoff.

  2. Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes

    cs.LG 2026-05 accept novelty 7.0 of 10

    The paper proves negative weight drift at initialization under MSE or cross-entropy with asymmetric activations, links it to up to 90% sparsity in GPT-nano, maps the sparsity-accuracy cliff across 79 configurations, a...

  3. RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    RT-Lynx shifts DiT sparsity from weights to activations, reports up to 1.55x linear-layer speedup while preserving generation quality across multiple diffusion models.

  4. VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    VisMMoE exploits visual-expert affinity via token pruning to achieve up to 2.68x faster VL-MoE inference on memory-constrained hardware while keeping accuracy competitive.

  5. Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity

    cs.LG 2025-12 unverdicted novelty 6.0 of 10

    SPON adds learnable persistent activation anchors trained via distribution matching to restore LLM accuracy under high activation sparsity by preventing representational distribution shifts.

  6. Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity

    cs.LG 2025-12 unverdicted novelty 5.0 of 10

    SPON adds a small set of trainable input-independent activation vectors as representational anchors, trained by distribution matching, to stabilize sparse activation in LLMs and recover performance lost to hidden-stat...

  7. Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches

    cs.LG 2025-09 unverdicted novelty 5.0 of 10

    Post-training N:M activation pruning preserves generative performance in LLMs better than equivalent weight pruning, with the 8:16 pattern emerging as a practical hardware-friendly choice.

  8. Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models

    cs.LG 2026-07 conditional novelty 4.5 of 10

    Sensitivity-aware gate-threshold calibration (SATS) plus token-identity routing beat percentile sparsification and static sparse execution on quality–throughput for open 8B LLMs.

Pith tools