REVIEW 22 cited by
Liger Kernel: Efficient Triton Kernels for LLM Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Liger Kernel: Efficient Triton Kernels for LLM Training
read the original abstract
Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands and the need for enhanced performance. In this work, we introduce Liger-Kernel, an open-sourced set of Triton kernels developed specifically for LLM training. With kernel optimization techniques like kernel operation fusing and input chunking, our kernels achieve on average a 20% increase in training throughput and a 60% reduction in GPU memory usage for popular LLMs compared to HuggingFace implementations. In addition, Liger-Kernel is designed with modularity, accessibility, and adaptability in mind, catering to both casual and expert users. Comprehensive benchmarks and integration tests are built in to ensure compatibility, performance, correctness, and convergence across diverse computing environments and model architectures. The source code is available under a permissive license at: github.com/linkedin/Liger-Kernel.
Forward citations
Cited by 22 Pith papers
-
Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation
Animation2Code benchmark with 1,069 videos tests VLMs on generating animation code, showing persistent failures in temporal consistency despite good visual matches.
-
CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs
CODA re-expresses most non-attention Transformer computations as GEMM-plus-epilogue programs using a constrained set of composable primitives to keep intermediate results on-chip and cut global memory traffic.
-
CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs
CODA re-expresses non-attention Transformer computations as GEMM epilogues using composable primitives for scaling, reductions, and transformations.
-
Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation
Decoupling prefix source from token-level KL direction in autoregressive sequence KL yields four objectives unifying SFT, DAgger, offline RL and OPD, with KL mixing and entropy-gated curriculum improving math reasonin...
-
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
Introduces SemanticSeg dataset with 30k+ instances and block distillation using sink tokens, dropout, and weighted loss to reach near full-attention performance with block attention.
-
Graphs of Research: Citation Evolution Graphs as Supervision for Research Idea Generation
GoR extracts citation DAGs using position, frequency, predecessor links and time, then fine-tunes Qwen2.5-7B on 498 seed papers to generate ideas, claiming SOTA over gpt-4o baselines via LLM judges.
-
BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
BOSCH decomposes attention-head selection for short-context hybridization into layer probing, adaptive ratio assignment, and grouped binary optimization, yielding better efficiency-performance tradeoffs than static or...
-
Bayesian Preference Learning for Test-Time Steerable Reward Models
ICRM casts reward modeling as amortized variational inference over a latent preference probability with a Beta prior, enabling test-time adaptation to unseen preferences and improving benchmark performance.
-
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
A new SemanticSeg dataset and block distillation with sink tokens, block dropout, and weighted loss let block-attention models reach near full-attention performance on long texts.
-
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
Introduces SemanticSeg dataset with over 30k instances and a block distillation framework to achieve near full-attention performance with automatic block segmentation.
-
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
A new 30k-instance semantic segmentation dataset plus block distillation with sink tokens, dropout, and weighted loss lets block-attention models reach near full-attention performance on long texts.
-
An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU
SlideFormer uses layer-sliding async offloading, pre-allocated heterogeneous memory, and fused Triton kernels to fine-tune 123B+ models on one RTX 4090 with 1.4–6.3× higher throughput and roughly half the memory of pr...
-
The Appeal and Reality of Recycling LoRAs with Adaptive Merging
Adaptive merging of recycled LoRAs gives little benefit over training a target-task LoRA, and randomly initialized LoRAs work as well as real ones once the target LoRA is in the pool.
-
Faster and Memory-Efficient Training of Sequential Recommendation Models for Large Catalogs
CCE- is a Triton kernel implementation of cross-entropy loss with negative sampling that reduces memory by more than 10x and accelerates training by up to 2x for large-catalog sequential recommenders.
-
LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification
LongSpec achieves up to 3.26x speedup over Flash Attention baselines on long-context datasets via memory-efficient drafting and verification techniques.
-
SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD
An Ascend-NPU training stack reaches 34.22% MFU on DeepSeek-V4-Pro, and a solver-verified CPT+SFT recipe raises OR benchmark averages to 71.81% (Flash) and 77.33% (Pro).
-
LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratories
LabVLA uses RoboGenesis simulation data and a two-stage FAST pretraining plus flow matching recipe on a Qwen3-VL backbone to achieve the highest success rates on LabUtopia under in- and out-of-distribution conditions.
-
RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment
A fully open 518M Arabic-specialized LLM, built by vocabulary injection and standard post-training on Qwen2.5-0.5B, beats same-class multilingual baselines and ships at 398 MB quantized.
-
Genome-Factory: A Library for Tuning, Deploying, and Interpreting Genomic Foundation Models
Genome-Factory is an open-source Python library that integrates data pipelines, model tuning, inference, benchmarks, and biological interpretation for genomic foundation models.
-
NVILA: Efficient Frontier Visual Language Models
NVILA improves on VILA with a scale-then-compress visual token strategy and full-lifecycle efficiency optimizations, matching or exceeding leading VLMs on image and video benchmarks while reducing training cost 1.9-5....
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A comprehensive reference book organizing existing techniques for agentic AI systems across LLM substrate, reasoning, agent design patterns, inter-agent coordination, and production deployment.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.