REVIEW 5 cited by
Enhancing Training Efficiency Using Packing with Flash Attention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Enhancing Training Efficiency Using Packing with Flash Attention
read the original abstract
Padding is often used in tuning LLM models by adding special tokens to shorter training examples to match the length of the longest sequence in each batch. While this ensures uniformity for batch processing, it introduces inefficiencies by including irrelevant padding tokens in the computation and wastes GPU resources. Hugging Face SFT trainer has always offered the option to use packing to combine multiple training examples, allowing for maximal utilization of GPU resources. However, up till now, it did not offer proper masking of each packed training example. This capability has been added to Hugging Face Transformers 4.44. We analyse this new feature and show the benefits across different variations of packing.
Forward citations
Cited by 5 Pith papers
-
Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation
RLSR trains source rewriters via RL with translation-quality improvement as the reward, outperforming prompt baselines at 4B scale while matching larger models.
-
FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training
FlashCP introduces Whole-Doc sharding, sharding-aware KV communication, and a heuristic for mixed sharding plans, claiming up to 1.63x speedup over prior CP methods for LLM training.
-
Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation
A 4B rewriting model trained with RL on downstream translation-quality gains outperforms no-rewriting and same-scale prompt-based rewriting, and roughly matches a 235B prompt-based rewriter.
-
LobRA: Multi-tenant Fine-tuning over Heterogeneous Data
LobRA reduces GPU seconds for multi-tenant LoRA fine-tuning by 45.03%-60.67% through heterogeneous FT replicas and per-step workload-balanced dispatching.
-
MAGI-1: Autoregressive Video Generation at Scale
MAGI-1 is a 24B-parameter autoregressive video world model that predicts denoised frame chunks sequentially with increasing noise to enable causal, scalable, streaming generation up to 4M token contexts.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.