Pith. sign in

REVIEW 5 cited by

Enhancing Training Efficiency Using Packing with Flash Attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.09105 v6 pith:FVEV65XE submitted 2024-07-12 cs.LG cs.AI

Achintya Kundu , Rhui Dih Lee , Laura Wynter , Raghu Kiran Ganti , Mayank Mishra This is my paper
classification cs.LGcs.AI
keywords trainingpackingbatchexamplesfacehuggingpaddingresources
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Padding is often used in tuning LLM models by adding special tokens to shorter training examples to match the length of the longest sequence in each batch. While this ensures uniformity for batch processing, it introduces inefficiencies by including irrelevant padding tokens in the computation and wastes GPU resources. Hugging Face SFT trainer has always offered the option to use packing to combine multiple training examples, allowing for maximal utilization of GPU resources. However, up till now, it did not offer proper masking of each packed training example. This capability has been added to Hugging Face Transformers 4.44. We analyse this new feature and show the benefits across different variations of packing.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    RLSR trains source rewriters via RL with translation-quality improvement as the reward, outperforming prompt baselines at 4B scale while matching larger models.

  2. FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training

    cs.DC 2026-06 unverdicted novelty 6.0 of 10

    FlashCP introduces Whole-Doc sharding, sharding-aware KV communication, and a heuristic for mixed sharding plans, claiming up to 1.63x speedup over prior CP methods for LLM training.

  3. Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation

    cs.CL 2026-06 conditional novelty 6.0 of 10

    A 4B rewriting model trained with RL on downstream translation-quality gains outperforms no-rewriting and same-scale prompt-based rewriting, and roughly matches a 235B prompt-based rewriter.

  4. LobRA: Multi-tenant Fine-tuning over Heterogeneous Data

    cs.DC 2025-09 accept novelty 6.0 of 10

    LobRA reduces GPU seconds for multi-tenant LoRA fine-tuning by 45.03%-60.67% through heterogeneous FT replicas and per-step workload-balanced dispatching.

  5. MAGI-1: Autoregressive Video Generation at Scale

    cs.CV 2025-05 unverdicted novelty 6.0 of 10

    MAGI-1 is a 24B-parameter autoregressive video world model that predicts denoised frame chunks sequentially with increasing noise to enable causal, scalable, streaming generation up to 4M token contexts.

Pith tools