PyTorch Fully Sharded Data Parallel enables training of significantly larger models than Distributed Data Parallel with comparable speed and near-linear TFLOPS scaling.
Dynamic tensor rematerialization,
2 Pith papers cite this work, alongside 2 external citations. Polarity classification is still indexing.
2
Pith papers citing it
2
external citations · external index
citation-role summary
background 1
citation-polarity summary
verdicts
UNVERDICTED 2roles
background 1polarities
background 1representative citing papers
Compares PEFT methods (LoRA, QLoRA, BitFit etc.) plus a new adaptive checkpointing strategy on ViT/Mamba vision models and VLMs, showing 20-30% energy cuts and 43-79% memory reduction at small accuracy cost on CIFAR-100/DTD.
citing papers explorer
-
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
PyTorch Fully Sharded Data Parallel enables training of significantly larger models than Distributed Data Parallel with comparable speed and near-linear TFLOPS scaling.
-
Efficient PEFT Methods with Adaptive Checkpointing for Vision Models and VLMs on Resource Constrained Consumer-GPUs
Compares PEFT methods (LoRA, QLoRA, BitFit etc.) plus a new adaptive checkpointing strategy on ViT/Mamba vision models and VLMs, showing 20-30% energy cuts and 43-79% memory reduction at small accuracy cost on CIFAR-100/DTD.