MTraining scales LLM training to 512K-token contexts on 32 A100 GPUs by integrating dynamic sparse training patterns with balanced and hierarchical sparse ring attention, achieving up to 6x throughput gains without accuracy loss on long-context benchmarks.
Zero: Memory optimiza- tions toward training trillion parameter models
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
representative citing papers
The authors hypothesize, with evidence from merging and freezing experiments, that the attention output projection is the primary locus of mathematical reasoning in large language models.
A reasoning LLM trained with GRPO on a multi-aspect reward (REMOR) produces peer reviews that score higher on that same reward than human reviews, but the evaluation is circular and partly in-sample.
OpenRLHF is a new open-source RLHF framework reporting 1.22x to 1.68x speedups and fewer lines of code than prior systems.
citing papers explorer
-
MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
MTraining scales LLM training to 512K-token contexts on 32 A100 GPUs by integrating dynamic sparse training patterns with balanced and hierarchical sparse ring attention, achieving up to 6x throughput gains without accuracy loss on long-context benchmarks.
-
Who Reasons in the Large Language Models?
The authors hypothesize, with evidence from merging and freezing experiments, that the attention output projection is the primary locus of mathematical reasoning in large language models.
-
REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning
A reasoning LLM trained with GRPO on a multi-aspect reward (REMOR) produces peer reviews that score higher on that same reward than human reviews, but the evaluation is circular and partly in-sample.
-
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
OpenRLHF is a new open-source RLHF framework reporting 1.22x to 1.68x speedups and fewer lines of code than prior systems.