REVIEW 9 cited by
NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe. However, building efficient tools to perform alignment can be challenging, especially for the largest and most competent LLMs which often contain tens or hundreds of billions of parameters. We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to a thousand GPUs for training the largest open-source LLMs such as Nemotron 4 340B and Llama 3.1 405B. NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for extensibility, allowing support for other alignment techniques with minimal effort. It is open-sourced with Apache 2.0 License and we invite community contributions at https://github.com/NVIDIA/NeMo-Aligner
Forward citations
Cited by 9 Pith papers
-
AReaL-DTE: Sparse Policy-Weight Transfer for Online Agentic Reinforcement Learning
AReaL-DTE is a snapshot-free delta transfer engine that reconstructs overwritten AdamW weights on demand, detects BF16-visible changes, and transfers receiver-ready sparse updates to speed up RL policy synchronization...
-
Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Training
BiDiRL raises disaggregated asynchronous LLM RL throughput up to 1.94× by hot-switching idle GPUs between rollout and training under a model-guided bidirectional scheduler.
-
TCIA: A Task-Centric Instruction Augmentation Method for Instruction Finetuning
TCIA augments task-specific instruction data by decomposing instructions into queries and constraints and using retrieval-guided breadth-first search to generate on-task variants, improving fine-tuned Llama-3.1-8B on ...
-
MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster
MindSpeed RL uses a distributed transfer dock and an allgather-swap method to raise RL training throughput on Ascend NPUs by 1.42 to 3.97 times over two open baselines.
-
DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
DistFlow decouples control and data flows in RL post-training, achieving near-linear scaling to 512 GPUs and up to 2.63x throughput vs. verl in tested workloads.
-
The Challenge of Teaching Reasoning to LLMs Without RL or Distillation
Twenty high-quality chain-of-thought examples from a reasoning model are enough to activate strong math reasoning in a 32B base model with lightweight fine-tuning.
-
When2Call: When (not) to Call Tools
When2Call measures when language models should call tools versus ask questions or refuse, and shows that RPO training substantially improves this decision-making.
-
AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset
A three-part recipe, built from a 540K-problem open dataset, long reasoning distillations, tool-integrated reasoning, and GenSelect, produces open-weight models that top most math-reasoning baselines and win AIMO-2.
-
AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
AsyncFlow combines a distributed streaming data queue with delayed parameter updates to improve RL post-training throughput by 1.59x on average over verl on Ascend clusters.
Discussion (0). Continue with ORCID to comment.