DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training

Dian Yang; Jiaming Xu; Jiarui Hu; Jinlong Hou; Liming Liu; Mingjun Zhang; Ping Zhang; Siyuan Feng; Tianyi Zhou; Tongyu Wang

REVIEW 4 major objections 5 minor 1 cited by

DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DISTFLOW claims that routing RL training data through one central node is the scaling bottleneck, and that a fully distributed data plane removes it, yielding near-linear scaling to 512 GPUs and up to 2.63x throughput.

desk verdict Solid engineering contribution to distributed RL post-training, but the headline speedup and OOM narrative depend on a baseline config that is under-reported and needs verification. read the letter →

arxiv 2507.13833 v4 pith:4RVOZWSS submitted 2025-07-18 cs.DC

Zhixin Wang , Jiaming Xu , Tianyi Zhou , Mingjun Zhang , Liming Liu , Jiarui Hu , Dian Yang , Tongyu Wang

show 5 more authors

Ping Zhang Jinlong Hou Siyuan Feng Yuan Qi Yuan Cheng

This is my paper · ORCID

classification cs.DC

keywords distributedRLtrainingcontrol-datadecouplingDAGexecutionmodelcolocatedarchitecturenear-linearscalabilityGRPOPPOLLMpost-training

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RL post-training — the stage that turns pretrained language models into reasoning and aligned systems — is currently limited less by compute than by data movement. The paper argues that mainstream frameworks make a single node responsible for both scheduling and shuttling the massive tensors that flow between generation, reward, and training stages, and that this central controller saturates as clusters grow. DISTFLOW is the paper's answer: split the loop so a DAG-based control plane carries only lightweight metadata while a distributed Data Coordinator moves data through per-GPU loaders, per-node buffers, local caching, load balancing, and double buffering. The paper reports that this removes the bottleneck: throughput tracks ideal linear scaling up to 512 GPUs, speedups over the centralized baseline reach 2.63x and grow with data intensity, and the framework completes 72B-parameter runs that make the baseline run out of memory.

What carries the argument

The load-bearing object is the decoupling of control flow from data flow, realized as two layers that never share a single node's bottleneck. The control plane is a DAG Planner that linearizes a user's logical workflow graph into a serialized task chain executed by per-GPU DAG workers that exchange only compact metadata. The data plane is the Data Coordinator, a parallelism-aware system of distributed dataloaders (one per GPU, each loading only its data-parallel-rank shard) and distributed databuffers (one per node) that moves large tensors, with four mechanisms doing the heavy lifting: local cache keeps data on the worker when the DP topology is unchanged, a constrained longest-processing-time (LPT) load-balancing heuristic assigns long sequences to the least-loaded workers under a cardinality cap, and an asynchronous double buffer swaps pointers in $O(1)$ while deallocating memory in the background. Together these ensure that costly remote redistribution happens only when a parallelism transition genuinely requires it.

What would settle it

Rerun the 72B-parameter runs on 32 GPUs and the 72B/32k-context runs with the baseline framework after enabling equivalent memory-saving optimizations (activation checkpointing, CPU offloading, sequence packing); if the baseline then completes those runs, the claim that its architecture cannot scale to those configurations is falsified, and the comparison would need to be redone at memory parity.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the colocated RL training loop can be re-architected so that data never has to pass through a single coordinating node. In the baseline, even a hybrid multi-controller design still routes all intermediate tensors — prompts, responses, log probabilities, rewards, advantages — through one controller's datapath, and the paper's profiling shows this dispatch cost grows linearly with GPU count and context length. DISTFLOW instead assigns the control plane to DAG workers that exchange only execution metadata and hands the entire data lifecycle to the Data Coordinator: distributed dataloaders shard the dataset by data-parallel rank, distributed databuffers redistribute tensors only when the parallel strategy actually changes (with a local-cache fast path otherwise), a constrained longest-processing-time heuristic balances variable-length sequences, and an atomic double-buffer swap hides reset overhead. The measured result is scaling efficiencies of 90.1 to 93.9 percent from 32 to 512 GPUs, end-to-end speedups of 1.09 to 2.63x over the baseline across the PPO and GRPO post-training algorithms, and completion of 72B/32-GPU and 72B/32k-context workloads that crash the baseline — all while reward and entropy curves match the baseline over 20 epochs of training.

Load-bearing premise

The comparison assumes the baseline framework was configured fairly, specifically that its out-of-memory failures on the 72B model at 32 GPUs and at 32k context length are inherent limits of its centralized architecture rather than missing memory-saving settings such as activation checkpointing, offloading, or sequence packing that could have been enabled.

Editorial extensions

If this is right

Adding GPUs to an RL post-training cluster translates almost directly into throughput rather than being consumed by coordination, with scaling efficiency holding at 90 to 94 percent from 32 to 512 GPUs.
The speedup over the centralized baseline grows with data intensity — 1.09 to 1.64x for PPO but up to 2.62x for the more data-heavy GRPO, and from 1.48x at 8k context to 2.03x at 64k — so the hardest data-moving workloads gain the most.
Configurations that defeat the baseline become runnable: 72B parameters on 32 GPUs and 72B at 32k context complete without out-of-memory errors.
Switching to the distributed data path does not change learning dynamics: reward and entropy trajectories match the baseline over 20 epochs while total time drops 21 percent, so no re-tuning is required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same decoupling pattern should generalize beyond RL: any distributed multi-stage pipeline with bulky tensor hand-offs, such as offline evaluation, dataset curation, or multi-phase fine-tuning, can adopt per-rank loading and a local-cache fast path to relieve the same single-node funnel.
The headline 2.63x depends on how the baseline was configured; if an equally memory-optimized baseline completed the 72B runs, the speedup would likely shrink, but the widening gap with GPU count and context length would remain the more robust signal.
The constrained LPT load balancer is a separable contribution: it could be extracted and benchmarked as a standalone sequence-balancing primitive for variable-length batches in any training framework.
Because the scaling test grows the global batch size proportionally with node count, the near-linear curve measures batch-throughput scaling; a constant-batch scaling test would probe a different, latency-bound regime and would show a smaller gain.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: this is a well-built systems paper that decouples control flow from data flow in RL post-training with a DAG scheduler and a distributed data coordinator, and the ablations and convergence check are genuinely useful. The other thing: the 2.63x speedup and the near-linear scaling claims rest on a verl baseline that OOMs at the two most demanding points, and the paper never shows that verl was given memory-saving settings like activation checkpointing or offloading. That is the load-bearing issue.

What the paper does well: the architecture is clearly motivated and described. Local caching, load balancing via a constrained LPT heuristic, and double buffering are sensible, and the ablation study isolates their contributions. The convergence test showing reward/entropy parity with verl is a good check that the throughput gains aren't bought by breaking training. Scaling efficiencies of 90-94% up to 512 GPUs are impressive if they hold up, and the step timeline breakdown in Appendix B is a nice piece of evidence.

Soft spots, in order of severity. First, baseline fairness. The OOM at 72B/32 GPUs and 72B/32k context is presented as proof of verl's centralized bottleneck, but without knowing verl's exact memory configuration, it could just be a missing activation checkpointing or offloading flag. That would turn the conclusion from architecture to setup. The paper should report the full config for both systems and ideally show verl running those points with standard memory optimizations. Second, no error bars anywhere. Throughput measurements are noisy, especially at 512 GPUs, and averages over a few iterations aren't enough to support the precision of a 2.63x claim. Third, the 'SOTA' claim is selective. They exclude asynchronous frameworks because of convergence concerns, but they don't compare convergence against those systems; they only claim parity with verl. That narrows the contribution to 'faster than one synchronous baseline', not 'better than SOTA'.

The math and engineering are sound on their own terms. The DAG decomposition, the load balancing algorithm, and the data buffer logic are all coherent. My concern is not that the architecture is fake; it's that the empirical evidence as presented is not yet conclusive. The central idea—decoupling control and data flow—is valuable and worth pursuing, and the paper deserves a serious referee. If the authors can show verl with equivalent memory optimizations, add error bars, and be honest about the comparison scope, this could be a solid contribution to the RL systems literature. I'd send it to review, but with the expectation of major revisions.

Referee Report

4 major / 5 minor

Summary. The paper presents DistFlow, a fully distributed RL post-training framework that decouples control flow from data flow. Control is handled by DAG Workers driven by a DAG Planner, while data movement is managed by a decentralized Data Coordinator containing a distributed dataloader, distributed databuffer, local cache, constrained-LPT load balancing, and an asynchronous double buffer. The authors report end-to-end PPO and GRPO experiments on 7B, 32B, and 72B models up to 128 GPUs, VLM scalability experiments up to 512 GPUs, ablations over the Data Coordinator components, and a 20-epoch convergence comparison. The headline claims are near-linear scaling up to 512 GPUs and up to a 2.63x throughput improvement over the chosen baseline, verl.

Significance. If the performance claims are reproducible, the core architectural idea is valuable: showing that moving data coordination off a central controller can remove a scaling bottleneck in RL post-training is a concrete systems contribution of interest to the ICML community. The paper includes several commendable elements: a clear motivation backed by profiling (Appendix B), a stepwise ablation that isolates the contributions of local cache, double buffering, load balancing, and the distributed buffer (Section 7.4), and a convergence check showing that the speedups do not visibly change reward or entropy curves (Section 7.5). The source code is provided. The main weakness is that the strongest quantitative claims rest on a single baseline whose configuration, especially its memory-related settings, is not documented in enough detail; combined with the absence of error bars, the reported speedups and OOM-based superiority arguments are not yet established at the standard expected for a systems paper.

major comments (4)

[§7.1, §7.2, Appendix A.1] The fairness of the verl baseline configuration is load-bearing and currently unsupported. Section 7.1 states only that both systems use vLLM and FSDP, but does not report the verl version/commit, FSDP sharding and activation-checkpointing settings, whether parameter or optimizer offloading was enabled, vLLM GPU-memory configuration, or whether sequence packing was used. The paper's OOM results for verl at 72B/32 GPUs (Figure 7c/8c) and at 72B/32k context (Figure 12c) are presented as evidence of an architectural ceiling, but if verl was run without equivalent memory optimizations, those OOMs are configuration artifacts rather than inherent limits. The 2.63x speedup claim and the scalability narrative depend on this comparison, so the authors must document the exact baseline configuration and demonstrate that verl was given the same memory-saving opportunities as DistFlow.
[§7.1] No statistical uncertainty is reported for any throughput measurement. The paper states that results are averaged over 'several iterations' after warm-up, but does not state the number of iterations, the variance, or whether any runs were repeated. The main quantitative claims, including the up-to-2.63x speedup and the scaling efficiencies of 90.1%, 93.9%, and 91.8% in Figure 9, are all presented without error bars. Given that the speedups vary widely across configurations (from 1.09x to 2.63x), the authors should report standard deviations or confidence intervals and specify exactly how many iterations and independent runs were used.
[§7.1, §8] The claim of 'state-of-the-art (SOTA)' throughput is not supported by the experimental design. The paper explicitly excludes asynchronous frameworks such as StreamRL and AReaL, and benchmarks only verl. Since the exclusion is based on the authors' judgment that asynchronous methods compromise convergence, the correct scope of the claim is 'over a synchronous colocated baseline,' not 'over SOTA frameworks.' The abstract and title should be qualified, or the authors should compare against at least one modern asynchronous baseline and report its convergence behavior as well as its throughput.
[§7.3] The scalability evaluation in Figure 9 reports only DistFlow's own throughput and defines scaling efficiency relative to DistFlow's smallest configuration. There is no verl curve in the scalability plots, and the statement that 'the baseline system could not complete the same linearity tests' is not backed by any shown data beyond the OOM points already discussed in Section 7.2. To support the claim that the near-linear scaling is a differentiator, the authors should show the baseline's behavior in the same setup or at least report the exact configurations and error messages for the baseline failures.

minor comments (5)

[§5.2] The term 'databuffer' is used with inconsistent capitalization; it should be 'DataBuffer' or 'databuffer' consistently throughout.
[Algorithm 1] The notation in lines 5-6 is difficult to parse: 'H← {(0, k)|k∈ {0, . . . , K−1}}' and 'P ← {∅}K−1 k=0' would benefit from a standard set-builder or tuple-vector formulation.
[References] The DeepScaleR reference is cited as a Notion blog with no stable identifier; an arXiv version or persistent URL should be provided if available.
[Introduction] The reference to 'Claude 4.5' in the introduction does not match the cited 'Claude 4 Sonnet Model Card'; please align the model name with the reference.
[Table 1] In the 'other overhead' row of Table 1, the total column for verl (10.781) does not appear to be the sum of the displayed entries, and the reader cannot tell which components are included; a footnote clarifying the arithmetic would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all central claims are direct measurements, the scaling-efficiency metric is a definition, and the ablation study is a legitimate stepwise comparison.

full rationale

This paper is an empirical systems paper whose central claims are measured throughputs and scaling curves, not quantities derived from fitted parameters. The scaling-efficiency metric in Eq. (1) is a normalization definition applied to directly measured throughput values, and it is not used to predict any result from the system's own components. The ablation study in Figure 10 is a legitimate stepwise comparison where each configuration is the same system plus one additional component, and the distributed-buffer ablation explicitly contrasts the fully distributed DataBuffer with a simulated centralized baseline, which is the correct experimental control. The baseline selection excludes asynchronous frameworks with a stated algorithmic-correctness rationale; this is a scope decision about which systems count as comparable, not a circular reduction in which the conclusion is equivalent to the input. The OOM events observed for verl at 72B/32 GPUs are presented as evidence of architectural limits, but whether that reflects an inherent design flaw or a missing memory optimization is a baseline-fairness and reproducibility question, not a case where the paper's output is determined by its input by construction. No self-citation is load-bearing: the verl/HybridFlow reference is external and used as a baseline engine, and no uniqueness theorem or prior result by the same authors is invoked to force the design choice. Therefore, no circular step can be exhibited from the paper's text, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

DistFlow introduces no new theoretical entities or fitted parameters. It relies on standard distributed-computing libraries (Ray, NCCL) and on the fairness of the baseline comparison, which is the most fragile assumption. The invented entities are software components (Data Coordinator, DAG Worker, Distributed Databuffer), not physical postulates.

assumptions (3)

domain assumption NCCL and Ray provide reliable, deadlock-free communication primitives as used in DistFlow's experiments.
The entire distributed design depends on these libraries behaving correctly at scale. The paper does not verify this beyond the reported runs (Sections 4 and 6).
domain assumption The verl baseline configuration is representative of its best-practice deployment, including memory optimizations such as activation checkpointing and recomputation.
The paper does not report verl's memory settings for the 72B/32 GPU and 72B/32k context runs that OOM. If verl was run without equivalent memory optimizations, the OOM does not reflect inherent architectural limits (Sections 7.2 and A.1).
domain assumption Throughput in tokens/s (global batch tokens divided by iteration time) is a valid end-to-end proxy for RL post-training efficiency across the compared systems.
The paper states results are averaged over several iterations after warm-up, excluding cold-start and initialization overheads. It may not fully capture stragglers or long-tail effects in a production run (Section 7.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training." pith.science (2026). https://pith.science/paper/4RVOZWSS

@misc{pith2026250713833,
  author       = {Pith},
  title        = {Pith review of: DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4RVOZWSS}},
  note         = {Machine review of arXiv:2507.13833}
}

read the original abstract

Effectively scaling Reinforcement Learning (RL) is crucial for enhancing the reasoning and alignment of Large Language Models. The massive data and complex execution flows inherent in these tasks require a distributed architecture capable of efficient scaling. However, to simplify programming and dependency management, mainstream frameworks often rely on a centralized architecture where a single node dispatches both control and data. This inherent coupling creates significant communication bottlenecks, severely limiting system scalability and efficiency. We present DISTFLOW, a novel, fully distributed RL framework that adopts a multi-controller paradigm. By decoupling data transmission from control dispatch, DISTFLOW establishes a parallelism-aware, decentralized Data Coordinator that leverages local caching, load balancing, and asynchronous double buffer to minimize communication overhead and mitigate straggler effects. For control logic, it introduces a task scheduler built upon Directed Acyclic Graph (DAG) that facilitates fine-grained, independent execution. Experimental results demonstrate that DISTFLOW achieves near-linear scalability up to 512 GPUs and delivers up to a 2.63x throughput improvement over state-of-the-art (SOTA) frameworks. The source code is available at: https://github.com/sii-research/siiRL.

Figures

Figures reproduced from arXiv: 2507.13833 by the authors.

**Figure 1.** Comparison of data dispatch paradigms. (a) Centralized Sequential Dispatch creates accumulated idle time (green). (b) Distributed Parallel Dispatch enables concurrent transfer, eliminating serialization bottlenecks. 1. Introduction The standard training paradigm for Large Language Models (LLMs) and Vision Language Models (VLMs) begins with pretraining (Vaswani et al., 2017) and Supervised FineTuning (SFT) (Ouyang… view at source ↗

**Figure 2.** Latency profiling of the centralized controller in verl with a 7B LLM. ” CTX ” denotes context length. the necessity of employing heterogeneous parallelization strategies across these distinct stages introduces significant complexity in coordinating data and control flows at scale. Traditional RL systems, such as OpenRLHF (Hu et al., 2024), employed a disaggregated architecture, partitioning the system into distinct… view at source ↗

**Figure 4.** Decomposing a user-defined DAG into a sequential execution pipeline. computational primitives characterized by Node ID, Role, Type, and Dependencies. This high-level abstraction enables the system to automatically derive the execution topology, shielding users from the complexities of distributed scheduling. DAG Decomposition. A primary challenge in executing a user-defined DAG is ensuring its efficient adaptation… view at source ↗

Figures from the paper (8 more)

**Figure 5.** Figure 5: Workflow of the Data Coordinator. The left panel illustrates the data flow between DAG nodes (e.g., Inference to Training) across changing parallelism strategies. The right panel details the internal mechanisms, including strategy-aware local caching, dynamic load bala…

**Figure 6.** Figure 6: Workflow of Distributed Dataloader. Each worker is only responsible for loading its own assigned piece of the total data. 7. Evaluation 7.1. Experimental Setup We conduct a series of experiments to evaluate DISTFLOW’s efficiency and scalability across four key scenari…

**Figure 7.** Figure 7: Throughput comparison of DISTFLOW and verl using the PPO algorithm. The results show that DISTFLOW is faster than the baseline for all tested model sizes and GPU counts. This speedup increases as more GPUs are added, and DISTFLOW can successfully complete large-scale t…

**Figure 8.** Figure 8: Throughput comparison of DISTFLOW and verl using the GRPO algorithm. With this more data-intensive algorithm, DISTFLOW’s speed advantage becomes even greater, as its distributed data system handles the increased data load more efficiently. is most pronounced in data-in…

**Figure 9.** Figure 9: Scalability evaluation of DISTFLOW. This experiment shows that DISTFLOW achieves near linear scalability on large clusters of up to 512 GPUs. This strong performance is attributed to its fully distributed architecture, which uniformly balances both computational and da…

**Figure 10.** Figure 10: Ablation study of the system with different batch sizes. of 1024, 2048, and 4096. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

**Figure 11.** Figure 11: Entropy (up) and Reward (down) curves comparison between verl and DISTFLOW. way for next-generation RL by enabling the training of frontier models with unprecedented scale and efficiency. Impact Statement This paper presents work whose goal is to advance the field of…

**Figure 12.** Figure 12: Long-context performance evaluation. The results show that DISTFLOW’s performance advantage over the baseline increases with longer context lengths. B. Step Timeline Breakdown [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
cs.CL 2025-09 conditional novelty 3.0 of 10

A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

8 extracted references · 2 canonical work pages · cited by 1 Pith paper

[2]

Fu, W., Gao, J., Shen, X., Zhu, C., Mei, Z., He, C., Xu, S., Wei, G., Mei, J., Wang, J., et al

URL https://arxiv.org/abs/2501.12948. Fu, W., Gao, J., Shen, X., Zhu, C., Mei, Z., He, C., Xu, S., Wei, G., Mei, J., Wang, J., et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning.Advances in Neural Information Processing Systems, 38:36256–36282,

arXiv
[4]

Meng, F., Du, L., Liu, Z., Zhou, Z., Lu, Q., Fu, D., Han, T., Shi, B., Wang, W., He, J., Zhang, K., Luo, P., Qiao, Y ., Zhang, Q., and Shao, W

Notion Blog. Meng, F., Du, L., Liu, Z., Zhou, Z., Lu, Q., Fu, D., Han, T., Shi, B., Wang, W., He, J., Zhang, K., Luo, P., Qiao, Y ., Zhang, Q., and Shao, W. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365,

arXiv
[5]

Accessed: 2025-09-17

URL https://cdn.openai.com/ gpt-5-system-card.pdf. Accessed: 2025-09-17. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow i...

work page 2025
[8]

Laminar: A scalable asynchronous rl post-training framework, 2025a

Sheng, G., Tong, Y ., Wan, B., Zhang, W., Jia, C., Wu, X., Wu, Y ., Li, X., Zhang, C., Peng, Y ., Lin, H., Liu, X., and Wu, C. Laminar: A scalable asynchronous rl post-training framework, 2025a. Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y ., Lin, H., and Wu, C. Hybridflow: A flexible and efficient rlhf framework. InProceedings of t...

arXiv
[2020]

ISBN 9781450379984

Association for Computing Machinery. ISBN 9781450379984. doi: 10.1145/3394486.3406703. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

arXiv
[2024]

Nemo-aligner: Scalable toolkit for efficient model align- ment.arXiv preprint arXiv:2405.01481,

Shen, G., Wang, Z., Delalleau, O., Zeng, J., Dong, Y ., Egert, D., Sun, S., Zhang, J., Jain, S., Taghibakhshi, A., et al. Nemo-aligner: Scalable toolkit for efficient model align- ment.arXiv preprint arXiv:2405.01481,

arXiv
[2025]

URL https://www-cdn.anthropic.com/ 4263b940cabb546aa0e3283f35b686f4f3b2ff47. pdf. Accessed: 2025-09-17. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from hu- man preferences. In Guyon, I., Luxburg, U. V ., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.),Advances in ...

work page 2025
[2026]

Open- RLHF: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143,

Hu, J., Wu, X., Wang, W., Zhang, D., and Cao, Y . Open- RLHF: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143,

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2]

Fu, W., Gao, J., Shen, X., Zhu, C., Mei, Z., He, C., Xu, S., Wei, G., Mei, J., Wang, J., et al

URL https://arxiv.org/abs/2501.12948. Fu, W., Gao, J., Shen, X., Zhu, C., Mei, Z., He, C., Xu, S., Wei, G., Mei, J., Wang, J., et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning.Advances in Neural Information Processing Systems, 38:36256–36282,

arXiv

[2] [4]

Meng, F., Du, L., Liu, Z., Zhou, Z., Lu, Q., Fu, D., Han, T., Shi, B., Wang, W., He, J., Zhang, K., Luo, P., Qiao, Y ., Zhang, Q., and Shao, W

Notion Blog. Meng, F., Du, L., Liu, Z., Zhou, Z., Lu, Q., Fu, D., Han, T., Shi, B., Wang, W., He, J., Zhang, K., Luo, P., Qiao, Y ., Zhang, Q., and Shao, W. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365,

arXiv

[3] [5]

Accessed: 2025-09-17

URL https://cdn.openai.com/ gpt-5-system-card.pdf. Accessed: 2025-09-17. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow i...

work page 2025

[4] [8]

Laminar: A scalable asynchronous rl post-training framework, 2025a

Sheng, G., Tong, Y ., Wan, B., Zhang, W., Jia, C., Wu, X., Wu, Y ., Li, X., Zhang, C., Peng, Y ., Lin, H., Liu, X., and Wu, C. Laminar: A scalable asynchronous rl post-training framework, 2025a. Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y ., Lin, H., and Wu, C. Hybridflow: A flexible and efficient rlhf framework. InProceedings of t...

arXiv

[5] [2020]

ISBN 9781450379984

Association for Computing Machinery. ISBN 9781450379984. doi: 10.1145/3394486.3406703. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

arXiv

[6] [2024]

Nemo-aligner: Scalable toolkit for efficient model align- ment.arXiv preprint arXiv:2405.01481,

Shen, G., Wang, Z., Delalleau, O., Zeng, J., Dong, Y ., Egert, D., Sun, S., Zhang, J., Jain, S., Taghibakhshi, A., et al. Nemo-aligner: Scalable toolkit for efficient model align- ment.arXiv preprint arXiv:2405.01481,

arXiv

[7] [2025]

URL https://www-cdn.anthropic.com/ 4263b940cabb546aa0e3283f35b686f4f3b2ff47. pdf. Accessed: 2025-09-17. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from hu- man preferences. In Guyon, I., Luxburg, U. V ., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.),Advances in ...

work page 2025

[8] [2026]

Open- RLHF: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143,

Hu, J., Wu, X., Wang, W., Zhang, D., and Cao, Y . Open- RLHF: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143,

arXiv