{"id":"86d80b0c-ba3a-4502-b26d-8c2b1c5d55fd","arxiv_id":"2607.09207","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"BiDiRL raises disaggregated asynchronous LLM RL throughput up to 1.94× by hot-switching idle GPUs between rollout and training under a model-guided bidirectional scheduler.","lead":"BiDiRL lets rollout and training GPU pools temporarily borrow each other during LLM reinforcement-learning post-training so idle bubbles shrink. On two 32-GPU clusters it raises training throughput by up to 1.94× versus strong baselines without changing how the model learns.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Short-horizon reward curves (60 steps) plus logical-preservation claims leave the 'no effect on convergence' half of the central claim under-supported.","rationale":"The reader's weakest_assumption correctly isolates the only material soft spot in the joint claim. Throughput evidence is multi-dimensional and internally consistent (planner validation median error ~3 %, hot-switch costs measured and gated, ablations isolate bidirectional + model-guided admission). The design arguments for logical preservation (partial resume, ordered merge, same layouts) make large long-run divergence unlikely, so the short horizon does not falsify the paper; it simply leaves one half of the strongest claim less firmly established than the other. No stronger load-bearing flaw (e.g., unfair baselines, broken models, or internal inconsistency) appears. Hence the ACCEPT verdict and low correctness risk stand; the concrete longer-horizon check would simply tighten the secondary claim without altering the systems contribution.","tokens_in":23947,"tokens_out":559,"duration_ms":17503,"concrete_test":"Re-run the exact Fig. 8 configuration (Qwen3VL-4B, Geo3K, same batch/staleness) for ≥300 steps under both BiDiRL and veRL (s=1 and s=2); report mean and final reward gaps plus any divergence after step 60. If mean gap exceeds ~0.05 or trajectories visibly separate, the 'no effect' clause weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim pairs a thoroughly measured throughput gain (up to 1.94×) with the assertion that bidirectional placement 'without affecting convergence behavior.' Throughput is backed by multi-workload, multi-hardware, multi-baseline runs plus ablations (Figs. 5–7, Table 5). Convergence rests only on Fig. 8 (first 60 steps of Geo3K reward under s=1 and s=2) plus the design claim that GRPO groups/samples are preserved by partial-rollout resume and ordered chunk merge (Secs. 5.2, 6.3, Table 2). Because borrowing changes physical placement, timing of weight sync, and can interrupt/resume partial groups or re-order chunk futures before merge, it is possible for the effective sample stream or gradient timing to differ from a pure primary-only run even when logical groups are eventually identical. Sixty steps is too short to rule out long-horizon divergence under the same staleness bounds that the system is designed to exploit. This is the softest link in the joint claim; the throughput numbers themselves do not depend on it.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"BiDiRL is a hybrid time–space multiplexing system for disaggregated, asynchronous LLM RL post-training. It keeps separate committed rollout and training pools but allows either pool to temporarily host the other stage via a hot-switch runtime, a scheduling-aware static planner that selects a hot-switch-compatible resource envelope from stage-time models, and a bidirectional runtime scheduler (Rollouter-on-TrainPoll and Trainer-on-RollPoll) that admits borrowing only when predicted benefit exceeds measured switch cost and splits work with online-calibrated models. On two 32-GPU testbeds (A6000 and H100), across response lengths, staleness bounds, resource budgets, model sizes, and text/multimodal datasets, the paper reports up to 1.94× training throughput over veRL, AReaL, and ROLL, with ablations attributing gains to model-guided bidirectional borrowing, and claims no effect on GRPO convergence.","tokens_in":24310,"tokens_out":1285,"duration_ms":22593,"significance":"If the results hold, BiDiRL addresses a concrete and recurring inefficiency in modern disaggregated RL stacks: residual idle windows that remain after asynchronous overlap and static partitioning. The combination of a hot-switch-compatible planner, measured switch costs, benefit-over-overhead admission, and two-sided borrowing under a fixed GPU budget is a clear systems contribution relative to one-sided elastic rollout or fixed-pool async designs (Table 1). Strengths include multi-baseline, multi-hardware end-to-end evaluation (Figure 5), ablations isolating both borrow directions and model-guided admission (Figure 6), validated stage-time models with median errors of ~3% (Figure 7), and explicit hot-switch cost measurements (Table 5). These make the throughput claim falsifiable and useful for the RL systems community.","major_comments":[{"comment":"The joint central claim pairs large throughput gains with “without affecting convergence behavior,” but the empirical support for the second half is thin. Figure 8 reports only the first 60 Geo3K reward steps under s=1 and s=2 versus veRL, with small last-point gaps (+0.017 / +0.000). Sections 5.2 and 6.3 and Table 2 argue that partial-rollout resume and ordered chunk merge preserve logical GRPO groups, yet borrowing still changes weight-sync timing, can interrupt/resume partial groups, and reorders chunk futures before merge. Under the same staleness bounds the system is designed to exploit, short-horizon reward agreement does not rule out long-run divergence of the effective sample stream or gradient timing. Please either (i) extend convergence runs to a substantially longer horizon (and, ideally, a second dataset/model) under the same s settings used in the throughput sweeps, or (ii)","section":null},{"comment":"End-to-end speedups in §7.2 are measured under “the same node-aligned rollout/trainer partitioning as the compared systems,” so the static planner’s selected envelope (Algorithm 1, §4) is not the primary driver of the headline 1.05×–1.94× numbers; its role is mainly to supply hot-switch-compatible layouts and stage models. Figure 7 shows that partition choice matters (up to 2.09× variation for 4B) and that the planner tracks the measured-best partition in the displayed sweeps, but the paper should more clearly separate (a) gains from bidirectional runtime scheduling under a fixed common partition from (b) gains from planner-chosen partitions. Without that separation, readers may over-attribute end-to-end speedups to static planning. A short table or paragraph that reports BiDiRL under the planner-selected partition versus the baseline-aligned partition would make the two contributions lo","section":null}],"minor_comments":[{"comment":"Abstract and §3.1 describe hot-switch overhead as “negligible,” while Table 5 reports C_in/C_out of several seconds (e.g., 3.58–7.70 s). §7.3 correctly treats these costs as non-negligible for short windows and gates admission on them. Align the abstract/intro wording with the measured costs and admission rule.","section":null},{"comment":"Figure 5 caption and §7.1 mark unsupported/OOM settings with ×/OOM, but a single consolidated table of which baseline configurations were excluded (and why) would improve reproducibility of the speedup ranges.","section":null},{"comment":"Notation for the resource envelope E in Eq. (2) introduces ρ_r, ρ_t and M_r, M_t; Algorithm 1 then returns d_r, d_t in the best tuple. A one-line clarification that d is induced from (g, ρ) would reduce minor ambiguity between layout and replica count.","section":null},{"comment":"In §5.2.1, prompt groups are split by replica capacity (Eq. 6) without length prediction; §8 notes this limitation. A brief quantitative note on how often interrupted auxiliary groups return partial prefixes would help readers judge resume overhead in practice.","section":null},{"comment":"Typos/polish: “1 .94×” spacing appears repeatedly in the abstract and §1; “Trainpoll/Rollpoll” capitalization is inconsistent with “TrainPoll/RollPoll” in Figure 3.","section":null}],"recommendation":"minor_revision","confidential_remarks":"This is a strong systems paper for a systems/DC venue. The throughput evaluation and ablations are above the usual bar; the only soft joint-claim element is long-horizon convergence. I would not reject on that basis, but I would not accept without either longer reward curves or a carefully narrowed convergence statement. Scope fit is good for cs.DC / systems for ML."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that BiDiRL actually ships the missing piece in disaggregated LLM RL: under a fixed GPU budget it can temporarily run trainers on the rollout pool and rollouters on the training pool, with a planner that only picks partitions that stay hot-switch compatible and a runtime that admits a borrow only when the stage-time model says the gain beats measured C_in/C_out/C_grad. That is not just “more async.” Prior systems either freeze the pools, expand only rollout, or harvest external machines. Same-budget two-way borrow is the real delta.\n\nThey execute it cleanly. Two 32-GPU platforms, response length / staleness / scale / model / modality sweeps, head-to-heads against veRL, AReaL, and ROLL, plus ablations that separate no-borrow, one-direction, and opportunistic policies from model-guided admission. Stage models land at ~3% median error; hot-switch costs are tabled; partial-rollout resume and ordered chunk merge are specified. Throughput up to 1.94× on fixed hardware is material for anyone paying for GRPO-style post-training.\n\nSoft spot, in proportion: the joint claim “faster without affecting convergence” rests on 60 Geo3K steps under two staleness values plus the design argument that logical groups and ordered merges are preserved. Borrowing changes placement, weight-sync timing, and can interrupt/resume partial groups, so long-horizon sample-stream or gradient-timing effects are not ruled out. That does not sink the systems result—the speedups stand without it—but the abstract over-sells the RL-behavior half. No public artifacts is a practical annoyance, not a soundness hole.\n\nThis is for people building or operating disaggregated RL stacks who care about utilization under staleness bounds. Math and citation pattern look honest; free parameters are the usual fitted stage coefficients and measured switch costs, not hidden knobs. I would send it to referees. Worth reading and, for systems work in this niche, worth citing.","headline":"Solid systems paper: same-budget bidirectional borrow under a hot-switch envelope is new relative to StreamRL/AReaL/ROLL, and the 1.05–1.94× throughput story is well measured; the “no effect on convergence” half is thinner than the abstract implies.","tokens_in":24925,"tokens_out":538,"would_cite":true,"duration_ms":8824,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"BiDiRL reclaims idle GPUs on both sides of asynchronous LLM RL, raising training throughput by up to 1.94× without changing how the model learns.","keywords":["LLM RL post-training","disaggregated architecture","asynchronous rollouts","bidirectional resource scheduling","hot-switch runtime","staleness-bounded training","resource bubbles","GPU scheduling"],"falsifier":"Run the same GRPO workload for several hundred steps with and without bidirectional borrowing; if final reward or sample statistics diverge once residual bubbles become large, the claim that placement is learning-neutral fails.","tokens_in":24858,"feed_emoji":"⚡","tokens_out":629,"duration_ms":11281,"temperature":0.7,"pith_summary":"Disaggregated, asynchronous reinforcement-learning post-training for large language models still leaves GPUs idle: a fixed split between generation (rollout) and weight-update (training) pools cannot keep pace with shifting response lengths, staleness limits, and parallelism constraints. BiDiRL treats those idle windows as a two-timescale scheduling problem. Before a job starts, a planner picks a resource envelope that roughly balances the two stages and keeps both pools hot-switchable. At runtime a lightweight switch mechanism lets the bottleneck stage borrow idle devices from the other pool, but only when a profiled time model predicts that the gain exceeds the measured switch cost. On two 32-GPU platforms the design lifts end-to-end token throughput by as much as 1.94 times relative to existing systems while preserving the logical samples that the learning algorithm consumes.","feed_headline":"Idle GPUs on both sides of LLM RL yield 1.94× more training","feed_subtitle":"BiDiRL lets rollout and training pools lend each other devices only when the switch pays off","key_machinery":"The hybrid time-space multiplexing stack: a hot-switch runtime that swaps rollout and training roles with measured overhead, a static planner that returns a hot-switch-compatible resource envelope, and a bidirectional scheduler that admits temporary borrowing only when stage-time models predict net benefit and then splits work between primary and auxiliary replicas.","core_discovery":"The paper shows that residual idle time in staleness-bounded disaggregated LLM RL is two-sided and can be harvested by bidirectional, model-guided borrowing inside a fixed GPU budget. With a hot-switch runtime, a scheduling-aware static partition, and an admission rule that compares predicted stage speedup against measured switch overhead, BiDiRL raises training throughput by up to 1.94× across workloads, models, and hardware without altering GRPO convergence behavior.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["BiDiRL harvests two-sided idle GPUs for 1.94× RL throughput","Bidirectional hot-switches cut idleness in disaggregated LLM RL","Planner plus runtime borrowing balances rollout-training GPU pools","Async RL gains 1.94× by lending idle devices both directions","Hot-switch runtime reclaims bubbles across RL resource pools"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That short early training curves plus the claim that logical sample groups are preserved under preemption are enough to guarantee that bidirectional placement never changes long-run learning behavior.","fun_headline_variants_meta":{"raw":{"variants":["BiDiRL harvests two-sided idle GPUs for 1.94× RL throughput","Bidirectional hot-switches cut idleness in disaggregated LLM RL","Planner plus runtime borrowing balances rollout-training GPU pools","Async RL gains 1.94× by lending idle devices both directions","Hot-switch runtime reclaims bubbles across RL resource pools"]},"model":"grok-4.5","effort":"low","cost_usd":0.010546,"raw_usage":{"total_tokens":2409,"prompt_tokens":881,"num_sources_used":0,"completion_tokens":95,"cost_in_usd_ticks":105460000,"prompt_tokens_details":{"text_tokens":881,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1433,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":881,"tokens_out":95,"duration_ms":12151,"temperature":1.0,"reasoning_tokens":1433,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T04:41:13.841615+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same GRPO workload for several hundred steps with and without bidirectional borrowing; if final reward or sample statistics diverge once residual bubbles become large, the claim that placement is learning-neutral fails.","supporting_citations":[],"review_version":1}