{"id":"79bb14f9-0b1a-4e88-bbe2-397318ef7553","arxiv_id":"2608.10402","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TideRL, a readiness-aware elastic RL system, raises agentic RL training goodput by up to 5.6x over synchronous and over 33% over asynchronous baselines by preserving rollout KV caches, pipelining reference and actor models, and migrating GPU ranks at weight-sync boundaries.","lead":"This paper presents TideRL, a system that schedules GPU work for reinforcement-learning training of AI agents, keeping useful data in memory and moving chips between rollout and training on the fly. TideRL reports up to 5.6x faster training than synchronous baselines and over 33% faster than asynchronous ones, which could cut the cost of training capable agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ERS's zero-overhead cache-free rank migration is invalid for evaluation tasks, which need anchored weights; the paper's own Limitations section admits eval phases disrupt the async pipeline, so ERS's claimed contribution and the end-to-end goodput numbers that include eval are not fully supported.","rationale":"The reader's weakest-assumption analysis correctly identifies the evaluation-task exception to ERS's cache-free migration. I agree, and I think this is the single most load-bearing vulnerability in the paper's argument, because ERS is advertised as 'zero-overhead elasticity' and is credited with a 35.7% throughput gain in the ablation (Table 2), yet the cost model and migration protocol in §5.2 assume that every KV cache on a migrated rollout rank is stale after a weight sync. Evaluation tasks are deliberately excluded from that assumption by the system's own priority rules (Table 1) and by the Limitations paragraph. The paper's suggested fix—relay weight synchronization as in Laminar—is explicitly future work, so the current system's measured results cannot claim to have solved evaluation-phase migration. This matters quantitatively: the step profile in Table 3 includes a large eval component, and the end-to-end runs evaluate every 20 steps, so evaluation phases are inside the measured wall-clock and waiting-time numbers. The right response is not to reject the paper: the architecture is coherent, the async baseline comparisons are reasonable, and the direction of the gains is credible. But the zero-overhead claim is stronger than the evidence, and the headline throughput numbers should be scoped to the non-evaluation portions of the loop, or the authors need to show that ERS during evaluation preserves both metric consistency and the measured gains. I also note the reader's other concerns: no code/data release, no error bars, and the Table 2 percentage arithmetic not reconciling exactly; those are verification issues that reinforce conditionality but do not change the main concern. Hence verdict remains CONDITIONAL, unchanged from the reader.","tokens_in":22899,"tokens_out":11195,"duration_ms":104526,"concrete_test":"Run the WebShop text-only 100-step experiment under two otherwise identical conditions: (A) ERS is allowed to migrate ranks that host evaluation tasks during the evaluation sub-phase, exactly as described in §5.2; (B) ERS is prohibited from migrating any rank that hosts an evaluation task (as Table 1's highest-priority tier implies). For each condition record eval BoN reward and pass rate at every 20-step checkpoint, total RWT+TWT, and training throughput over the full run, plus a no-ERS control. If condition A changes the eval metrics relative to the no-ERS control, the anchored-version premise is violated. If condition B eliminates most of ERS's waiting-time or throughput gain, the zero-overhead elasticity claim covers only the non-eval portion of the loop, and the headline numbers must be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ERS's 'Cache-Free Task Migration' (§5.2) argues that after every weight sync, historical KV caches are invalid for the new policy, so a migrated task can be resumed with a fresh prefill under the new weights at no cost. That argument is valid only for ordinary training rollout. Evaluation tasks are the exception the paper itself states: Table 1 gives evaluation tasks highest CTB priority because they 'require anchored weights,' and §7's Limitations says enforcing a frozen model version for eval 'can temporarily disrupt the asynchronous pipeline momentum' and leaves relay weight synchronization as future work. If ERS closes a rank hosting an eval task, CTB's reassignment would either (a) resume the task with the synchronized new weights, silently changing the evaluated model version and corrupting the 'similar task performance' numbers, or (b) keep the old-version weights/KV caches, which is exactly the cache transfer and memory cost the zero-overhead claim denies. Eval tasks are not a negligible corner: Table 3's step profile uses 200 eval groups vs 32 train groups. The ERS ablation (+35.7%) and the 77.6% RWT+TWT reduction therefore include phases where the load-bearing assumption of free cache discard does not hold, or ERS must be disabled during eval. Either way the central goodput claim is over-generalized.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TideRL, an asynchronous reinforcement learning (RL) system for multi-turn agentic workloads. Three co-designed mechanisms are introduced: Continuous Task Batching (CTB) for task-level KV-cache-aware rollout scheduling, Resource-Aware Ref-Actor Pipelining (RA2P) that switches between decoupled streaming and colocated aggregation based on data readiness, and Elastic Resource Scaling (ERS) that migrates ranks between rollout and training using the same readiness signals. The paper claims up to 5.6x training-goodput improvement over the synchronous VeRL baseline, over 33% over asynchronous baselines (AReaL, StreamRL), comparable task performance, a 1.58x KV cache hit-rate improvement, 44.3% per-step training time reduction, and up to 77.6% waiting-time reduction. The evaluation covers text-only (WebShop, AlfWorld) and multimodal (OSWorld, ScienceBoard) tasks with Qwen models on 32 H100 GPUs.","tokens_in":1480,"tokens_out":1542,"duration_ms":74694,"significance":"If the claims are validated, TideRL is a substantive systems contribution. The paper correctly identifies real bottlenecks in agentic RL (KV-cache preemption, ref-actor thrashing, static resource allocation) and proposes a coherent, well-motivated co-design. The goodput definition (training throughput excluding stale tokens) is appropriate for on-policy RL. The component-wise motivation is supported by measurements (Figures 2-4), and the implementation is modular. However, the empirical validation has gaps: no repeated runs or error bars, a quantitative inconsistency in the ablation table, and an unresolved tension between the zero-overhead elasticity claim and the paper's own limitations about evaluation tasks. These issues make the central quantitative claims not yet fully supported.","major_comments":[{"comment":"The improvement percentages in Table 2 do not match the listed throughput values. The baseline is 20.2 k token/s; adding RA2P gives 23.1, which is an increase of 14.4%, not 10.4%. Adding ERS to that gives 31.8, an increase of 37.7% over 23.1, not 35.7%. The 4.8% for CTB is close (33.3/31.8 = 1.047). This internal inconsistency affects the paper's component-wise attribution of gains and must be corrected, ideally with confidence intervals across repeated runs.","section":"Section 6.4, Table 2"},{"comment":"The zero-overhead cache-free migration claim is over-generalized for evaluation tasks. Section 5.2 states that after each weight sync, all KV caches are invalid and can be discarded for free, so migrated tasks resume with a fresh prefill under the new weights. However, Table 1 gives evaluation tasks the highest CTB priority because they 'require anchored weights,' and Section 7's Limitations admits that enforcing a frozen model version for evaluation 'can temporarily disrupt the asynchronous pipeline momentum' and leaves a relay synchronization mechanism to future work. Since the end-to-end measurements include evaluation phases (Table 3 profiles 200 eval groups vs. 32 train groups, and Section 6.4 explicitly includes the initial evaluation step), the paper must either (a) demonstrate that ERS never closes a rank hosting an active evaluation task and maintain zero overhead during evaluation phases, or (b) restrict the zero-overhead claim to training rollout and quantify the overheads incurred during evaluation.","section":"Section 5.2 and Section 7"},{"comment":"All reported throughput, training-time, and waiting-time numbers are single-point measurements with no error bars, no repeated runs, and no code/data release. Given the high variance in agentic workloads described in the paper (e.g., Figure 4 shows RWT/TWT crossing patterns even under a fixed ratio, and Section 6.3 mentions 'high variance in training workloads'), the central claims (5.6x, 33%, 77.6%, and the ablation percentages) require at least a small number of repeated runs with confidence intervals, or a clear statement of experimental variance, to be convincing.","section":"Section 6"}],"minor_comments":[{"comment":"The system is called 'TideRL' in the title and abstract but 'TIDERL' in the body; similarly, Section 5 uses 'RA²P' in the heading while the rest of the paper uses 'RA2P'. Please standardize the notation.","section":"Title/Abstract vs. Body"},{"comment":"The priority-score weights ω1–ω4 are introduced but no default values or sensitivity analysis are provided; since the CTB preemption hierarchy depends on this score, a specification of the default weights and a brief ablation would improve reproducibility.","section":"Section 4.2, Eq. (1)"},{"comment":"The caption 'CTB beats F10 and F9 on both metrics' is vague; it should state which metrics (hit rate and throughput) and under what configuration the comparison is made.","section":"Figure 12"},{"comment":"The formula for Cdec defines δi in a compressed manner; clarifying how N0 enters the definition (e.g., the range of i for which δ_i=0) would make the cost model easier to follow.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper builds closely on the authors' own AgentRL system (reference [44]) with significant author overlap, but the novelty relative to AgentRL is not crisply stated; please ask the authors to clarify the incremental contribution. Also note the absence of any code/data release statement, which is increasingly expected for systems papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TideRL is a genuine integration, not a repackaging: CTB, RA2P, and ERS are mechanisms that existing systems each touch in isolation, and the paper's contribution is making them share one readiness signal and testing that feedback loop end to end. That is worth a serious referee.\n\nThe motivation is well supported by measurements (the KV cache hit-rate collapse in Figure 2, the RWT/TWT crossover in Figure 4), the design is clearly explained, and the evaluation covers text and multimodal workloads, several model sizes, and three strong baselines with swept fixed partitions. The authors also report learning curves and get close to the synchronous baseline on task performance, which is the right check.\n\nBut three things keep me from taking the specific numbers on faith. First, no code, no data release, no repeated runs, no error bars. RL training is noisy, and a 1.33x improvement over the best asynchronous baseline is exactly the kind of effect that can move with the seed. Second, the ablation table has a small internal inconsistency: the +10.4% and +35.7% in Table 2 don't match the throughput values they sit next to. Likely a rounding or editing slip, but it makes me want to re-derive every number. Third, the zero-overhead elasticity claim is over-generalized. The cache-free migration argument is elegant for training rollout, where weight sync invalidates the KV cache anyway. But the paper's own CTB priority gives evaluation tasks top billing precisely because they need anchored weights, and the Limitations section admits that enforcing a frozen version for eval can disrupt the async pipeline. If ERS moves a rank hosting eval tasks, either it keeps the old weights and caches (which is not cache-free), or it resubmits with the new weights and corrupts the eval. The paper leaves this as future work, which is honest, but the Abstract and Section 5.2 claim no wasted KV cache without that caveat. Since eval tasks are part of the measured workload, the end-to-end goodput numbers are not fully supported as stated.\n\nWho is this for? Anyone building async RL training for agents will find the RA2P loss-deferral trick and the migration-at-sync idea useful even without adopting the whole stack. The paper deserves peer review. I'd accept it conditionally and ask for code, error bars, a fixed ablation table, and a section that either scopes ERS to training-only tasks or explains how eval tasks survive migration.","headline":"A credible, well-engineered systems integration whose end-to-end numbers I'd trust only after code and error bars, and whose zero-overhead migration claim needs explicit scoping for evaluation tasks.","tokens_in":23755,"tokens_out":3411,"would_cite":false,"duration_ms":31010,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TideRL claims that agentic RL training goodput—the rollout tokens that actually reach the trainer and advance an update—can rise by up to 5.6× over synchronous baselines and by more than 33% over asynchronous ones when rollout…","keywords":["agentic reinforcement learning","RL goodput","continuous task batching","KV cache management","elastic resource scaling","asynchronous RL training","reference-actor pipelining"],"falsifier":"Run one training step that contains live rollout groups and an in-flight evaluation batch pinned to an anchored frozen model, and trigger a rank migration while that batch is active. If the migration forces the pipeline to suspend, rebuilds a communication group, or reloads any KV cache for the evaluation tasks, the zero-overhead elasticity claim is falsified; the wall-clock cost of the migration itself is the observable.","tokens_in":22674,"feed_emoji":"⚡","tokens_out":9632,"duration_ms":72113,"temperature":0.7,"pith_summary":"TideRL sets out to fix reinforcement learning for multi-turn agentic large language models, where tasks pause for external environments, resume with growing contexts, and finish at wildly different times. The paper's central claim is that the right scheduling unit is task readiness rather than raw GPU occupancy: keep useful rollout state resident, choose between two reference-actor execution modes from the backlog already at hand and the pace at which new micro-batches arrive, and move ranks between rollout and training as the bottleneck shifts. On that basis TideRL reports RL training goodput—the rollout tokens that actually reach the trainer and advance an update—improved by up to 5.6× over synchronous baselines and by more than 33% over asynchronous baselines on text-only and multi-modal agentic workloads, while staying within a 0.01 best-of-N reward deficit and a 0.5% pass-rate deficit of the best synchronous result after 100 steps. If true, that matters because it attacks the real cost of agentic RL: GPU waiting, repeated prefill recomputation, and model-swap thrashing, which grow with trajectory variance and environment latency rather than with model size.","feed_headline":"Agentic AI training up to 5.6x faster with ready-aware scheduler","feed_subtitle":"Cuts GPU waiting and recomputation while keeping task rewards within 0.01 of the synchronous baseline.","key_machinery":"The central object is the readiness-signal pair (RAS, TPRM): RAS is the number of micro-batches already waiting in the global buffer when a training step begins, and TPRM is the interval at which additional micro-batches become ready during the step. Everything in the system is coupled to this pair: Continuous Task Batching shapes when trajectories become ready, RA2P selects decoupled streaming versus colocated aggregation from the pair, and Elastic Resource Scaling reallocates GPU ranks to shrink whichever side is waiting. The load-bearing identity is the natural weight-sync boundary of on-policy RL: once new weights arrive, all prior KV caches are mathematically stale and must be flushed, so a migrating rank can discard its cached context for free, join the weight broadcast, and resume with the new policy without a pipeline suspension or an NCCL group rebuild.","core_discovery":"TideRL's central claim is that the three inefficiencies of agentic RL training—KV-cache preemption while tasks wait on environments, trainer stalls or reference-actor swap thrashing while micro-batches arrive unevenly, and static GPU partitions that cannot follow a shifting producer-consumer bottleneck—share one root cause: nobody schedules from task-level readiness. The proposed mechanism is a single feedback loop. Continuous Task Batching keeps useful rollout state resident by admitting, pausing, and resuming tasks with token-aware priorities that favor evaluation boundaries, near-complete GRPO groups, active trajectories, and long contexts. Resource-Aware Ref-Actor Pipelining consumes ready micro-batches through two modes selected by readiness signals: decoupled streaming, with loss computation deferred into the backward pass so reference forward, actor forward, and actor backward overlap with zero model swaps, and colocated aggregation with zero-copy shared-memory transfer when data is sparse. Elastic Resource Scaling moves individual ranks between rollout and training using those same signals, and because on-policy RL already invalidates every KV cache at each weight-sync boundary, the migration can discard caches for free and hide weight movement under the synchronization broadcast. With this loop, the paper reports training-throughput gains up to 5.6× over the best synchronous baseline and more than 1.33× over the asynchronous baselines on text and multi-modal agentic workloads, with task performance within 0.01 best-of-N reward and 0.5% pass rate of the synchronous reference.","pith_inferences":["Beyond the paper, the same RAS/TPRM mode-selection logic could address the evaluation-phase gap the authors leave for future work: a relay weight-synchronization path that keeps a frozen evaluation model separate would let the zero-overhead migration premise extend to steps containing evaluation tasks.","Beyond the paper, readiness-driven scheduling of this kind could transfer to interactive agent serving, where cache pinning and prefill decisions are made at task granularity; the difference is that serving lacks the weight-sync boundary that makes TideRL's migration free, so elasticity there would need a different cost argument.","Beyond the paper, the free-migration argument depends on the algorithm being on-policy with periodic full weight synchronization; for any future RL variant that allows stale KV caches to remain useful across weight updates, the zero-overhead elasticity claim would need to be re-derived."],"forward_implications":["Synchronous phase barriers for agentic RL sacrifice most of the available goodput; adopting readiness-aware scheduling recovers up to 5.6× training throughput on text workloads and over 33% on multimodal workloads at near-identical task performance.","Elevating rollout scheduling from requests to tasks raises KV cache hit rate by 1.58× and generation throughput by 1.15×, because paused multi-turn tasks keep their context instead of being evicted while waiting for environment feedback.","Reference-actor execution does not have to choose between computation stalls and model-swap thrashing: decoupled streaming with deferred loss suits abundant ready data, and ready-batch aggregation with zero-copy transfer suits sparse data, cutting per-step training time by up to 44.3%.","Rank migration between rollout and training can be made effectively free by aligning it with the post-update cache flush, so a fixed GPU pool can absorb shifting bottlenecks and cut total waiting time by up to 77.6%.","Asynchronous RL systems can go faster without adding GPUs; they need to react to readiness signals, since elastic reallocation within a fixed allocation beats the best static rollout fraction."],"supporting_citations":[{"why":"Supplies the GRPO objective and the on-policy staleness constraint that make old KV caches invalid after each weight sync, the premise behind ERS's zero-overhead migration.","marker":"[28]"},{"why":"Synchronous RL baseline (VeRL) enforcing rollout-trainer barriers; provides the 5.6× comparison point and the long-tail stall problem TideRL targets.","marker":"[30]"},{"why":"Asynchronous RL baseline (AReaL) that decouples rollout from training; comparison point for the over-33% gains and for bounded-staleness rollout.","marker":"[5]"},{"why":"Asynchronous streaming baseline (StreamRL) that streams micro-batches to the trainer; comparison point for RA2P's anti-thrashing design.","marker":"[49]"},{"why":"Seer's first-turn probing idea is adapted by CTB's token-aware admission; also the reference whose suffix decoding TideRL rejects for multi-turn workloads.","marker":"[23]"},{"why":"APRIL's active partial rollouts bound off-policy staleness in asynchronous RL and inform CTB's dispatch throttling and the staleness bound.","marker":"[52]"},{"why":"AgentRL supplies the disaggregated architecture with environments and rewards on an external CPU cluster that TideRL builds on.","marker":"[44]"}],"fun_headline_variants":["Readiness-aware scheduling boosts agentic RL up to 5.6x","TideRL: Cut GPU idle, speed agentic RL 5.6x","Agentic RL training 5.6x faster with ready-aware scheduler","Scheduling by readiness makes agentic RL 5.6x more efficient"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme assumes that every rollout task's cached context becomes worthless the instant new model weights arrive, so a rank can be moved without saving anything; the paper's own limitations concede that evaluation tasks need an anchored, frozen model version, and keeping that version in sync can disturb the asynchronous pipeline—the one place where that free-discard assumption does not obviously hold.","fun_headline_variants_meta":{"raw":{"variants":["Readiness-aware scheduling boosts agentic RL up to 5.6x","TideRL: Cut GPU idle, speed agentic RL 5.6x","Agentic RL training 5.6x faster with ready-aware scheduler","Scheduling by readiness makes agentic RL 5.6x more efficient"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1473,"prompt_tokens":1071,"completion_tokens":402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":318}},"tokens_in":687,"tokens_out":402,"duration_ms":3928,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:21:38.346106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one training step that contains live rollout groups and an in-flight evaluation batch pinned to an anchored frozen model, and trigger a rank migration while that batch is active. If the migration forces the pipeline to suspend, rebuilds a communication group, or reloads any KV cache for the evaluation tasks, the zero-overhead elasticity claim is falsified; the wall-clock cost of the migration itself is the observable.","supporting_citations":[{"cited_title":"Hybridflow: A flexible and efficient rlhf framework","cited_arxiv_id":null,"evidence_quote":"Synchronous RL baseline (VeRL) enforcing rollout-trainer barriers; provides the 5.6× comparison point and the long-tail stall problem TideRL targets."},{"cited_title":"AREAL: A large-scale asynchronous reinforcement learning system for language reasoning","cited_arxiv_id":null,"evidence_quote":"Asynchronous RL baseline (AReaL) that decouples rollout from training; comparison point for the over-33% gains and for bounded-staleness rollout."}],"review_version":1}