{"id":"c120fd03-cdb6-4a99-9edf-df45ac7a8afd","arxiv_id":"2608.09444","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Continuous depth batching schedules each loop iteration of a depth-adaptive looped language model separately, reaching up to 99% of the theoretical adaptive-depth speedup.","lead":"The paper builds a scheduler that lets each token in a looped language model exit after a different number of thinking steps while keeping the GPU batch full. It reports 1.5 to 1.9 times higher throughput and 45 to 90 percent lower latency on two test models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ouro speedups are measured under a shared KV cache that Appendix C.1 shows collapses Ouro's GSM8K accuracy from 77.86% to 0.23%, so the headline 1.5–1.9x claim may not transfer to the actual model.","rationale":"I read the paper in good faith; the scheduler design is genuinely novel and the FLOP bound and roofline analysis are useful tools. The most load-bearing weakness is the internal inconsistency between the experimental setup (shared KV cache) and the model's demonstrated accuracy under that cache. This is the reader's weakest assumption, and I agree with it. Replay of pre-recorded exit traces is a standard benchmarking simplification and is not by itself disqualifying, though it should be disclosed. The absence of released code is a reproducibility concern, not a correctness one. The concern is addressable by a rerun with an accurate cache layout, so CONDITIONAL remains the right verdict and no change to the reader's verdict is needed.","tokens_in":17296,"tokens_out":4844,"duration_ms":46026,"concrete_test":"Rerun the Ouro 1.4B offline throughput and online latency experiments of Section 6 using the last-exited KV cache (or a full depth-indexed cache with depth-aware padding) in place of the shared cache, keeping all scheduler code, replay traces, and thresholds identical. Report GSM8K accuracy and the CDB-vs-CB speedups. If Ouro accuracy returns to approximately 77.86% and the CDB speedups stay near the 94–99% of the FLOP bound reported in Figure 5, the concern is resolved; if accuracy is restored only at materially lower speedups, the abstract's claim must be scoped to cache-preserving configurations for Ouro.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main serving experiments for Ouro 1.4B use the single-slot shared KV cache (Section 6.1: 'Both models use a shared KV cache'). The paper's own Appendix C.1, Table 1, shows this cache destroys Ouro's task accuracy: GSM8K drops from 77.86% (full depth-indexed) to 0.23% (single shared slot), while Huginn is insensitive. Because the shared cache deliberately changes attention semantics (Section 4.2), the tokens in the Ouro throughput/latency runs are not being processed by a model that performs the benchmarked task. The abstract's claim that CDB achieves 'up to 99% of the theoretical maximum speed-up' and '1.5–1.9x higher offline throughput' therefore rests on an internal inconsistency with the accuracy appendix. The paper's own Limitations section concedes that only the shared KV cache was evaluated; the last-exited cache described in Section 4.2 has different memory and copy costs and was not benchmarked. This is a load-bearing experimental gap, not a theoretical flaw in the scheduler.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes continuous depth batching (CDB), a serving technique for depth-adaptive looped language models. CDB decomposes decode into separate prelude, recurrent-core, and coda queues, introduces depth-aware KV-cache layouts, and hides scheduling overhead with asynchronous batch preparation and a lookahead exit gate. The authors derive a FLOP-based upper bound on the speed-up from adaptive depth and a roofline latency model, then evaluate CDB on Ouro 1.4B and Huginn 3.5B, reporting up to 99% of the theoretical maximum speed-up, 1.5-1.9x offline throughput gains, and 45-90% lower normalized latency under dynamic serving load.","tokens_in":17596,"tokens_out":5293,"duration_ms":53894,"significance":"If the experimental claims held, this would be a meaningful systems contribution: it would show that depth-adaptive looped LMs can be served efficiently end to end and would provide a queue-based scheduling recipe applicable beyond the two benchmarked models. The paper has clear strengths: the FLOP bound in Eq. (2) is simple and internally consistent; the roofline model in Section 5 is fitted to careful latency measurements; the scheduler design addresses a real and previously unimplemented problem; and the authors evaluate two architecturally distinct looped LMs. However, the headline serving results are currently undermined by two experimental choices: the shared KV cache used for Ouro is shown in the paper itself to collapse Ouro's task accuracy, and exit decisions are replayed from precomputed traces rather than produced live by the gate. These issues are load-bearing for the speedup claims, so the paper requires major revision.","major_comments":[{"comment":"Section 6.1 states that 'Both models use a shared KV cache,' but Appendix C.1, Table 1 shows that the single-slot shared layout collapses Ouro's GSM8K accuracy from 77.86% to 0.23%. The Ouro throughput and latency results in Figures 5 and 6 are therefore measured on a configuration of Ouro that does not perform the benchmarked task, and the abstract's 1.5-1.9x speedup claim cannot be assumed to transfer to the actual Ouro model. The paper should either re-run the serving experiments with a cache layout that preserves accuracy (e.g., full depth-indexed or first-then-shared, which retains 71.34%), or report accuracy for the served configuration and explicitly argue why the speedup transfers despite the accuracy collapse.","section":"Section 6.1 / Appendix C.1, Table 1"},{"comment":"Appendix B.2 states that 'we do not use the exit gate during our experiments' and that per-token exits are recorded once in advance and then replayed. This means the lookahead gate described in Section 4.3 is not exercised end to end in the serving benchmarks; the reported throughput, latency, and 0.67% idle-time figures characterize trace-replay execution only, not live adaptive inference. Because per-token exit decisions are the defining adaptive component of CDB, the current experiments cannot validate the claim that CDB realizes the adaptive-depth speedup. Please run the serving experiments with live gate decisions under the same KV-cache layout used for serving, or clearly separate trace-replay benchmarks from live-gate validation and temper the claims accordingly.","section":"Appendix B.2 / Section 4.3"},{"comment":"The comparison to the FLOP bound in Section 6.2, where CDB is reported to reach 94-99% of the theoretical maximum for Ouro, uses the upper bound of Eq. (2), which by construction ignores prefill, scheduling, KV-cache overhead, and boundary-stage costs. Combined with trace replay, this metric mostly characterizes queue mechanics rather than adaptive inference. The paper should state explicitly that Eq. (2) is an idealized decode-only reference, and it should report the actual mean exit depth and the accuracy of the served model under the benchmarked configuration, since the bound's value depends directly on \\bar{d}.","section":"Section 6.2 / Eq. (2)"}],"minor_comments":[{"comment":"No code or artifact link is provided; for a systems paper whose main contribution is an implementation, releasing the scheduler, kernels, and replay traces would substantially aid reproducibility.","section":"General"},{"comment":"The definition of 'per-step device idle (%)' and the measurement conditions should be clarified, including whether the reported 0.67% idle time was measured with live gate decisions or with replayed traces.","section":"Figure 3"},{"comment":"The paper notes that the minimum coda batch size for Huginn 'should thus scale with the decode batch size,' but this parameter is not swept; a sensitivity analysis would clarify how robust the Huginn speedups are to this choice.","section":"Section 6.2"},{"comment":"The catastrophic Ouro accuracy drop under single-slot sharing (0.23%) is only in the appendix; it should be highlighted in the main text wherever the shared cache is described as the standard serving configuration.","section":"Appendix C.1"},{"comment":"The roofline model fit in Appendix A is reported as essentially perfect (R^2 >= 0.999) for the latency-versus-depth fit; reporting confidence intervals or residual plots for the batch-size fits would strengthen the claim that B* is estimated reliably.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The core scheduling idea is promising and the analytical framework is internally sound, but the experimental validation is currently not sufficient to support the abstract's headline claims. The shared-KV accuracy collapse and the use of replayed exit traces are fixable in principle, but they require either re-running the benchmarks in a configuration that preserves model accuracy and exercises the live gate, or substantially reframing the claims. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is the first actual end-to-end serving stack for depth-adaptive looped LMs, and the queue-based scheduler plus lookahead gate are real engineering contributions. But the headline Ouro speedup is measured under a shared KV cache that the paper's own Appendix C.1 shows collapses Ouro from 77.86% to 0.23% GSM8K. So the 1.5-1.9x claim is not yet supported for Ouro; Huginn's results are more trustworthy.\n\nThe genuinely new piece is the implementation: separate priority queues for prelude, recurrent core, and coda, a one-step lookahead exit gate that hides scheduling latency, and two KV-cache layouts with a clear analysis of their trade-offs. The FLOP bound is simple but useful, and it is derived independently of the measurements, so the fact that CDB gets within a few percent of it tells you something real. The roofline model is fitted, but it's used to explain when refill pays rather than to manufacture the main speedups, so I don't see circularity there. The authors are also transparent: they state in the limitations that only the shared KV cache was evaluated.\n\nThe soft spots are real but not fatal. The shared-cache issue is load-bearing for Ouro: the tokens in the throughput and latency runs are not being processed by a model that performs the benchmarked task. That's a big gap. The exit decisions are replayed from pre-recorded traces rather than produced live by the gate, so the scheduler isn't tested against the actual gate's behavior, and the lookahead gate is only validated against teacher exits. There's no code release, which makes it hard to verify the scheduler overhead claims. These are fixable. The Huginn results, where the shared cache does not hurt accuracy, are much more solid.\n\nThis paper is for people building inference systems for recurrent or looped architectures. It deserves a serious referee, but the revision needs to either benchmark Ouro with a cache that preserves accuracy or re-scope the claims to Huginn. I'd bring it to reading group — the experimental assumptions are a good case study.","headline":"First real serving stack for depth-adaptive looped LMs, but the Ouro speedup claim rests on a cache layout that destroys the model's accuracy; Huginn results are the solid half.","tokens_in":18087,"tokens_out":2394,"would_cite":false,"duration_ms":23550,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that depth-adaptive looped language models — which spend fewer loops on easy tokens — can be served at up to 99% of the theoretical speed-up, once inference schedules each loop iteration instead of the whole forward pass.","keywords":["looped language models","depth-adaptive inference","continuous depth batching","early-exit scheduling","KV cache design","LLM serving systems","roofline model","adaptive compute"],"falsifier":"Serve Ouro 1.4B under CDB with the first-then-shared or full depth-indexed KV cache — the layouts that keep GSM8K accuracy near 71–78% — and compare throughput and normalized latency against the same engine running full depth: if the reported 1.5–1.9× throughput gain and 45–90% latency reduction do not appear, the central claim fails for the model on which it is strongest. A second check is to measure the recurrent-step latency curve on hardware with much cheaper weight reload and confirm whether the refill advantage disappears as $B^*$ shrinks, as the roofline model predicts.","tokens_in":17095,"feed_emoji":"⚙️","tokens_out":14669,"duration_ms":129055,"temperature":0.7,"pith_summary":"Looped language models promise depth-adaptive inference: by iterating a shared block of layers a variable number of times per token, easy tokens consume less compute than hard ones. The paper's claim is that this promise has been blocked by the serving stack, not by the models, because variable-depth tokens break the uniform forward pass that standard batching requires. It introduces continuous depth batching (CDB), which schedules each loop iteration as its own unit, routes prelude and coda stages through separate queues, and hides scheduling behind GPU execution with a lookahead gate. On Ouro 1.4B and Huginn 3.5B, the paper reports that CDB reaches up to 99% of the theoretical maximum speed-up from adaptive depth, delivering 1.5–1.9× higher offline throughput and 45–90% lower normalized latency under dynamic load. If the paper is right, depth-adaptive looped LMs become practical to serve, and the queue-based recipe transfers to any model with variable per-token depth.","feed_headline":"Per-loop batching hits 99% of the adaptive-depth speed limit","feed_subtitle":"Scheduling each loop iteration as its own batch turns easy-token savings into 1.5–1.9× throughput.","key_machinery":"The load-bearing mechanism is a queue-based decode scheduler that treats the prelude, recurrent core, and coda as independent stages and batches tokens per loop iteration rather than per token generation. Three pieces carry the argument: the refill mode, which exploits the recurrent core's shared weights to admit new tokens into slots freed by early exits; the lookahead gate, which reads the exit signal one step early so batch preparation overlaps GPU execution, cutting measured per-step device idle time to 0.67%; and depth-aware KV caches — a last-exited layout that copies a token's final state into the depth slots it skipped, and a shared layout that keeps one KV slot per layer and overwrites it at every loop, trading attention semantics for an $r_{\\max}$-fold memory saving. The analytical machinery is the FLOP bound $\\frac{F_0 + r_{\\max}F_r}{F_0 + \\bar d F_r} \\le \\frac{r_{\\max}}{\\bar d}$, where $\\bar d$ is the mean exit depth, together with the roofline model $t(N_c, N_b) = \\delta + \\gamma N_b + \\alpha N_c$, whose saturation batch size $B^* = \\delta/(\\gamma + \\alpha L)$ marks the boundary between the memory-bound regime where refill pays and the compute-bound regime where early exits save time directly.","core_discovery":"The paper's central claim is that depth-adaptive decoding of looped language models can be served efficiently, and that continuous depth batching is the first end-to-end implementation to do so. CDB decomposes generation into four queues — prefill, prelude, recurrent core, and coda — and schedules at the granularity of individual loop iterations, so tokens at different depths share the same forward pass even though they exit at different times. A lookahead gate makes the exit decision one loop step in advance, giving the CPU a full step to prepare the next batch while the GPU runs the current one, and a refill mode fills slots freed by early exits with fresh tokens to keep the recurrent batch large. Evaluating with exit traces replayed from Alpaca and ShareGPT, the paper finds that CDB realizes up to 99% of the theoretical FLOP-bound speed-up on Ouro 1.4B and Huginn 3.5B, translating to 1.5–1.9× higher offline throughput than continuous batching at full depth and 45–90% lower normalized latency under Poisson arrivals. The paper further derives a FLOP-based upper bound and a roofline latency model that locate the benefit of depth adaptivity in the memory-bound regime, where a step's cost is dominated by reloading the recurrent core's weights rather than by batch size.","pith_inferences":["All serving numbers use the single-slot shared KV cache; because the paper's own ablation shows that cache collapses Ouro's GSM8K accuracy from 77.86% to 0.23%, a direct test is CDB with a first-then-shared cache — which keeps accuracy near 71% — to see whether the 1.5–1.9× throughput gain survives an accuracy-preserving layout.","The roofline model implies a hardware prediction the paper does not test: on accelerators where weight reload is cheap (small $\\delta$), $B^*$ shrinks, the memory-bound regime nearly disappears, and refill should lose most of its throughput advantage.","The paper replays fixed exit traces for reproducibility; in a live deployment the gate would react to queue state, and the interaction between exit-depth distribution and serving dynamics is not covered by the reported gains.","The stage-queue abstraction invites composition with chunked prefill and speculative decoding, but those combinations are untested; in particular, a speculative head would add a second per-token routing decision inside the loop that the lookahead gate would have to cover."],"forward_implications":["On the fully looped Ouro architecture, CDB with refill reaches 94–99% of the FLOP bound and stays within 2% of the baseline when no early exit occurs, so the scheduler's own overhead is negligible.","On Huginn, with transformer layers in the prelude and coda, CDB still reaches up to 1.9× throughput but only 73–92% of the FLOP bound, so heavy boundary stages are the main structural factor that erodes adaptive-depth gains.","Refill's advantage is largest at batch sizes below $B^*$, where decode is memory-bound and shrinking the batch saves no wall-clock time; above $B^*$ the two modes converge, so no-refill is preferable when simplicity matters.","The lookahead gate imposes a minimum depth $r_{\\min}=2$, which caps the achievable speed-up for models with small $r_{\\max}$; the paper shows the one-step-earlier decision matches the original gate's accuracy.","CDB executes any given exit rule without changing model predictions, so the scheduler is compatible with learned gates, training-free convergence criteria, and depth-routing methods, not tied to a particular model."],"supporting_citations":[{"why":"Supplies the paged-attention serving baseline whose codebase, kernels, and admission policy all three compared engines share.","marker":"Kwon et al. 2023"},{"why":"Defines continuous batching, the token-level baseline that CDB is normalized against and whose uniform forward pass motivates the method.","marker":"Yu et al. 2022"},{"why":"Provides the Huginn model, the prelude-core-coda nomenclature, and the training-free convergence exit criterion that the experiments replay.","marker":"Geiping et al. 2025"},{"why":"Provides the Ouro model, its learned exit gate, and the near-lossless KV-sharing claim that the accuracy ablation checks.","marker":"Zhu et al. 2025"},{"why":"Introduced the continuous-depth-batching refill concept that this paper implements end to end for the first time.","marker":"Bae et al. 2025a"},{"why":"Supplies the roofline model that grounds the recurrent-step latency fit and the saturation batch size $B^*$.","marker":"Williams et al. 2009"},{"why":"Provides the Alpaca workload whose prompt and output length distributions drive the throughput and latency measurements.","marker":"Taori et al. 2023"},{"why":"Provides the ShareGPT workload, a long and heavy-tailed request distribution used in both offline and online evaluations.","marker":"ShareGPT Team 2023"}],"fun_headline_variants":["Depth batching nears the adaptive-speed limit","Loop-level scheduling closes the adaptive gap","CDB: 99% of the depth-adaptive speed cap","Adaptive-depth LMs get a batching breakthrough","Per-iteration batching speeds up looped LMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single-slot shared KV cache, used in every serving experiment, is an acceptable configuration for the benchmarked models — yet the paper's own Appendix C.1 shows this cache collapses Ouro's GSM8K accuracy from 77.86% to 0.23%, so the headline speed-ups may not hold under the cache layout that preserves Ouro's quality.","fun_headline_variants_meta":{"raw":{"variants":["Depth batching nears the adaptive-speed limit","Loop-level scheduling closes the adaptive gap","CDB: 99% of the depth-adaptive speed cap","Adaptive-depth LMs get a batching breakthrough","Per-iteration batching speeds up looped LMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1389,"prompt_tokens":1088,"completion_tokens":301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":225}},"tokens_in":704,"tokens_out":301,"duration_ms":4041,"temperature":1.0,"reasoning_tokens":225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:30.564319+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Serve Ouro 1.4B under CDB with the first-then-shared or full depth-indexed KV cache — the layouts that keep GSM8K accuracy near 71–78% — and compare throughput and normalized latency against the same engine running full depth: if the reported 1.5–1.9× throughput gain and 45–90% latency reduction do not appear, the central claim fails for the model on which it is strongest. A second check is to measure the recurrent-step latency curve on hardware with much cheaper weight reload and confirm whether the refill advantage disappears as $B^*$ shrinks, as the roofline model predicts.","supporting_citations":[{"cited_title":"Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein","cited_arxiv_id":null,"evidence_quote":"Provides the Huginn model, the prelude-core-coda nomenclature, and the training-free convergence exit criterion that the experiments replay."},{"cited_title":"Hashimoto","cited_arxiv_id":null,"evidence_quote":"Provides the Alpaca workload whose prompt and output length distributions drive the throughput and latency measurements."},{"cited_title":"ShareGPT , 2023","cited_arxiv_id":null,"evidence_quote":"Provides the ShareGPT workload, a long and heavy-tailed request distribution used in both offline and online evaluations."}],"review_version":1}