{"id":"02e95019-bbe9-4a5a-afb8-de87d1a902b0","arxiv_id":"2506.22950","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A GRPO decoding framework that cuts memory via micro-batched KV-cache reuse and improves decoding-round efficiency with predicted-length scheduling, at the cost of serialization.","lead":"Infinite Sampling splits each group of generated answers into memory-safe micro batches and schedules decoding by predicted sequence length. It reports halving peak memory and cutting decoding rounds by up to 45 percent, but does not run the full training loop to verify training stability.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central throughput claim rests on 'Running Steps' as a proxy for wall-clock, but that metric is inconsistent with the paper's own definition in Table 1 and ignores variable per-step cost plus scheduler/BERT overhead; no wall-clock or end-to-end training data are reported.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: Running Steps is used as the sole throughput metric with no wall-clock validation, and the 'stable GRPO training' claim lacks training experiments. My reading of the full text reinforces this. The memory reduction claim is plausible and has direct measurements in Figure 4, and the scheduling ideas are coherent, so I would not reject the paper outright. However, the central efficiency claim in the abstract is quantified in decoding steps, and Section 5.1's definition of Running Steps cannot be reconciled with Table 1's Dynamic-slot row: 360 steps at g=4 would require 1,440 token-slots, but 32 completions of average length 21 contain only 672 tokens. Even if the metric were internally consistent, autoregressive decode step cost is not constant across a variable-length batch; SJF and continuous sampling change batch composition, so step reductions do not directly translate to wall-clock gains. The paper also provides no measurement of BERT predictor and scheduler overhead, both of which are nontrivial at k=16 prefixes per sample. Finally, the paper's title and abstract commit to 'stable GRPO training', yet no GRPO training run, reward curve, or downstream accuracy is reported; the only evidence is average sequence length preservation, which is not a proxy for optimization stability. These are addressable with additional experiments rather than fundamental errors, so the reader's CONDITIONAL verdict is appropriate and remains unchanged.","tokens_in":20218,"tokens_out":8094,"duration_ms":96603,"concrete_test":"On the same A100-80GB setup, instrument the full decoding pipeline for Naive Micro Group, Fixed-slot, Dynamic-slot, and Infinite Sampling over the same prompt set, and report end-to-end wall-clock decoding time including prefix sampling (k=16), BERT length prediction, FPTAS/SJF bookkeeping, and decode kernel time. Then compute cost-weighted steps (e.g., sum over steps of active-sequence KV lengths) and compare with Table 1. If wall-clock reductions track the 25-45% step reductions and scheduler/BERT overhead is below 5% of decode time, the throughput claim stands; otherwise the abstract's percentages must be revised. Separately, run at least one full GRPO training loop (e.g., GSM8K, G=32, g=4) and report reward or accuracy versus training steps; without this, 'stable GRPO training' remains untested.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 defines Running Steps as one token-generation round across all g active slots, i.e., g tokens generated per step. Under this definition, Table 1 is internally inconsistent: GSM8K/Qwen3-8B Dynamic-slot reports 360 steps with g=4, implying 1,440 token-slots, while 32 completions at average length 21 total only 672 tokens. If slots are not always full, then the metric is not the one defined. More importantly, Running Steps is not a valid throughput proxy even if measured consistently: the cost of a decode step depends on the number of active sequences and their KV-cache lengths, and continuous sampling plus SJF deliberately changes which sequences are batched together at each step. A step containing one long sequence is more expensive than one containing several short sequences, so a 25-45% reduction in steps does not imply a comparable wall-clock speedup. The paper reports no wall-clock time, no profiled cost of the BERT length predictor described in Section 4.2, and no scheduler overhead, yet the abstract's throughput claims ('over 25%', 'up to 45% reduction') are stated in terms of steps. Finally, the claimed 'stable GRPO training' is never tested: Section 5 contains decoding ablations only, with no reward, loss, or final accuracy from an actual GRPO run, so the stability half of the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Infinite Sampling, a decoding-side framework for GRPO training that decouples the sample group size G from GPU memory by (1) decomposing large groups into micro sampling groups that reuse a shared prompt KV cache, (2) interleaving token generation across micro groups via continuous sampling, and (3) using a length-aware scheduler that combines a BERT-based prefix-conditioned length predictor with a two-stage FPTAS/SJF scheduling scheme. The experiments on Qwen3-1.7B/8B across GSM8K, MATH, and KK report peak-memory reductions of over 50% versus full-group decoding and reductions in 'Running Steps' of up to 45% relative to naive micro group decoding, while preserving average sequence length. The paper also claims that this yields efficient and stable GRPO training under tight memory budgets.","tokens_in":20457,"tokens_out":5017,"duration_ms":59711,"significance":"If the memory and throughput claims hold, this is a practically valuable contribution: large-group GRPO is indeed memory-bound, and a decoding scheme that makes the memory footprint independent of group size would help practitioners train with larger groups on limited hardware. The paper credits itself with a concrete system design, clear ablations, and an oracle lower bound, which is useful framing. However, the current evidence is not sufficient to support the central claims. The memory measurements are the strongest part of the paper. The throughput claim is based exclusively on a step-count metric that is not a constant-cost proxy for wall-clock time, and the stability half of the central claim is never evaluated: there is no actual GRPO training run, no reward or loss curve, and no downstream accuracy result. The predictor evaluation in Figure 5 is isolated from the scheduling pipeline, and Algorithm 2's FPTAS claim is not accompanied by a proof or even a correct algorithmic description. The paper is well written and addresses an important problem, but the load-bearing empirical claims need substantial additional validation.","major_comments":[{"comment":"The central throughput claim rests on 'Running Steps' as a proxy for wall-clock time, but this metric is not a constant-cost unit. A decoding step with a long active sequence and a long KV cache can cost substantially more than a step with several short sequences, and the proposed SJF scheduler deliberately changes which sequences are batched together at each step. The paper reports no wall-clock time, no measured cost of BERT inference, prefix sampling, or SJF bookkeeping, and no end-to-end generation time. Therefore the abstract's claims of 'over 25%' and 'up to 45%' throughput improvement are not established by the data in Table 1.","section":"Section 5.1, Table 1"},{"comment":"Table 1 appears internally inconsistent under the paper's own definition of Running Steps. Section 5.1 defines one running step as one token-generation round across all g active decoding slots, i.e., g tokens per step. For GSM8K with Qwen3-8B and the default g=4, Dynamic-slot reports 360 steps, which would correspond to 1,440 token-slots; but 32 completions with average length 21 yield only 672 generated tokens. If slots are not always full, then the metric is not the one defined in Section 5.1, and the comparisons in Tables 1 and 2 are based on an inconsistent unit. The paper should either reconcile the definition, state the actual average number of active slots, or replace this metric with wall-clock time.","section":"Table 1, Section 5.1"},{"comment":"The claim that Infinite Sampling enables 'stable GRPO training' is not tested. Section 5 reports only decoding-side ablations: memory, running steps, and average sequence length. There is no actual GRPO training run, no reward or loss curve, no evaluation of final task accuracy, and no comparison of training stability against a baseline. The statement in Table 1's 'Stability vs. Efficiency' bullet is therefore an interpretation, not a result. The micro-batched reward and backpropagation design in Section 3.4 is described but never validated, so the stability half of the central claim is unsupported by the experiments.","section":"Section 5.2, Section 5.3, Section 3.4"},{"comment":"The length predictor is central to the scheduler, but its evaluation is incomplete. The predictor is fine-tuned on LMSYS-Chat and then applied to GSM8K, MATH, and KK, yet Figure 5 reports only prediction error as a function of prefix length k, with no baseline (e.g., a prompt-only predictor or no predictor), no error bars, no stated error metric, and no measurement of the overhead of BERT inference and prefix sampling during training. The paper's assertion that the predictor runs with 'negligible runtime overhead' is not measured. Without such data, it is unclear how much of the step reduction in Tables 1 and 2 transfers to real throughput, and whether the predictor transfers across the distribution shift from chat data to math reasoning tasks.","section":"Section 4.2, Figure 5"},{"comment":"Algorithm 2 is described as an FPTAS for micro-group assignment, but the algorithm is a greedy list-scheduling heuristic with scaled lengths, not an FPTAS. No approximation guarantee is derived, and the loop 'for each sample ... for n=1 to N ... break' can terminate without assigning all samples if no group has enough remaining capacity, with no fallback or feasibility check. The paper's statement that FPTAS 'ensures formal guarantees on group balance' is therefore not supported by the presented algorithm. This is not merely a terminology issue: the theoretical contribution claimed in Section 3.3 and the feasibility of the decoding plan depend on this algorithm. The authors should either provide a correct FPTAS with a proof, or relabel the method as a heuristic and verify its feasibility guarantees empirically.","section":"Algorithm 2, Section 3.3"}],"minor_comments":[{"comment":"The paper states that all metrics are averaged over all prompts, but Table 1 does not report standard deviations or the number of prompts considered; for a systems paper, this makes it hard to distinguish signal from noise.","section":"Section 5.1"},{"comment":"Figure 5 does not specify the error metric plotted on the vertical axis, and no error bars or repeated-run variation are shown; please add the definition and variability information.","section":"Figure 5"},{"comment":"References [15] and [16] are the same InstructGPT paper listed twice with slightly different formatting; one duplicate should be removed.","section":"References"},{"comment":"In Algorithm 3, the condition C = {i | i not in F and i not in mask} is confusing because, after Stage 1, every sample has a mask entry; please clarify whether 'mask' means 'already scheduled' or 'already completed', and define the refill candidate set accordingly.","section":"Algorithm 3"},{"comment":"The expansion 'FPTAS' is written as 'fixed-point approximation scheme' in the introduction, but the standard expansion is 'fully polynomial-time approximation scheme'; please correct this in the text.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"I believe the underlying idea is worthwhile and the memory numbers are credible, but the paper currently overclaims on both throughput and training stability. Before sending this back to the authors, I would ask the editor to require wall-clock measurements, a consistent efficiency metric, and at least one small end-to-end GRPO training comparison. The FPTAS label should also be corrected or supported with a real proof, since the current Algorithm 2 does not match the claimed theoretical property."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a systems paper that gets the memory story right but the throughput and stability story wrong. The micro sampling group idea is just sequential batching with KV cache reuse, and they've adapted continuous batching to single-prompt GRPO with a shared prefix cache. That's a legitimate and useful adaptation for memory-constrained RLHF; the 50%+ memory reduction figures are structurally consistent with the approach, and they show an oracle lower bound that their scheduler nearly hits. They also preserve average sequence length, which matters for GRPO.\n\nThe soft spots are concentrated where they matter most. The paper's headline claim is 'efficient and stable GRPO training', but no GRPO training run appears anywhere. There are no reward curves, no final accuracy, no policy gradient updates. 'Stable' is asserted, not shown. The throughput claim is measured in 'Running Steps', defined as one decode round across all g active slots. That metric is not a valid proxy for wall-clock: a step scheduling one long sequence costs more than a step batching several short ones, and the whole point of the SJF scheduler is to change which sequences are batched together. So a 25-45% step reduction does not imply a comparable speedup. They also never measure the BERT length predictor or the scheduler overhead.\n\nThere is also an internal inconsistency in Table 1. For GSM8K/Qwen3-8B Dynamic-slot, 360 steps at g=4 implies 1,440 token-slots, but average length 21 over 32 samples totals 672 tokens. Either the metric is not what they defined or the average is wrong. Minor, but it shakes confidence in the other numbers.\n\nThe single-prompt limitation (B=1) is acknowledged in the Conclusion but never analyzed; multi-prompt compatibility is left as future work, which is fine for a first system but means the scope is narrow.\n\nI'm not bothered by the novelty. It's an engineering combination of known techniques, and the FPTAS+SJF scheduling with a token-conditioned length predictor is a reasonable fit for GRPO decoding. Citation pattern is fine.\n\nBottom line: this deserves a serious referee, not a desk reject, because the memory-constrained decoding problem for large-group GRPO is real and the paper addresses it directly. But the referee should require wall-clock measurements, a real GRPO training run (even small), and corrected metrics before acceptance. I wouldn't cite it yet, but I'd read the revision.","headline":"Real memory savings for large-group GRPO decoding, but the throughput and stability claims are resting on a metric that doesn't survive scrutiny.","tokens_in":21029,"tokens_out":2646,"would_cite":false,"duration_ms":27765,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that GRPO training can scale to much larger sample groups on a fixed GPU by decomposing each group into memory-bounded micro groups, interleaving their decoding, and scheduling them by predicted length.","keywords":["GRPO","group reinforcement learning","KV cache","micro sampling groups","continuous sampling","length-aware scheduling","shortest-job-first","memory-efficient LLM training"],"falsifier":"Run the full pipeline on an A100 with Qwen3-8B, GSM8K prompts, group size 32 and micro group size 4, measuring wall-clock time end to end including the BERT length predictor and scheduling bookkeeping; if the wall-clock ratio of Infinite Sampling to naive micro groups is not close to 1769/3250, or if measured peak memory during the entire pipeline (not just decoding) exceeds the reported bound, the central claims are refuted. Separately, train GRPO with both methods and compare reward curves or held-out accuracy to test the stability claim.","tokens_in":19958,"feed_emoji":"⚡","tokens_out":7199,"duration_ms":74259,"temperature":0.7,"pith_summary":"Infinite Sampling argues that large-group GRPO training does not have to pay memory proportional to the group size. The paper proposes splitting each prompt's group of completions into memory-sized micro groups that share a single prompt KV cache, then interleaving token generation across those micro groups so decoding slots stay busy, and finally scheduling completions by predicted length to cut idle waiting. The reported result is that peak decoding memory drops by over 50% (21.55 GB to 10.64 GB on Qwen3-1.7B with group size 32), and decoding rounds drop by up to 45% at the same average sequence length. A sympathetic reader should care because this is a direct path to training with much larger GRPO groups on a fixed GPU budget, if the step-reduction metric survives a wall-clock check.","feed_headline":"Half the memory, 45 percent fewer decoding steps for group RL","feed_subtitle":"Split completions into shared-cache micro-groups and schedule by predicted length to fit bigger GRPO groups on one GPU.","key_machinery":"The load-bearing mechanism is a shared, bounded KV-cache pool: the prompt's prefill cache is computed once and reused by every completion, and a pool sized for g active sequences is recycled across micro groups. On top of that, continuous sampling interleaves token-level decoding across micro groups, and a length-aware scheduler first packs estimated-length samples into balanced groups with an FPTAS bin-packing approximation, then refills freed slots with the shortest remaining predicted job. A BERT-based predictor, conditioned on the first k decoded tokens concatenated to the prompt, supplies the length estimates that make the packing possible.","core_discovery":"On the paper's own terms, the discovery is that the KV cache, not the optimizer state, is the bottleneck that limits group size in GRPO, and that this bottleneck can be decoupled from the group size. The paper shows that decoding G completions in a fixed-size KV pool shared across sequential micro groups makes peak memory depend only on the micro group size g, and that a token-conditioned length predictor feeding a two-stage scheduler (FPTAS global grouping plus shortest-job-first slot refill) recovers most of the throughput lost to serialization. The central measurement is 'running steps,' the number of token-generation rounds across g active slots, which the paper reports as a proxy for decoding throughput. In the main experiments the full framework matches an oracle lower bound within about 1% while preserving the average sequence length of the naive micro-group baseline.","pith_inferences":["The same memory-bounded, length-aware scheduling could be dropped into other multi-sample decoding workloads, such as best-of-n sampling or self-consistency evaluation, where every completion shares a prompt.","The paper's stability claim is not yet evidenced: a direct GRPO training run comparing reward curves, KL, and downstream accuracy against full-group decoding would settle whether preserving average length preserves training quality.","Because the step metric ignores scheduler and predictor cost, the real speedup depends on these overheads staying under a few percent; a wall-clock measurement is the decisive next experiment.","An obvious extension is to layer KV-cache compression on top of the pool to shrink the micro group further, which would compound the memory savings rather than trade against them."],"forward_implications":["Group size becomes a memory-free hyperparameter: a user can set G large while peak decoding memory stays pinned to the micro group size.","On a fixed GPU, the framework supports GRPO runs that would otherwise be infeasible, at the cost of partial serialization.","Decoding rounds are reduced by up to 45% (3250 to 1770 on GSM8K) while average sequence length is unchanged, so fewer rounds do not come from truncated outputs.","The dynamic-slot variant is fastest but biases toward short sequences, so it is unsuitable for reward-based training despite its low step count.","The practical scheduler lands within about 1% of the oracle that knows true lengths before decoding."],"supporting_citations":[{"why":"supplies the GRPO objective and group-normalized advantage estimation that the framework makes scalable","marker":"[25]"},{"why":"provides the continuous-batching technique that continuous sampling adapts to single-prompt training-time decoding","marker":"[12]"},{"why":"provides the proxy-model sequence-length prediction approach extended to token-conditioned estimation","marker":"[18]"},{"why":"provides the encoder architecture used for the length regression head","marker":"[7]"},{"why":"supplies the training corpus for the length predictor","marker":"[32]"},{"why":"supplies the GSM8K task used to measure decoding steps and average length","marker":"[3]"},{"why":"supplies the MATH task used to measure decoding steps and average length","marker":"[13]"},{"why":"supplies the KK task used to measure decoding steps and average length","marker":"[30]"}],"fun_headline_variants":["Micro-groups halve GRPO memory, add 25% throughput","KV cache shared across micro-groups shrinks GRPO memory","Infinite Sampling: decouple group size from GPU memory","Length-aware scheduling fits bigger GRPO groups on one GPU","Predict lengths to schedule GRPO groups for efficient training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a 'running step' is a constant-cost unit of work and that equal average sequence length means equal GRPO sample quality, so the reported step and length numbers alone carry the efficiency and stability conclusions.","fun_headline_variants_meta":{"raw":{"variants":["Micro-groups halve GRPO memory, add 25% throughput","KV cache shared across micro-groups shrinks GRPO memory","Infinite Sampling: decouple group size from GPU memory","Length-aware scheduling fits bigger GRPO groups on one GPU","Predict lengths to schedule GRPO groups for efficient training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3532,"prompt_tokens":951,"completion_tokens":2581,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2507}},"tokens_in":567,"tokens_out":2581,"duration_ms":75717,"temperature":1.0,"reasoning_tokens":2507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:55:12.343861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on an A100 with Qwen3-8B, GSM8K prompts, group size 32 and micro group size 4, measuring wall-clock time end to end including the BERT length predictor and scheduling bookkeeping; if the wall-clock ratio of Infinite Sampling to naive micro groups is not close to 1769/3250, or if measured peak memory during the entire pipeline (not just decoding) exceeds the reported bound, the central claims are refuted. Separately, train GRPO with both methods and compare reward curves or held-out accuracy to test the stability claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the continuous-batching technique that continuous sampling adapts to single-prompt training-time decoding"}],"review_version":1}