{"id":"f793fb25-10bd-4eeb-9306-e6aed205c3ec","arxiv_id":"2607.13394","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Replacing the learned partition function in GFlowNet-style RL with an in-batch Monte Carlo estimate yields GFlowRL, a stable distribution-matching algorithm that beats GRPO/PPO/FlowRL on math, code, and red-teaming and scales to 235B MoE models.","lead":"A Microsoft Research team replaces the learned \"partition function\" used in GFlowNet-style RL for language models with a simple average computed from the rollout batch, removing an auxiliary network and its instability. The resulting algorithm, GFlowRL, beats PPO, GRPO, and the prior GFlowNet method FlowRL on math, code, and jailbreak benchmarks and trains stably on MoE models up to 235B parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The actual GFlowRL loss (Eq. 8) length-normalizes log-ratios while the fixed-point proof (Prop. B.1) omits length normalization; Remark B.4 admits the realized target is length-weighted unless within-group lengths are near-constant, an unmeasured assertion on which the distribution-matching claim re","rationale":"The reader's weakest-assumption diagnosis is exactly the gap I find most load-bearing: the theory (Prop. B.1) is stated for the unnormalized, unclipped population loss, while all experiments use length normalization. The clipping issue is less concerning because Remark B.3 correctly observes it is inactive at a zero-residual fixed point, but length normalization is present even at the fixed point and provably changes the stationary distribution. The paper's defense is an unquantified assertion about within-group length homogeneity. This is not an external-consensus disagreement; it is an internal consistency issue between the theoretical claim and the trained objective. Other weaknesses (single runs, no released code, 235B trained for only 30 steps) affect the strength of the empirical evidence but do not strike at the conceptual core the way the length-normalization gap does. The authors' honesty in Remark B.4 is a credit, but the missing measurement is precisely what must be supplied before the distribution-matching claim can be accepted. Since the reader's verdict is already CONDITIONAL and this concern reinforces it, no verdict adjustment is needed.","tokens_in":26811,"tokens_out":6188,"duration_ms":81070,"concrete_test":"Instrument the rollout logs from the Qwen2.5-7B math run (or a fresh short run with all G=16 rollouts saved per prompt) and compute per-prompt within-group response-length statistics: mean, standard deviation, coefficient of variation (CV = std/mean), and P90/P10 ratio. If the median within-group CV exceeds roughly 0.1, or a substantial fraction of prompts have P90/P10 > 1.2, then Remark B.4's 'vary only modestly' is contradicted. This single measurement settles whether the actual Eq. (8) fixed point is materially different from the intended p*(y|x), and thus whether the central distribution-matching claim applies to the trained model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GFlowRL preserves the reward-distribution-matching objective by replacing the learned partition function with an in-batch Monte Carlo estimate. Proposition B.1 characterizes the fixed point of the population loss without length normalization and with clipping inactive. However, every reported experiment uses Eq. (8), which divides each log-ratio by |y|. Unlike clipping (Remark B.3), length normalization is active at the fixed point: Remark B.4 explicitly derives the stationary distribution as pi_theta(y|x) proportional to pi_ref(y|x) exp(|y|(beta r(x,y) - Zt(x))), which equals the intended p*(y|x) only when all responses in a rollout group have the same length. The paper asserts in Remark B.4 that 'response lengths vary only modestly within rollout groups on reasoning benchmarks' but provides no measurement of within-group length variation. If lengths vary materially, the trained policy optimizes a length-weighted reward distribution rather than the claimed softmax target. This gap is internal to the paper's theory-vs-method consistency: the proof is correct for Eq. (9), but Eq. (8) is what is trained. The empirical gains could still hold, but the central 'preserving distribution-matching' narrative would not be supported for the actual algorithm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GFlowRL, a GFlowNet-style RL algorithm for LLM post-training that replaces the learned prompt-conditional partition function Z_phi with an in-batch Monte Carlo estimate Z_t computed from the rollout group already sampled for policy gradient training. The algorithm adds two stabilizers, an importance-sampling weight and an asymmetric flow-gap clip, and is claimed to preserve the reward-distribution-matching objective of trajectory balance while removing the auxiliary partition network and its optimizer/synchronization overhead. Empirical results are reported for dense models (Qwen2.5-7B/32B, DeepSeek-R1-Distill-Qwen-7B/14B) on math, code, and adversarial red-teaming benchmarks, and for MoE models up to Qwen3-235B-A22B, where the paper reports stable training and gains over GRPO/PPO/FlowRL, including a Codeforces rating of 2048 at 14B.","tokens_in":27082,"tokens_out":6513,"duration_ms":79104,"significance":"If the central claim holds, this is a practically significant contribution: it would be the first GFlowNet-style RL method to scale stably to large MoE models, while simplifying the training infrastructure to be GRPO-compatible. The paper has real strengths: a clear diagnostic for the learned partition function, gradient-norm statistics, a synthetic distribution-matching benchmark, extensive benchmark tables at multiple scales, and qualitative Codeforces examples. However, the theoretical guarantee in Proposition B.1 is proved for a population, unnormalized, unclipped loss, whereas the deployed loss in Eq. (8) is finite-batch, length-normalized, and clipped; Remark B.4 concedes a length-dependent fixed-point distortion. Because the distribution-matching claim is the paper's central narrative, this gap must be addressed before the result can be regarded as supported.","major_comments":[{"comment":"The central claim that GFlowRL 'preserves the reward-distribution-matching objective' is proved only for the population, unnormalized, unclipped loss in Eq. (9), while every experiment uses Eq. (8), which divides the log-ratio terms by |y|. Remark B.4 explicitly derives the resulting fixed point as pi_theta(y|x) proportional to pi_ref(y|x) exp(|y|(beta r(x,y) - Z_t(x))), which equals the claimed p*(y|x) only when all responses in a rollout group have equal length. The paper asserts without measurement that 'response lengths vary only modestly within rollout groups on reasoning benchmarks.' This is load-bearing: if within-group length variation is non-negligible, the trained policy optimizes a length-weighted distribution, not the softmax target. Please report within-group length statistics for the actual benchmark groups, quantify the induced fixed-point distortion, or modify the estimat","section":"Appendix B, Prop. B.1 / Remark B.4, and Eq. (8)"},{"comment":"The diagnostic claim that 'replacing Z_phi with random noise yields comparable performance' is contradicted by the paper's own 32B results. In Table 2 (7B), FlowRL-RandomLogZ averages 36.19 vs. FlowRL 35.63, supporting the claim; but in Table 15 (32B), FlowRL-RandomLogZ averages 44.73 vs. FlowRL 48.39, with large drops on AIME25, Minerva, and Olympiad. This inconsistency weakens the motivation that the learned partition function 'contributes little.' Please reconcile the two scales or temper the claim.","section":"Sec. 3.1 and Table 15"},{"comment":"The text states that for Qwen3-235B-A22B, GFlowRL 'is trained for only 30 steps vs. 100 for GRPO,' but Table 13 lists a training schedule '[0,100]' for that model and for Qwen3-30B-A3B. Please clarify the actual number of update steps and the schedule range. If the 235B run is only 30 steps, the claim that 'the recipe transfers' rests on a very short run; provide training curves or early-plateau evidence to support the scaling conclusion.","section":"Sec. 4.3 and Table 13"},{"comment":"The proof of Proposition B.1 imposes normalization over all y in Step 2, but the zero-loss condition is only established for y in the support of pi_theta*. If the target p* has support outside the current policy's support, equality on the support does not imply equality as distributions over Y. Please state the support assumption explicitly (e.g., full support of the autoregressive policy) or revise the proof to show that the fixed point is p* on the relevant support.","section":"Appendix B, Prop. B.1 proof"}],"minor_comments":[{"comment":"Notation is inconsistent: Eq. (5) uses pi_theta_ref while Eq. (6) uses pi_ref for the same reference policy. Please unify.","section":"Eqs. (5) and (6)"},{"comment":"The text reports the learned partition function result as 35.61, while Table 2 lists FlowRL as 35.63. Also, the claim that random replacement 'slightly improves' should be qualified by the 32B results in Table 15, where random replacement is worse.","section":"Sec. 3.1 and Table 2"},{"comment":"The hyperparameter table lists 'Clip ratio epsilon_low 0.2, clip ratio epsilon_high 0.28' alongside an 'IS threshold 2.0.' It is unclear whether the clip ratios refer to the flow-gap clip in Eq. (7) or to a policy clip ratio. Please clarify the naming and distinguish the two clipping mechanisms.","section":"Table 9"},{"comment":"The qualitative comparison states 'All evaluations use Pass@16 with temperature 0.6 and top-p=0.95,' but the main code tables report both Pass@16 and Avg@16. Please specify which sampling parameters apply to which table and metric.","section":"Appendix I"},{"comment":"The reference list contains a typo: 'HugginfaceH4' should be 'HuggingFaceH4.'","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper reports an impressive and unusually broad empirical evaluation, including 235B MoE scaling, and the proposed method appears practically promising. However, the theory-vs-method gap around length normalization, the internal contradiction in the random-logZ diagnostic between 7B and 32B, and the inconsistent reporting of the 235B training steps all need to be resolved before publication. The central idea is defensible, but the distribution-matching claim is not currently supported for the actual trained loss."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: GFlowRL replaces the learned partition function in trajectory-balance RL with an in-batch Monte Carlo estimate from the rollout group, and shows this trains stably on dense and MoE models up to 235B. That is a genuinely useful engineering result, and the paper is refreshingly candid about its own caveats. You should read it if you work on RL post-training for reasoning models.\n\nWhat is actually new: the diagnosis that a freshly initialized Z_phi is mostly noise in the short-horizon LLM regime, the estimator itself, and the derived fixed-point characterization. The empirical scope is real: math, code, red-teaming, dense and sparse architectures, with FlowRL failing to converge in several regimes where GFlowRL succeeds. The paper also flags its own limitations—Remark B.4, Appendix A—which is more than most do.\n\nThe soft spots are real but proportionate. The big one: Proposition B.1 is proved for the population loss without length normalization and with clipping inactive, but every experiment uses Eq. (8), which length-normalizes the log-ratios. Remark B.4 admits this changes the stationary distribution to a length-weighted variant, and the paper asserts without measuring that within-group lengths vary only modestly. That assertion is load-bearing for the distribution-matching narrative. The empirical gains might survive even if lengths vary a lot, but the paper does not currently support the claim that the trained policy matches the intended softmax target. This is the main thing a referee should push on.\n\nSmaller issues: the headline comparisons are single runs without error bars; beta and clip thresholds are tuned on the target benchmarks; the 235B run uses 30 steps versus GRPO's 100; and there is a numeric inconsistency in F.2 (text says 49.40, table says 50.42). Code is promised but not released, and the FlowRL failure claims rely on the authors' own prior work.\n\nNone of this sinks the paper. The core recipe—drop the learned partition network, use a group-based baseline, clip the flow gap—is plausible and has enough empirical support to justify a serious referee. Who this is for: people building practical RL pipelines for reasoning LLMs, and GFlowNet researchers interested in when the partition function is actually necessary. I would send it to review, but require code release, error bars, and either a theory that covers the length-normalized loss or a measurement of within-group length variation.","headline":"A serious, mostly well-executed scaling recipe for GFlowNet-style RL, but the paper's own theory does not cover the length-normalized loss actually trained, so the distribution-matching claim is weaker than it looks.","tokens_in":27690,"tokens_out":1557,"would_cite":true,"duration_ms":19713,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The learned partition function in GFlowNet-style RL can be replaced by a Monte Carlo estimate from the existing rollout batch, preserving the reward-proportional fixed point while stabilizing training.","keywords":["GFlowNet-style RL","trajectory balance","partition function estimation","distribution-matching reinforcement learning","LLM reasoning","mixture-of-experts","policy gradient stability","adversarial red-teaming"],"falsifier":"Run a trained GFlowRL model on prompts with verifiable rewards and compare the empirical response distribution to π_ref(y|x) exp(β r(x,y))/Z(x). If long responses are sampled more often than that target predicts, and the discrepancy tracks exp(|y| β r), the length-normalized loss is not reproducing the claimed fixed point. A simpler check: compute within-group response-length variance on the training data; the theory predicts the distortion grows with it.","tokens_in":26597,"feed_emoji":"🎯","tokens_out":6792,"duration_ms":69264,"temperature":0.7,"pith_summary":"This paper tries to establish that the learned per-prompt normalizer known as the partition function, long treated as essential to trajectory-balance training, is actually the main source of instability when GFlowNet-style RL is scaled to LLMs. It proposes replacing that network with an in-batch Monte Carlo estimate computed from the same group of rollouts already sampled for each prompt. If the paper is right, distribution-matching RL—which samples reasoning paths in proportion to reward rather than maximizing it—becomes stable enough to train dense and mixture-of-experts models at scale, with no extra network, optimizer state, or synchronization overhead. The authors prove the simplified objective retains the trajectory-balance fixed point, where the policy is proportional to the reference policy times an exponentially weighted reward, and report consistent gains over reward-maximizing baselines across math, code, and adversarial red-teaming.","feed_headline":"One rollout batch replaces the learned partition function in LLM RL","feed_subtitle":"Reward-matching RL keeps its fixed point while training stably on dense and MoE LLMs up to 235B parameters.","key_machinery":"The load-bearing object is the in-batch Monte Carlo estimator Zt(x), which replaces the learned partition network. It is the average over the G rollouts of βr + log π_ref − log π_old; because it enters the loss only through a stop-gradient operator, it acts as a centering baseline with no parameters and no gradient. The other machinery is the flow-gap clip: the per-rollout residual is split into a flow gap evaluated at the rollout policy plus a policy-update term, and the gap is clipped asymmetrically so positive corrections have more room than negative ones.","core_discovery":"The central claim is that log Z(x), the per-prompt normalizer that trajectory-balance objectives learn as an auxiliary network, can be replaced by Zt(x): the mean over a prompt's rollout group of β r(x,y) + log π_ref(y|x) − log π_old(y|x), inserted as a stop-gradient baseline. With this substitution, a zero-loss self-consistent fixed point of the GFlowRL loss satisfies π_θ(y|x) ∝ π_ref(y|x) exp(β r(x,y)), the same target as trajectory balance, so no distribution-matching fidelity is lost. Two stabilizers—importance-sampling weights that correct rollout/trainer drift and asymmetric clipping of the flow gap—keep outlier rollouts from destabilizing updates. The paper reports that this design tr","pith_inferences":["Not tested in the paper: if the learned partition term is as useless as the random-replacement diagnostic suggests, then a constant or group-mean baseline might capture most of the method's benefit; an ablation that replaces Zt(x) with the group mean of rewards would separate the estimator's centering effect from its distribution-matching effect.","The proof's dependence on equal response lengths makes a concrete prediction: on tasks with high within-group length variance, the trained policy should be length-tilted (weighted by exp(|y| β r)); measuring this tilt would tell whether the length-normalized loss is actually matching the claimed target.","The estimator's variance is O(1/G), so reducing the group size G should degrade stability and accuracy; this is a testable scaling law for deployment in low-throughput settings.","The same estimator-centric design could generalize to agentic or multimodal RL, where rollouts are more heterogeneous and rewards noisier, but the paper does not claim that and leaves it open."],"forward_implications":["GFlowNet-style RL can run on the same infrastructure as standard group-based policy optimization; the auxiliary partition network and its optimizer state disappear.","At convergence the trained policy should sample responses with probability proportional to the reference-policy likelihood times exp(β r), so multiple high-reward reasoning paths are retained instead of collapsing to one mode.","Gradient norms during training stay at the same scale as reward-maximizing baselines, so the divergence pattern seen in prior distribution-matching methods is avoidable.","The recipe transfers unchanged from dense models to MoE models up to 235B parameters, opening distribution-matching RL to sparse architectures."],"fun_headline_variants":["Drop learned partition: GFlowRL scales reward-matching to 235B","In-batch MC estimate replaces partition net in GFlowNet RL","GFlowRL: kills partition net, hits 235B MoE stably","No more learned Z: GFlowRL matches rewards at 235B scale","First GFlowNet-style RL stable on dense and MoE LLMs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The fixed-point proof is stated for the loss without length normalization and with clipping inactive, while the actual loss divides every term by response length; the load-bearing premise is that response lengths within a rollout group are nearly equal, otherwise the policy converges to a length-weighted distribution different from the intended reward-proportional target.","fun_headline_variants_meta":{"raw":{"variants":["Drop learned partition: GFlowRL scales reward-matching to 235B","In-batch MC estimate replaces partition net in GFlowNet RL","GFlowRL: kills partition net, hits 235B MoE stably","No more learned Z: GFlowRL matches rewards at 235B scale","First GFlowNet-style RL stable on dense and MoE LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000339,"raw_usage":{"total_tokens":1778,"prompt_tokens":886,"completion_tokens":892,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":793}},"tokens_in":630,"tokens_out":892,"duration_ms":9108,"temperature":1.0,"reasoning_tokens":793,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T05:18:48.965332+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a trained GFlowRL model on prompts with verifiable rewards and compare the empirical response distribution to π_ref(y|x) exp(β r(x,y))/Z(x). If long responses are sampled more often than that target predicts, and the discrepancy tracks exp(|y| β r), the length-normalized loss is not reproducing the claimed fixed point. A simpler check: compute within-group response-length variance on the training data; the theory predicts the distortion grows with it.","supporting_citations":[],"review_version":1}