Pith. sign in

REVIEW 4 major objections 5 minor 30 references

The learned partition function in GFlowNet-style RL can be replaced by a Monte Carlo estimate from the existing rollout batch, preserving the reward-proportional fixed point while stabilizing training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:18 UTC pith:E7GJIDC2

load-bearing objection A serious, mostly well-executed scaling recipe for GFlowNet-style RL, but the paper's own theory does not cover the length-normalized loss actually trained, so the distribution-matching claim is weaker than it looks. the 4 major comments →

arxiv 2607.13394 v1 pith:E7GJIDC2 submitted 2026-07-15 cs.CL cs.LG

GFlowRL: Scaling Distribution-Matching RL to Large Language Models

Xiaodong Liu , Michael Xu , Jack W. Stokes , Paul Smolensky , Doug Burger , Jianfeng Gao This is my paper
classification cs.CL cs.LG
keywords GFlowNet-style RLtrajectory balancepartition function estimationdistribution-matching reinforcement learningLLM reasoningmixture-of-expertspolicy gradient stabilityadversarial red-teaming
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the learned per-prompt normalizer known as the partition function, long treated as essential to trajectory-balance training, is actually the main source of instability when GFlowNet-style RL is scaled to LLMs. It proposes replacing that network with an in-batch Monte Carlo estimate computed from the same group of rollouts already sampled for each prompt. If the paper is right, distribution-matching RL—which samples reasoning paths in proportion to reward rather than maximizing it—becomes stable enough to train dense and mixture-of-experts models at scale, with no extra network, optimizer state, or synchronization overhead. The authors prove the simplified objective retains the trajectory-balance fixed point, where the policy is proportional to the reference policy times an exponentially weighted reward, and report consistent gains over reward-maximizing baselines across math, code, and adversarial red-teaming.

Core claim

The central claim is that log Z(x), the per-prompt normalizer that trajectory-balance objectives learn as an auxiliary network, can be replaced by Zt(x): the mean over a prompt's rollout group of β r(x,y) + log π_ref(y|x) − log π_old(y|x), inserted as a stop-gradient baseline. With this substitution, a zero-loss self-consistent fixed point of the GFlowRL loss satisfies π_θ(y|x) ∝ π_ref(y|x) exp(β r(x,y)), the same target as trajectory balance, so no distribution-matching fidelity is lost. Two stabilizers—importance-sampling weights that correct rollout/trainer drift and asymmetric clipping of the flow gap—keep outlier rollouts from destabilizing updates. The paper reports that this design tr

What carries the argument

The load-bearing object is the in-batch Monte Carlo estimator Zt(x), which replaces the learned partition network. It is the average over the G rollouts of βr + log π_ref − log π_old; because it enters the loss only through a stop-gradient operator, it acts as a centering baseline with no parameters and no gradient. The other machinery is the flow-gap clip: the per-rollout residual is split into a flow gap evaluated at the rollout policy plus a policy-update term, and the gap is clipped asymmetrically so positive corrections have more room than negative ones.

Load-bearing premise

The fixed-point proof is stated for the loss without length normalization and with clipping inactive, while the actual loss divides every term by response length; the load-bearing premise is that response lengths within a rollout group are nearly equal, otherwise the policy converges to a length-weighted distribution different from the intended reward-proportional target.

What would settle it

Run a trained GFlowRL model on prompts with verifiable rewards and compare the empirical response distribution to π_ref(y|x) exp(β r(x,y))/Z(x). If long responses are sampled more often than that target predicts, and the discrepancy tracks exp(|y| β r), the length-normalized loss is not reproducing the claimed fixed point. A simpler check: compute within-group response-length variance on the training data; the theory predicts the distortion grows with it.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • GFlowNet-style RL can run on the same infrastructure as standard group-based policy optimization; the auxiliary partition network and its optimizer state disappear.
  • At convergence the trained policy should sample responses with probability proportional to the reference-policy likelihood times exp(β r), so multiple high-reward reasoning paths are retained instead of collapsing to one mode.
  • Gradient norms during training stay at the same scale as reward-maximizing baselines, so the divergence pattern seen in prior distribution-matching methods is avoidable.
  • The recipe transfers unchanged from dense models to MoE models up to 235B parameters, opening distribution-matching RL to sparse architectures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: if the learned partition term is as useless as the random-replacement diagnostic suggests, then a constant or group-mean baseline might capture most of the method's benefit; an ablation that replaces Zt(x) with the group mean of rewards would separate the estimator's centering effect from its distribution-matching effect.
  • The proof's dependence on equal response lengths makes a concrete prediction: on tasks with high within-group length variance, the trained policy should be length-tilted (weighted by exp(|y| β r)); measuring this tilt would tell whether the length-normalized loss is actually matching the claimed target.
  • The estimator's variance is O(1/G), so reducing the group size G should degrade stability and accuracy; this is a testable scaling law for deployment in low-throughput settings.
  • The same estimator-centric design could generalize to agentic or multimodal RL, where rollouts are more heterogeneous and rewards noisier, but the paper does not claim that and leaves it open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GFlowRL, a GFlowNet-style RL algorithm for LLM post-training that replaces the learned prompt-conditional partition function Z_phi with an in-batch Monte Carlo estimate Z_t computed from the rollout group already sampled for policy gradient training. The algorithm adds two stabilizers, an importance-sampling weight and an asymmetric flow-gap clip, and is claimed to preserve the reward-distribution-matching objective of trajectory balance while removing the auxiliary partition network and its optimizer/synchronization overhead. Empirical results are reported for dense models (Qwen2.5-7B/32B, DeepSeek-R1-Distill-Qwen-7B/14B) on math, code, and adversarial red-teaming benchmarks, and for MoE models up to Qwen3-235B-A22B, where the paper reports stable training and gains over GRPO/PPO/FlowRL, including a Codeforces rating of 2048 at 14B.

Significance. If the central claim holds, this is a practically significant contribution: it would be the first GFlowNet-style RL method to scale stably to large MoE models, while simplifying the training infrastructure to be GRPO-compatible. The paper has real strengths: a clear diagnostic for the learned partition function, gradient-norm statistics, a synthetic distribution-matching benchmark, extensive benchmark tables at multiple scales, and qualitative Codeforces examples. However, the theoretical guarantee in Proposition B.1 is proved for a population, unnormalized, unclipped loss, whereas the deployed loss in Eq. (8) is finite-batch, length-normalized, and clipped; Remark B.4 concedes a length-dependent fixed-point distortion. Because the distribution-matching claim is the paper's central narrative, this gap must be addressed before the result can be regarded as supported.

major comments (4)
  1. [Appendix B, Prop. B.1 / Remark B.4, and Eq. (8)] The central claim that GFlowRL 'preserves the reward-distribution-matching objective' is proved only for the population, unnormalized, unclipped loss in Eq. (9), while every experiment uses Eq. (8), which divides the log-ratio terms by |y|. Remark B.4 explicitly derives the resulting fixed point as pi_theta(y|x) proportional to pi_ref(y|x) exp(|y|(beta r(x,y) - Z_t(x))), which equals the claimed p*(y|x) only when all responses in a rollout group have equal length. The paper asserts without measurement that 'response lengths vary only modestly within rollout groups on reasoning benchmarks.' This is load-bearing: if within-group length variation is non-negligible, the trained policy optimizes a length-weighted distribution, not the softmax target. Please report within-group length statistics for the actual benchmark groups, quantify the induced fixed-point distortion, or modify the estimat
  2. [Sec. 3.1 and Table 15] The diagnostic claim that 'replacing Z_phi with random noise yields comparable performance' is contradicted by the paper's own 32B results. In Table 2 (7B), FlowRL-RandomLogZ averages 36.19 vs. FlowRL 35.63, supporting the claim; but in Table 15 (32B), FlowRL-RandomLogZ averages 44.73 vs. FlowRL 48.39, with large drops on AIME25, Minerva, and Olympiad. This inconsistency weakens the motivation that the learned partition function 'contributes little.' Please reconcile the two scales or temper the claim.
  3. [Sec. 4.3 and Table 13] The text states that for Qwen3-235B-A22B, GFlowRL 'is trained for only 30 steps vs. 100 for GRPO,' but Table 13 lists a training schedule '[0,100]' for that model and for Qwen3-30B-A3B. Please clarify the actual number of update steps and the schedule range. If the 235B run is only 30 steps, the claim that 'the recipe transfers' rests on a very short run; provide training curves or early-plateau evidence to support the scaling conclusion.
  4. [Appendix B, Prop. B.1 proof] The proof of Proposition B.1 imposes normalization over all y in Step 2, but the zero-loss condition is only established for y in the support of pi_theta*. If the target p* has support outside the current policy's support, equality on the support does not imply equality as distributions over Y. Please state the support assumption explicitly (e.g., full support of the autoregressive policy) or revise the proof to show that the fixed point is p* on the relevant support.
minor comments (5)
  1. [Eqs. (5) and (6)] Notation is inconsistent: Eq. (5) uses pi_theta_ref while Eq. (6) uses pi_ref for the same reference policy. Please unify.
  2. [Sec. 3.1 and Table 2] The text reports the learned partition function result as 35.61, while Table 2 lists FlowRL as 35.63. Also, the claim that random replacement 'slightly improves' should be qualified by the 32B results in Table 15, where random replacement is worse.
  3. [Table 9] The hyperparameter table lists 'Clip ratio epsilon_low 0.2, clip ratio epsilon_high 0.28' alongside an 'IS threshold 2.0.' It is unclear whether the clip ratios refer to the flow-gap clip in Eq. (7) or to a policy clip ratio. Please clarify the naming and distinguish the two clipping mechanisms.
  4. [Appendix I] The qualitative comparison states 'All evaluations use Pass@16 with temperature 0.6 and top-p=0.95,' but the main code tables report both Pass@16 and Avg@16. Please specify which sampling parameters apply to which table and metric.
  5. [References] The reference list contains a typo: 'HugginfaceH4' should be 'HuggingFaceH4.'

Circularity Check

0 steps flagged

No significant circularity: GFlowRL's fixed-point claim is derived from a stated objective, and the main empirical results are external and falsifiable.

full rationale

The derivation chain is self-contained. Eq. (4) defines Z_t(x) as an in-batch mean of per-sample trajectory-balance targets; Proposition B.1 then derives the fixed point of the population loss (Eq. 9) from the zero-residual condition, and Step 3 verifies that the definition of Z_t is consistent with log Z. This is a fixed-point characterization of a stated objective, not a fitted parameter relabeled as a prediction; the estimator is not tuned to any benchmark. The empirical claims (Tables 2–5) are tested against external math, code, and safety benchmarks, and against FlowRL/GRPO/PPO baselines, so they are externally falsifiable. The chief self-citations (FlowRL, SEMA) are prior work used as baselines; the load-bearing instability claim is supported by in-paper diagnostics (random-logZ ablation, gradient norm statistics), not by those citations alone. The main acknowledged gap — Remark B.4's unmeasured assertion that response lengths vary only modestly within rollout groups — is a limitation of the theoretical claim (the actual Eq. 8 length-normalizes and therefore has a length-weighted fixed point unless lengths are equal), but this is a correctness/assumption issue, not circularity. No circular step is identified.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical or architectural entities are postulated: Z_t(x) is a batch statistic (a mean of per-sample TB targets), not a new parameter or latent. The free parameters are the temperature beta, the clip bounds, the IS constants, and the length normalization, all chosen by hand or small sweeps; the axioms are mostly standard GFlowNet background plus two domain assumptions (length uniformity, expressivity) that are asserted rather than verified.

free parameters (4)
  • beta (inverse temperature) = 8
    Sharpness of the target distribution in Eqs. 2-4; selected by sweeping beta in {1,5,8,10,15} on the six math test benchmarks, peaking at 38.2 at beta=8 (Table 6a). The scale of Z_t and the whole objective depend on it.
  • flow-gap clip bounds (eps_low, eps_high) = 0.2, 0.28
    Asymmetric clip in Eq. 7; hand-set, justified heuristically as allowing faster upward corrections on under-sampled correct solutions.
  • importance-sampling cap (IS threshold 2.0; RS thresholds 1.01/0.99) = 2.0; 1.01/0.99
    Caps w(i) = min(pi_theta/pi_old, 1+eps) and ratio re-scaling; hand-set stabilizer constants in Table 9.
  • length-normalization divisor 1/|y|
    Chosen to prevent long sequences from dominating the loss; directly causes the fixed-point distortion admitted in Remark B.4.
axioms (5)
  • standard math Trajectory-balance optimum characterization (Malkin et al. 2022)
    Invoked in Eq. 3 and Prop. B.1: zero TB loss implies pi_theta proportional to pi_ref exp(beta r).
  • domain assumption Policy class expressivity: p*(y|x) proportional to pi_ref(y|x) exp(beta r(x,y)) is representable
    Explicit assumption of Prop. B.1; unproven for the actual LLM policies used in the experiments.
  • domain assumption Response lengths vary only modestly within rollout groups on reasoning benchmarks
    Remark B.4; load-bearing for the claim that length normalization does not materially distort the distribution-matching target; asserted, not measured.
  • ad hoc to paper Flow-gap clip is inactive in a neighborhood of the optimum
    Remark B.3 presumes the optimization reaches the clip-inactive region; the paper admits updates away from the fixed point are biased by design, and Proposition B.1 is only a characterization, not a convergence guarantee.
  • domain assumption Reward verifier r(x,y) is a faithful oracle
    Binary correctness, unit-test pass rates, and jailbreak-classifier outputs are treated as ground-truth rewards (Sec. 2, Eqs. 1-2).

pith-pipeline@v1.3.0-alltime-deepseek · 26314 in / 21181 out tokens · 196835 ms · 2026-08-02T05:18:48.965332+00:00 · methodology

0 comments
read the original abstract

Generative Flow Networks (GFlowNets) offer a promising alternative to reward-maximizing reinforcement learning (RL) for large reasoning models, encouraging diverse reasoning paths by matching reward distributions rather than collapsing to dominant modes. Recent work shows promise on math and code, but scaling GFlowNet-style RL to modern post-training pipelines remains difficult: as model size, rollout horizon, reward noise, and distributed-systems complexity grow together, a learned prompt-conditional partition function becomes a source of gradient instability and engineering overhead rather than a useful normalizer. Through systematic analysis, we find that the learned partition function, previously treated as essential, can be replaced by an in-batch Monte Carlo estimate computed from the rollout group already required for training. We propose GFlowRL, a streamlined GFlowNet-style RL algorithm that removes the auxiliary partition network entirely while preserving the reward-distribution-matching objective, completed by two stabilizers: importance-sampling correction for rollout/trainer drift and asymmetric flow-gap clipping for outlier residuals. GFlowRL exceeds all counterparts on math, code, and adversarial red-teaming benchmarks, reaching a Codeforces rating of 2048 at the 14B scale (within 25 Elo of o3-mini) and attaining the highest average ASR@1 on AdvBench and HarmBench, outperforming the previous SOTA multi-turn attacker in a regime where FlowRL, a prior GFlowNet-style method, diverges. The same recipe transfers to all evaluated MoE configurations up to 235B parameters, where FlowRL again fails to converge. To our knowledge, GFlowRL is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures. Code will be at: https://github.com/microsoft/gflowrl

Figures

Figures reproduced from arXiv: 2607.13394 by Doug Burger, Jack W. Stokes, Jianfeng Gao, Michael Xu, Paul Smolensky, XiaoDong Liu.

Figure 1
Figure 1. Figure 1: Comparison of distribution matching across GRPO, FlowRL-RandomLogZ, FlowRL, and [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Codeforces rating comparison across Dense and Sparse MoE model families. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Gradient norm over training steps on Qwen2.5-7B math reasoning tasks. FlowRL suffers [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The course grained diversity scores are computed across the 16 solutions for each problem [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Prompt used for the LLM-as-judge diversity evaluation. [PITH_FULL_IMAGE:figures/full_fig_p029_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of distribution matching across GRPO, FlowRL-RandomLogZ, FlowRL, and [PITH_FULL_IMAGE:figures/full_fig_p030_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of distribution matching across GRPO, FlowRL-RandomLogZ, FlowRL, and [PITH_FULL_IMAGE:figures/full_fig_p030_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of distribution matching across GRPO, FlowRL-RandomLogZ, FlowRL, and [PITH_FULL_IMAGE:figures/full_fig_p031_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of distribution matching across GRPO, FlowRL-RandomLogZ, FlowRL, and [PITH_FULL_IMAGE:figures/full_fig_p031_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 8 linked inside Pith

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen

    URL https://openreview.net /forum?id=Ouj6p4ca60. Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: Stabilizing critic-free policy optimization with global advantage normalization.arXiv preprint arXiv:2501.03262,

    Pith/arXiv arXiv

  3. [5]

    Math-500

    HugginfaceH4. Math-500. URL https://huggingface.co/datasets/HuggingFaceH4/MATH -500. Naman Jain, Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu, e...

    2025

  4. [6]

    Koray Kavukcuoglu

    URL https://proceedings.iclr.cc/paper_files/paper/2025/file/94074dd5a072 d28ff75a76dabed43767-Paper-Conference.pdf. Koray Kavukcuoglu. Gemini 2.5: Our most intelligent AI model,

    2025

  5. [7]

    Google Blog (The Keyword), Published Mar

    URL https://blog.goo gle/technology/google-deepmind/gemini-model-thinking-updates-march-2025/ . Google Blog (The Keyword), Published Mar. 25,

    2025

  6. [8]

    Kimi k1.5: Scaling reinforcement learning with llms

    Kimi-Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599,

    Pith/arXiv arXiv

  7. [11]

    Automated red teaming with GOAT: the generative offensive agent tester

    Maya Pavlova, Erik Brinkman, Krithika Iyer, Vítor Albiero, Joanna Bitton, Hailey Nguyen, Cris- tian Canton Ferrer, Ivan Evtimov, and Aaron Grattafiori. Automated red teaming with GOAT: the generative offensive agent tester. InICLR 2025 Workshop on Building Trust in Language Models and Applications,

    2025

  8. [12]

    URLhttps://aclanthology.org/2025.acl-long.1207/

    Association for Computational Linguistics. URLhttps://aclanthology.org/2025.acl-long.1207/. Lorenz Richter, Ayman Boustati, Nikolas Nüsken, Francisco Ruiz, and Omer Deniz Akyildiz. Vargrad: A low-variance gradient estimator for variational inference. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information ...

    2025

  9. [13]

    D.6 Diagnostic and Synthetic Setup Random-logZ replacement.For the diagnostic in Sec. 4.2, we replace logZ ϕ(x) at every training step with an independent sample logZ∼ N(µ, σ2), where (µ, σ2) are matched to the empirical distribution of logZ ϕ(x) observed under standard FlowRL training [Zhu et al., 2026]. We use µ= 0.5 and σ= 1 . All other settings (optim...

    2026

  10. [14]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    USENIX Association. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  11. [16]

    Thinking Machines

    URLhttps://arxiv.org/abs/2410.18469. Thinking Machines. Defeating nondeterminism in llm inference. https://thinkingmachines.a i/blog/defeating-nondeterminism-in-llm-inference/,

    arXiv

  12. [17]

    URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ebd6d2f5d60ff 9afaeda1a81fc53e2d0-Paper.pdf. Siddarth Venkatraman, Minsu Kim, Luke Rowe, Moksh Jain, Marcin Sendera, Sarthak Mittal, Luca Scimeca, Mohsin Hasan, Pablo Lemos, Emmanuel Bengio, Alexandre Adam, Jarrid Rector-Brooks, Yoshua Bengio, Glen Berseth, and Nikolay Malkin. Amortizing intra...

    2017

  13. [19]

    Ronald J Williams

    URL https://aclanthology.org/2025.emnlp-main.100/. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine Learning, 8(3):229–256,

    2025

  14. [20]

    13 Hao Yang, Lizhen Qu, Ehsan Shareghi, and Gholamreza Haffari

    URLhttps://www.pnas.org/doi/abs/10.1073/pnas.2504966122. 13 Hao Yang, Lizhen Qu, Ehsan Shareghi, and Gholamreza Haffari. Jigsaw puzzles: Splitting harmful questions to jailbreak large language models, 2024a. URL https://arxiv.org/abs/2410.1

    doi:10.1073/pnas.2504966122

  15. [22]

    Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin

    Work in progress. Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Training LLMs for divergent reasoning with minimal examples. InForty-second International Conference on Machine Learning, 2025a. URLhttps://openreview.net/forum?id=qyMxunrR2j. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan D...

    2025

  16. [23]

    mlr.press/v162/zhang22v.html

    URL https://proceedings. mlr.press/v162/zhang22v.html. Yifan Zhang and Team Math-AI. Minerva math. URL https://huggingface.co/datasets/ma th-ai/minervamath. Yifan Zhang and Team Math-AI. American mathematics competitions (amc) 2023,

    2023

  17. [25]

    American invitational mathematics examination (aime) 2025,

    Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025,

    2025

  18. [26]

    14 A Limitations and Broader Impacts GFlowRL’s in-batch Monte Carlo estimate of logZ can have higher variance in principle, especially when the group size is small. In practice, our two stabilizers, importance-sampling correction and flow-gap clipping, help control this variance, and the resulting streamlined estimator consistently outperforms the learned...

    2022

  19. [27]

    17 Table 8:Dataset statistics.Training and evaluation datasets used in our experiments, spanning mathematical reasoning, code generation, and AI security. Domain Split Dataset # Examples License Math Train DAPO-Math-17K [BytedTsinghua-SIA, 2025] 17,398 Apache 2.0 Test AIME 2024 [Zhang and Math-AI, 2024] 30 Apache 2.0 AIME 2025 [Zhang and Math-AI, 2025] 30...

    2025

  20. [29]

    Instruct

    with Qwen2.5-7B as the backbone. Method Backbone ConstantLogZ FOR GFlowRL Average Acc 23.02 25.34 29.53 40.92 F.2 Dense 32B Math Results Table 15:Results on math reasoning benchmarks for dense models.We report Avg@16 accuracy with relative improvements over the backbone shown as subscripts. Positive gains are shown in green and negative changes in red. GF...

    2023

  21. [30]

    to Maximum Diversity (score = 5), asEVALUATION CRITERIA inthe prompt. 22 H Distribution Matching Comparison To evaluate distribution matching performance, we construct a synthetic target distribution defined as a mixture of three Gaussian components with distinct centers. We compare four algorithms: GRPO (a maximum-reward method), FlowRL, FlowRL-RandomLog...

    2025

  22. [1459]

    Chain of attack: a semantic-driven contextual multi-turn attacker for llm, 2024b

    Xikang Yang, Xuehai Tang, Songlin Hu, and Jizhong Han. Chain of attack: a semantic-driven contextual multi-turn attacker for llm, 2024b. URLhttps://arxiv.org/abs/2405.05610. Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training. https://fengyao.notion.sit ...

    Pith/arXiv arXiv

  23. [2001]

    Foot-in-the-door: A multi-turn jailbreak for LLMs

    Zixuan Weng, Xiaolong Jin, Jinyuan Jia, and Xiangyu Zhang. Foot-in-the-door: A multi-turn jailbreak for LLMs. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1939–1950. Association for Computational Linguistics,

    2025

  24. [2017]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  25. [2020]

    12 Mark Russinovich, Ahmed Salem, and Ronen Eldan

    URL https://proceedings.neurips.cc/paper_fil es/paper/2020/file/9c22c0b51b3202246463e986c7e205df-Paper.pdf. 12 Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: the crescendo multi-turn llm jailbreak attack. InProceedings of the 34th USENIX Conference on Security Symposium, SEC ’25, USA,

    2020

  26. [2021]

    URL https://doi.org/10.1007/s4 2985-021-00102-x. OpenAI. Introducing openai o1. https://openai.com/o1/ ,

    doi:10.1007/s4

  27. [2022]

    URL https://proceedings.neurips.cc/paper _files/paper/2022/file/27b51baca8377a0cf109f6ecc15a0f70-Paper-Conference. pdf. 11 Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward J Hu, Katie E Everett, Dinghuai Zhang, and Yoshua Bengio. GFlownets and variational inference. InThe Eleventh International Conference on Learning Representations,

    2022

  28. [2023]

    Yifan Zhang and Team Math-AI

    URL https://huggingface.co/datasets/math-ai/amc23. Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024,

    2024

  29. [2024]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al

    URL https://arxiv.org/abs/2407.21783. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.Nature, 645:633–638, 2025a. Weiyang Guo, Zesheng Shi, Zhuo Li, Yequan Wang, Xuebo Liu, Wenya Wang, Fangming Li...

    Pith/arXiv arXiv

  30. [2025]

    URL https://huggingface.co/datasets/Byte dTsinghua-SIA/DAPO-Math-17k. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick ...

    Pith/arXiv arXiv