arxiv: 2605.11403 · v1 · submitted 2026-05-12 · 💻 cs.LG · cs.AI· cs.CL

Recognition: no theorem link

fg-expo: Frontier-guided exploration-prioritized policy optimization via adaptive kl and gaussian curriculum

Mingxiong Lin , Zhangquan Gong , Maowen Tang , Qian Li , Chuangchuang Wang , Jian Ma , Sutian Huang , Kai Tang

show 1 more author

Haonan Lu

Authors on Pith no claims yet

Pith reviewed 2026-05-13 02:54 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL

keywords reinforcement learningpolicy optimizationmathematical reasoningcurriculum samplingKL penaltylarge language modelsGRPO

0 comments

The pith

FG-ExPO enhances GRPO for LLM math reasoning by scaling the KL penalty with batch accuracy and sampling questions around moderate difficulty.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies that GRPO's fixed KL coefficient limits necessary exploration when the policy needs to change substantially and that uniform question sampling misses the most useful training signals from moderately difficult problems. It introduces two components to address this: Accuracy-Conditioned KL Scaling adjusts the penalty strength nonlinearly based on current batch accuracy, relaxing it during poor performance and tightening it during good performance, while Gaussian Curriculum Sampling weights problems according to a Gaussian distribution centered near 0.5 accuracy to focus on the learning frontier. If these changes work as intended, models should explore more effectively early in training and refine more efficiently later, producing higher pass rates especially when multiple samples are allowed at inference time. The reported results show this occurring on two model scales across six benchmarks, with the biggest lift on the hardest dataset.

Core claim

FG-ExPO augments GRPO by replacing its fixed KL coefficient with Accuracy-Conditioned KL Scaling, which applies a smooth nonlinear function of batch average accuracy to loosen the constraint when performance is low and strengthen it when performance is satisfactory, together with Gaussian Curriculum Sampling that assigns higher weights to questions whose accuracy lies near 0.5. On DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base this yields consistent gains over vanilla GRPO, including an absolute rise of 13.34 points on AIME 2025 pass@32 from 63.33 percent to 76.67 percent and an average pass@32 improvement of 2.66 on the 8B model, with substantially larger benefits at pass@32 than at pass@1

What carries the argument

Accuracy-Conditioned KL Scaling (AKL) and Gaussian Curriculum Sampling (GCS), which together adapt the exploration constraint and the data distribution to the model's current performance frontier.

If this is right

Larger relative gains on pass@32 than on pass@1 indicate that the method expands the set of distinct correct solutions reachable under a fixed inference budget.
The components are lightweight and can be added to existing RLVR pipelines without major changes to the training loop.
Improvements appear across model sizes from 1.5B to 8B parameters and across multiple standard mathematical reasoning benchmarks.
The biggest lifts occur on harder problems such as AIME, suggesting the approach is especially useful when the model must discover non-obvious reasoning paths.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same accuracy-triggered adjustment of constraints could be tested in other policy optimization algorithms or non-math domains such as code generation.
Replacing the fixed Gaussian center with a slowly adapting mean that tracks the model's current average accuracy might produce further gains.
If the accuracy level that defines the learning frontier shifts with model scale, the sampling distribution could be made scale-dependent without additional hyperparameters.

Load-bearing premise

The observed gains are produced by the AKL and GCS mechanisms themselves rather than by differences in implementation details, hyperparameter choices, or random variation.

What would settle it

An ablation that disables AKL and GCS while matching every other training detail and observes whether the gains over vanilla GRPO disappear, or evaluation on a new model and benchmark where no improvement occurs.

read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard paradigm for LLM mathematical reasoning, with Group Relative Policy Optimization (GRPO) serving as the dominant algorithm. We identify two overlooked inefficiencies inherent in GRPO. First, a fixed KL coefficient overly restricts policy exploration at moments when the model needs to diverge significantly from the reference policy. Second, uniform question sampling overlooks that moderately difficult problems produce the most informative gradient signals. We propose FG-ExPO, short for Frontier-Guided Exploration-Prioritized Policy Optimization, which integrates two lightweight components. Accuracy-Conditioned KL Scaling (AKL) adjusts the KL penalty strength through a smooth nonlinear function of batch average accuracy, loosening the constraint when the model performs poorly and strengthening it when the model achieves satisfactory results. Gaussian Curriculum Sampling (GCS) assigns sampling weights to questions following a Gaussian distribution centered at a moderate accuracy level around 0.5, focusing model training on its learning frontier. We conduct evaluations on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base across six mainstream mathematical reasoning benchmarks. Experimental results demonstrate that FG-ExPO consistently outperforms vanilla GRPO. It delivers an absolute improvement of 13.34 on the AIME 2025 pass@32 metric, rising from 63.33 percent to 76.67 percent, and obtains an average pass@32 gain of 2.66 on the 8B model. The substantially larger performance gains observed on pass@32 compared to pass@1 verify that FG-ExPO enlarges the model's effective exploration space under a fixed inference budget.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

FG-ExPO reports measurable gains over GRPO on math benchmarks but the attribution to AKL and GCS remains unverified without ablations or matched controls.

read the letter

FG-ExPO adds accuracy-conditioned KL scaling and Gaussian curriculum sampling to GRPO for LLM math reasoning. It reports a 13-point absolute lift on AIME 2025 pass@32 and a 2.66 average gain on the 8B model, with larger benefits at pass@32 than pass@1. The motivation is straightforward: fixed KL can choke exploration when accuracy is low, and uniform sampling wastes effort on questions that are either too easy or too hard. The two tweaks target those spots directly and stay lightweight enough to drop into existing pipelines. The paper evaluates on two model sizes across six benchmarks and shows the pattern holds, which is useful data for anyone iterating on RLVR. The pass@32 emphasis is a nice touch because it directly ties to the exploration claim under fixed inference budgets. The soft spot is exactly what the stress-test flags. KL strength and sampling distribution are both highly sensitive in this regime, yet the abstract gives no component ablations, no re-tuned GRPO baseline at equivalent exploration level, and no mention of run counts or variance. A 13-point jump is large enough that it could reflect implementation details or hyperparameter luck rather than the proposed mechanisms. If the full paper supplies those controls and code, the attribution strengthens considerably; otherwise the central claim stays provisional. This is for researchers actively tuning RL fine-tuning for reasoning models who need practical next steps on exploration. A reader already working with GRPO would get concrete ideas and benchmark numbers to test. It deserves peer review because the techniques are well-motivated, the results are concrete, and the gaps are fixable with standard experimental additions rather than fundamental redesign.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes FG-ExPO as an enhancement to Group Relative Policy Optimization (GRPO) for RLVR in LLM mathematical reasoning. It introduces Accuracy-Conditioned KL Scaling (AKL), which modulates the KL penalty via a nonlinear function of batch-average accuracy, and Gaussian Curriculum Sampling (GCS), which weights questions by a Gaussian centered near accuracy 0.5 to focus on the learning frontier. Experiments on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base across six math benchmarks report consistent gains over vanilla GRPO, including a 13.34-point absolute increase on AIME 2025 pass@32 (63.33% to 76.67%) and a 2.66 average pass@32 gain on the 8B model, interpreted as evidence of enlarged exploration under fixed inference budgets.

Significance. If the gains prove robustly attributable to AKL and GCS, the approach supplies a simple, low-overhead heuristic for improving exploration in GRPO-style RLVR without introducing new hyperparameters or architectural changes. The differential improvement on pass@32 versus pass@1 is a potentially useful signal for exploration quality. The work is entirely empirical and heuristic, with no parameter-free derivations or machine-checked proofs.

major comments (2)

[Abstract and Experimental Results] Abstract and Experimental Results: The central claim that the 13.34-point AIME 2025 pass@32 gain (and 2.66 average on 8B) arises specifically from AKL and GCS is unsupported by component ablations, matched hyperparameter re-tuning of the GRPO baseline, or statistical tests. In GRPO-style RLVR the KL coefficient and sampling distribution are known to be highly sensitive; without these controls the attribution to the proposed mechanisms cannot be verified and the reported improvements could reflect implementation differences or noise.
[Method] Method description: AKL is described as a 'smooth nonlinear function of batch average accuracy' and GCS as Gaussian weights centered at accuracy 0.5, but the manuscript supplies neither the explicit functional form (e.g., the precise mapping from accuracy to KL scale) nor pseudocode or implementation details sufficient for exact reproduction.

minor comments (2)

[Experimental Results] The abstract states evaluations on 'six mainstream mathematical reasoning benchmarks' but does not list them; the full experimental section should enumerate the benchmarks and report per-benchmark metrics with standard deviations.
[Experimental Results] Pass@32 results are highlighted as evidence of exploration gains, yet the paper does not clarify whether the same number of samples (32) and decoding strategy were used for both FG-ExPO and the GRPO baseline.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful and constructive feedback. We address each major comment below and will revise the manuscript to strengthen the empirical support and reproducibility of the work.

read point-by-point responses

Referee: [Abstract and Experimental Results] The central claim that the 13.34-point AIME 2025 pass@32 gain (and 2.66 average on 8B) arises specifically from AKL and GCS is unsupported by component ablations, matched hyperparameter re-tuning of the GRPO baseline, or statistical tests. In GRPO-style RLVR the KL coefficient and sampling distribution are known to be highly sensitive; without these controls the attribution to the proposed mechanisms cannot be verified and the reported improvements could reflect implementation differences or noise.

Authors: We agree that component ablations, explicit confirmation of matched hyperparameters, and statistical tests are required to rigorously attribute the gains to AKL and GCS. In the revised manuscript we will add (i) ablations isolating AKL alone and GCS alone, (ii) a statement that the GRPO baseline was run with identical hyperparameters and training settings, and (iii) standard deviations and significance tests across multiple random seeds. These additions will directly address the attribution concern while preserving the reported results. revision: yes
Referee: [Method] Method description: AKL is described as a 'smooth nonlinear function of batch average accuracy' and GCS as Gaussian weights centered at accuracy 0.5, but the manuscript supplies neither the explicit functional form (e.g., the precise mapping from accuracy to KL scale) nor pseudocode or implementation details sufficient for exact reproduction.

Authors: We acknowledge that the current description is insufficient for exact reproduction. The revised manuscript will provide the explicit functional form of the Accuracy-Conditioned KL Scaling (the precise nonlinear mapping from batch-average accuracy to the KL coefficient), the exact mean, variance, and normalization details of the Gaussian used in Gaussian Curriculum Sampling, and pseudocode for the full FG-ExPO training loop. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical heuristics without self-referential reduction

full rationale

The paper proposes two heuristic components (AKL and GCS) to modify GRPO based on observed inefficiencies in exploration and sampling difficulty. These are defined directly via the stated functional forms (nonlinear accuracy-dependent KL scaling and Gaussian weighting around accuracy 0.5) and evaluated on external benchmarks. No derivation chain, equation, or prediction is shown that reduces claimed gains to quantities fitted or defined inside the paper itself; the results remain externally falsifiable experimental outcomes rather than tautological outputs of the method's own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No explicit free parameters, axioms, or invented entities are stated in the abstract; the method relies on standard RL assumptions and the existence of a smooth nonlinear function for KL scaling whose exact form is unspecified here.

pith-pipeline@v0.9.0 · 5632 in / 1103 out tokens · 110821 ms · 2026-05-13T02:54:37.592926+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages · 11 internal anchors

[1]

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, et al. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787,

work page internal anchor Pith review Pith/arXiv arXiv
[2]

(2025), Gpg: A simple and strong reinforcement learning baseline for model reasoning, arXiv preprint arXiv:2504.02546

Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. GPG: A simple and strong reinforcement learning baseline for model reasoning.arXiv preprint arXiv:2504.02546,

work page arXiv
[4]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Distilled checkpoint released alongside the DeepSeek-R1 report. Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

OpenAI o1 System Card

Aaron Jaech et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

work page internal anchor Pith review Pith/arXiv arXiv
[6]

Understanding R1-Zero-Like Training: A Critical Perspective

Zichen Liu, Changyu Chen, Wenjun Li, et al. Understanding R1-Zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

work page internal anchor Pith review Pith/arXiv arXiv
[7]

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

work page internal anchor Pith review Pith/arXiv arXiv
[8]

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

work page internal anchor Pith review Pith/arXiv arXiv
[9]

HybridFlow: A Flexible and Efficient RLHF Framework

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256,

work page internal anchor Pith review Pith/arXiv arXiv
[10]

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

work page internal anchor Pith review Pith/arXiv arXiv
[11]

Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs

Xumeng Wen, Zihan Liu, Shun Zheng, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs.arXiv preprint arXiv:2506.14245,

work page internal anchor Pith review arXiv
[12]

Qwen3 Technical Report

An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

work page internal anchor Pith review Pith/arXiv arXiv
[13]

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

Qiying Yu, Zheng Zhang, Ruofei Zhu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476,

work page internal anchor Pith review Pith/arXiv arXiv