Pith. sign in

REVIEW 23 cited by

Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2508.16949 v7 pith:LZD3HADY submitted 2025-08-23 cs.LG cs.AI

Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning

Yang Zhou , Sunzhu Li , Shunyu Liu , Wenkai Fang , Kongcheng Zhang , Jiale Zhao , Jingwen Yang , Yihe Zhou
show 6 more authors
Jianwei Lv Tongya Zheng Hengtong Lu Wei Chen Yan Xie Mingli Song
This is my paper
classification cs.LG cs.AI
keywords reasoningexplorationlearningruscarlreinforcementrubricsbottleneckcannot
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Recent advances in Large Language Models (LLMs) have underscored the potential of Reinforcement Learning (RL) to facilitate the emergence of reasoning capabilities. Despite the encouraging results, a fundamental dilemma persists as RL improvement relies on learning from high-quality samples, yet the exploration for such samples remains bounded by the inherent limitations of LLMs. This, in effect, creates an undesirable cycle in which what cannot be explored cannot be learned. In this work, we propose Rubric-Scaffolded Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework designed to break the exploration bottleneck for open-ended reasoning. Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit scaffolding for exploration during rollout generation, where different rubrics are provided as external guidance within task instructions to steer diverse high-quality responses. This guidance is gradually decayed over time, encouraging the model to internalize the underlying reasoning patterns; (2) verifiable rewards for exploitation during model training, where we can obtain robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL on open-ended reasoning tasks. Extensive experiments demonstrate the superiority of the proposed RuscaRL across various benchmarks, effectively expanding reasoning boundaries under the Best-of-N evaluation. Our code is available at https://github.com/IANNXANG/RuscaRL.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 7.0

    EvoRubrics introduces adversarial co-evolution between policy LLM and rubric generator to produce dynamic, adaptive rewards for reinforcement learning on open-ended tasks.

  2. Visual Preference Optimization with Rubric Rewards

    cs.CV 2026-04 unverdicted novelty 7.0

    rDPO uses offline-built rubrics to generate on-policy preference data for DPO, raising benchmark scores in visual tasks over outcome-based filtering and style baselines.

  3. Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategy

    cs.LG 2026-03 unverdicted novelty 7.0

    ARL-RR alternates optimization over rubric meta-classes with dynamic selection to avoid fixed scalarization, outperforming baselines on HealthBench.

  4. Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation

    cs.AI 2026-08 conditional novelty 6.0

    Adding behavioral strategy hints to RL training prompts, then distilling only correct hint-conditioned answers into the unconditioned model, lifts Qwen3-1.7B math pass@1 by 5.0% over DAPO.

  5. CriPO: Enhancing Rubric-based RL via Self-Distillation

    cs.LG 2026-07 conditional novelty 6.0

    CriPO augments GRPO with localized self-distillation and token-level advantage flipping to address both unexplored and suppressed rubric criteria, improving rubric-based RL on medicine and science QA.

  6. CriPO: Enhancing Rubric-based RL via Self-Distillation

    cs.LG 2026-07 conditional novelty 6.0

    CriPO combines GRPO with on-policy self-distillation: it injects missing rubric behaviors through forward-KL and flips token-level advantages for suppressed criteria, outperforming baseline rubric RL.

  7. Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teacher

    cs.CV 2026-06 unverdicted novelty 6.0

    T2R proposes a self-improving Teacher and competition-guided Reasoner with case-wise rewards to provide more effective supervision for CoT optimization in CXR VQA.

  8. RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review

    cs.CL 2026-06 conditional novelty 6.0

    A rubric-first LLM pipeline that splits peer review into rubric generation, rubric-conditioned review writing, and final scoring outperforms existing AI reviewers on alignment with human judgments in a 200-paper test.

  9. Deep Research as Rubric for Reinforcement Learning

    cs.CL 2026-05 unverdicted novelty 6.0

    DR-rubric is a two-stage framework using iterative agentic search to generate atomic verifiable constraints for GRPO-based RL, achieving competitive performance on 6 benchmarks with 1K-3K examples via bootstrap or fro...

  10. RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains

    cs.LG 2026-05 unverdicted novelty 6.0

    RUBRIC-ARROW is an alternating rubric generator and judge framework that uses probability-based scoring and pairwise preferences to improve pointwise reward modeling accuracy for LLM post-training in non-verifiable domains.

  11. ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning

    cs.CL 2026-05 unverdicted novelty 6.0

    ARES generates 100K rubric-annotated QA instances from raw documents and shows rubric-based RL trained on them outperforms continual pretraining, SFT, and binary-reward RL on seven benchmarks.

  12. Reward Hacking in Rubric-Based Reinforcement Learning

    cs.AI 2026-05 unverdicted novelty 6.0

    Rubric-based RL verifiers can be gamed via partial criterion satisfaction and implicit-to-explicit tricks, yielding proxy gains that do not improve quality under rubric-free judges; stronger verifiers reduce but do no...

  13. Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning

    cs.AI 2026-05 unverdicted novelty 6.0

    Rubric-grounded RL with LLM judges on document-derived criteria raises Llama-3.1-8B normalized reward to 71.7% on held-out rubrics and improves performance on GSM8K, MATH, and GPQA benchmarks.

  14. Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text

    cs.CL 2026-04 unverdicted novelty 6.0

    POP bootstraps post-training signals for open-ended LLM tasks by synthesizing rubrics during self-play on pretraining corpus, yielding performance gains on Qwen-2.5-7B across healthcare QA, creative writing, and instr...

  15. Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

    cs.CL 2026-04 unverdicted novelty 6.0

    Vocabulary dropout prevents diversity collapse in LLM co-evolution by masking proposer logits, yielding average +4.4 point solver gains on mathematical reasoning benchmarks at 8B scale.

  16. Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks

    cs.CL 2026-04 unverdicted novelty 6.0

    RTT bridges response-level rubrics to token-level rewards via a relevance discriminator and intra-sample group normalization, yielding higher instruction and rubric accuracy than baselines.

  17. QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

    cs.CL 2026-06 unverdicted novelty 5.0

    QUBRIC co-designs queries and rubrics via teacher key points, contrastive generation, and learnability filtering to support GRPO training, yielding +5.5 on ArenaHard and +6.3 average transfer to legal/moral/narrative ...

  18. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  19. Reinforcement Learning with Robust Rubric Rewards

    cs.CV 2026-05 unverdicted novelty 5.0

    RLR³ extends RLVR to criterion-level rubric verification via dual execution paths, minimal exposure masking, hierarchical aggregation, and saturation mitigation, delivering 4.7-point gains over base on 15 benchmarks w...

  20. ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning

    cs.CL 2026-05 unverdicted novelty 5.0

    ARES generates 100K rubric-annotated QA instances from raw documents and demonstrates superior rubric-based RL performance over baselines on open-ended benchmarks.

  21. Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

    cs.AI 2026-05 unverdicted novelty 5.0

    Mid-training LLMs on self-generated diverse reasoning paths improves subsequent RL performance on mathematical benchmarks and OOD tasks.

  22. Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

    cs.CL 2026-04 accept novelty 5.0

    LLM post-training is unified as off-policy or on-policy interventions that expand support for useful behaviors, reshape policies within reachable states, or consolidate behavior across training stages.

  23. Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

    cs.CL 2026-04 conditional novelty 5.0

    Randomly masking a proposer's output vocabulary during training and generation sustains curriculum diversity and improves solver accuracy by +4.4 points at 8B in LLM co-evolution.