Pith. sign in

REVIEW 5 major objections 6 minor 53 references

This paper claims that supervising a model only on whether its chain of thought agrees with its final answer — using a small, general-purpose language model — produces multiple-choice reasoning that is both more consistent and more accurate

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:28 UTC pith:UFQ4DJRI

load-bearing objection A cheap consistency-reward recipe for MCQ RL that mostly delivers, but the headline accuracy gain is the compound metric, the consistency checker's precision is unmeasured, and 'consistency alone' overstates the stage-2 reward. the 5 major comments →

arxiv 2510.09278 v2 pith:UFQ4DJRI submitted 2025-10-10 cs.CL cs.AI

CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts

Jiuheng Lin , Cong Jiang , Zirui Wu , Jiarui Sun , Yansong Feng This is my paper
classification cs.CL cs.AI
keywords consistency-aware RLmultiple-choice question trainingreasoning consistencyreward hackingprocess reward modeldata reformulationlaw and medicine LLMssmall-model supervision
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard reinforcement learning on multiple-choice data can raise final-answer accuracy while degrading the reasoning behind it; the paper reports logical inconsistency rising from 7% to 31% in a pilot. CLARITY is proposed as a remedy: a small, general-purpose language model reads the model's chain of thought, extracts the options the reasoning endorses, and penalizes any mismatch with the final answer. A two-stage refine-then-monitor scheme and a reformulation of easy questions into harder ones push consistency and accuracy up, with reported gains of 16.5% and 7.5% over baselines. The broader claim is that reasoning consistency alone — cheap to check, with no domain expertise — is enough of a signal to train expert models in law and medicine and to transfer to open-ended tasks.

Core claim

The paper's central claim is that the consistency of a model's judgment about each option — not the factual correctness of the reasoning itself — is a sufficient reward signal for MCQ reinforcement learning in expert domains. Using only a 7B general-purpose LLM to parse the believed-correct options from the chain of thought and penalize disagreement with the final answer, CLARITY reports a 16.5% improvement in response consistency and a 7.5% improvement in reliable accuracy over outcome-based RL baselines. The paper also claims that this consistency-only supervision transfers to out-of-domain open-ended benchmarks and improves human-rated professionalism, coherence, and readability. The larg

What carries the argument

The load-bearing object is the consistency-aware reward: a separate small LLM parses the believed-correct options from the policy's chain of thought, and the reward penalizes any mismatch between that parsed set and the final answer. Around it, CLARITY wraps a two-stage pipeline — stage one using a structural reward to force transparent option-wise reasoning, stage two dropping that constraint while applying the consistency reward plus a strict answer reward — and a dynamic data reformulation that deconstructs easy questions into propositions, polishes and diversifies them, and regroups them into new harder questions. The consistency signal is what makes a weak checker sufficient: it only ne

Load-bearing premise

The whole method hinges on the small 7B checker reliably extracting the options the policy's reasoning endorses and doing so without drifting on domain-specific phrasing; if it misreads those endorsements, the reward punishes style rather than bad reasoning.

What would settle it

Take 100 responses from a CLARITY-trained model on held-out MCQs; have human experts mark each response as consistent or not, and compute the agreement between human labels and the small checker's consistency verdict. If agreement falls far below the paper's reported human-reference-model correlation (r ≈ 0.61), or if a stronger checker flags many consistent-but-wrong answers that received full reward, the core mechanism — consistency as a faithful proxy for reasoning quality — is not doing the work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Outcome-only RL on MCQs can make answers more accurate while making reasoning less consistent, so accuracy alone is an unsafe training signal.
  • A 7B general-purpose LLM can substitute for large or domain-trained process reward models when the supervision target is consistency rather than step-level correctness.
  • Consistency supervision improves both consistency (16.5%) and reliable accuracy (7.5%), where 'reliable' means the final answer is both correct and consistent with the reasoning.
  • Trained models generalize to open-ended out-of-domain tasks, indicating the benefit is not mere memorization of MCQ format.
  • The two-stage refine-then-monitor design prevents the reward hacking that appears when the checker is too weak (0.5B/1.5B models).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If consistency is the causal mechanism, the same supervision should transfer to open-ended tasks with a checkable final answer, such as evidence-based fact-checking or code with tests, where a small model could check that intermediate claims support the final claim; the paper only demonstrates MCQ-to-open-ended transfer.
  • Because weaker checkers fail badly, a natural extension is to distill the 7B checker's consistency judgments into a smaller classifier; if the distilled checker preserves the 7B model's detection rate, the training overhead could drop well below the paper's reported 8-hour-to-over-1-day increase.
  • Measured against human consistency labels rather than the same kind of checker that provides the reward, the reported 16.5% gain could shrink or grow; a direct comparison would clarify whether the reward teaches genuine logical coherence or checker-pleasing phrasing.
  • The data reformulation strategy's alpha schedule is a coarse knob; one could test whether adaptively varying the number of options or mixing domains further improves generalization, which the paper lists as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CLARITY, a reinforcement-learning framework for multiple-choice question (MCQ) training in expert domains such as law and medicine. CLARITY replaces expensive process reward models with a small general-purpose LLM (Qwen-2.5-7B) that parses the policy's believed-correct options from its chain of thought and penalizes mismatches between those parsed judgments and the final answer. The training uses a two-stage refine-then-monitor pipeline with a structure reward in stage 1 and a consistency reward plus answer reward in stage 2, together with a dynamic data-reformulation strategy that deconstructs easy examples into propositions and regroups them into harder questions. Experiments on JEC-QA and MedQA report improvements in a compound "Acc+" (correct-and-consistent) metric of 7.5 percentage points and in consistency of 16.5 percentage points, plus gains on out-of-domain open-ended tasks and in human evaluations of coherence and professionalism. The authors claim this shows that small, general-purpose LLMs can guide expert-model training using only consistency supervision, without large teacher models or expert-annotated corpora.

Significance. If the central claim is sustained, CLARITY is a valuable contribution: it offers a relatively cheap, domain-agnostic way to inject reasoning-quality supervision into MCQ reinforcement learning, where outcome-only rewards are known to produce superficially correct but internally inconsistent chains of thought. The paper includes useful analyses of failure modes (over-exclusion, over-selection, dissociated answers), open-source code, and human evaluation. However, the evidence as presented does not fully support the strongest claims. The headline accuracy gain is a compound metric, not raw accuracy; the reliability of the 7B consistency checker is measured only indirectly; and the pipeline still relies on DeepSeek-V3 for data augmentation and evaluation, undercutting the "small-model-only" framing. These issues are load-bearing because the method's value depends on the consistency reward being both accurate and non-gameable.

major comments (5)
  1. [Abstract, §5.1, Table 1] The headline "accuracy +7.5%" is the compound Acc+ (correct and consistent) metric, not raw accuracy. In Table 1, CLARITY-7B's average raw accuracy is 61.3 vs. 61.1 for Standard RL (+0.2 pp), and legal accuracy drops from 58.5 to 56.4. The abstract and §5.1 should state explicitly that the 7.5-point gain is in Acc+, and the paper should report raw accuracy and significance separately.
  2. [Algorithm 1 lines 32–35; §5.3; Appendix B.1] The consistency reward's reliability is not established. The paper reports that Qwen-2.5-7B detects 94% of DeepSeek-V3-flagged inconsistencies, and that human–DeepSeek agreement is only r=0.610 (Pearson). Neither number measures the 7B checker's precision or false-positive rate on valid reasoning. If the 7B checker false-flags legitimate legal/medical phrasing, stage-2 reward (line 35) punishes correct reasoning and the policy can learn to satisfy the checker's heuristics rather than reason. Please provide a precision/false-positive measurement against human-annotated labels and demonstrate robustness to paraphrasing and domain-specific language.
  3. [Section 4; Algorithm 1 line 4] The claim that CLARITY uses only a small, general-purpose LLM is not accurate for the full pipeline. Section 4 states that DeepSeek-V3 is used for "data augmentation and consistency evaluation," and Algorithm 1 line 4 invokes LLM_Refine for reformulation. Thus the method still requires a large LLM for data reformulation; only the consistency reward itself is 7B. This should be scoped carefully in the abstract and conclusion, or the augmentation component should be ablated to show what can be achieved with the 7B checker alone.
  4. [Tables 1–2, §5] No error bars, multiple seeds, or significance tests are reported. Given that the raw-accuracy gap between CLARITY-7B and Standard RL is only 0.2 pp on average, the headline gains in Acc+ and consistency could reflect run-to-run variance. Please report at least 3 seeds with standard deviations and, where possible, paired significance tests.
  5. [§5.1, Tables 1–2] The sentence "models that even surpass large commercial systems like GPT-4o" is not supported by the tables. In Table 1, CLARITY-7B has lower average Acc+ (58.9 vs. 61.3) and much lower medical Acc+ (63.3 vs. 83.0) than GPT-4o; in Table 2 the average Acc+ is 52.0 vs. 52.2. The claim should be tempered or substantiated with a specific comparison.
minor comments (6)
  1. [Abstract] Typo: "to to better exploit" should be "to better exploit."
  2. [Related Work] Typo: "reinforencement" should be "reinforcement."
  3. [Table 1 caption] The metric "Acc+" is defined only in §4; please define it in the table caption or a footnote, and clarify that it is not the same as raw accuracy.
  4. [Figures 5–6] The axes of Figures 5 and 6 are not labeled. In particular, the y-axis of Figure 6 (response frequency/length) should specify what is plotted.
  5. [Table 7] The table captions "True Value" and "False Value" are awkward; use "Correct" and "Incorrect" or similar. Also, the absence of an answer-reward row for Stage-1 is confusing given Algorithm 1 lines 23–24; consider aligning the notation.
  6. [Appendix C.1] Typo: "underoptimal" should be "suboptimal." Also, "Rouge-L" should be "ROUGE-L."

Circularity Check

0 steps flagged

No construction-level circularity: the consistency signal is internal to the policy's own CoT vs. final answer, headline metrics are measured by a different model (DeepSeek-V3) and against external ground truth, and no load-bearing self-citation or imported ansatz is used.

full rationale

The load-bearing reward, R_consistency = I[Ans_parsed = Ans_gen] (Algorithm 1, lines 32-33), compares a frozen external checker's parse of the policy's believed-correct options to the policy's own final answer; no ground-truth label enters this term, so the consistency 'prediction' is not the fit of any parameter. The reported consistency gains (16.5%) are measured with DeepSeek-V3, a different model from the Qwen-2.5-7B reward checker, with a reported human-model correlation (Pearson r=0.610, Appendix B.1), so the evaluation is not the training instrument by construction. Accuracy gains are measured by exact match against gold answers and are also trained on gold answers in stage 2 (Algorithm 1, line 34) — standard RL supervision, not a circular reduction; the title's 'consistency alone' is thus an overclaim relative to the algorithm, but overclaiming is not circularity. Out-of-domain generalization rests on external benchmarks (LexEval, PubMedQA) scored by Rouge-L. There is no load-bearing self-citation: citations (GRPO, DeepSeek-V3, Qwen-2.5, JEC-QA, MedQA) are external, and no uniqueness theorem or fitted ansatz is imported from the authors' prior work. The remaining concerns are confounds or validity risks, not circularity: (i) Qwen-2.5-7B computes the initial pass rate that selects data for reformulation and is also the base of CLARITY-7B, but the pass rate is not reported as a prediction, it only gates data difficulty; (ii) DeepSeek-V3 both polishes reformulated data and evaluates consistency, though evaluation is on the untouched validation split; (iii) the 7B checker's precision against valid reasoning is unmeasured, so false-flagging could teach style rather than reasoning — this is the legitimate correctness risk flagged by the skeptic, not a definitional reduction, and the paper itself acknowledges the reward-hacking regime for weaker checkers (Figure 5; Section 5.4). None of these reduces a stated prediction to an input by construction. Per the rubric, a method tied to external benchmarks and a different-model evaluator receives 0-2; assigning 1 for the minor same-family/evaluator confounds.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim leans on three unevidenced-in-paper premises: consistency-as-proxy for reasoning quality, the 7B checker's parse reliability, and option-decomposability of MCQs; plus hand-set hyperparameters (alpha, stage lengths, reward weights) with no sensitivity analysis. No new entities are introduced.

free parameters (4)
  • reformulation proportion alpha = 0.5 (starts at step 100; 'gradually increasing' schedule not specified)
    Controls what fraction of the easiest instances are deconstructed into harder multi-option questions; Section 4 and Algorithm 1. No sensitivity analysis reported.
  • stage-1 sample budget and epochs = 500 instances, 2 epochs; stage-2: remaining data, 8 epochs
    Refine-then-monitor split defined in Section 4 / Appendix C.1. Chosen by hand, no ablations over the split.
  • reward weights = answer 1.0; consistency true 0.0 / false -1.0 (weight 1.0); format 0.1/-1.0; structure 1.0/0.0
    Table 7. Hand-set without sensitivity analysis; the reward balance is load-bearing for the two-stage behavior.
  • rollouts per sample and temperature = 6 rollouts, temperature 1.0, nucleus p=1.0
    Table 6. Exploration settings that affect the consistency signal's coverage; not varied.
axioms (5)
  • domain assumption Consistency between CoT option-judgments and final answer is a valid proxy for reasoning reliability
    Section 3.1: 'making consistency a useful proxy for reasoning reliability.' If false, the reward optimizes a correlate, not reasoning quality.
  • domain assumption A 7B general-purpose LLM can reliably parse believed-correct options from CoT without domain expertise
    Section 3.1 claims this requires only 'fundamental semantic understanding'; Appendix B bounds it: 94% inconsistency detection vs DeepSeek, r=0.610 human agreement.
  • domain assumption DeepSeek-V3 judgments are a reliable ground truth for consistency and data polishing
    Used for the consistency metric, the pilot study, and the data augmentation; validated once with r=0.610 (Appendix B).
  • domain assumption MCQ options are decomposable into independent, self-contained propositions
    Dynamic data reformulation concatenates stem+option and determines correctness by the original answer (Algorithm 1); this holds poorly for case-analysis questions where options are mutually dependent.
  • domain assumption Regex-detected per-option structure is a robust scaffold that transfers to free-form reasoning
    Stage-1 structure reward uses regex over option labels (Algorithm 1 line 23); assumes explicit option-wise format is the right intermediate objective and that the regex captures it.

pith-pipeline@v1.3.0-alltime-deepseek · 26264 in / 14600 out tokens · 119728 ms · 2026-08-04T10:28:42.470481+00:00 · methodology

0 comments
read the original abstract

Training expert LLMs in domains with scarce data is difficult, often relying on multiple-choice questions (MCQs). However, standard outcome-based reinforcement learning (RL) on MCQs is risky. While it may improve accuracy, we observe it often degrades reasoning quality such as logical consistency. Existing solutions to supervise reasoning, such as large-scale Process Reward Models (PRMs), are prohibitively expensive. To address this, we propose CLARity, a cost-effective RL framework that enhances reasoning quality using only a small, general-purpose LLM. CLARity integrates a consistency-aware reward mechanism with a 2-stage refine-then-monitor training pipeline to enhance reasoning consistency, and a dynamic data reformulation strategy to to better exploit limited data. Experiments demonstrate that CLARity improves response consistency by 16.5% and accuracy by 7.5% over baselines. Human evaluations further confirm holistic improvements in coherence and professionalism. Thus, CLARity offers a generalizable solution that enables smaller models to effectively guide expert models by reasoning consistency. Our code is open sourced at: https://github.com/Infinite-set/CLARity

Figures

Figures reproduced from arXiv: 2510.09278 by Cong Jiang, Jiarui Sun, Jiuheng Lin, Yansong Feng, Zirui Wu.

Figure 1
Figure 1. Figure 1: Illustration of risks in MCQ RL: rewarding [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Response quality dynamics under GRPO train [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of CLARITY, an efficient MCQ RL framework that trains expert models using only small, general-purpose LLMs. It combines a consistency mechanism for detecting inconsistencies, a refine-then-monitor training pipeline for improving reasoning quality, and a dynamic data reformulation for maximizing data utility. 3.1 Consistency-Aware Learning Mechanism Our pilot study shows that outcome-based RL on MC… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of three inconsistency [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Reasoning quality comparison between CLARITY and different baselines: without Stage-1, using Qwen-1.5B as the consistency reward model, and the vanilla Qwen2.5-7B-Instruct. 0 100 200 300 Training Steps 0 200 400 600 800 1000 Instance Num As-A-Whole Num 0 100 200 300 Training Steps 100 150 200 250 300 # Tokens Average Response Len CLARity W/o Stage-1 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics of as-a-whole response frequency and response length. In CLARITY, the first 15 steps correspond to Stage-1. to mitigate this, which we evaluate through experi￾ments to demonstrate its necessity. Refine-then-monitor pipeline offers flexible rea￾soning and avoids hacking. Our 2-stage refine￾then-monitor pipeline first encourages explicit rea￾soning for each option and then uses a consistenc… view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics of CLARITY in Stage-1 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Training dynamics of CLARITY in Stage-2. accuracy on the validation set, and shows almost no improvement in out-of-domain tasks. The rule￾based approach forces the model to generate re￾sponses with a fixed format designed to match pre￾defined patterns rather than encouraging deeper reasoning, which makes the model more prone to overfitting during training, producing shorter re￾sponses that overly focused o… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 39 linked inside Pith

  1. [1]

    Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. 2025. https://hkunlp.github.io/blog/2025/Polaris Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models

    2025

  2. [2]

    Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. https://arxiv.org/abs/2503.08679 Chain-of-thought reasoning in the wild is not always faithful . Preprint, arXiv:2503.08679

    Pith/arXiv arXiv 2025

  3. [3]

    Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi

    Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. https://arxiv.org/abs/2503.11926 Monitoring reasoning models for misbehavior and the risks of promoting obfuscation . Preprint, arXiv:2503.11926

    Pith/arXiv arXiv 2025

  4. [4]

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024 a . https://arxiv.org/abs/2412.18925 Huatuogpt-o1, towards medical complex reasoning with llms . Preprint, arXiv:2412.18925

    Pith/arXiv arXiv 2024

  5. [5]

    Bowman, Jan Leike, Jared Kaplan, and Ethan Perez

    Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. 2025. https://arxiv.org/abs/2505.05410 Reasoning models don't always say what they think . Preprint, arXiv:2505.05410

    Pith/arXiv arXiv 2025

  6. [6]

    Yanda Chen, Chandan Singh, Xiaodong Liu, Simiao Zuo, Bin Yu, He He, and Jianfeng Gao. 2024 b . https://arxiv.org/abs/2401.13986 Towards consistent natural-language explanations via explanation-consistency finetuning . Preprint, arXiv:2401.13986

    Pith/arXiv arXiv 2024

  7. [7]

    Bowman, Julian Michael, Ethan Perez, and Miles Turpin

    James Chua, Edward Rees, Hunar Batra, Samuel R. Bowman, Julian Michael, Ethan Perez, and Miles Turpin. 2025. https://arxiv.org/abs/2403.05518 Bias-augmented consistency training reduces biased reasoning in chain-of-thought . Preprint, arXiv:2403.05518

    Pith/arXiv arXiv 2025

  8. [8]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 179 others. 2024. https://api.semanticscholar.org/CorpusID:275118643 Deepseek-v3 technical report . ArXiv, abs/2412.19437

    Pith/arXiv arXiv 2024

  9. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    Pith/arXiv arXiv 2019

  10. [10]

    Lishui Fan, Yu Zhang, Mouxiang Chen, and Zhongxin Liu. 2025. https://arxiv.org/abs/2508.05170 Posterior-grpo: Rewarding reasoning processes in code generation . Preprint, arXiv:2508.05170

    Pith/arXiv arXiv 2025

  11. [11]

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. https://doi.org/10.48550/arXiv.2504.11536 ReTool : Reinforcement Learning for Strategic Tool Use in LLMs . Preprint, arXiv:2504.11536

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.11536 2025

  12. [12]

    Pedro Ferreira, Wilker Aziz, and Ivan Titov. 2025. https://arxiv.org/abs/2504.05294 Truthful or fabricated? using causal attribution to mitigate reward hacking in explanations . Preprint, arXiv:2504.05294

    Pith/arXiv arXiv 2025

  13. [13]

    GLM, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, and 152 others. 2025. https://arxiv.org/abs/2508.06471 Glm-4.5: Agentic, reasoning, and coding (arc) foundation models . Preprint, arX...

    Pith/arXiv arXiv 2025

  14. [14]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. https://arxiv.org/abs/2411.15594 A survey on llm-as-a-judge . Preprint, arXiv:2411.15594

    Pith/arXiv arXiv 2025

  15. [15]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  16. [16]

    Bolei He, Xinran He, Run Shao, Shanfu Shu, Xianwei Xue, Mingquan Cheng, Haifeng Li, and Zhenhua Ling. 2025. https://arxiv.org/abs/2508.15213 Select to know: An internal-external knowledge self-selection framework for domain-specific question answering . Preprint, arXiv:2508.15213

    arXiv 2025

  17. [17]

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang. 2025. Open-reasoner-zero: An open source approach to scaling reinforcement learning on the base model. https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero

    2025

  18. [18]

    Jerry Huang, Siddarth Madala, Risham Sidhu, Cheng Niu, Hao Peng, Julia Hockenmaier, and Tong Zhang. 2025. https://arxiv.org/abs/2503.12759 Rag-rl: Advancing retrieval-augmented generation via rl and curriculum learning . Preprint, arXiv:2503.12759

    Pith/arXiv arXiv 2025

  19. [19]

    HuggingFace. 2025. https://github.com/huggingface/open-r1 Open r1: A fully open reproduction of deepseek-r1

    2025

  20. [20]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. https://arxiv.org/abs/2503.09516 Search-r1: Training llms to reason and leverage search engines with reinforcement learning . Preprint, arXiv:2503.09516

    Pith/arXiv arXiv 2025

  21. [21]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. https://arxiv.org/abs/2009.13081 What disease does this patient have? a large-scale open domain question answering dataset from medical exams . Preprint, arXiv:2009.13081

    Pith/arXiv arXiv 2020

  22. [22]

    Cohen, and Xinghua Lu

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019. https://arxiv.org/abs/1909.06146 Pubmedqa: A dataset for biomedical research question answering . Preprint, arXiv:1909.06146

    Pith/arXiv arXiv 2019

  23. [23]

    Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599

    Pith/arXiv arXiv 2025

  24. [24]

    Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. 2024. https://arxiv.org/abs/2407.13692 Prover-verifier games improve legibility of llm outputs . Preprint, arXiv:2407.13692

    Pith/arXiv arXiv 2024

  25. [25]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  26. [26]

    Fangyu Lei, Jinxiang Meng, Yiming Huang, Tinghong Chen, Yun Zhang, Shizhu He, Jun Zhao, and Kang Liu. 2025. https://arxiv.org/abs/2506.01710 Reasoning-table: Exploring reinforcement learning for table reasoning . Preprint, arXiv:2506.01710

    Pith/arXiv arXiv 2025

  27. [27]

    Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. 2025 a . Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. Advances in Neural Information Processing Systems, 37:25061--25094

    2025

  28. [28]

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025 b . https://arxiv.org/abs/2502.11886 Limr: Less is more for rl scaling . Preprint, arXiv:2502.11886

    Pith/arXiv arXiv 2025

  29. [29]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    Pith/arXiv arXiv 2019

  30. [30]

    Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. 2025. https://arxiv.org/abs/2412.04984 Frontier models are capable of in-context scheming . Preprint, arXiv:2412.04984

    Pith/arXiv arXiv 2025

  31. [31]

    OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, and 400 others. 2024 a . https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, ar...

    Pith/arXiv arXiv 2024

  32. [32]

    OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, and 243 others. 2024 b . https://arxiv.org/abs/2412.16720 Openai o1 system card ...

    Pith/arXiv arXiv 2024

  33. [33]

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. 2025. Tinyzero. https://github.com/Jiayi-Pan/TinyZero. Accessed: 2025-01-24

    2025

  34. [34]

    Edwin W Patterson. 1951. Case method in american legal education: Its origins and objectives, the. J. Legal Educ., 4:1

    1951

  35. [35]

    Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. https://arxiv.org/abs/2402.13950 Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning . Preprint, arXiv:2402.13950

    Pith/arXiv arXiv 2024

  36. [36]

    Qwen. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: The power of scaling rl

    2025

  37. [37]

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    Pith/arXiv arXiv 2025

  38. [38]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://api.semanticscholar.org/CorpusID:28695052 Proximal policy optimization algorithms . ArXiv, abs/1707.06347

    Pith/arXiv arXiv 2017

  39. [39]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. https://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Pith/arXiv arXiv 2024

  40. [40]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. https://doi.org/10.1145/3689031.3696075 Hybridflow: A flexible and efficient rlhf framework . In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297. ACM

    arXiv 2025

  41. [41]

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. https://arxiv.org/abs/2503.05592 R1-searcher: Incentivizing the search capability in llms via reinforcement learning . Preprint, arXiv:2503.05592

    Pith/arXiv arXiv 2025

  42. [42]

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. 2025. https://arxiv.org/abs/2503.23829 Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains . Preprint, arXiv:2503.23829

    Pith/arXiv arXiv 2025

  43. [43]

    Sijie Wang, Quanjiang Guo, Kai Zhao, Yawei Zhang, Xin Li, Xiang Li, Siqi Li, Rui She, Shangshu Yu, and Wee Peng Tay. 2025 a . https://arxiv.org/abs/2508.05242 Codeboost: Boosting code llms by squeezing knowledge from code snippets with rl . Preprint, arXiv:2508.05242

    Pith/arXiv arXiv 2025

  44. [44]

    Zezhong Wang, Xingshan Zeng, Weiwen Liu, Yufei Wang, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2025 b . https://arxiv.org/abs/2406.16144 Chain-of-probe: Examining the necessity and accuracy of cot step-by-step . Preprint, arXiv:2406.16144

    Pith/arXiv arXiv 2025

  45. [45]

    Lilian Weng. 2024. https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ Reward hacking in reinforcement learning. lilianweng.github.io

    2024

  46. [46]

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. 2025. https://doi.org/10.48550/arXiv.2502.14768 Logic- RL : Unleashing LLM Reasoning with Rule-Based Reinforcement Learning . Preprint, arXiv:2502.14768

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.14768 2025

  47. [47]

    Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang. 2025. https://arxiv.org/abs/2502.06772 Reasonflux: Hierarchical llm reasoning via scaling thought templates . Preprint, arXiv:2502.06772

    Pith/arXiv arXiv 2025

  48. [48]

    Hongzhou Yu, Tianhao Cheng, Yingwen Wang, Wen He, Qing Wang, Ying Cheng, Yuejie Zhang, Rui Feng, and Xiaobo Zhang. 2025. https://arxiv.org/abs/2501.09213 Finemedlm-o1: Enhancing medical knowledge reasoning ability of llm from supervised fine-tuning to test-time training . Preprint, arXiv:2501.09213

    Pith/arXiv arXiv 2025

  49. [49]

    Low, Eugene Ho Hong Zhuang, and Daren Zong Loong Tan

    Yang Zhao, Chengxiao Dai, Wei Zhuo, Tan Chuan Fu, Yue Xiu, Dusit Niyato, Jonathan Z. Low, Eugene Ho Hong Zhuang, and Daren Zong Loong Tan. 2025. https://arxiv.org/abs/2508.01815 Agentict ^2 s:robust text-to-sparql via agentic collaborative reasoning over heterogeneous knowledge graphs for the circular economy . Preprint, arXiv:2508.01815

    Pith/arXiv arXiv 2025

  50. [50]

    Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Jec-qa: a legal-domain question answering dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9701--9708

    2020

  51. [51]

    Ruochen Zhou, Minrui Xu, Shiqi Chen, Junteng Liu, Yunqi Li, Xinxin Lin, Zhengyu Chen, and Junxian He. 2025. https://doi.org/10.48550/arXiv.2507.04391 Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning ? Preprint, arXiv:2507.04391

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.04391 2025

  52. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  53. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...