Pith. sign in

REVIEW 4 major objections 5 minor 72 references

J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training with verifiable rewards, not fine-tuning, gives LLM judges a test-time scaling trend, and a 7B model trained this way outperforms prior open judges by 4.8%.

desk verdict A useful recipe for making a 7B judge improve with test-time compute, with a genuinely interesting RL-attribution finding — but the missing dedup analysis between the RL data and the eval benchmarks is the first thing to check. read the letter →

arxiv 2505.11875 v1 pith:6Y3UMYT7 submitted 2025-05-17 cs.LG cs.CL

Chi-Min Chan , Chunpu Xu , Jiaming Ji , Zhen Ye , Pengcheng Wen , Chunyang Jiang , Yaodong Yang , Wei Xue
show 2 more authors
Sirui Han Yike Guo
This is my paper · ORCID
classification cs.LGcs.CL
keywords LLM-as-a-Judgetest-timescalingreinforcementlearningverifiablerewardsreflectionpreferenceevaluationrewardmodelJ1-7B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that giving an LLM judge more inference-time compute is only useful if the model has been trained to use it. It claims that the ability to benefit from Simple Test-Time Scaling — forcing the model to reflect by appending "wait" tokens — is not innate, does not come from supervised fine-tuning on reflection examples, and is acquired mainly through reinforcement learning with a reward that simply checks the final verdict. To demonstrate this, the authors build J1-7B, a 7B judge trained with SFT on rejection-sampled reflection data followed by RL on the RISE English DPO subset. They report that J1-7B outperforms the previous best open-source judge by 4.8% across RewardBench, RewardMath, Anthropic Harmless, and CodePrefBench, and exhibits a 5.1% stronger scaling trend under STTS. If correct, this means test-time scaling is a trainable capability for evaluators, and the recipe is: verifiable-reward RL after reflective cold-start.

What carries the argument

The engine of the paper is Simple Test-Time Scaling (STTS), a budget-forcing trick adopted from prior work (Muennighoff et al., 2025) that replaces the closing </think> token with a reflective prompt such as "Wait," so the judge generates additional reasoning before committing to a verdict. The training pipeline that makes this effective consists of (1) rejection sampling with DeepSeek-R1 over HelpSteer2, OffsetBias, WildGuard, and Magpie to build an SFT set of correct reflection trajectories, including trajectories fixed by reflection; (2) RL (default Reinforce++, with PPO and GRPO ablations) on the RISE English DPO subset, with a reward of 1 if the predicted preference matches the ground truth and 0 otherwise; and (3) the paper's measurement device, the relative improvement metric ΔRelative%, which normalizes accuracy gains by the remaining headroom to 100% accuracy. The argument that RL is the source of scaling ability rests on tracking the STTS slope (r-values from linear regression of relative improvement across attempts) across RL checkpoints.

What would settle it

Run the same two-stage training recipe with an RL set constructed to be disjoint from RewardBench, RewardMath, Anthropic Harmless, and CodePrefBench (e.g., by n-gram dedup or by sourcing from different preference corpora) and check whether the 4.8% overall gain and the 5.1% scaling-trend advantage persist; alternatively, scan the RISE English DPO subset for near-duplicates of evaluation examples.

Watch

Extended reading notes

Core claim

The paper's central claim is that existing LLM-as-a-Judge models do not inherently exhibit a scaling trend under Simple Test-Time Scaling (STTS), and that fine-tuning on reflection-enhanced data alone is insufficient; significant scaling emerges primarily during RL training with a verifiable reward, and this is what makes J1-7B outperform previous open-source judges by 4.8% average accuracy while showing a 5.1% stronger scaling trend. The three key findings are (1) existing judges do not naturally scale with added reflective tokens, (2) SFT on reflection-enhanced datasets preserves that weak scaling, and (3) the scaling trend grows as RL training progresses, measured by the slope of relative improvement across checkpoints.

Load-bearing premise

The RL training data (the English DPO subset of RISE) is assumed not to overlap with the four evaluation benchmarks, but the paper reports no deduplication or leakage check; any overlap would inflate the reported 4.8% gain and the 5.1% scaling-trend advantage.

Editorial extensions

If this is right

  • A judge trained with verifiable-reward RL can outperform far larger or closed-source models on preference discrimination at the same inference budget.
  • STTS scaling shows diminishing or negative returns after roughly two "wait" tokens on some benchmarks (e.g., RewardBench), suggesting the budget must be tuned per task.
  • Because GRPO-trained models already produce long reflective chains, additional forced reflection adds little; algorithm choice affects whether test-time scaling pays off.
  • The scaling trend emerges progressively across RL checkpoints, so early stopping or insufficient RL steps would miss the STTS benefit.
  • The 5.1% stronger scaling trend refers to relative improvement over remaining headroom, not raw accuracy, so it is best read with the initial accuracy gap in mind.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not check for overlap between its RL training set (RISE English DPO subset) and the four evaluation benchmarks; if any examples appear in both, the 4.8% and 5.1% figures would be inflated. Auditing and re-measuring on a disjoint set is a concrete next step.
  • The same recipe could be tested on other verifiable supervision tasks, e.g., judging factuality with a retrieval check, where a 'verdict plus explanation' reward might produce even steeper scaling than 'wait' tokens alone.
  • The case studies show reflection sometimes flips a correct verdict to wrong by changing criteria; a policy that decides when to reflect (e.g., only when the model's confidence is low) could outperform the fixed budget-forcing schedule.
  • A testable extension is to run the same two-stage training on a leakage-free, independently sourced RL set (e.g., synthetic preference pairs not drawn from any evaluation benchmark) and confirm the gains persist.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes J1-7B, a 7B-parameter LLM-as-a-judge built on Qwen2.5-7B-Base, trained in two stages: first SFT on rejection-sampled, reflection-augmented preference data, then RL with a verifiable accuracy reward. At inference, the authors apply Simple Test-Time Scaling (STTS) by repeatedly injecting reflective tokens such as "wait" before the final verdict. Experiments on RewardBench, RewardMath, Anthropic Harmless, and CodePrefBench report that J1-7B surpasses the previous open-source state of the art, RISE-Judge, by 4.8% overall and shows a 5.1% stronger scaling trend under STTS. The paper also claims three findings: existing judges do not naturally scale with STTS, SFT alone does not impart this scaling, and the scaling capability emerges mainly during the RL phase. The appendix includes data construction statistics, additional ablations on RL data mixtures and algorithms, checkpoint analyses, case studies, and the prompt template.

Significance. If the central claims hold, the practical contribution is meaningful: a compact 7B judge that outperforms larger closed models on preference-pair accuracy, produces interpretable reasoning traces, and benefits from cheap test-time scaling. The checkpoint analysis in Section 4.7 is a useful empirical exploration of when STTS capability is acquired. The paper is also candid about limitations such as unfaithful reasoning and decision flips during reflection. However, the claimed 4.8% improvement and the RL-phase attribution rest on assumptions about train/eval separation and metric comparability that are not verified in the manuscript. The paper would be significantly strengthened by reporting confidence intervals, defining the scaling-trend measure, and addressing the overlap risk between RL training data and evaluation benchmarks. The significance is therefore conditional on these checks.

major comments (4)
  1. [§3.3, §4.1, Table 1] The paper never establishes that the RISE English DPO subset used for RL training is disjoint from the four evaluation benchmarks. RISE is an aggregated preference corpus, and RewardBench and Anthropic Harmless are widely used preference corpora, so overlap is a concrete risk. If any RewardBench, RewardMath, Anthropic Harmless, or CodePrefBench tuples appear in the RL training set, Eq. (3) rewards the model for matching test labels, which would inflate the Table 1 accuracy gain (75.98 vs. 71.17) and confound the Section 4.7 claim that STTS capability emerges during RL. The paper should list the sources and sizes of RISE's English DPO subset, perform exact-match and near-duplicate deduplication against all eval sets, and rerun the main experiments on a verifiably disjoint training set. Appendix F.1 ablates data mixtures but does not address evaluation overlap.
  2. [Appendix C, Table 1] The RewardMath evaluation deviates from the benchmark's standard scoring: the paper computes an instance-wise accuracy rather than problem-wise accuracy, calling the standard metric 'overly strict.' This raises the question of whether the baseline RewardMath numbers in Table 1 (e.g., RISE-Judge 81.69) were recomputed under the same instance-wise protocol or taken from prior work using problem-wise scoring. If the baselines were not recomputed, the comparison is unfair and the headline 4.8% improvement may be an artifact of the metric change. The authors should state explicitly which scoring protocol was used for each model, provide both instance-wise and problem-wise results, and re-evaluate the baselines under the same protocol.
  3. [Abstract, §4.4, Eq. (7)] The '5.1% stronger scaling trend' is never defined. Equation (7) defines relative improvement per STTS attempt, but no equation or procedure specifies what 'scaling trend' means (e.g., slope of relative improvement versus attempt count, area under the curve, or average gain per token) or how 5.1% is computed from the four benchmark curves. Without this definition, the abstract's central quantitative claim is unverifiable. The paper should define the scaling-trend metric in the main text and show the per-benchmark values that lead to 5.1%.
  4. [§4.7, Figure 6] The evidence for the claim that 'significant scaling trend emerges primarily during the RL phase' is statistically fragile. Figure 6b reports per-checkpoint Pearson correlations across only five STTS attempts, and these r-values fluctuate widely (e.g., for Anthropic Harmless, from -0.751 to 0.980) with a regression slope of 0.10 and r=0.47 across checkpoints. No confidence intervals, standard errors, or repeated-run variability are provided. The checkpoint trend is not clearly monotonic (Figure 6c also varies non-monotonically). The authors should provide error bars or confidence intervals, define what 'significant' means here, and apply a formal trend test across checkpoints. Otherwise the third key finding is not sufficiently supported.
minor comments (5)
  1. [§4.2, Table 1] The model name 'IntermLM2-7B-Reward' appears to be a typo for 'InternLM2-7B-Reward'; please correct it.
  2. [Figure 3] The figure labels are garbled, e.g., 'Δ = 1 . 3 5 5 . 0' and the y-axis formatting is inconsistent across panels. Please redraw the figure with clean labels and consistent axes.
  3. [§4.4 and §4.5] The notation 'Attempt 1' through 'Attempt 5' is used without a precise description of how many reflective tokens are appended at each attempt; please state the mapping explicitly (e.g., Attempt k means k-1 'wait' tokens appended) and report token counts.
  4. [Appendix A] The claim that STTS outputs are 'significantly greater diversity' than parallel decoding is supported only by PCA visualizations of 100 instances with no quantitative diversity metric or statistical test; please add a numerical measure (e.g., embedding distance distribution) and report the sample size.
  5. [Appendix H] The prompt template says 'Please first analysis both of the answer step by step' — this should be 'analyze'. Also ensure the template is verbatim identical across all baselines and describe how parsing of the '[[A]]' verdict was implemented.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: J1-7B's training and STTS claims are empirical and evaluated against external benchmarks; self-citations are not load-bearing.

full rationale

The paper's central claims are empirical: J1-7B is trained via SFT on rejection-sampled reflection data and RL with a verifiable reward, and then evaluated on four preference benchmarks. No equation in the paper defines the evaluation metric in terms of the training objective, and no fitted parameter is renamed as a prediction. The RL reward in Eq. (3) is a standard accuracy indicator on training tuples, while the STTS gains are measured on separate benchmark sets. Self-citations such as RQ-RAG and LLaSA appear only in related-work enumerations and do not support any load-bearing premise. The 'STTS emerges during RL' finding is supported by checkpoint-level regressions and ablations, not by construction. A possible data-contamination risk between the RISE English DPO subset and the evaluation benchmarks is a correctness or leakage concern, not circularity, because the paper does not demonstrate a specific overlap and no claimed prediction reduces to a training-set label by definition.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claims rest on the reliability of human preference labels, the absence of train/eval overlap between the RISE RL set and the four benchmarks, the modified RewardMath metric, and the assumption that 'wait' token injection produces genuine reflection. No mathematical derivation or invented entities are involved.

assumptions (4)
  • domain assumption Human preference labels in the SFT and RL datasets and in the evaluation benchmarks are correct and reliable ground truth for response quality.
    The verifiable reward in Eq. (3) is the indicator of agreement with human labels; if the labels are noisy or biased, accuracy numbers and RL training signals lose meaning.
  • domain assumption The RISE English DPO subset used for RL does not overlap with the four evaluation benchmarks.
    No deduplication is described between Section 3.3's RL data and Section 4.1's benchmarks; contamination would inflate the reported gains.
  • domain assumption Instance-wise scoring on RewardMath is a valid accuracy measure for judge preference.
    Appendix C states the paper deviates from the standard problem-wise RewardMath protocol; this changes the metric's denominator and can make results non-comparable to prior published RewardMath numbers.
  • domain assumption Appending 'wait' tokens at the end of the initial thinking phase elicits additional genuine reflection.
    Section 3.4 relies on this mechanism from Muennighoff et al. (2025) to attribute performance gains to test-time reflection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge." pith.science (2026). https://pith.science/paper/6Y3UMYT7

@misc{pith2026250511875,
  author       = {Pith},
  title        = {Pith review of: J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6Y3UMYT7}},
  note         = {Machine review of arXiv:2505.11875}
}
abstract

The current focus of AI research is shifting from emphasizing model training towards enhancing evaluation quality, a transition that is crucial for driving further advancements in AI systems. Traditional evaluation methods typically rely on reward models assigning scalar preference scores to outputs. Although effective, such approaches lack interpretability, leaving users often uncertain about why a reward model rates a particular response as high or low. The advent of LLM-as-a-Judge provides a more scalable and interpretable method of supervision, offering insights into the decision-making process. Moreover, with the emergence of large reasoning models, which consume more tokens for deeper thinking and answer refinement, scaling test-time computation in the LLM-as-a-Judge paradigm presents an avenue for further boosting performance and providing more interpretability through reasoning traces. In this paper, we introduce $\textbf{J1-7B}$, which is first supervised fine-tuned on reflection-enhanced datasets collected via rejection-sampling and subsequently trained using Reinforcement Learning (RL) with verifiable rewards. At inference time, we apply Simple Test-Time Scaling (STTS) strategies for additional performance improvement. Experimental results demonstrate that $\textbf{J1-7B}$ surpasses the previous state-of-the-art LLM-as-a-Judge by $ \textbf{4.8}$\% and exhibits a $ \textbf{5.1}$\% stronger scaling trend under STTS. Additionally, we present three key findings: (1) Existing LLM-as-a-Judge does not inherently exhibit such scaling trend. (2) Model simply fine-tuned on reflection-enhanced datasets continues to demonstrate similarly weak scaling behavior. (3) Significant scaling trend emerges primarily during the RL phase, suggesting that effective STTS capability is acquired predominantly through RL training.

Figures

Figures reproduced from arXiv: 2505.11875 by the authors.

Figure 1
Figure 1. Comparison of Bradley-Terry model and LLM-as-a-Judge under different scaling strategies. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of J1-7B. We first curate the SFT dataset through rejection sampling and subse￾quently apply RL training to integrate STTS capabilities into J1-7B . of alignment. We believe this approach represents a key step toward achieving scalable oversight in AI systems. Furthermore, it offers a more interpretable evaluation framework, as it traces the model’s thought process and how its reasoning evolves, providing v… view at source ↗
Figure 3
Figure 3. Scaling trend for STTS on four different tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Cold start on reasoning-intensive data improves STTS. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Scaling trend for STTS on Anthropic Harmless and CodePrefBench with different RL [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Scaling behaviour of different checkpoints on Anthropic Harmless. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Distribution patterns between STTS and Parallel Decoding. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Length distribution of the curated dataset. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Reflective words frequency of the curated dataset. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Word cloud of the curated dataset. • Anthropic Harmless (Bai et al., 2022) is a benchmark used for evaluating models’ harm￾lessness. During the data collection process, human annotators are encouraged to provoke LLMs into generating harmful responses and to identify w…
Figure 11
Figure 11. Figure 11: Scaling trend for STTS on RewardBench and RewardMATH with different RL algorithms. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Scaling behaviour of different checkpoints on CodePrefBench. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Scaling behaviour of different checkpoints on RewardBench. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Scaling behaviour of different checkpoints on RewardMATH. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Data mixture ablation on Anthropic Harmless and CodePrefBench. [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Data mixture ablation on RewardBench and RewardMATH. [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Decision change during STTS. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Reflective words frequency on four benchmark. [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Performance conditioned on differenct reflective counts. [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: Case study 1 for Initial Wrong → Correct after reflection. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]
Figure 21
Figure 21. Figure 21: Case study 2 for Initial Wrong → Correct after reflection. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]
Figure 22
Figure 22. Figure 22: Case study 1 for Initial Correct → Wrong after reflection. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_22.png]
Figure 23
Figure 23. Figure 23: Case study 2 for Initial Correct → Wrong after reflection. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_23.png]
Figure 24
Figure 24. Figure 24: Case study 1 for Consistently Correct. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_24.png]
Figure 25
Figure 25. Figure 25: Case study 2 for Consistently Correct. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_25.png]
Figure 26
Figure 26. Figure 26: Case study 1 for Consistently Wrong [PITH_FULL_IMAGE:figures/full_fig_p032_26.png]
Figure 27
Figure 27. Figure 27: shows the prompt template that we use for the evaluation. By default, we use it for all the models we evaluate in this paper. It is derived from RewardBench (Lambert et al., 2024). Prompt Template [PITH_FULL_IMAGE:figures/full_fig_p033_27.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 8 canonical work pages

  1. [1]

    Critique-out-loud reward models

    Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprint arXiv:2408.11791, 2024

    arXiv 2024

  2. [2]

    Chain-of-thought reasoning in the wild is not always faithful

    Iv \'a n Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild is not always faithful. arXiv preprint arXiv:2503.08679, 2025

    arXiv 2025

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  4. [4]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  5. [5]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

  6. [6]

    Internlm2 technical report, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, S...

    2024

  7. [7]

    Rq-rag: Learning to refine queries for retrieval augmented generation

    Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.00610, 2024

    arXiv 2024

  8. [8]

    Codemonkeys: Scaling test-time compute for software engineering

    Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher R \'e , and Azalia Mirhoseini. Codemonkeys: Scaling test-time compute for software engineering. arXiv preprint arXiv:2501.14723, 2025

    arXiv 2025

Show all 72 references
  1. [9]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp.\ 10835--10866. PMLR, 2023

    2023

  2. [10]

    Gemini 2.0 flash thinking mode

    Google. Gemini 2.0 flash thinking mode. https://cloud.google.com/vertex-ai/generative-ai/docs/thinking-mode, December 2024. Accessed: 2025-04-29

    2024

  3. [11]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [12]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint arXiv:2406.18495, 2024

    2024 arXiv

  7. [15]

    Metrics for explainable ai: Challenges and prospects

    Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. Metrics for explainable ai: Challenges and prospects. arXiv preprint arXiv:1812.04608, 2018

    2018 arXiv

  8. [16]

    Human feedback is not gold standard

    Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard. arXiv preprint arXiv:2309.16349, 2023

    2023 arXiv

  9. [17]

    Reinforce++: A simple and efficient approach for aligning large language models

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  10. [18]

    open-r1/openr1-math-220k

    HuggingFace. open-r1/openr1-math-220k. https://huggingface.co/datasets/open-r1/OpenR1-Math-220k, 2025. Accessed: 2025-04-29

    2025

  11. [19]

    Pku-saferlhf: Towards multi-level safety alignment for llms with human preference

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513, 2024

    2024 arXiv

  12. [20]

    On scalable oversight with weak llms judging strong llms

    Zachary Kenton, Noah Siegel, J \'a nos Kram \'a r, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah Goodman, et al. On scalable oversight with weak llms judging strong llms. Advances in Neural Information Processing Systems, 3...

    2024

  13. [21]

    Evaluating robustness of reward models for mathematical reasoning

    Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. Evaluating robustness of reward models for mathematical reasoning. arXiv preprint arXiv:2410.01729, 2024

    2024 arXiv

  14. [22]

    The past, present and better future of feedback learning in large language models for subjective human preferences and values

    Hannah Rose Kirk, Andrew M Bean, Bertie Vidgen, Paul R \"o ttger, and Scott A Hale. The past, present and better future of feedback learning in large language models for subjective human preferences and values. arXiv preprint arXiv:2310.07629, 2023

    arXiv

  15. [23]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  16. [24]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594, 2024

    2024

  17. [25]

    Generative judge for evaluating alignment

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023

    2023 arXiv

  18. [26]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  19. [27]

    Skywork-reward: Bag of tricks for reward modeling in llms

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451, 2024 a

    2024 arXiv

  20. [28]

    Video-t1: Test-time scaling for video generation

    Fangfu Liu, Hanyang Wang, Yimo Cai, Kaiyan Zhang, Xiaohang Zhan, and Yueqi Duan. Video-t1: Test-time scaling for video generation. arXiv preprint arXiv:2503.18942, 2025 a

    2025 arXiv

  21. [29]

    Learning code preference via synthetic evolution

    Jiawei Liu, Thanh Nguyen, Mingyue Shang, Hantian Ding, Xiaopeng Li, Yu Yu, Varun Kumar, and Zijian Wang. Learning code preference via synthetic evolution. arXiv preprint arXiv:2410.03837, 2024 b

    2024 arXiv

  22. [30]

    Inference-time scaling for generalist reward modeling

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495, 2025 b

    2025

  23. [31]

    Principal components analysis (pca)

    Andrzej Ma \'c kiewicz and Waldemar Ratajczak. Principal components analysis (pca). Computers & Geosciences, 19 0 (3): 0 303--342, 1993

    1993

  24. [32]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36: 0 46534--46594, 2023

    2023

  25. [33]

    Llm critics help catch llm bugs

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215, 2024

    2024 arXiv

  26. [34]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  27. [35]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/, September 2024. Accessed: 2025-04-29

    2024

  28. [36]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  29. [37]

    Offsetbias: Leveraging debiased data for tuning evaluators

    Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators. arXiv preprint arXiv:2407.06551, 2024

    2024 arXiv

  30. [38]

    Codeforces cots

    Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces cots. https://huggingface.co/datasets/open-r1/codeforces-cots, 2025

    2025

  31. [39]

    Learning to plan & reason for evaluation with thinking-llm-as-a-judge

    Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-llm-as-a-judge. arXiv preprint arXiv:2501.18099, 2025

    2025 arXiv

  32. [40]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  33. [41]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  34. [42]

    Rethinking reflection in pre-training

    Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre-training. arXiv preprint arXiv:2504.04022, 2025

    2025 arXiv

  35. [43]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  36. [44]

    Skywork critic model series

    Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model series. https://huggingface.co/Skywork, September 2024. URL https://huggingface.co/Skywork

    2024

  37. [45]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  38. [46]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  39. [47]

    Openr: An open source framework for advanced reasoning with large language models

    Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671, 2024 a

    arXiv

  40. [48]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33: 0 5776--5788, 2020

    2020

  41. [49]

    Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023

    2023 arXiv

  42. [50]

    Helpsteer2: Open-source dataset for training top-performing reward models

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024 b

    2024 arXiv

  43. [51]

    Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences

    Martin Weyssow, Aton Kamanda, Xin Zhou, and Houari Sahraoui. Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences. arXiv preprint arXiv:2403.09032, 2024

    2024 arXiv

  44. [52]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594, 2024 a

    2024 arXiv

  45. [53]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024 b

    2024 arXiv

  46. [54]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024

    2024

  47. [55]

    Llava-o1: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440, 2024 a

    2024 arXiv

  48. [56]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024 b

    2024 arXiv

  49. [57]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  50. [58]

    Mastering complex control in moba games with deep reinforcement learning

    Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp....

    2020

  51. [59]

    Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis

    Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi DAI, et al. Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis. arXiv preprint arXiv:2502.04128, 2025

    2025 arXiv

  52. [60]

    Improving reward models with synthetic critiques

    Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gall \'e . Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.20850, 2024

    2024 arXiv

  53. [61]

    Improve llm-as-a-judge ability as a general ability

    Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li. Improve llm-as-a-judge ability as a general ability. arXiv preprint arXiv:2502.11689, 2025 a

    2025 arXiv

  54. [62]

    Self-generated critiques boost reward modeling for language models

    Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.16646, 2024

    2024 arXiv

  55. [63]

    Z1: Efficient test-time scaling with code

    Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Z1: Efficient test-time scaling with code. arXiv preprint arXiv:2504.00810, 2025 b

    2025 arXiv

  56. [64]

    Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning

    Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884, 2024 a

    arXiv

  57. [65]

    Openprm: Building open-domain process-based reward models with preference trees

    Kaiyan Zhang, Jiayuan Zhang, Haoxin Li, Xuekai Zhu, Ermo Hua, Xingtai Lv, Ning Ding, Biqing Qi, and Bowen Zhou. Openprm: Building open-domain process-based reward models with preference trees. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  58. [66]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024 b

    2024 arXiv

  59. [67]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  60. [68]

    Judgelm: Fine-tuned large language models are scalable judges

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023

    2023 arXiv

  61. [69]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  62. [70]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  63. [71]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  64. [72]

    wait" step across four benchmark tasks. Because the proportion of responses altered after adding each

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.