{"id":"fd2691b6-4ec7-412c-8846-c5931322af1f","arxiv_id":"2505.11875","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"J1-7B, a judge LLM trained with supervised fine-tuning and reinforcement learning, improves when forced to reflect with 'wait' tokens, and the scaling ability emerges during the RL phase.","lead":"The paper trains a 7-billion-parameter language model judge, J1-7B, that gets better at picking the preferred answer when forced to 'wait' and reconsider during inference. It reports that this test-time scaling ability mostly comes from reinforcement learning, not just supervised fine-tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No deduplication between RL data (RISE English DPO) and the four eval benchmarks; if RISE contains RewardBench/Harmless tuples, the 4.8% gain and the RL-phase scaling finding are inflated.","rationale":"The reader's weakest assumption is exactly the concern I would put first: train/eval overlap through the RISE English DPO subset used for RL. This is the most load-bearing because it directly attacks the headline 4.8% improvement and the RL-phase attribution, not just presentation details. The paper's own text shows the RL reward (Eq. 3) is a 0/1 match to r_true on the RISE English DPO subset, and the eval sets include standard preference corpora whose examples could plausibly be inside RISE. No deduplication is reported anywhere, including Appendix F.1. If overlap exists, Table 1's comparison against RISE-Judge is unfair in J1's favor, and the checkpoint progression in Figures 6 and 12-14 could be explained by memorization rather than learned reflection. I do not treat this as evidence of fraud; it is an unaddressed methodological risk. The correct response is to keep the paper conditional pending the overlap check. I would not move to REJECT because overlap is not established, and I would not move to ACCEPT because the check is missing. Thus the verdict remains conditional, matching the reader's assessment. Other issues, such as the undefined 'scaling trend' metric, the modified RewardMath instance-wise scoring, the absence of error bars, and the unreleased code and weights, are real but secondary; the deduplication check is the one that can settle the central claim.","tokens_in":30943,"tokens_out":3838,"duration_ms":38400,"concrete_test":"Download the RISE English DPO subset and the four benchmark test sets; canonicalize prompts and response pairs (normalize whitespace, strip formatting, and apply MinHash or exact hashing for near-duplicate detection); flag all exact and near-duplicate matches. Then recompute Table 1 and Figure 3 after removing overlapping instances from the evaluation sets. If overlap is nonzero and J1-7B's 4.8% overall gap or its 5.1% STTS scaling-trend advantage shrinks materially when the overlapping instances are excluded, the headline results are contaminated by train/test leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the RL training distribution does not contain the evaluation labels that Eq. (3) rewards. Section 3.3 says the RL phase uses 'the english DPO subset from the RISE dataset' and Section 4.1 evaluates on RewardBench, RewardMath, Anthropic Harmless, and CodePrefBench, but the paper gives no list of RISE's constituent sources and no deduplication or leakage analysis. RISE is an aggregated preference corpus, and RewardBench and Anthropic Harmless are standard preference corpora, so overlap is a concrete risk. If even a fraction of RewardBench or Harmless tuples appear in the English DPO subset, the reward signal during RL is partly computed from test labels. That would inflate the Table 1 overall accuracy (J1-7B SFT+RL 75.98 vs RISE-Judge 71.17; the claimed 4.8% improvement) and would also confound the paper's third finding (Section 4.7) that STTS capability emerges during RL: the 'learning' could be memorization of eval instances rather than acquisition of reflective judgment. This concern is checkable and is not resolved by the paper's appendices; Appendix F.1 only ablates RL data mixtures, not eval overlap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes J1-7B, a 7B-parameter LLM-as-a-judge built on Qwen2.5-7B-Base, trained in two stages: first SFT on rejection-sampled, reflection-augmented preference data, then RL with a verifiable accuracy reward. At inference, the authors apply Simple Test-Time Scaling (STTS) by repeatedly injecting reflective tokens such as \"wait\" before the final verdict. Experiments on RewardBench, RewardMath, Anthropic Harmless, and CodePrefBench report that J1-7B surpasses the previous open-source state of the art, RISE-Judge, by 4.8% overall and shows a 5.1% stronger scaling trend under STTS. The paper also claims three findings: existing judges do not naturally scale with STTS, SFT alone does not impart this scaling, and the scaling capability emerges mainly during the RL phase. The appendix includes data construction statistics, additional ablations on RL data mixtures and algorithms, checkpoint analyses, case studies, and the prompt template.","tokens_in":31197,"tokens_out":3632,"duration_ms":39212,"significance":"If the central claims hold, the practical contribution is meaningful: a compact 7B judge that outperforms larger closed models on preference-pair accuracy, produces interpretable reasoning traces, and benefits from cheap test-time scaling. The checkpoint analysis in Section 4.7 is a useful empirical exploration of when STTS capability is acquired. The paper is also candid about limitations such as unfaithful reasoning and decision flips during reflection. However, the claimed 4.8% improvement and the RL-phase attribution rest on assumptions about train/eval separation and metric comparability that are not verified in the manuscript. The paper would be significantly strengthened by reporting confidence intervals, defining the scaling-trend measure, and addressing the overlap risk between RL training data and evaluation benchmarks. The significance is therefore conditional on these checks.","major_comments":[{"comment":"The paper never establishes that the RISE English DPO subset used for RL training is disjoint from the four evaluation benchmarks. RISE is an aggregated preference corpus, and RewardBench and Anthropic Harmless are widely used preference corpora, so overlap is a concrete risk. If any RewardBench, RewardMath, Anthropic Harmless, or CodePrefBench tuples appear in the RL training set, Eq. (3) rewards the model for matching test labels, which would inflate the Table 1 accuracy gain (75.98 vs. 71.17) and confound the Section 4.7 claim that STTS capability emerges during RL. The paper should list the sources and sizes of RISE's English DPO subset, perform exact-match and near-duplicate deduplication against all eval sets, and rerun the main experiments on a verifiably disjoint training set. Appendix F.1 ablates data mixtures but does not address evaluation overlap.","section":"§3.3, §4.1, Table 1"},{"comment":"The RewardMath evaluation deviates from the benchmark's standard scoring: the paper computes an instance-wise accuracy rather than problem-wise accuracy, calling the standard metric 'overly strict.' This raises the question of whether the baseline RewardMath numbers in Table 1 (e.g., RISE-Judge 81.69) were recomputed under the same instance-wise protocol or taken from prior work using problem-wise scoring. If the baselines were not recomputed, the comparison is unfair and the headline 4.8% improvement may be an artifact of the metric change. The authors should state explicitly which scoring protocol was used for each model, provide both instance-wise and problem-wise results, and re-evaluate the baselines under the same protocol.","section":"Appendix C, Table 1"},{"comment":"The '5.1% stronger scaling trend' is never defined. Equation (7) defines relative improvement per STTS attempt, but no equation or procedure specifies what 'scaling trend' means (e.g., slope of relative improvement versus attempt count, area under the curve, or average gain per token) or how 5.1% is computed from the four benchmark curves. Without this definition, the abstract's central quantitative claim is unverifiable. The paper should define the scaling-trend metric in the main text and show the per-benchmark values that lead to 5.1%.","section":"Abstract, §4.4, Eq. (7)"},{"comment":"The evidence for the claim that 'significant scaling trend emerges primarily during the RL phase' is statistically fragile. Figure 6b reports per-checkpoint Pearson correlations across only five STTS attempts, and these r-values fluctuate widely (e.g., for Anthropic Harmless, from -0.751 to 0.980) with a regression slope of 0.10 and r=0.47 across checkpoints. No confidence intervals, standard errors, or repeated-run variability are provided. The checkpoint trend is not clearly monotonic (Figure 6c also varies non-monotonically). The authors should provide error bars or confidence intervals, define what 'significant' means here, and apply a formal trend test across checkpoints. Otherwise the third key finding is not sufficiently supported.","section":"§4.7, Figure 6"}],"minor_comments":[{"comment":"The model name 'IntermLM2-7B-Reward' appears to be a typo for 'InternLM2-7B-Reward'; please correct it.","section":"§4.2, Table 1"},{"comment":"The figure labels are garbled, e.g., 'Δ = 1 . 3 5 5 . 0' and the y-axis formatting is inconsistent across panels. Please redraw the figure with clean labels and consistent axes.","section":"Figure 3"},{"comment":"The notation 'Attempt 1' through 'Attempt 5' is used without a precise description of how many reflective tokens are appended at each attempt; please state the mapping explicitly (e.g., Attempt k means k-1 'wait' tokens appended) and report token counts.","section":"§4.4 and §4.5"},{"comment":"The claim that STTS outputs are 'significantly greater diversity' than parallel decoding is supported only by PCA visualizations of 100 instances with no quantitative diversity metric or statistical test; please add a numerical measure (e.g., embedding distance distribution) and report the sample size.","section":"Appendix A"},{"comment":"The prompt template says 'Please first analysis both of the answer step by step' — this should be 'analyze'. Also ensure the template is verbatim identical across all baselines and describe how parsing of the '[[A]]' verdict was implemented.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and useful topic, and the core idea is plausible, but the two main quantitative claims currently hinge on unresolved leakage and metric-comparability questions. I recommend major revision rather than rejection because the issues are checkable: the authors can audit RISE's composition, deduplicate, recompute RewardMath under a consistent protocol, and define the scaling-trend measure. I would also encourage the editor to ask for confidence intervals on the headline numbers before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper has one genuinely new and useful finding: training a judge with SFT on reflection-augmented data and then RL with a verifiable reward produces a model whose accuracy improves when you append \"wait\" tokens, and the scaling ability appears during the RL stage, not the SFT stage. Second, the headline numbers should be treated as provisional until the authors show that the RISE English DPO subset does not overlap with the four evaluation benchmarks. That is a real, load-bearing concern, not an edge case.\n\nWhat the paper does well: it takes s1-style test-time scaling and RLVR, both known ingredients, and applies them to a domain where they hadn't been systematically tested. The checkpoint analysis in Section 4.7 — tracking STTS slope/correlation across RL training — is a direct way to test the attribution claim, and the ablations on RL algorithms and data mixtures are informative. The case studies are honest: they show unfaithful reasoning and cases where reflection flips a correct answer to wrong. For a 7B model, the empirical breadth is solid.\n\nThe soft spots, in proportion. The missing dedup/leakage analysis is the main one. RISE is an aggregated preference corpus, and Anthropic Harmless and RewardBench are standard preference collections; the paper gives no source list for RISE and no overlap check. If even a fraction of those eval tuples are in the RL training set, Eq. (3)'s reward is partly computed from test labels, which would inflate the 4.8% gain and confound the \"RL imparts scaling\" finding. This is checkable and the appendices don't address it. Second, the RewardMath metric is instance-wise rather than problem-wise (Appendix C); baselines in Table 1 may not have been scored the same way. The paper should state clearly whether all models were evaluated with the same protocol. Third, the \"5.1% stronger scaling trend\" is a headline claim but no formal definition is given; define the slope/metric in the main text. Fourth, no confidence intervals anywhere, which matters for a scaling claim. Finally, no code or weights are released; for a recipe paper that is a real limitation, though not a scientific error.\n\nThe central approach holds up. The RL-phase attribution is plausible and well-motivated, not circular. The paper deserves a serious referee. I would make the dedup analysis a mandatory part of the revision, and the metric definitions and error bars minor/moderate fixes. It will be useful for people working on LLM-as-judge and scalable oversight, provided the numbers survive scrutiny.","headline":"A useful recipe for making a 7B judge improve with test-time compute, with a genuinely interesting RL-attribution finding — but the missing dedup analysis between the RL data and the eval benchmarks is the first thing to check.","tokens_in":31762,"tokens_out":3801,"would_cite":true,"duration_ms":38950,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training with verifiable rewards, not fine-tuning, gives LLM judges a test-time scaling trend, and a 7B model trained this way outperforms prior open judges by 4.8%.","keywords":["LLM-as-a-Judge","test-time scaling","reinforcement learning","verifiable rewards","reflection","preference evaluation","reward model","J1-7B"],"falsifier":"Run the same two-stage training recipe with an RL set constructed to be disjoint from RewardBench, RewardMath, Anthropic Harmless, and CodePrefBench (e.g., by n-gram dedup or by sourcing from different preference corpora) and check whether the 4.8% overall gain and the 5.1% scaling-trend advantage persist; alternatively, scan the RISE English DPO subset for near-duplicates of evaluation examples.","tokens_in":30767,"feed_emoji":"⚖️","tokens_out":5978,"duration_ms":53606,"temperature":0.7,"pith_summary":"The paper sets out to show that giving an LLM judge more inference-time compute is only useful if the model has been trained to use it. It claims that the ability to benefit from Simple Test-Time Scaling — forcing the model to reflect by appending \"wait\" tokens — is not innate, does not come from supervised fine-tuning on reflection examples, and is acquired mainly through reinforcement learning with a reward that simply checks the final verdict. To demonstrate this, the authors build J1-7B, a 7B judge trained with SFT on rejection-sampled reflection data followed by RL on the RISE English DPO subset. They report that J1-7B outperforms the previous best open-source judge by 4.8% across RewardBench, RewardMath, Anthropic Harmless, and CodePrefBench, and exhibits a 5.1% stronger scaling trend under STTS. If correct, this means test-time scaling is a trainable capability for evaluators, and the recipe is: verifiable-reward RL after reflective cold-start.","feed_headline":"RL, not fine-tuning, makes LLM judges scale at test time","feed_subtitle":"A 7B judge trained with verifiable rewards beats prior open critics and gains more from extra 'wait' tokens.","key_machinery":"The engine of the paper is Simple Test-Time Scaling (STTS), a budget-forcing trick adopted from prior work (Muennighoff et al., 2025) that replaces the closing </think> token with a reflective prompt such as \"Wait,\" so the judge generates additional reasoning before committing to a verdict. The training pipeline that makes this effective consists of (1) rejection sampling with DeepSeek-R1 over HelpSteer2, OffsetBias, WildGuard, and Magpie to build an SFT set of correct reflection trajectories, including trajectories fixed by reflection; (2) RL (default Reinforce++, with PPO and GRPO ablations) on the RISE English DPO subset, with a reward of 1 if the predicted preference matches the ground truth and 0 otherwise; and (3) the paper's measurement device, the relative improvement metric ΔRelative%, which normalizes accuracy gains by the remaining headroom to 100% accuracy. The argument that RL is the source of scaling ability rests on tracking the STTS slope (r-values from linear regression of relative improvement across attempts) across RL checkpoints.","core_discovery":"The paper's central claim is that existing LLM-as-a-Judge models do not inherently exhibit a scaling trend under Simple Test-Time Scaling (STTS), and that fine-tuning on reflection-enhanced data alone is insufficient; significant scaling emerges primarily during RL training with a verifiable reward, and this is what makes J1-7B outperform previous open-source judges by 4.8% average accuracy while showing a 5.1% stronger scaling trend. The three key findings are (1) existing judges do not naturally scale with added reflective tokens, (2) SFT on reflection-enhanced datasets preserves that weak scaling, and (3) the scaling trend grows as RL training progresses, measured by the slope of relative improvement across checkpoints.","pith_inferences":["The paper does not check for overlap between its RL training set (RISE English DPO subset) and the four evaluation benchmarks; if any examples appear in both, the 4.8% and 5.1% figures would be inflated. Auditing and re-measuring on a disjoint set is a concrete next step.","The same recipe could be tested on other verifiable supervision tasks, e.g., judging factuality with a retrieval check, where a 'verdict plus explanation' reward might produce even steeper scaling than 'wait' tokens alone.","The case studies show reflection sometimes flips a correct verdict to wrong by changing criteria; a policy that decides when to reflect (e.g., only when the model's confidence is low) could outperform the fixed budget-forcing schedule.","A testable extension is to run the same two-stage training on a leakage-free, independently sourced RL set (e.g., synthetic preference pairs not drawn from any evaluation benchmark) and confirm the gains persist."],"forward_implications":["A judge trained with verifiable-reward RL can outperform far larger or closed-source models on preference discrimination at the same inference budget.","STTS scaling shows diminishing or negative returns after roughly two \"wait\" tokens on some benchmarks (e.g., RewardBench), suggesting the budget must be tuned per task.","Because GRPO-trained models already produce long reflective chains, additional forced reflection adds little; algorithm choice affects whether test-time scaling pays off.","The scaling trend emerges progressively across RL checkpoints, so early stopping or insufficient RL steps would miss the STTS benefit.","The 5.1% stronger scaling trend refers to relative improvement over remaining headroom, not raw accuracy, so it is best read with the initial accuracy gap in mind."],"supporting_citations":[{"why":"Supplies the budget-forcing technique (appending 'wait' tokens) that the paper adopts as Simple Test-Time Scaling for judges.","marker":"Muennighoff et al., 2025"},{"why":"Provides the RISE English DPO subset used as RL training data and the RISE-Judge baseline that J1-7B surpasses.","marker":"Yu et al., 2025a"},{"why":"Generates the reflection trajectories used to construct the SFT dataset via rejection sampling.","marker":"Guo et al., 2025"},{"why":"Contributes RewardBench, one of the four evaluation benchmarks, and the prompt template used for pairwise judging.","marker":"Lambert et al., 2024"},{"why":"Contributes Anthropic Harmless, the benchmark used to measure judge accuracy on safety preferences.","marker":"Bai et al., 2022"},{"why":"Contributes RewardMath, the benchmark for judge accuracy on mathematical reasoning preferences.","marker":"Kim et al., 2024"},{"why":"Contributes CodePrefBench, the benchmark for judge accuracy on code preferences.","marker":"Liu et al., 2024b"}],"fun_headline_variants":["RL unlocks test-time scaling for LLM judges, not SFT","Only RL makes LLM-as-a-judge scale with extra tokens","J1-7B: RL gives judges a 5.1% stronger scaling trend","Test-time scaling for judges emerges only after RL","Why fine-tuning alone can't give LLM judges test-time scaling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The RL training data (the English DPO subset of RISE) is assumed not to overlap with the four evaluation benchmarks, but the paper reports no deduplication or leakage check; any overlap would inflate the reported 4.8% gain and the 5.1% scaling-trend advantage.","fun_headline_variants_meta":{"raw":{"variants":["RL unlocks test-time scaling for LLM judges, not SFT","Only RL makes LLM-as-a-judge scale with extra tokens","J1-7B: RL gives judges a 5.1% stronger scaling trend","Test-time scaling for judges emerges only after RL","Why fine-tuning alone can't give LLM judges test-time scaling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3349,"prompt_tokens":1006,"completion_tokens":2343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":2265}},"tokens_in":622,"tokens_out":2343,"duration_ms":17085,"temperature":1.0,"reasoning_tokens":2265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:46:04.203938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two-stage training recipe with an RL set constructed to be disjoint from RewardBench, RewardMath, Anthropic Harmless, and CodePrefBench (e.g., by n-gram dedup or by sourcing from different preference corpora) and check whether the 4.8% overall gain and the 5.1% scaling-trend advantage persist; alternatively, scan the RISE English DPO subset for near-duplicates of evaluation examples.","supporting_citations":[],"review_version":1}