Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read A 7B judge trained with EIS-GRPO achieves order-invariance by pooling rewards across swapped-order versions of each comparison, and this makes it outperform much larger judges on reasoning benchmarks.

desk verdict A solid, useful judge-training recipe with a new benchmark; the one real soft spot is a missing local-only ablation that leaves the mechanism claim under-supported. read the letter →

arxiv 2505.13346 v3 pith:RNC25EC6 submitted 2025-05-19 cs.CL cs.AI

classification cs.CLcs.AI
keywords LLM-as-judgereinforcementlearningGRPOpositionalbiaspairwiseevaluationreasoningstateequivalenceJudgeBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a judge model's poor performance on reasoning tasks is largely positional inconsistency, and that a reinforcement-learning objective can remove that inconsistency directly. The proposed algorithm, EIS-GRPO, treats the two orderings of a pairwise comparison as equivalent initial states, samples a group of judgments from each ordering, and computes advantages jointly across both groups before updating the policy. Trained this way, a 7B judge called J4R-CJ-7B reaches 50.35 average accuracy across three reasoning-evaluation benchmarks, higher than every judge at or below 14B parameters and close to a 32B GRPO-trained judge. On JudgeBench, EIS-GRPO improves accuracy over a balanced-data GRPO baseline by 9.4% and over a duplicated-data GRPO baseline by 11.2%, while raising consistency from 68.86 to 81.14. If the claim holds, order-invariance is not a side effect of good judging but a cause of it.

What carries the argument

The central mechanism is the two-part advantage estimate of EIS-GRPO (Eq. 4). Starting from an input $q$, the algorithm forms $L$ transformed versions $T_\ell(q)$ (for pairwise judging, the two orderings of the candidate responses), samples $G/L$ outputs from each, and gives output $(i,\ell)$ the advantage $$\hat{A}^{(i,\ell)} = \frac{$R^{{(i,\ell)}}$-\bar{R}_{[L]}}{\sigma_{R_{[L]}}} + \frac{$R^{{(i,\ell)}}$-\bar{R}_{\ell}}{\sigma_{R_\ell}},$$ where the first term is normalized across all subgroups and the second within subgroup $\ell$. The policy update is conditioned on $T_\ell(q)$, so the model is rewarded for correct judgments under each ordering rather than for matching outputs across orderings. The two terms do complementary work: the global term asserts that the transformed states are equivalent, and the local term prevents a weak subgroup's best responses from being undervalued because stronger responses exist in another subgroup.

What would settle it

Build a held-out set of pairwise samples in which the two orderings genuinely are not equivalent, for instance a rule stating that when both responses are acceptable the one listed first should win, and train two judges on the same data, one with EIS-GRPO and one with plain GRPO. If the EIS-GRPO judge is less accurate on that set because it was forced to treat the two states as equivalent, the central premise is refuted; if it still matches or beats GRPO, the equivalence assumption passes a stricter test than the paper's order-swap design.

Watch

Extended reading notes

Core claim

The paper's central claim is that judges fail in reasoning evaluation in a specific, measurable way: when the order of two candidate responses is swapped, the judge's verdict often swaps too, and training with EIS-GRPO turns this failure into a learning signal. For a pairwise sample $(x, y_1, y_2)$, the better response is fixed by an outcome label, and the two input orderings $(A=y_1, B=y_2)$ and $(A=y_2, B=y_1)$ are the equivalent initial states; the correct answer relabels from $A$ to $B$ across them. The model is updated with an advantage computed jointly across both orderings, so it must treat the two states as equivalent to be rewarded. The paper reports that J4R-CJ-7B, a 7B judge initialized from CompassJudger-7B, scores 50.35 average accuracy across PPE, JudgeBench, and ReasoningJudgeBench, the best among all judges at or below 14B and within 1.5 points of the best 32B judge evaluated; it also introduces ReasoningJudgeBench, a 1,483-sample benchmark spanning math, multi-hop, domain-specific, and everyday reasoning, where most existing judges fail to reach 45%.

Load-bearing premise

The load-bearing premise is that flipping which response is presented first and which is presented second leaves the judging task substantively unchanged, so the two orderings can be pooled in one reward computation; if position itself changed the correct verdict, EIS-GRPO would reward the model for ignoring a real signal.

Editorial extensions

If this is right

  • If EIS-GRPO is right, judge training no longer needs a doubled dataset: swapped-order responses are used as subgroups of the same batch, so the model sees both orders without extra storage or wall-clock cost beyond sampling $G$ responses.
  • The same subgroup-advantage structure is a direct recipe for other transformation-invariant RLVR tasks, including paraphrases, multiple-choice option orders, and any input transformation whose correct answer relabels consistently.
  • Because the training signal punishes disagreement between orderings, judge accuracy and order-swap consistency should keep rising together on harder reasoning splits, not just on benchmark averages.
  • Under FLOP-matched inference, a small trained judge becomes the better compute choice: majority voting over J4R-CJ-7B responses raises JudgeBench accuracy from 56.86 to 64.94, above several larger judges matched on inference compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only tests response-order and multiple-choice relabeling as transformations; a natural extension would apply EIS-GRPO to meaning-preserving paraphrases of the prompt itself, which would reveal whether the learned invariance is about position specifically or about input equivalence generally.
  • A judge that is perfectly order-consistent could still be directionally wrong, for example by always preferring the first response; reporting accuracy separately for trials where the correct answer appears first versus second would separate true invariance from a learned default.
  • Because EIS-GRPO compares responses across subgroups, increasing group size $G$ resembles increasing a contrastive batch size; the paper's observed sweet spot at $G=32$ may be task-dependent, and tuning $G$ against task difficulty is a cheap, testable next step.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes EIS-GRPO, an extension of GRPO in which each prompt is transformed into L equivalent versions and G/L responses are sampled from each version; the advantage in Eq. (4) combines a global term computed over all G responses with a local term computed within each version. Applied to pairwise judge training, the two versions are the two orderings of the candidate responses, and the algorithm is used to train J4R-CJ-7B from CompassJudger-7B using outcome rewards on roughly 10K MATH/ReClor-derived pairs. The authors also introduce ReasoningJudgeBench, a 1,483-sample pairwise benchmark covering math, multi-hop, everyday, and domain-specific reasoning, and report that J4R-CJ-7B outperforms all small judge baselines and roughly matches 32B judges on JudgeBench and ReasoningJudgeBench. The paper includes ablations against SFT, DPO, balanced data, data duplication, global-only advantage, and group size, as well as a FLOP-matched comparison with larger judges.

Significance. If the EIS-GRPO mechanism holds, this is a meaningful methodological contribution: it is one of the first RL training algorithms designed specifically for judge models, and the state-equivalence construction is a clean way to reduce positional bias without additional training data. The empirical results are strong and the evaluation is largely independent of the training data, since training uses MATH and ReClor pairs while the main benchmarks are PPE, JudgeBench, and the new ReasoningJudgeBench. The paper also provides detailed hyperparameters and a reproducible training protocol. However, the central mechanistic claim that the global advantage term, rather than paired co-update of both orderings, drives the gains is not yet supported by the reported ablations, and the FLOP-matched comparison conflates self-consistency with training-method quality. With a local-only control added and the FLOP comparison corrected, this would be a solid contribution.

major comments (3)
  1. [Sec. 5.2, Table 2, Eq. (4)] The paper concludes in Sec. 5.2 that "including both global and local information" in the advantage is necessary, and Sec. 3.2 attributes the state-equivalence mechanism to the global term. However, the ablations never run the local-only variant with the same subgroup structure: G=32 split into two 16-response order subgroups, both updated together, with advantage computed as (R(i,l) - Rbar_l)/sigma_l only. The closest baselines, GRPO w/ [Dup] G=16 and G=32, differ in normalization and/or do not guarantee that the two orderings of the same comparison are co-updated in a single optimization step. If a local-only-with-paired-updates baseline matches EIS-GRPO, the observed gains come from seeing both orderings and co-updating, not from the equivalence-enforcing global term, and the conceptual novelty reduces to paired data duplication. Please add this control and report accuracy and consistency on both JudgeBench and ReasoningJudgeBench.
  2. [Sec. 5.2, Table 3] The "FLOP-matched" comparison is not actually matched: the baseline accuracies in Table 3 are single-sample evaluations, while the J4R-CJ-7B accuracy is obtained by majority voting over delta_inf samples. Self-consistency is a test-time scaling technique available to any judge, so the comparison conflates gains from the training method with gains from majority voting. To support the equal-compute claim, the same sampling-and-majority-vote procedure should be applied to the baselines; if that is infeasible, the table should be presented as a single-sample J4R comparison at equal compute and the self-consistency numbers should be removed.
  3. [App. D.1, Eq. (7)] The illustrative calculation for "Global advantage only" contains an arithmetic error that undermines the stated explanation. Under global normalization alone, all reward-1.0 responses receive the same global advantage regardless of subgroup; with the paper's own numbers, that value is 0.285, not 1.732 as printed for subgroup B. The corrected value still supports the qualitative point that the best response in the weak subgroup is under-reinforced relative to the local baseline, but the text and numbers need to be corrected.
minor comments (4)
  1. [Sec. 5.1, Table 1] The main results are single-run numbers with no error bars or significance tests; given that several comparisons differ by only a few points, some repeated-run variance or significance reporting would substantially strengthen the claims.
  2. [Abstract and Sec. 1] The statement that J4R "outperforms GPT-4o and the next best small judge by 6.7% and 9%" should specify whether these are relative or absolute improvements, since the same numbers could be read either way.
  3. [Sec. 4.1 and App. C] Because ReasoningJudgeBench is a new benchmark, a short quality-control section would help: for example, human spot-checks of generated pairs, label-agreement statistics, or a contamination check against the training data would clarify what the benchmark guarantees.
  4. [Fig. 3] The caption says marker size indicates model size, but the figure has no legend; an explicit legend would make the size scale interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EIS-GRPO is trained on external rule-based rewards and evaluated on independent benchmarks, with no fitted parameter renamed as a prediction.

full rationale

No circular step is present. The paper's central chain is: EIS-GRPO defines the objective L_EIS-GRPO (Eq. 5) with advantage A(i,l) = global term + local term (Eq. 4); rewards are R(i,l)=R_j+R_f, where R_j checks parse(o) against the externally known answer label a(l) (Eq. 6). The training signal is therefore rule-based verification on MATH/ReClor pairs, not a quantity defined by the benchmark results. Headline accuracy and consistency numbers in Tables 1-3 are measured on PPE, JudgeBench, and the newly constructed ReasoningJudgeBench; these benchmarks were not used to set any hyperparameter or fit any parameter reported in the claims. The ablations in Table 2 compare EIS-GRPO against SFT, DPO, balanced-data GRPO, duplicated-data GRPO, and global-advantage-only GRPO under matched normalization and time, so the reported improvement is an empirical comparison, not a tautology. Self-citations (e.g., Zhou et al. 2025 for judge difficulty, Wang et al. 2024a for the DPO recipe, Ke et al. 2025 survey) are contextual or baseline-related and do not carry the load of the equivalence claim; no uniqueness theorem from the authors is invoked. The missing local-only ablation noted by a skeptical reader is a genuine experimental confound for isolating the global term, but it is not circularity: the paper nowhere defines the predicted gain to be identical to the training objective by construction. Thus no significant circularity is found.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central claim depends on standard RL/GRPO background, on domain assumptions about pairwise preference being order-invariant and outcome labels being reliable, and on manually chosen hyperparameters that were only partially ablated. No invented entities are introduced. The heaviest assumptions are the order-equivalence premise and the reliability of the automatically generated training labels.

free parameters (8)
  • Group size G = 32
    Chosen by hand; Table 5 ablation shows accuracy varies from 40.66 to 45.04 across G=16/24/32/64.
  • Number of augmented states L = 2
    Only the candidate-order swap is used; no ablation of L.
  • KL divergence coefficient beta = 0.0001
    Standard GRPO hyperparameter, chosen by hand.
  • Clip epsilon = 0.2
    Standard PPO/GRPO hyperparameter, chosen by hand.
  • Learning rate = 1e-6
    Chosen by hand.
  • Format reward magnitude = 0.5
    Reward R_f set to +0.5/-0.5 by hand; no ablation.
  • Sampled responses per prompt = 20
    Data construction choice; used to form correct/incorrect pairs.
  • Rollout batch size / train batch size = 256 / 128
    Computational choices reported in Sec. 4.
assumptions (6)
  • standard math Policy gradient theorem and GRPO objective are valid optimization targets (Eq. 3).
    Background from Schulman et al. 2017 and Shao et al. 2024; the paper builds on this without re-deriving it.
  • domain assumption Swapping the order of candidate responses creates a substantively equivalent initial state with a relabeled answer.
    Core premise of EIS-GRPO; enters in Sec. 4 where a(1)=A and a(2)=B.
  • domain assumption The correct/incorrect outcome labels from MATH and ReClor define an unambiguous better response for every training pair.
    Used to construct training pairs in Sec. 4; if labels are noisy, the RL reward is noisy.
  • domain assumption The judge's final label parse(o) is the only behavior that matters; the chain-of-thought critique is not directly rewarded.
    Reward in Eq. 6 only checks the parsed label; critique quality is assumed to improve implicitly through RL.
  • domain assumption Ground-truth labels in PPE, JudgeBench, and ReasoningJudgeBench are correct and define the accuracy metric.
    Evaluation setup in Sec. 5; benchmark correctness is taken from source datasets.
  • standard math Self-consistency majority voting improves accuracy for the FLOP-matched evaluation.
    Standard technique from Wang et al. 2022, used in Sec. 5.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization." pith.science (2026). https://pith.science/paper/RNC25EC6

@misc{pith2026250513346,
  author       = {Pith},
  title        = {Pith review of: J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNC25EC6}},
  note         = {Machine review of arXiv:2505.13346}
}
read the original abstract

To keep pace with the increasing pace of large language models (LLM) development, model output evaluation has transitioned away from time-consuming human evaluation to automatic evaluation, where LLMs themselves are tasked with assessing and critiquing other model outputs. LLM-as-judge models are a class of generative evaluators that excel in evaluating relatively simple domains, like chat quality, but struggle in reasoning intensive domains where model responses contain more substantive and challenging content. To remedy existing judge shortcomings, we explore training judges with reinforcement learning (RL). We make three key contributions: (1) We propose the Equivalent Initial State Group Relative Policy Optimization (EIS-GRPO) algorithm, which allows us to train our judge to be robust to positional biases that arise in more complex evaluation settings. (2) We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work. (3) We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%, matching or exceeding the performance of larger GRPO-trained judges on both JudgeBench and ReasoningJudgeBench.

Figures

Figures reproduced from arXiv: 2505.13346 by the authors.

Figure 1
Figure 1. Overview of our three contributions: (1) We propose EIS-GRPO, a RL training algorithm [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of Equivalent Initial State GRPO: Each input [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Judge pairwise consistency is inversely correlated with task diffi￾culty, with very poor consistency in reasoning tasks. Plot of 11 judges, with marker size indicating model size across 3 non-reasoning (HHH, LFQA, InstruSum) and 1 reasoning benchmark (JudgeBench). When judging reasoning, are judges just guessing? Past work [Li et al., 2023, Wang et al., 2023a] has shown that judge models are sensitive to the order o… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Reward and length dynamics of training J4R-CJ-7B. [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A topic-F1 reward measuring alignment between summary and source-document topics, combined with GRPO training, improves multi-document summarization over several baselines.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    - So, \(17_b = b + 7\)

    **Convert \(17_b\)**: - In base \(b\), \(17_b\) means \(1 \cdot b + 7\). - So, \(17_b = b + 7\)

  2. [2]

    - So, \(97_b = 9b + 7\)

    **Convert \(97_b\)**: - In base \(b\), \(97_b\) means \(9 \cdot b + 7\). - So, \(97_b = 9b + 7\). ### Divisibility Condition We require \(17_b\) to be a divisor of \(97_b\). Thus: \[97_b = k \times 17_b\] for some integer \(k\). Substituting the values we found: \[9b + 7 = k(b + 7)\] Expanding the right side: \[9b + 7 = kb + 7k\] ### Solving the Equation ...

  3. [3]

    **If \(k = 2\):** \[b = \frac{7(2 - 1)}{9 - 2} = \frac{7}{7} = 1 \quad (\text{Not valid since } b > 9)\]

  4. [4]

    **If \(k = 3\):** \[b = \frac{7(3 - 1)}{9 - 3} = \frac{14}{6} = \frac{7}{3} = 2.\overline{3} \quad (\text{Not valid})\]

  5. [5]

    **If \(k = 4\):** \[b = \frac{7(4 - 1)}{9 - 4} = \frac{21}{5} = 4.2 \quad (\text{Not valid})\]

  6. [6]

    **If \(k = 5\):** \[b = \frac{7(5 - 1)}{9 - 5} = \frac{28}{4} = 7 \quad (\text{Not valid})\]

  7. [7]

    **If \(k = 6\):** \[b = \frac{7(6 - 1)}{9 - 6} = \frac{35}{3} = 11.\overline{6} \quad (\text{Not valid as an integer})\]

  8. [8]

    Thus, \(b = 21\) is a base greater than 9 for which \(17_b\) divides \(97_b\) exactly

    **If \(k = 7\):** \[b = \frac{7(7 - 1)}{9 - 7} = \frac{42}{2} = 21\] 22 A valid integer \(b\) is 21. Thus, \(b = 21\) is a base greater than 9 for which \(17_b\) divides \(97_b\) exactly. ### Conclusion Having checked for divisibility circumstances and calculations, we conclude that the only base \(b > 9\) that satisfies the condition is: \[b = 21\] There...

Show all 21 references
  1. [9]

    Therefore, in decimal, \( 17_b = b + 7 \)

    **Convert \( 17_b \) into decimal:** The number \( 17_b \) represents \( 1 \times b + 7 \). Therefore, in decimal, \( 17_b = b + 7 \)

  2. [10]

    Therefore, in decimal, \( 97_b = 9b + 7 \)

    **Convert \( 97_b \) into decimal:** The number \( 97_b \) represents \( 9 \times b + 7 \). Therefore, in decimal, \( 97_b = 9b + 7 \). The problem states that \( 17_b \) (which is \( b + 7 \)) is a divisor of \( 97_b \) (which is \( 9b + 7 \)). This implies that: \[ 9b + 7 = ...

  3. [11]

    **\( k = 1 \):** \[ b = \frac{7(1-1)}{9-1} = \frac{0}{8} = 0 \,\text{(not valid since } b > 9) \]

  4. [12]

    **\( k = 2 \):** \[ b = \frac{7(2-1)}{9-2} = \frac{7}{7} = 1 \,\text{(not valid since } b > 9) \]

  5. [13]

    **\( k = 3 \):** \[ b = \frac{7(3-1)}{9-3} = \frac{14}{6} \,\text{(not an integer)} \]

  6. [14]

    **\( k = 4 \):** \[ b = \frac{7(4-1)}{9-4} = \frac{21}{5} \,\text{(not an integer)} \]

  7. [15]

    **\( k = 5 \):** \[ b = \frac{7(5-1)}{9-5} = \frac{28}{4} = 7 \,\text{(not valid since } b > 9) \]

  8. [16]

    **\( k = 6 \):** \[ b = \frac{7(6-1)}{9-6} = \frac{35}{3} \,\text{(not an integer)} \]

  9. [17]

    **\( k = 7 \):** \[ b = \frac{7(7-1)}{9-7} = \frac{42}{2} = 21 \,\text{(valid)} \]

  10. [18]

    **\( k = 8 \):** \[ b = \frac{7(8-1)}{9-8} = \frac{49}{1} = 49 \,\text{(valid)} \]

  11. [19]

    his" uncle, which suggests that the uncle belongs to one of the two friends. The next part of the sentence states,

    **\( k = 9 \):** \[ b = \frac{7(9-1)}{9-9} \] is undefined because this involves division by zero. The valid integer bases \( b \) for which \( b > 9 \) are \( b = 21 \) and \( b = 49 \). Therefore, their sum is: \[ 21 + 49 = 70 \] Answer: 70 Judge explanation ================...

  12. [20]

    Assistant A is better: [[A>B]]

  13. [21]

    Verdict: [[A>B]] or [[B>A]]

    Assistant B is better: [[B>A]] Use the following template: Explanation: Your detailed thought process as an explanation. Verdict: [[A>B]] or [[B>A]]. ### User Prompt <|User Prompt|> {question} <|The Start of Assistant A’s Answer|> {response_a} <|The End of Assistant A’s Answer...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.