{"id":"f6e6e44d-4ba4-491f-818c-4edf267a53cd","arxiv_id":"2505.13346","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A 7B judge trained with a new RL method, EIS-GRPO, that enforces answer-order invariance, beats GPT-4o and larger judges on reasoning evaluation benchmarks.","lead":"The paper introduces EIS-GRPO, a reinforcement learning tweak that trains a small language model to judge which of two competing answers is better regardless of which is listed first. The resulting 7B judge, J4R, scores higher on reasoning evaluation benchmarks than much larger open judges and OpenAI's GPT-4o.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EIS-GRPO's global-advantage mechanism is not isolated: the paper ablates global-only but never local-only, so the claimed state-equivalence effect may be confounded with paired data duplication.","rationale":"The paper is a strong, self-contained empirical contribution: EIS-GRPO is clearly specified, the ablations are extensive, ReasoningJudgeBench is a useful new resource, and the main result that J4R-CJ-7B is a competitive small judge is supported by the reported numbers. My concern is narrower than the reader's: rather than questioning whether the two orderings are substantively equivalent (for pairwise comparison this equivalence is definitionally true), I question whether the joint advantage term is the active mechanism. The global-only negative control shows that the global term alone is harmful, but this does not establish that the global term is beneficial when added to local advantages. A local-only control is the missing experiment that would settle this. The concern does not overturn the empirical recipe, but it affects how the method should be interpreted and presented: if local-only is as good as EIS-GRPO, the 'learned state equivalence' narrative in Sec. 3.2 and the title's emphasis on the group-relative joint advantage would need to be revised, while the practical contribution of a well-tuned judge training recipe would remain. Therefore I keep the reader's CONDITIONAL verdict unchanged, with this ablation as a concrete condition to resolve the mechanistic ambiguity.","tokens_in":27555,"tokens_out":16660,"duration_ms":174483,"concrete_test":"Train the local-only ablation with the exact J4R-CJ-7B setup from Sec. 4 (G=32, two subgroups of 16, same rewards and KL), but set the advantage to Ahat(i,l) = (R(i,l) - Rbar_l)/sigma_l, dropping the global term in Eq. (4). Evaluate on JudgeBench and ReasoningJudgeBench using the consistent-accuracy protocol, and compare against the EIS-GRPO row of Table 2 with a paired bootstrap across benchmark items. If the 95% confidence intervals overlap EIS-GRPO, the global term is not the active ingredient; if local-only falls to the level of the Dup G=16 baseline, the global term is confirmed as essential.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanistic claim is that the joint (global) advantage in Eq. (4) is the component that teaches state equivalence, with Sec. 5.2 concluding that 'including both global and local information' is necessary. The experimental support for that necessity is incomplete. Table 2 ablates the global-only variant (global term without local, Eq. (7) in App. D.1) and shows it fails, but it never ablates the local-only variant: identical subgroup sampling and paired co-update of both orderings, with advantage computed only as (R(i,l) - Rbar_l)/sigma_l. The closest reported baseline, 'GRPO w/ [Dup], G=16', uses duplicated data and local normalization, but it does not guarantee that the two orderings of the same comparison are updated together, and it uses G=16 normalization, not the two-subgroups-within-G=32 structure of EIS-GRPO. If local-only matches EIS-GRPO, the observed gains come from seeing both orderings and co-updating, not from the equivalence-enforcing global term, and the method's conceptual novelty reduces to data duplication with paired updates. Because the title, abstract, and Sec. 3.2 all attribute the robustness gain to the joint advantage, this missing control is load-bearing for the central claim, even though the final model's empirical performance is not itself in question.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EIS-GRPO, an extension of GRPO in which each prompt is transformed into L equivalent versions and G/L responses are sampled from each version; the advantage in Eq. (4) combines a global term computed over all G responses with a local term computed within each version. Applied to pairwise judge training, the two versions are the two orderings of the candidate responses, and the algorithm is used to train J4R-CJ-7B from CompassJudger-7B using outcome rewards on roughly 10K MATH/ReClor-derived pairs. The authors also introduce ReasoningJudgeBench, a 1,483-sample pairwise benchmark covering math, multi-hop, everyday, and domain-specific reasoning, and report that J4R-CJ-7B outperforms all small judge baselines and roughly matches 32B judges on JudgeBench and ReasoningJudgeBench. The paper includes ablations against SFT, DPO, balanced data, data duplication, global-only advantage, and group size, as well as a FLOP-matched comparison with larger judges.","tokens_in":27820,"tokens_out":8220,"duration_ms":80995,"significance":"If the EIS-GRPO mechanism holds, this is a meaningful methodological contribution: it is one of the first RL training algorithms designed specifically for judge models, and the state-equivalence construction is a clean way to reduce positional bias without additional training data. The empirical results are strong and the evaluation is largely independent of the training data, since training uses MATH and ReClor pairs while the main benchmarks are PPE, JudgeBench, and the new ReasoningJudgeBench. The paper also provides detailed hyperparameters and a reproducible training protocol. However, the central mechanistic claim that the global advantage term, rather than paired co-update of both orderings, drives the gains is not yet supported by the reported ablations, and the FLOP-matched comparison conflates self-consistency with training-method quality. With a local-only control added and the FLOP comparison corrected, this would be a solid contribution.","major_comments":[{"comment":"The paper concludes in Sec. 5.2 that \"including both global and local information\" in the advantage is necessary, and Sec. 3.2 attributes the state-equivalence mechanism to the global term. However, the ablations never run the local-only variant with the same subgroup structure: G=32 split into two 16-response order subgroups, both updated together, with advantage computed as (R(i,l) - Rbar_l)/sigma_l only. The closest baselines, GRPO w/ [Dup] G=16 and G=32, differ in normalization and/or do not guarantee that the two orderings of the same comparison are co-updated in a single optimization step. If a local-only-with-paired-updates baseline matches EIS-GRPO, the observed gains come from seeing both orderings and co-updating, not from the equivalence-enforcing global term, and the conceptual novelty reduces to paired data duplication. Please add this control and report accuracy and consistency on both JudgeBench and ReasoningJudgeBench.","section":"Sec. 5.2, Table 2, Eq. (4)"},{"comment":"The \"FLOP-matched\" comparison is not actually matched: the baseline accuracies in Table 3 are single-sample evaluations, while the J4R-CJ-7B accuracy is obtained by majority voting over delta_inf samples. Self-consistency is a test-time scaling technique available to any judge, so the comparison conflates gains from the training method with gains from majority voting. To support the equal-compute claim, the same sampling-and-majority-vote procedure should be applied to the baselines; if that is infeasible, the table should be presented as a single-sample J4R comparison at equal compute and the self-consistency numbers should be removed.","section":"Sec. 5.2, Table 3"},{"comment":"The illustrative calculation for \"Global advantage only\" contains an arithmetic error that undermines the stated explanation. Under global normalization alone, all reward-1.0 responses receive the same global advantage regardless of subgroup; with the paper's own numbers, that value is 0.285, not 1.732 as printed for subgroup B. The corrected value still supports the qualitative point that the best response in the weak subgroup is under-reinforced relative to the local baseline, but the text and numbers need to be corrected.","section":"App. D.1, Eq. (7)"}],"minor_comments":[{"comment":"The main results are single-run numbers with no error bars or significance tests; given that several comparisons differ by only a few points, some repeated-run variance or significance reporting would substantially strengthen the claims.","section":"Sec. 5.1, Table 1"},{"comment":"The statement that J4R \"outperforms GPT-4o and the next best small judge by 6.7% and 9%\" should specify whether these are relative or absolute improvements, since the same numbers could be read either way.","section":"Abstract and Sec. 1"},{"comment":"Because ReasoningJudgeBench is a new benchmark, a short quality-control section would help: for example, human spot-checks of generated pairs, label-agreement statistics, or a contamination check against the training data would clarify what the benchmark guarantees.","section":"Sec. 4.1 and App. C"},{"comment":"The caption says marker size indicates model size, but the figure has no legend; an explicit legend would make the size scale interpretable.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is within scope and the empirical contribution is likely strong. The main issue is the missing local-only control, which is straightforward to run and directly tests the paper's central mechanism, plus the unfair FLOP-matched comparison. Both are fixable with additional experiments and revised claims; I do not see grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a good empirical contribution. EIS-GRPO is cleanly specified, and the trained 7B judge is genuinely strong: it beats all small judge baselines on the combined benchmark, roughly matches RM-R1-32B, and the gains are large (56.86 vs. 51.71 over duplicated-data GRPO with the same G=32 budget on JudgeBench). ReasoningJudgeBench is a real resource, with sensible coverage of reasoning types beyond math and code, and the authors are transparent about reproducing baselines with their own prompt.\n\nWhat is actually new: EIS-GRPO is not just GRPO with data duplication. It samples subgroups from transformed versions of the same input, computes advantages jointly across subgroups and locally within them, and updates the policy conditioned on each transformed state. That last point distinguishes it from concurrent NoisyRollout, which only diversifies the group. The ablations are helpful: balanced vs. duplicated data, group size, initialization, and SFT/DPO comparisons. The training objective rewards correct judgments from external ground-truth labels, and the main numbers come from independent benchmarks, so circularity is not a concern here.\n\nThe soft spots are real but manageable. The main one is the missing local-only ablation. The paper claims the joint global advantage teaches state equivalence, and it shows that global-only fails. But it never runs EIS-GRPO with the same subgroup pairing and only the local advantage. The closest baseline, GRPO with duplicated data and G=16, does not guarantee that both orderings of the same comparison are co-updated within one group of 32. Without that control, the observed gains could come from paired co-updating of both orderings rather than from the cross-subgroup normalization. That matters because the title and abstract attribute robustness to the joint advantage. It is fixable, but it is load-bearing for the mechanism claim, not for the full recipe's empirical performance.\n\nAlso minor: no error bars or significance tests on the headline numbers, and the benchmark construction has an ambiguous sentence about sampling prompts where the model gets all 20 responses correct or incorrect. The intent is probably a filter, but the phrasing is unclear and could affect perceived difficulty.\n\nOverall, this deserves a serious peer review. The method is new, the benchmark is useful, and the empirical result is credible on its face. Ask for the local-only ablation, error bars or repeated seeds, and ideally training code. Not a desk reject.","headline":"A solid, useful judge-training recipe with a new benchmark; the one real soft spot is a missing local-only ablation that leaves the mechanism claim under-supported.","tokens_in":744,"tokens_out":983,"would_cite":true,"duration_ms":31205,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B judge trained with EIS-GRPO achieves order-invariance by pooling rewards across swapped-order versions of each comparison, and this makes it outperform much larger judges on reasoning benchmarks.","keywords":["LLM-as-judge","reinforcement learning","GRPO","positional bias","pairwise evaluation","reasoning evaluation","state equivalence","ReasoningJudgeBench"],"falsifier":"Build a held-out set of pairwise samples in which the two orderings genuinely are not equivalent, for instance a rule stating that when both responses are acceptable the one listed first should win, and train two judges on the same data, one with EIS-GRPO and one with plain GRPO. If the EIS-GRPO judge is less accurate on that set because it was forced to treat the two states as equivalent, the central premise is refuted; if it still matches or beats GRPO, the equivalence assumption passes a stricter test than the paper's order-swap design.","tokens_in":27337,"feed_emoji":"⚖️","tokens_out":8897,"duration_ms":86852,"temperature":0.7,"pith_summary":"This paper tries to establish that a judge model's poor performance on reasoning tasks is largely positional inconsistency, and that a reinforcement-learning objective can remove that inconsistency directly. The proposed algorithm, EIS-GRPO, treats the two orderings of a pairwise comparison as equivalent initial states, samples a group of judgments from each ordering, and computes advantages jointly across both groups before updating the policy. Trained this way, a 7B judge called J4R-CJ-7B reaches 50.35 average accuracy across three reasoning-evaluation benchmarks, higher than every judge at or below 14B parameters and close to a 32B GRPO-trained judge. On JudgeBench, EIS-GRPO improves accuracy over a balanced-data GRPO baseline by 9.4% and over a duplicated-data GRPO baseline by 11.2%, while raising consistency from 68.86 to 81.14. If the claim holds, order-invariance is not a side effect of good judging but a cause of it.","feed_headline":"A 7B judge beats larger evaluators by ignoring order","feed_subtitle":"Order-swap RL training lifts a 7B judge above GPT-4o on reasoning-heavy evaluation benchmarks.","key_machinery":"The central mechanism is the two-part advantage estimate of EIS-GRPO (Eq. 4). Starting from an input $q$, the algorithm forms $L$ transformed versions $T_\\ell(q)$ (for pairwise judging, the two orderings of the candidate responses), samples $G/L$ outputs from each, and gives output $(i,\\ell)$ the advantage $$\\hat{A}^{(i,\\ell)} = \\frac{$R^{{(i,\\ell)}}$-\\bar{R}_{[L]}}{\\sigma_{R_{[L]}}} + \\frac{$R^{{(i,\\ell)}}$-\\bar{R}_{\\ell}}{\\sigma_{R_\\ell}},$$ where the first term is normalized across all subgroups and the second within subgroup $\\ell$. The policy update is conditioned on $T_\\ell(q)$, so the model is rewarded for correct judgments under each ordering rather than for matching outputs across orderings. The two terms do complementary work: the global term asserts that the transformed states are equivalent, and the local term prevents a weak subgroup's best responses from being undervalued because stronger responses exist in another subgroup.","core_discovery":"The paper's central claim is that judges fail in reasoning evaluation in a specific, measurable way: when the order of two candidate responses is swapped, the judge's verdict often swaps too, and training with EIS-GRPO turns this failure into a learning signal. For a pairwise sample $(x, y_1, y_2)$, the better response is fixed by an outcome label, and the two input orderings $(A=y_1, B=y_2)$ and $(A=y_2, B=y_1)$ are the equivalent initial states; the correct answer relabels from $A$ to $B$ across them. The model is updated with an advantage computed jointly across both orderings, so it must treat the two states as equivalent to be rewarded. The paper reports that J4R-CJ-7B, a 7B judge initialized from CompassJudger-7B, scores 50.35 average accuracy across PPE, JudgeBench, and ReasoningJudgeBench, the best among all judges at or below 14B and within 1.5 points of the best 32B judge evaluated; it also introduces ReasoningJudgeBench, a 1,483-sample benchmark spanning math, multi-hop, domain-specific, and everyday reasoning, where most existing judges fail to reach 45%.","pith_inferences":["The paper only tests response-order and multiple-choice relabeling as transformations; a natural extension would apply EIS-GRPO to meaning-preserving paraphrases of the prompt itself, which would reveal whether the learned invariance is about position specifically or about input equivalence generally.","A judge that is perfectly order-consistent could still be directionally wrong, for example by always preferring the first response; reporting accuracy separately for trials where the correct answer appears first versus second would separate true invariance from a learned default.","Because EIS-GRPO compares responses across subgroups, increasing group size $G$ resembles increasing a contrastive batch size; the paper's observed sweet spot at $G=32$ may be task-dependent, and tuning $G$ against task difficulty is a cheap, testable next step."],"forward_implications":["If EIS-GRPO is right, judge training no longer needs a doubled dataset: swapped-order responses are used as subgroups of the same batch, so the model sees both orders without extra storage or wall-clock cost beyond sampling $G$ responses.","The same subgroup-advantage structure is a direct recipe for other transformation-invariant RLVR tasks, including paraphrases, multiple-choice option orders, and any input transformation whose correct answer relabels consistently.","Because the training signal punishes disagreement between orderings, judge accuracy and order-swap consistency should keep rising together on harder reasoning splits, not just on benchmark averages.","Under FLOP-matched inference, a small trained judge becomes the better compute choice: majority voting over J4R-CJ-7B responses raises JudgeBench accuracy from 56.86 to 64.94, above several larger judges matched on inference compute."],"supporting_citations":[{"why":"Defines the base GRPO objective, group-sampled advantages, and rule-based reward setting that EIS-GRPO modifies.","marker":"[Shao et al., 2024]"},{"why":"Supplies JudgeBench, one of the three evaluation suites, and the two-order accuracy protocol used for all judges.","marker":"[Tan et al., 2024]"},{"why":"Provides CompassJudger-7B, the initialization for J4R-CJ-7B and a direct SFT-trained baseline.","marker":"[Cao et al., 2024]"},{"why":"JudgeLRM is the prior GRPO-for-judges study that this paper builds on and compares against.","marker":"[Chen et al., 2025b]"},{"why":"RM-R1 is the concurrent GRPO judge-training pipeline used as a comparison baseline.","marker":"[Chen et al., 2025c]"},{"why":"Documents judge positional bias and supplies the consistency metric that motivates the order-equivalence transformation.","marker":"[Li et al., 2023]"},{"why":"Establishes RL from verifiable rewards with chain-of-thought, the training paradigm used here and a prompted baseline (DeepSeek-R1).","marker":"[Guo et al., 2025]"},{"why":"Qwen2.5-7B is the weaker initialization for J4R-Qwen-Inst and a prompted instruct baseline.","marker":"[Yang et al., 2024]"}],"fun_headline_variants":["Order-swap RL makes 7B judge beat GPT-4o on reasoning","Small 7B judge beats GPT-4o via order-swap RL","Order-swap training gives 7B judge edge over GPT-4o","7B judge trained on swapped orders tops GPT-4o","Order swap teaches 7B judge to outrank GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that flipping which response is presented first and which is presented second leaves the judging task substantively unchanged, so the two orderings can be pooled in one reward computation; if position itself changed the correct verdict, EIS-GRPO would reward the model for ignoring a real signal.","fun_headline_variants_meta":{"raw":{"variants":["Order-swap RL makes 7B judge beat GPT-4o on reasoning","Small 7B judge beats GPT-4o via order-swap RL","Order-swap training gives 7B judge edge over GPT-4o","7B judge trained on swapped orders tops GPT-4o","Order swap teaches 7B judge to outrank GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4101,"prompt_tokens":1026,"completion_tokens":3075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":2987}},"tokens_in":642,"tokens_out":3075,"duration_ms":20714,"temperature":1.0,"reasoning_tokens":2987,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:15:27.997076+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a held-out set of pairwise samples in which the two orderings genuinely are not equivalent, for instance a rule stating that when both responses are acceptable the one listed first should win, and train two judges on the same data, one with EIS-GRPO and one with plain GRPO. If the EIS-GRPO judge is less accurate on that set because it was forced to treat the two states as equivalent, the central premise is refuted; if it still matches or beats GRPO, the equivalence assumption passes a stricter test than the paper's order-swap design.","supporting_citations":[],"review_version":1}