REVIEW 3 major objections 2 minor
GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs
T0 review · 3 major / 2 minor · reviewed 2026-05-23 · grok-4.3
Pith's one-line read Asking LLMs to select the worse response instead of the better one improves pairwise judgment accuracy.
desk verdict GRP gives a zero-cost prompt reversal that lifts strict-consistency accuracy on JudgeBench by 4-6 points across judges, but the positive-bias story is not directly tested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Goal-Reversed Prompting (GRP), the prompt change that inverts the selection goal to identify the inferior response and recovers the preference by elimination.
What would settle it
Measure whether GRP still raises accuracy on a judge model that has been explicitly fine-tuned or prompted to eliminate positive-leaning bias.
Extended reading notes
Core claim
Goal-Reversed Prompting (GRP) is a one-line prompt modification that asks the judge to identify the worse of two candidate answers rather than the better one, then recovers the original preference by elimination. GRP requires no extra inference rounds and composes with any existing template. On JudgeBench under strict consistency (correct only when both orderings match the gold label), GRP raises GPT-4o from 61.71% to 66.23% and Claude-3.5-Sonnet from 60.00% to 66.00% on GPT-4o-generated pairs using Arena-Hard SOP, with larger gains on reasoning and mathematics; the lift persists when pairs come from Claude-3.5-Sonnet and when the template is reduced to direct prompting.
Load-bearing premise
Instruction-tuned judges carry a positive-leaning bias that reversal specifically offsets without introducing new biases or order effects of similar size.
Editorial extensions
If this is right
- GRP can be added to any existing pairwise template without changing the rest of the evaluation pipeline.
- Gains appear largest on reasoning and mathematics categories.
- The benefit remains when response pairs are generated by a different model and when SOP scaffolding is removed.
- Stronger base judges receive larger absolute improvements than weaker ones.
Reading between the lines
- The same reversal tactic could be tested on open-source judges to check whether the bias pattern is tied to scale or alignment method.
- If reversal works by exposing latent capacity, it might combine with other light prompt adjustments such as explicit bias warnings.
- The approach might generalize to other LLM decision tasks that involve comparative selection under bias, such as preference tuning data curation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Goal-Reversed Prompting (GRP), a one-line change to pairwise LLM-as-a-judge prompts that asks the model to identify the worse of two responses rather than the better one, recovering the preference by elimination. Motivated by reverse reasoning in human problem-solving and a posited positive-leaning bias in instruction-tuned judges, GRP is shown to compose with existing templates (direct, CoT, Arena-Hard SOP) without extra inference cost. On JudgeBench under a strict consistency protocol (both orderings must match gold), GRP yields consistent gains for GPT-4o and Claude-3.5-Sonnet across GPT-4o- and Claude-generated pairs, with largest improvements on Reasoning and Mathematics; stronger judges benefit more.
Significance. If the empirical gains hold under proper statistical controls, GRP offers a simple, zero-extra-cost intervention that improves zero-shot pairwise evaluation on a public benchmark. The use of multiple judges, pair sources, and templates, plus the strict consistency protocol, strengthens the result relative to typical LLM-judge papers; the observation that gains persist when SOP scaffolding is removed is a positive indicator that the effect is not template-specific.
major comments (3)
- [Abstract / Experiments] Abstract and Experiments section: the central motivation—that GRP specifically counteracts a positive-leaning bias in instruction-tuned judges without introducing comparable new biases or order effects—is stated but not directly measured. No baseline preference-rate analysis, error-type breakdown by polarity, or comparison against non-instruction-tuned models is reported, leaving the mechanism correlational rather than causal.
- [Results] Results (e.g., the +4.52 and +6.00 point lifts on Arena-Hard SOP): the reported percentages lack sample counts, variance estimates, or statistical significance tests. Without these, it is impossible to determine whether the observed differences exceed what would be expected from prompt variation alone under the strict consistency protocol.
- [Experiments] Experiments: while the paper notes that gains persist across minimal direct prompts and different pair generators, an ablation that isolates the reversal from other prompt-phrasing changes (e.g., explicit “worse” wording versus rephrased “better” instructions) is not presented, leaving open the alternative that the lift arises from more effective phrasing rather than bias counteraction.
minor comments (2)
- [Method] The description of the strict consistency protocol would benefit from an explicit equation or pseudocode showing how a judgment is counted as correct only when both orderings agree with gold.
- [Results] Table or figure captions should state the exact number of JudgeBench items used for each reported percentage.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed comments. We address each major comment below, indicating revisions where appropriate to improve the manuscript.
read point-by-point responses
-
Referee: [Abstract / Experiments] Abstract and Experiments section: the central motivation—that GRP specifically counteracts a positive-leaning bias in instruction-tuned judges without introducing comparable new biases or order effects—is stated but not directly measured. No baseline preference-rate analysis, error-type breakdown by polarity, or comparison against non-instruction-tuned models is reported, leaving the mechanism correlational rather than causal.
Authors: We acknowledge that the current evidence for the posited mechanism is correlational, relying on empirical gains and prior literature on LLM biases rather than direct measurements. We will revise the Experiments section to include a baseline preference-rate analysis on JudgeBench and an error-type breakdown by polarity. A comparison against non-instruction-tuned models is outside the scope of this work due to model access constraints and will be noted as future work. revision: partial
-
Referee: [Results] Results (e.g., the +4.52 and +6.00 point lifts on Arena-Hard SOP): the reported percentages lack sample counts, variance estimates, or statistical significance tests. Without these, it is impossible to determine whether the observed differences exceed what would be expected from prompt variation alone under the strict consistency protocol.
Authors: We agree this information is necessary for rigorous interpretation. The JudgeBench set comprises 1000 examples; we will explicitly report this, add bootstrap variance estimates, and include statistical significance tests (such as McNemar's test for paired judgments) in the revised Results section. revision: yes
-
Referee: [Experiments] Experiments: while the paper notes that gains persist across minimal direct prompts and different pair generators, an ablation that isolates the reversal from other prompt-phrasing changes (e.g., explicit “worse” wording versus rephrased “better” instructions) is not presented, leaving open the alternative that the lift arises from more effective phrasing rather than bias counteraction.
Authors: The GRP change is a minimal one-line modification that reverses the goal while keeping all other template elements identical. We will add explicit clarification in the Experiments section that the only difference is the goal reversal. A broader ablation on alternative phrasings of the 'better' instruction would require new experiments; we will instead discuss this alternative explanation as a limitation while noting that the effect holds across multiple templates. revision: partial
Circularity Check
No circularity: empirical evaluation on external benchmark
full rationale
The paper introduces GRP as a prompting modification and validates it through direct empirical measurements on the external JudgeBench benchmark under a strict consistency protocol. No equations, fitted parameters, self-referential definitions, or load-bearing self-citations appear in the derivation chain; the reported accuracy lifts (e.g., +4.52 for GPT-4o) are computed from model judgments against gold preferences and do not reduce to any quantity defined by the method itself. The bias-counteraction motivation is stated as background but is not invoked as a mathematical premise that forces the results.
Assumptions & free parameters
assumptions (2)
- domain assumption JudgeBench provides reliable gold-standard preferences for response pairs.
- domain assumption The strict consistency protocol (agreement on both orderings) is an appropriate measure of judge performance.
Cite this review
Pith. "Pith review of GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs." pith.science (2026). https://pith.science/paper/2503.06139
@misc{pith2026250306139,
author = {Pith},
title = {Pith review of: GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/2503.06139}},
note = {Machine review of arXiv:2503.06139}
}
read the original abstract
Pairwise LLM-as-a-judge evaluation asks the judge to identify the \emph{better} of two candidate answers. We study a one-line modification that asks for the \emph{worse} answer instead and recovers the preference by elimination, a procedure we call Goal-Reversed Prompting (GRP). GRP introduces no extra inference rounds, composes with any prompt template (direct, chain-of-thought, or Arena-Hard SOP), and leaves the rest of the evaluation pipeline untouched. Two observations motivate the reversal. Reverse reasoning is a recurring strategy in human problem solving, and modern instruction-tuned judges exhibit a positive-leaning bias that asking for the worse answer can counteract. On JudgeBench under a strict consistency protocol that counts a judgment as correct only when both response orderings agree with the gold preference, GRP improves all three closed-source judges we test across both response-pair sources. With GPT-4o-generated pairs, the Arena-Hard SOP baseline improves from 61.71\% to 66.23\% for GPT-4o (+4.52) and from 60.00\% to 66.00\% for Claude-3.5-Sonnet (+6.00), with the largest absolute gains on Reasoning and Mathematics. The lift persists when response pairs come from Claude-3.5-Sonnet and when the SOP scaffolding is stripped to a minimal direct-prompting template, suggesting that goal reversal acts on the underlying judging behavior rather than on a particular rubric. Stronger judges benefit more than weaker ones, suggesting that goal reversal exposes additional reasoning capacity rather than compensating for its absence.
Reviewed May 23, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.