Direct-Scoring NLG Evaluators Can Use Pairwise Comparisons Too

Alexander Shelton; Ashton Williamson; Logan Lawrence

REVIEW 4 major objections 6 minor 14 references

Direct-Scoring NLG Evaluators Can Use Pairwise Comparisons Too

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A direct-scoring evaluator matches pairwise LLM judges using synthetic comparison ladders.

desk verdict A neat, clearly described method for giving pairwise comparison signal to direct scorers, but the headline gains are small, error bars are absent, and the anchor-ordering assumption is untested. read the letter →

arxiv 2509.05440 v1 pith:FUTCSY4G submitted 2025-09-05 cs.CL cs.AIcs.LG

Logan Lawrence , Ashton Williamson , Alexander Shelton This is my paper

classification cs.CLcs.AIcs.LG

keywords NLGevaluationdirectscoringpairwisecomparisonsyntheticin-contextexamplesLLM-as-a-judgesample-levelcorrelationmeta-evaluationsummarization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that an LLM-based evaluator can assign absolute scores to generated text without giving up the alignment with human judgment that pairwise-comparison evaluators are known for. Its method builds, for each input and quality dimension, five synthetic summaries of increasing quality, then asks the evaluator to compare the target text to each of those five anchors and converts the Better/Worse/Similar probabilities into a weighted absolute score. On the SummEval, TopicalChat, and HANNA meta-evaluation benchmarks, this direct-scoring method lands within a few hundredths of state-of-the-art pairwise evaluators (axis-averaged sample-level Spearman correlation +0.03, -0.03, and +0.05 relative to those baselines). If the claim holds, systems that need a score for thresholding or sorting no longer have to choose between absolute ratings and pairwise-grade accuracy.

What carries the argument

The synthetic quality ladder: for each source text and quality dimension, the LLM generates a worst and best example, then recursively generates intermediates between adjacent rungs, yielding five in-context summaries ordered by quality. At inference the target is compared to each rung with a forced-choice Better/Worse/Similar prompt, and the softmax probabilities are combined as a weighted sum over rung indices to give an absolute score. This machinery converts a relative pairwise signal into an absolute score without external references or human preference data.

What would settle it

Human annotators rank the five synthetic summaries for a sample of articles on, say, SummEval consistency; if the generated order (worst to best) matches human order in only a minority of cases, or if scores computed from human-corrected ladders correlate no better with human quality ratings, the method's central mechanism is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that direct scoring can be implemented as a pairwise comparison against synthetic anchors. Instead of asking the LLM to rate a summary on a numeric scale, the method first generates a five-rung ladder of summaries for the same source text—worst, best, then three intermediate ones produced recursively—and then asks 'better, worse, or similar' for each rung. The softmaxed probabilities of those three tokens are weighted by the rung index to produce an absolute score. Across three benchmarks and two 7–8B backbones, the resulting absolute scores correlate with human judgments about as well as dedicated pairwise evaluators like PairS-beam and ZEPO; the authors also find t

Load-bearing premise

The five synthetic summaries for each article are generated to be increasing in quality, but the paper does not check against human judgments that this ordering actually holds; if it does not, the computed absolute score is built on a distorted ladder.

Editorial extensions

If this is right

Absolute scores can be obtained with sample-level correlation comparable to pairwise evaluators, so downstream thresholding and filtering no longer require switching to a weaker metric.
The same synthetic-anchor recipe transfers across summarization (SummEval), dialog (TopicalChat), and story generation (HANNA).
Probability aggregation over comparison tokens is necessary for the gain: increasing the number of sampled judgments helps, but stays below the softmax-based score.
More instruction-following backbones improve the evaluator, suggesting the method's ceiling is tied to backbone capability rather than to the prompt design.
Releasing the synthetic in-context summaries gives other researchers a ready-made ranking ladder per article and dimension.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The synthetic ladder is generated by the same LLM that will later be used as the evaluator, so the method could inherit that model's own quality bias; generating anchors with a different, stronger model and evaluating with another would test whether the result is a general ranking ability or a self-preference artifact.
A natural extension is to replace the fixed five rungs with a continuous ladder or with multiple ladders per dimension, which the authors' own N=9 experiment suggests could keep improving scores.
Because the anchors are per-article and per-dimension, the approach could be adapted to other scalar rating schemes (e.g., 1–10) and other attributes such as safety or style, not just quality.
The comparison-prompt design effectively turns a regression problem into a set of local preference judgments; this may transfer to reward-model training, where pairwise preference data are abundant but scalar labels are not.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the core idea is neat and clearly presented, but the empirical case is thin and the method rests on an unverified assumption about anchor ordering.

What's actually new: instead of asking an LLM for a score, this paper builds synthetic reference summaries at five quality levels by recursively prompting for 'intermediate' examples between worst and best, then compares the candidate summary pairwise against each anchor and aggregates via a probability-weighted sum. That gives you an absolute score while keeping pairwise comparison signal. They release code and the synthetic data, which is good practice.

What works: the method is competitive across three benchmarks. On SummEval it beats the next best by 0.03-0.05 average correlation; on HANNA it leads by 0.05; on TopicalChat it trails by 0.03. Gains are modest, but the ablations are informative: probability aggregation beats sampling, and performance improves with stronger LLMs.

The big soft spot is the monotonicity assumption. The recursive prompt asks the LLM for a summary 'between' a worse and better one, but nothing checks that the five anchors are actually strictly ordered along the dimension. The weighted sum in eq. 2 assigns positive weight to 'Better' and negative to 'Worse' for each anchor; if an anchor's true quality is out of order, that weight is misaligned and the score is distorted. The paper shows one encouraging example but no human or independent model validation of the ordering. That is load-bearing, and should be tested.

Second, Table 1 has no error bars. With differences of 0.03-0.05, the headline results could be noise. The paper's own ablations report standard deviations, so this is fixable.

Third, NewsRoom is dropped from the main results, and their method is not reported on it at all. Their reasoning about BERTScore and axis correlations is plausible, but a referee will want to see the numbers.

Verdict: this is a serious, citable contribution. It deserves peer review. I'd ask for anchor-order validation and error bars before accepting, but the method and the release make it worth engaging with.

Referee Report

4 major / 6 minor

Summary. The paper proposes a direct-scoring NLG evaluator that derives absolute scores from pairwise comparisons against synthetic in-context examples. For each source item, five anchors (icl_1...icl_5) are generated by recursive contrastive prompting: the LLM is first asked for the worst and best possible summaries, then for intermediate qualities between existing anchors. At test time, the target summary is compared with each anchor via a "Better/Worse/Similar" prompt, the softmax probabilities are computed, and the final score is the weighted sum s(·) = Σ_i i·[p(Better|i) − p(Worse|i)]. Experiments on SummEval, TopicalChat, and HANNA report sample-level Spearman correlations that are claimed to be comparable to pairwise evaluators (axis-averaged differences of +0.03, −0.03, and +0.05 versus the best baseline). Ablations examine sampling versus probability-based prediction, the number of anchors N, and the choice of generation/prediction LLMs. The paper releases code, prompts, and generated data.

Significance. If the reported results are robust, the method offers a simple way to obtain direct scores with pairwise-style alignment, which would be useful for threshold-based applications. The method is clearly described, the code and data are released, and the ablation study (Table 2 and Table 3) provides useful diagnostic information about prediction methods and model choices. However, the central claim rests on an unvalidated monotonicity assumption about the synthetic anchors, and the main results lack uncertainty quantification. Because these issues directly affect the credibility of the headline comparisons, the contribution is currently suggestive rather than fully established.

major comments (4)

[§2.1, §2.2 (score equation)] The score s(·) = Σ_{i=1..5} i·[p(Better|i) − p(Worse|i)] assumes that icl_1 ≺ icl_2 ≺ ... ≺ icl_5 along the evaluated dimension. The generation procedure in §2.1 prompts for the worst and best summaries and then recursively asks for intermediate summaries, but the manuscript provides no systematic evidence that the resulting chain is monotonically ordered. Figure 1 is a single illustrative example; no human validation, no independent-model check, and no analysis of the released synthetic data are reported. If any adjacent pair is inverted or tied, the weights i and −i are misapplied and the final score is distorted. Since every downstream comparison and headline result depend on this ordering, this is a load-bearing assumption. Please add a validation study (e.g., human or an independent judge rating the five anchors per article and reporting inversion/tie rates), or at minimum analyze t
[Table 1, §4] The headline results are differences of +0.03, −0.03, and +0.05 in axis-averaged Spearman correlation. Table 1 reports point estimates only, with no bootstrap confidence intervals, standard errors, or significance tests. Given that these differences are small, the reader cannot determine whether 'performs comparably' is a real effect or within sampling noise. The paper's own ablation in Table 2 shows non-negligible variance for sampling-based variants, and the correlation estimates on finite document/machine sets also carry uncertainty. Please report confidence intervals (e.g., bootstrap) or paired significance tests against the key baselines (notably PairS-beam and G-Eval) for the three datasets.
[§3, Table 4] NewsRoom is omitted from the main results, and Table 4 in the appendix—which reports NewsRoom performance for other methods—does not include the proposed method at all. The stated reason (BERTScore with roberta-large attains high performance on NewsRoom) does not explain why 'Ours' is absent. As a result, the paper cannot support a claim of general comparability to pairwise evaluators across the standard benchmark suite. Please add the proposed method to Table 4 (and ideally to Table 1), or provide a substantive, method-specific justification for the omission.
[§1, §5] The paper motivates direct scoring by the need for absolute scores in threshold-based filtering and sorting. However, all experiments only measure rank correlation (Spearman ρ), which is invariant to monotone transformations of the score. No analysis is provided of whether the proposed scores are calibrated to the 1–5 human scale or whether applying a threshold to s(·) would select high-quality outputs. To support the claimed practical advantage, please include a calibration analysis (e.g., mean predicted score per human rating bin, or a threshold-based selection experiment).

minor comments (6)

[Abstract] Typo: 'Forsample-level' should be 'For sample-level'.
[§2.1] 'inter-ratingconsistency' is missing a space: should be 'inter-rating consistency'.
[§6 Limitations] 'we ablate on on a selection' contains a duplicated 'on'.
[Table 1] The baseline named 'Direct Scoring' can be confused with the proposed method (which is also a direct-scoring evaluator). Consider renaming the baseline to something like 'Direct Scoring (G-Eval-style)' or 'Linear Scoring' for clarity.
[References] References to 'Liu et al. 2024a' and '2024b' appear to describe the same work (identical titles); consolidate or distinguish them. Also 'Fabbri et al., 2021b,b' has a duplicated 'b'.
[Tables 5-6] The axis abbreviations (COH, FLU, CON, REL, INF) are defined in the captions, but the definitions are easy to miss; consider adding a footnote with the full names in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the synthetic-anchor scoring method is evaluated against external human benchmarks and involves no fitted-input-as-prediction or self-citation chain.

full rationale

The derivation chain is self-contained. Synthetic in-context summaries are produced by a prompted LLM (Section 2.1); the final score is defined in Section 2.2 as a weighted sum of the same LLM's comparison probabilities against those anchors: s(·)=Σ_{i=1..5} i·p(Better|i) − i·p(Worse|i). No parameter is fitted to the human labels of SummEval, TopicalChat, or HANNA; the reported correlations are computed after the fact against external judgments. The ablations (Tables 2–3) select among prompt schemes and N, but this is ordinary model selection, not a circular reduction: the score formula would be the same without those choices. The main limitation—that the monotone ordering of the synthetic anchors is assumed rather than validated, and that the same model family generates and judges—is a genuine validity/robustness concern, but it is not a case where a prediction is equal to its input by construction or where a load-bearing claim rests on a self-citation. Therefore no circular step is established.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The method introduces no new physical or theoretical entities; it creates synthetic text examples as internal reference points.

free parameters (1)

N (number of synthetic in-context examples) = 5 (default), ablation tested 2,3,5,9
The number of synthetic summaries per article/dimension is a design choice. The ablation in Table 2 shows N=9 gives higher correlation (37.80 vs 37.43), but N=5 is kept for simplicity and alignment with common rating scales, making it a hand-chosen hyperparameter.

assumptions (2)

domain assumption LLM-generated synthetic summaries are monotonically increasing in quality with the assigned score level
Section 2.1 aims for 'inter-rating consistency' but provides no human validation that the generated summaries actually improve from score 1 to 5.
domain assumption Softmax probabilities over 'Better/Worse/Similar' comparisons against the anchors aggregate into a meaningful continuous score
The scoring formula s(.) = sum_i [i,-i,0] * [pB,pW,pS] assumes the probability-weighted sum is a valid proxy for quality; no calibration analysis is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Direct-Scoring NLG Evaluators Can Use Pairwise Comparisons Too." pith.science (2026). https://pith.science/paper/FUTCSY4G

@misc{pith2026250905440,
  author       = {Pith},
  title        = {Pith review of: Direct-Scoring NLG Evaluators Can Use Pairwise Comparisons Too},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FUTCSY4G}},
  note         = {Machine review of arXiv:2509.05440}
}

read the original abstract

As large-language models have been increasingly used as automatic raters for evaluating free-form content, including document summarization, dialog, and story generation, work has been dedicated to evaluating such models by measuring their correlations with human judgment. For \textit{sample-level} performance, methods which operate by using pairwise comparisons between machine-generated text perform well but often lack the ability to assign absolute scores to individual summaries, an ability crucial for use cases that require thresholding. In this work, we propose a direct-scoring method which uses synthetic summaries to act as pairwise machine rankings at test time. We show that our method performs comparably to state-of-the-art pairwise evaluators in terms of axis-averaged sample-level correlations on the SummEval (\textbf{+0.03}), TopicalChat (\textbf{-0.03}), and HANNA (\textbf{+0.05}) meta-evaluation benchmarks, and release the synthetic in-context summaries as data to facilitate future work.

Figures

Figures reproduced from arXiv: 2509.05440 by the authors.

**Figure 1.** Overview of Method. (Left) First an LLM is prompted to generate summaries reflecting various levels of quality using a contrastive scheme (orange), then compared to the machine summary to generate probabilities over comparative language (green), eg. "Better", "Similar", and "Worse". (Right) We show an example of summaries (blue) of increasing quality (scores 1 through 5) for a SummEval article (grey) on the "consist… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

[1]

Better"|i), ...: Table 13 (d)p(

SummEval (a) Best/Worst (Scores 1,5): Table 7 (b) Recursive (Scores 2,3,4): Table 10 (c)p("Better"|i), ...: Table 13 (d)p("Yes"|i), p("No"|i): Table 16

work page
[2]

Better"|i), ...: Table 14 (d)p(

TopicalChat (a) Best/Worst (Scores 1,5): Table 8 (b) Recursive (Scores 2,3,4): Table 11 (c)p("Better"|i), ...: Table 14 (d)p("Yes"|i), p("No"|i): Table 17

work page
[3]

Better"|i), ...: Table 15 (d)p(

HANNA (a) Best/Worst (Scores 1,5): Table 9 (b) Recursive (Scores 2,3,4): Table 12 (c)p("Better"|i), ...: Table 15 (d)p("Yes"|i), p("No"|i): Table 18 You will be given a source document and an evaluation dimension for a summary. Your task is to write the {{ worst_best : str } possible summary you can think of with regards to this dimension. Your response s...

work page
[4]

Better"

Respond with only one of the following: "Better" "Worse" or "Similar" which indicate whether the target summary is better than, worse than, or similar to the reference summary. Original Article: {{ article : str }} Reference Summary: {{ icl_summary : str }} Target Summary: {{ target_summary : str }} Table 13:SummEvalp("Better"), ...Prediction Prompt Templ...

work page
[6]

Read the news article carefully and identify the main facts and details it presents

work page
[7]

Compare them to the article

Read the target summary and example summary. Compare them to the article

work page
[8]

Compare the quality of the target summary to reference summary with respect to {{ col : str }}

work page
[13]

Better"

Respond with only one of the following: "Better" "Worse" or "Similar" which indicate whether the evaluation story is better than, worse than, or similar to the reference story. Story Idea: {{ story_prompt : str }} Reference Story: {{ icl_summary : str }} Evaluation Story: {{ target_summary : str }} Table 14:TopicalChatp("Better"), ...Prediction Prompt Tem...

work page

Show all 14 references

[14]

Read the story carefully
[15]

Compare them to the idea

Read the reference story and evaluation story. Compare them to the idea
[16]

Compare the quality of the evaluation story to the reference story with respect to {{ col : str }}
[17]

Better"

Respond with only one of the following: "Better" "Worse" or "Similar" which indicate whether the evaluation story is better than, worse than, or similar to the reference story. Story Idea: {{ story_prompt : str }} Reference Story: {{ icl_summary : str }} Evaluation Story: {{ t...
[2020]

coherence,

Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lél...

2009 arXiv
[2024]

Michael Denkowski and Alon Lavie

Do language models enjoy their own stories? prompting large language models for automatic story evaluation.Transactions of the Association for Com- putational Linguistics, 12:1122–1142. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation eva...

2014 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Better"|i), ...: Table 13 (d)p(

SummEval (a) Best/Worst (Scores 1,5): Table 7 (b) Recursive (Scores 2,3,4): Table 10 (c)p("Better"|i), ...: Table 13 (d)p("Yes"|i), p("No"|i): Table 16

work page

[2] [2]

Better"|i), ...: Table 14 (d)p(

TopicalChat (a) Best/Worst (Scores 1,5): Table 8 (b) Recursive (Scores 2,3,4): Table 11 (c)p("Better"|i), ...: Table 14 (d)p("Yes"|i), p("No"|i): Table 17

work page

[3] [3]

Better"|i), ...: Table 15 (d)p(

HANNA (a) Best/Worst (Scores 1,5): Table 9 (b) Recursive (Scores 2,3,4): Table 12 (c)p("Better"|i), ...: Table 15 (d)p("Yes"|i), p("No"|i): Table 18 You will be given a source document and an evaluation dimension for a summary. Your task is to write the {{ worst_best : str } possible summary you can think of with regards to this dimension. Your response s...

work page

[4] [4]

Better"

Respond with only one of the following: "Better" "Worse" or "Similar" which indicate whether the target summary is better than, worse than, or similar to the reference summary. Original Article: {{ article : str }} Reference Summary: {{ icl_summary : str }} Target Summary: {{ target_summary : str }} Table 13:SummEvalp("Better"), ...Prediction Prompt Templ...

work page

[5] [6]

Read the news article carefully and identify the main facts and details it presents

work page

[6] [7]

Compare them to the article

Read the target summary and example summary. Compare them to the article

work page

[7] [8]

Compare the quality of the target summary to reference summary with respect to {{ col : str }}

work page

[8] [13]

Better"

Respond with only one of the following: "Better" "Worse" or "Similar" which indicate whether the evaluation story is better than, worse than, or similar to the reference story. Story Idea: {{ story_prompt : str }} Reference Story: {{ icl_summary : str }} Evaluation Story: {{ target_summary : str }} Table 14:TopicalChatp("Better"), ...Prediction Prompt Tem...

work page