arxiv: 2604.09497 · v1 · submitted 2026-04-10 · 💻 cs.CL · cs.AI

Recognition: unknown

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

Hippolyte Gisserot-Boukhlef , Nicolas Boizard , Emmanuel Malherbe , C\'eline Hudelot , Pierre Colombo

Authors on Pith no claims yet

Pith reviewed 2026-05-10 16:42 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords LLM evaluationBERTsemantic correctnessreference-based evaluationlexical methodsencoder modelsgenerative AI assessment

0 comments

The pith

A fine-tuned BERT model judges LLM answer correctness as reliably as large models but at far lower cost.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper demonstrates that lexical methods for scoring LLM outputs, which rely on exact word matches, align poorly with human views of whether an answer is semantically correct. Through tests across 36 models and 15 tasks, it establishes that these rigid checks often penalize valid responses that vary in phrasing. To fix this, the authors train a BERT encoder on synthetic question-candidate-reference examples to assess meaning rather than format. The resulting system runs efficiently and reaches performance levels close to those of much larger LLM-based judges. If the approach holds, it would allow fast, scalable evaluation of generative models without the expense of heavy inference.

Core claim

BERT-as-a-Judge is an encoder-driven approach for assessing answer correctness in reference-based generative settings. It requires only lightweight training on synthetically annotated question-candidate-reference triplets, proves robust to variations in output phrasing, and consistently outperforms lexical baselines while matching the performance of much larger LLM judges.

What carries the argument

BERT-as-a-Judge, a fine-tuned encoder model that scores whether a candidate answer matches a reference in meaning after training on synthetic triplets.

If this is right

Evaluation of new LLMs becomes feasible at scale without high compute budgets.
Developers can assess semantic correctness rather than forcing strict output formats.
Insights from the experiments guide practical choices of training data and model size for similar judges.
The method supports reliable comparisons across diverse downstream tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Similar encoder-based judges could be trained on other base models to trade off speed and accuracy further.
Production systems that generate many outputs might adopt this for ongoing quality monitoring where full LLM judges are impractical.
The success of synthetic data here raises the question of how well such training generalizes to entirely new domains or languages.

Load-bearing premise

Synthetically generated training examples accurately reflect how humans would judge semantic correctness without introducing systematic errors or biases.

What would settle it

Human ratings on a fresh set of LLM outputs where BERT-as-a-Judge scores diverge from or underperform both lexical baselines and larger LLM judges.

Figures

Figures reproduced from arXiv: 2604.09497 by C\'eline Hudelot, Emmanuel Malherbe, Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Pierre Colombo.

**Figure 2.** Figure 2: Quantification of regex parsing failures. Values represent the failure rate, defined [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Comparison between encoder-based evaluation and LLM judges from the Qwen-3 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: BERT-as-a Judge evaluation quality across different training budgets. BERT-as-a-Judge is training-efficient. By default, we train encoder models on 1M question-candidate-reference triplets. In this experiment, we evaluate lighter configurations: 500K, 200K, and 100K samples ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 5.** Figure 5: Effect of score thresholding on BERT-as-a-Judge downstream assessment accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

**Figure 6.** Figure 6: Sensitivity of the AH estimate to variations in AS and ρ. 12In Equation 4, we assume (Yˆ,YS) ⊥ (YH = YS), in line with our empirical observations. Intuitively, this means that agreement between the predicted and synthetic labels, Yˆ and YS, is independent of whether the synthetic label YS matches the human label YH. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗

read the original abstract

Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model's true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judgments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthetically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling tradeoff between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge's performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

BERT-as-a-Judge gives a practical lightweight alternative to LLM judges for reference-based eval, backed by a solid lexical study but resting on unverified synthetic-to-human transfer.

read the letter

The main thing to know is that this paper first runs a large comparison showing lexical metrics like exact match or F1 align poorly with human judgments across 36 models and 15 tasks, then trains a BERT encoder on synthetic question-candidate-reference triplets to score semantic correctness. It reports beating the lexical baselines while matching the accuracy of much larger LLM judges at far lower cost, and they release the artifacts.

Referee Report

1 major / 1 minor

Summary. The paper introduces BERT-as-a-Judge, an encoder-based model trained on synthetically annotated (question, candidate, reference) triplets for reference-based evaluation of LLM generative outputs. It reports a large-scale empirical study across 36 models and 15 tasks showing that lexical methods correlate poorly with human judgments, and claims that the proposed lightweight BERT approach outperforms lexical baselines while matching the performance of much larger LLM judges, offering an efficient and scalable alternative.

Significance. If the results hold, the work provides a practical tradeoff for LLM evaluation by delivering semantic assessment at lower computational cost than LLM judges while avoiding the inaccuracies of lexical methods. The release of all project artifacts supports reproducibility and downstream use. The systematic study of lexical limitations across many models and tasks adds empirical value to the evaluation literature.

major comments (1)

[Abstract] Abstract: The central claim that BERT-as-a-Judge 'matches the performance of much larger LLM judges' and provides a 'robust' alternative relies on evaluation against LLM judges using synthetically annotated data; however, no direct correlation study with independent human judgments is reported for the trained BERT model on held-out real annotations. This is load-bearing because synthetic labels (typically LLM-generated) may not proxy human semantic judgments without bias, as noted in the weakest assumption.

minor comments (1)

[Abstract] Abstract: The summary of results from 36 models and 15 tasks provides no specific metrics, statistical tests, data splits, or numerical performance values, which limits the ability to verify the outperformance claims.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. The major comment correctly identifies a gap in our validation strategy for BERT-as-a-Judge. We address it point-by-point below and commit to revisions that improve transparency without overstating our results.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim that BERT-as-a-Judge 'matches the performance of much larger LLM judges' and provides a 'robust' alternative relies on evaluation against LLM judges using synthetically annotated data; however, no direct correlation study with independent human judgments is reported for the trained BERT model on held-out real annotations. This is load-bearing because synthetic labels (typically LLM-generated) may not proxy human semantic judgments without bias, as noted in the weakest assumption.

Authors: We agree this is a substantive limitation. Our large-scale study (36 models, 15 tasks) demonstrates that lexical methods correlate poorly with human judgments, establishing the need for semantic evaluation. BERT-as-a-Judge is trained and evaluated on synthetic (question, candidate, reference) triplets and is shown to match LLM judges in agreement metrics on held-out synthetic data. We do not report a new direct human correlation study for the trained BERT model on independent real annotations, as the referee notes. This reliance on synthetic labels and LLM judges as proxy is acknowledged in the paper's assumptions section, but the abstract claim could be read as stronger than the evidence supports. We will revise the abstract to state more precisely that BERT-as-a-Judge matches LLM judges on synthetic test sets while outperforming lexical baselines. We will also expand the discussion and limitations sections to explicitly address potential biases in synthetic annotations, reference prior work on LLM-human agreement, and clarify the scope of our claims. These changes will be made in the next version. revision: yes

Circularity Check

0 steps flagged

No significant circularity; relies on empirical comparisons to external baselines

full rationale

The paper's core claims rest on a large-scale empirical study across 36 models and 15 tasks showing lexical methods correlate poorly with humans, followed by training BERT-as-a-Judge on synthetically annotated triplets and reporting that it outperforms lexical baselines while matching larger LLM judges. These are performance measurements against independent external references (lexical metrics, human correlations for baselines, and LLM-judge outputs), not derivations that reduce by construction to fitted parameters or self-referential definitions. No equations or steps equate the model's output to its training inputs via self-definition, and no load-bearing uniqueness theorem or ansatz is imported from the authors' prior work. Minor self-citation risk is possible in the broader literature but is not load-bearing here.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based solely on the abstract; the method rests on the assumption that synthetic annotations serve as valid training signals for semantic correctness. No explicit free parameters or invented entities are described.

axioms (1)

domain assumption Synthetically generated annotations accurately reflect human notions of answer correctness
The training process depends on these annotations being reliable proxies for human judgment.

pith-pipeline@v0.9.0 · 5546 in / 1103 out tokens · 62185 ms · 2026-05-10T16:42:22.631507+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 1 canonical work pages · 1 internal anchor

[1]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...
[2]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...
[3]

Holistic Evaluation of Language Models

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.5281/zenodo.12608602 2022
[4]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...