REVIEW 7 cited by
The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The "LLM-as-an-annotator" and "LLM-as-a-judge" paradigms employ Large Language Models (LLMs) as annotators, judges, and evaluators in tasks traditionally performed by humans. LLM annotations are widely used, not only in NLP research but also in fields like medicine, psychology, and social science. Despite their role in shaping study results and insights, there is no standard or rigorous procedure to determine whether LLMs can replace human annotators. In this paper, we propose a novel statistical procedure, the Alternative Annotator Test (alt-test), that requires only a modest subset of annotated examples to justify using LLM annotations. Additionally, we introduce a versatile and interpretable measure for comparing LLM annotators and judges. To demonstrate our procedure, we curated a diverse collection of ten datasets, consisting of language and vision-language tasks, and conducted experiments with six LLMs and four prompting techniques. Our results show that LLMs can sometimes replace humans with closed-source LLMs (such as GPT-4o), outperforming the open-source LLMs we examine, and that prompting techniques yield judges of varying quality. We hope this study encourages more rigorous and reliable practices.
Forward citations
Cited by 7 Pith papers
-
ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering
An LLM-based proxy can reproduce human relevance judgments about topic model outputs closely enough to substitute for an average human annotator, and classical LDA remains competitive under this test.
-
Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe
Instruction-tuned LLMs collapse per-call outputs onto a single answer while still being able to state the target distribution accurately in one call; prompt perturbation recovers some variation.
-
Multi-Domain Explainability of Preferences
A concept-discovery plus hierarchical regression pipeline explains human, LLM-judge, and reward-model preferences at local and global levels across eight domains.
-
Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
Hidden-state factuality probes trained on synthetic statements do not generalize to LLM-generated factual statements, despite reproducing prior results on original datasets.
-
Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
Tool-augmented LLM annotators improve agreement with ground-truth preferences on long-form factual and coding tasks, with mixed results on math, compared to standard LLM-as-a-judge baselines.
-
Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications
A practical framework for application-level LLM safety testing: organization-specific taxonomies plus black-box adversarial evaluation, illustrated by a Singapore government pilot.
-
Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods
LLMs extend, rather than replace, classical social science methods, with a proposed three-tier bias framework for LLM-augmented surveys.
Discussion (0). Sign in to comment.