REVIEW 53 cited by
Can Large Language Models Be an Alternative to Human Evaluations?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans. However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms. Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation. We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation. We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: open-ended story generation and adversarial attacks. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs. We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer. We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.
Forward citations
Cited by 53 Pith papers
-
Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models
LLM probability estimates violate the law of total probability across partitions, and subgroup-aggregated estimates often beat direct population-level estimates (the macro fallacy).
-
Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models
An agentic framework and a distilled 3B model evaluate text-to-image and text-to-video generators with roughly 10% of the samples of standard benchmarks, at the cost of only partial accuracy.
-
VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP
The VIVID benchmark shows that current LLMs, including GPT-4o, interpret Vietnamese idioms and proverbs at less than half of the maximum score, exposing a cultural competence gap.
-
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.
-
Capital Markets LLM Reliability Score (CM-LRS): From Plausible to Bankable
CM-LRS is a workflow-output-layer reliability score for capital-markets LLM outputs; on five public-document workflows, frontier closed-source models score 4.09–4.31 and Llama 3.3 70B scores 3.15 under four LLM judges.
-
Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration
A multi-agent prompt-refinement system using pairwise AI judging and targeted edit signals outperforms prior automated methods on complex text-to-image tasks.
-
AdsQA: Towards Advertisement Video Understanding
AdsQA adds an ad-video question-answering benchmark and ReAd-R, a GRPO-trained model that beats 7B baselines but not larger closed models.
-
Rule2Text: A Framework for Generating and Evaluating Natural Language Explanations of Knowledge Graph Rules
Rule2Text generates and evaluates natural language explanations of knowledge graph rules, finding that chain-of-thought prompting with entity types works best and that fine-tuning Zephyr on LLM-built ground truth shar...
-
CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation
A large-scale LLM-driven urban simulator with recursive planning, memory, and belief modules, claimed to reproduce real-world time use, travel, and crowd patterns better than prior agent frameworks.
-
The MedPerturb Dataset: What Non-Content Perturbations Reveal About Human and Clinical LLM Decision Making
MedPerturb finds that LLMs are more sensitive to gender and style changes in clinical text, while medical students are more sensitive to LLM-generated summaries and dialogues, in triage decisions.
-
How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG
A new GraphRAG evaluation framework using graph-grounded questions and bias-correction yields much smaller win rates than earlier reports, casting doubt on reported GraphRAG gains.
-
Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge
A generalised Product-of-Experts framework with a new 'probability of reordering' selection metric that reduces the number of LLM comparisons needed for ranking by about 50%.
-
LLM-based Evaluation Policy Extraction for Ecological Modeling
APEF learns interpretable evaluation policies for ecological time-series models by combining an LLM-driven weight optimizer with human or predefined pairwise preference annotations.
-
A Case Study Investigating the Role of Generative AI in Quality Evaluations of Epics in Agile Software Development
An LLM-based evaluator for agile epics was built from a new rubric and tested with 17 product managers, who found it useful but limited by lack of domain knowledge and rigid scoring.
-
SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation
An LLM-based user simulator with persona matching, memory, and visual perception aligns more closely with real users than earlier agent models and can support offline A/B testing for recommender systems.
-
Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity
Multi-agent debate mostly underperforms simple chain-of-thought baselines when tested broadly, while randomly mixing different models into the debate reliably improves performance.
-
A Benchmark for the Detection of Metalinguistic Disagreements between LLMs and Knowledge Graphs
Metalinguistic disagreements, where the dispute is over word meaning rather than facts, appear in LLM fact-checking against knowledge graphs, based on a 250-triple pilot study.
-
Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
Twenty-nine interviews show AI practitioners rely on synthetic data across nearly every pipeline stage while validation remains mostly manual spot-checking.
-
Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course
In a semester-long bioinformatics course, LLM graders with hand-tuned rubrics and examples graded written answers in 85 to 90 percent agreement with human TAs, but the accuracy was measured on the same set used to des...
-
Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models
Evaluation Agent is an LLM-agent framework that evaluates visual generative models with a handful of samples per round, claiming a 10x time reduction while keeping conclusions within one tier of full-benchmark results...
-
Generative Adversarial Reviews: When LLMs Become the Critic
A new LLM-agent framework, GAR, generates peer reviews from a graph representation of manuscripts and predicts conference acceptance decisions, reportedly matching or exceeding human reviewer performance.
-
Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
A controlled, multi-family study shows the relative generation-verification gap grows with pretraining flops for stable verification methods, and iterative self-improvement saturates quickly.
-
Multi-Facet Blending for Faceted Query-by-Example Retrieval
A synthetic-data augmentation method that decomposes documents into facets and recombines LLM-written similar and dissimilar fragments improves faceted query-by-example retrieval without citation labels.
-
Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension
HuCoSC scores generated code by recursively decomposing code into sub-blocks, storing dependency semantics, and using an LLM to compare the semantic descriptions of reference and generated code.
-
Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensiveness
Across five offensiveness and hate speech datasets, LLM alignment with human annotators is inconsistent for gender and ethnicity, with the only rounded-average consistency being lower alignment with Black than White a...
-
Scaling Point-in-Time Language Models
Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.
-
Game Theory Driven Multi-Agent Framework Mitigates Language Model Hallucination
A game-framed multi-agent system synthesizes large chemistry CoT/QA corpora and trains OmniChem-7B to near GPT-4o-mini performance with a large reported drop in hallucinations.
-
RESBev: Making BEV Perception More Robust
A latent world model predicts clean BEV semantic features from sequential observations to recover existing Lift-Splat-Shoot pipelines under natural and adversarial corruption with few-shot fine-tuning.
-
LaQual: An Automated Framework for LLM App Quality Evaluation
LaQual automates LLM app-store quality evaluation through scenario classification, static indicator filtering, and LLM-generated dynamic metrics, with Spearman correlations of about 0.6 against human ratings.
-
Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation
Fine-tuning video generators on driving data can improve visual fidelity while degrading how accurately the model predicts the movement of cars and pedestrians.
-
Generative Artificial Intelligence Extracts Structure-Function Relationships from Plants for New Materials
A generative AI framework that reads plant structure-function literature, generates hypotheses, and produces a lab-validated pollen-based adhesive with measured shear strength.
-
Balancing Information Accuracy and Response Timeliness in Networked LLMs
For binary questions, combining m specialized LLMs with a Bayesian majority rule improves accuracy, and the paper derives the optimal m that trades accuracy against system delay.
-
Evaluating Causal Explanation in Medical Reports with LLM-Based and Human-Aligned Metrics
On a small set of NTCIR-18 reports, GPT-based evaluators, especially GPT-Black, tracked expert judgments of causal medical explanations better than similarity metrics such as BERTScore and cosine similarity.
-
Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation
PrefBERT, a 150M-parameter reward model trained on human quality ratings, outperforms ROUGE-L and BERTScore as a GRPO reward signal for open-ended long-form generation.
-
OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software
A framework using LLM agents to simulate developers at four experience levels and automatically evaluate open source platform usability from generated code.
-
An Empirical Study of Evaluating Long-form Question Answering
In long-form QA, LLM-based evaluators correlate with human judgments better than ROUGE or BERTScore, but they are biased by answer length, question type, self-reinforcement, and rare-word usage, and fine-grained promp...
-
A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
Relevance labels produced by LLMs depend on the prompt and the judge model; LLM-written prompts are more consistent than expert-written prompts across three assessment paradigms, and graded 0-to-3 judgments are unstab...
-
Image Embedding Sampling Method for Diverse Captioning
A training-free hierarchical embedding sampling method (HBoP) lets a small BLIP model generate captions as diverse as human ones, beating much larger VLMs on diversity metrics.
-
Can Large Language Models Predict the Outcome of Judicial Decisions?
Fine-tuning a small LLaMA model on a new Arabic legal dataset yields near-par performance with a larger model, but the generalization claim is tested on the same instructions used during training.
-
The Impostor is Among Us: Can Large Language Models Capture the Complexity of Human Personas?
Participants distinguished human-written from GPT-4o-generated personas, rating AI personas higher on informativeness, positivity, consistency, and clarity but also higher on stereotypicality.
-
Can Large Language Models Serve as Evaluators for Code Summarization?
An LLM prompt that makes the model role-play a code reviewer scores code summaries with 81.59% Spearman correlation with human judgment, outperforming BLEU and BERTScore on a 300-sample benchmark.
-
ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation
ReFINE is a fine-tuned Llama3 reward model that scores radiology reports on multiple criteria through a margin-based loss, showing higher correlation with human ratings than prior metrics.
-
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
Multi-LLM pairwise Elo ranking with adjustable consensus thresholds produces rankings of competency profiles that average Spearman ρ≈0.83 with expert judgments.
-
Integrating Generative AI into Cybersecurity Education: A Study of OCR and Multimodal LLM-assisted Instruction
An OCR-and-LLM pipeline integrated into a legacy cybersecurity lab platform delivers comparable instructional value to multimodal models on text-heavy slides with lower cost.
-
Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Counterfactual Voting Adjustment fits an answer-level logistic voting model and averages over rank and prior-vote distributions to estimate quality, yielding rankings that align better with GPT-4o helpfulness judgment...
-
CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation
CCRS is a zero-shot LLM-as-a-judge framework whose five metrics discriminate between RAG systems on BioASQ with comparable or better power than RAGChecker at lower compute.
-
OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
The paper advocates protecting and leveraging OpenReview's peer review corpus as a community asset for LLM-based review assistance, benchmarks, and alignment.
-
EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta
EQUATOR uses vector search to fetch a human reference answer and a binary LLM grader to score open-ended LLM responses, producing far lower scores than standard benchmarks.
-
Do LLMs Agree on the Creativity Evaluation of Alternative Uses?
Four LLMs show high agreement when scoring and ranking alternative uses and do not favor their own outputs, but the accuracy benchmark is derived from the generation prompts rather than human judgment.
-
SRSA: A Cost-Efficient Strategy-Router Search Agent for Real-world Human-Machine Interactions
SRSA, a router that picks between direct, parallel, and planning searches for each query, improves informativeness and completeness on a new contextual-query benchmark while using fewer LLM inference steps than a ReAct agent.
-
The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.
-
(WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges
Fine-tuning PHI-3 on TruthfulQA with a hand-tuned prompt yields a reported 90.8% accuracy, but the paper describes no held-out evaluation.
-
The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?
A review of known vulnerabilities in LLM benchmarks (contamination, overfitting, human and LLM judge bias) with a sketch of a proposed zero-day evaluation framework.
Discussion (0). Continue with ORCID to comment.