REVIEW 19 cited by
Is ChatGPT a Good NLG Evaluator? A Preliminary Study
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regard ChatGPT as a human evaluator and give task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruction to prompt ChatGPT to evaluate the generated results of NLG models. We conduct experiments on five NLG meta-evaluation datasets (including summarization, story generation and data-to-text tasks). Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases. In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets. For the meta-evaluation datasets which are created greatly depending on the reference and thus are biased, the ChatGPT evaluator might lose its effectiveness. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.
Forward citations
Cited by 19 Pith papers
-
Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm
A trigger-tag watermark embedded by fine-tuning lets modified LLMs mark their own phishing outputs for cheap detection.
-
Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts
HAMLET introduces a three-level key-fact hierarchy with automated query-focused summarization to evaluate LLM recall and faithfulness in book-length contexts, achieving over 90% agreement with experts.
-
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.
-
RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model
RetiBridge/GROK is a 7B multimodal LLM trained to turn quantitative retinal biomarkers into qualitative diagnoses, outperforming larger open models and OpenAI-o3 on its self-built, LLM-judged benchmark.
-
Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge
A generalised Product-of-Experts framework with a new 'probability of reordering' selection metric that reduces the number of LLM comparisons needed for ranking by about 50%.
-
R-TOFU: Unlearning in Large Reasoning Models
R-TOFU shows that answer-level unlearning is insufficient for large reasoning models because residual knowledge persists in chain-of-thought traces.
-
Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance
A benchmark of 19 small language models on 2,000 news articles shows that the best small models nearly match 70B-parameter LLMs in summary quality while producing shorter summaries.
-
Optimization is Better than Generation: Optimizing Commit Message Leveraging Human-written Commit Message
A commit-message optimization method that starts from human-written messages and uses GPT-4 plus automated evaluators outperforms message generation and completion methods on three of four quality metrics.
-
Retrieval-Augmented Recommendation Explanation Generation with Hierarchical Aggregation
REXHA improves recommendation explanation generation by hierarchically summarizing all reviews into user and item profiles and using pseudo-document queries to retrieve relevant reviews for a language model.
-
$I^2G$: Generating Instructional Illustrations via Text-Conditioned Diffusion
A pairwise-conditioned diffusion model generates instructional illustrations from procedural text and is finetuned with a text-image alignment reward.
-
LLMs to Support a Domain Specific Knowledge Assistant
A synthetic QA dataset for IFRS sustainability reporting is created with LLMs and used to build and evaluate two QA pipelines, with the fully LLM-based pipeline scoring highest.
-
Reason4Rec: Deliberative User Preference Alignment of Large Language Models for Recommendation
An LLM recommender that decomposes preference reasoning into summarization, matching, and prediction steps, trained on review text, improves rating accuracy and explanation quality on Amazon and Yelp benchmarks.
-
Can LLMs Ask Good Questions?
LLM-generated questions mostly ask for descriptions and long answers, and they use the source text more evenly than human-authored questions.
-
Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM
SumAutoEval is an LLM-based entity-level summarization evaluator with four dimensions; its claimed human-correlation advantage is not consistently supported by the experiments.
-
Can Large Language Models Serve as Evaluators for Code Summarization?
An LLM prompt that makes the model role-play a code reviewer scores code summaries with 81.59% Spearman correlation with human judgment, outperforming BLEU and BERTScore on a 300-sample benchmark.
-
LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Data
A lightweight RGB-D cross-attention network is proposed for rail defect detection, but the SOTA accuracy and generalization claims are internally inconsistent and the implementation is not public.
-
Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons
A knockout tournament of iterative pairwise LLM comparisons improves agreement with human expert scores by 0.07 Pearson on average across exam grading and MT evaluation.
-
The Performance of the LSTM-based Code Generated by Large Language Models (LLMs) in Forecasting Time Series Data
This paper reports that LLM-generated LSTM forecast code can reach RMSE values close to a generic manually built LSTM on 10 stock datasets, but the comparison is weakened by cherry-picked best results and no repeated ...
-
Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Method
Fine-tuning Stable Diffusion with DreamBooth-style knowledge and hypernetwork-guided crack control maps can synthesize substation meter defect images that boost a YOLOv8 defect detector's mAP when added to the training set.
Discussion (0). Continue with ORCID to comment.