Pith. sign in

REVIEW 20 cited by

LLM-based NLG Evaluation: Current Status and Challenges

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.01383 v3 pith:WMGEVX4O submitted 2024-02-02 cs.CL

Mingqi Gao , Xinyu Hu , Jie Ruan , Xiao Pu , Xiaojun Wan This is my paper · ORCID
classification cs.CL
keywords evaluationllmslanguagediscussllm-basedmethodsmetricsnatural
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Evaluating natural language generation (NLG) is a vital but challenging problem in natural language processing. Traditional evaluation metrics mainly capturing content (e.g. n-gram) overlap between system outputs and references are far from satisfactory, and large language models (LLMs) such as ChatGPT have demonstrated great potential in NLG evaluation in recent years. Various automatic evaluation methods based on LLMs have been proposed, including metrics derived from LLMs, prompting LLMs, fine-tuning LLMs, and human-LLM collaborative evaluation. In this survey, we first give a taxonomy of LLM-based NLG evaluation methods, and discuss their pros and cons, respectively. Lastly, we discuss several open problems in this area and point out future research directions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models

    cs.CL 2025-01 conditional novelty 7.0 of 10

    MDEval scores Markdown Awareness as the normalized edit distance between a model's HTML-tagged output and a GPT-4o rewrite, and reports human-alignment accuracy of 84.1% when ties are excluded.

  2. From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs

    cs.CL 2026-07 conditional novelty 6.5 of 10

    On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.

  3. AI Analyst: Framework and Comprehensive Evaluation of Large Language Models for Financial Time Series Report Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LLMs such as GPT-4o can generate coherent financial reports from time series data, and a proposed highlighting system categorizes report segments by whether they stem from data, reasoning, or external knowledge.

  4. BriefMe: A Legal NLP Benchmark for Assisting with Legal Briefs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    BriefMe introduces a legal brief benchmark with argument summarization, argument completion, and case retrieval, and shows LLMs beat human headings on the first two but struggle on the latter two.

  5. LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learning

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A fine-tuned multimodal LLM can rate slide-based lecture quality in line with human judges, outperforming generic metrics and prompted LLMs on a new annotated dataset.

  6. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Output-based LLM-as-a-judge methods using large LLMs achieve near-human correlation with human scores for code translation and generation, but not for code summarization or pairwise comparisons.

  7. Optimization is Better than Generation: Optimizing Commit Message Leveraging Human-written Commit Message

    cs.SE 2025-01 conditional novelty 6.0 of 10

    A commit-message optimization method that starts from human-written messages and uses GPT-4 plus automated evaluators outperforms message generation and completion methods on three of four quality metrics.

  8. Natural Language Reinforcement Learning

    cs.LG 2024-11 conditional novelty 6.0 of 10

    NLRL replaces scalar RL values with LLM-generated language narratives, trains language critics with language MC/TD, and improves policies via LLM-based policy iteration, outperforming PPO on four small agentic tasks.

  9. FlashDP: Private Training Large Language Models with Efficient DP-SGD

    cs.LG 2025-07 conditional novelty 5.0 of 10

    FlashDP fuses per-sample gradient computation, norm calculation, clipping, and noise addition into a cache-friendly block-wise all-reduce workflow that avoids explicit per-sample gradient storage and redundant recomputation.

  10. Enterprise Large Language Model Evaluation Benchmark

    cs.AI 2025-06 reject novelty 5.0 of 10

    A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...

  11. Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving

    eess.AS 2025-05 conditional novelty 5.0 of 10

    Multi-task behavior imitation with speech-text interleaving improves speech LLM generalization on prompts and zero-shot tasks using only paired speech and transcripts.

  12. FactsR: A Safer Method for Producing High Quality Healthcare Documentation

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FactsR decomposes clinical note generation into real-time fact extraction and recursive refinement, reporting improved completeness and conciseness over a few-shot baseline on the 57-encounter Primock57 benchmark, tho...

  13. Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Listen, Perceive, Express (LPE) uses two-stage ASR/SER training plus chain-of-thought prompting to let a frozen LLM generate empathetic responses from speech without question-answer fine-tuning data.

  14. Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation

    cs.CL 2025-01 conditional novelty 5.0 of 10

    REST-PG trains LLMs to reason over user profiles and self-train on high-reward outputs, improving personalized long-form generation by 14.5% over SFT on LongLaMP.

  15. Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM

    cs.CL 2024-12 reject novelty 5.0 of 10

    SumAutoEval is an LLM-based entity-level summarization evaluator with four dimensions; its claimed human-correlation advantage is not consistently supported by the experiments.

  16. Engineering AI Judge Systems

    cs.SE 2024-11 conditional novelty 5.0 of 10

    A constitution-based, search-driven development framework for AI judge systems improves judged accuracy by up to 6.2% on commit message generation, with about 58% of general principles reused across five languages.

  17. Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis

    cs.CL 2024-11 reject novelty 5.0 of 10

    Fine-tuning Llama-2 and Mistral 7B on a purpose-built dataset of meta-analysis abstracts improves the relevance of generated meta-analysis abstracts, but the gains rest on a small human evaluation and a questionable l...

  18. Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)

    cs.CL 2026-06 unverdicted novelty 4.0 of 10

    A graph-augmented RAG system with vector and graph query tools halves hallucinations and raises factual correctness scores on the MoNaCo complex QA benchmark.

  19. Improve LLM-based Automatic Essay Scoring with Linguistic Features

    cs.CL 2025-02 conditional novelty 4.0 of 10

    Adding handcrafted linguistic features to zero-shot LLM prompts modestly improves automatic essay scoring for Mistral-7B on ASAP and ELLIPSE, but GPT-4 shows no benefit on ASAP and no significance tests are provided.

  20. The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.

Pith tools