Pith. sign in

REVIEW 5 cited by

Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.07103 v2 pith:7NRFTMGA submitted 2024-01-13 cs.CL

Zhen Li , Xiaohan Xu , Tao Shen , Can Xu , Jia-Chen Gu , Yuxuan Lai , Chongyang Tao , Shuai Ma This is my paper · ORCID
classification cs.CL
keywords evaluationlanguageassessingchallengeslargeleveragingllm-basedllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. This paper aims to provide a thorough overview of leveraging LLMs for NLG evaluation, a burgeoning area that lacks a systematic analysis. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. Our detailed exploration includes critically assessing various LLM-based methodologies, as well as comparing their strengths and limitations in evaluating NLG outputs. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RewardAnything: Generalizable Principle-Following Reward Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...

  2. Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation

    cs.CL 2025-01 conditional novelty 5.0 of 10

    REST-PG trains LLMs to reason over user profiles and self-train on high-reward outputs, improving personalized long-form generation by 14.5% over SFT on LongLaMP.

  3. ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models

    cs.CL 2024-12 conditional novelty 5.0 of 10

    A branch-merge and reward-token DPO-based multimodal evaluator that surpasses GPT-4-Turbo on image-text medical QA, but relies on LLM-generated labels.

  4. Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation

    cs.LG 2024-11 conditional novelty 5.0 of 10

    On legacy MUMPS code, LLM-generated line comments are rated almost as good as human-written ones; on IBM assembly they are rated low, and standard automated metrics do not predict human quality scores.

  5. Reference-free Evaluation Metrics for Text Generation: A Survey

    cs.CL 2025-01 conditional novelty 4.0 of 10

    The survey classifies reference-free NLG evaluation metrics into learning from human judgments, pseudo-judgments, context-hypothesis correspondence, and peer evaluation.

Pith tools