Pith. sign in

REVIEW 35 cited by

A Survey on Evaluation of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.03109 v9 pith:SM2TNWV4 submitted 2023-07-06 cs.CL cs.AI

Yupeng Chang , Xu Wang , Jindong Wang , Yuan Wu , Linyi Yang , Kaijie Zhu , Hao Chen , Xiaoyuan Yi
show 8 more authors
Cunxiang Wang Yidong Wang Wei Ye Yue Zhang Yi Chang Philip S. Yu Qiang Yang Xing Xie
This is my paper · ORCID
classification cs.CLcs.AI
keywords llmsevaluationevaluatelanguagetasksapplicationsbetterdevelopment
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, educations, natural and social sciences, agent applications, and other areas. Secondly, we answer the `where' and `how' questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 35 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 200 citations worldwide. Full citation record

  1. Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents

    cs.CR 2026-07 conditional novelty 7.5 of 10

    Malicious tools can systematically extract isolated LLM-agent long-term memory via persistence, pure-anchor retrieval steering, and reactivation payloads, reaching 80% extraction with unlimited triggers and 47% with 20.

  2. Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures

    cs.AI 2025-06 conditional novelty 7.0 of 10

    Xent Games formalize a large family of LLM evaluation tasks as games whose rewards and constraints are signed cross-entropy sums, and propose using them to build general capability measures.

  3. LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A benchmark for LLM agents in partially observable joint decision-making reveals that deliberation challenges current models but can enable reflection and error correction.

  4. EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new Persian-Islamic trustworthiness benchmark ranks Claude highest and Qwen lowest across eight LLMs and finds safety is the weakest dimension.

  5. Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Mask-GCG uses learnable masks to prune a minority of low-impact tokens from GCG attack suffixes, slightly improving speed while showing most tokens are necessary.

  6. Basic Reading Distillation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 564M-parameter model trained on teacher-generated NER and QA annotations over generic text matches or beats a 13B teacher on some benchmarks, though the comparison is not always clean.

  7. Exploring Gender Bias Beyond Occupational Titles

    cs.CL 2025-07 conditional novelty 6.0 of 10

    The paper presents GenderLexicon and a ClozeGender score, reporting that action verbs and object nouns carry gender bias beyond occupational stereotypes in English and Japanese language models.

  8. TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Models that ace hard math benchmarks score much lower on a new benchmark of simple strategic board games, exposing a gap in basic reasoning.

  9. RewardAnything: Generalizable Principle-Following Reward Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...

  10. STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 100-problem benchmark of difficult math derivations from academic papers, where the best LLMs score below 5% and small-scale fine-tuning yields modest gains on standard math benchmarks.

  11. ExpeTrans: LLMs Are Experiential Transfer Learners

    cs.CL 2025-05 reject novelty 6.0 of 10

    ExpeTrans transfers textual task-solving experience from source tasks to target tasks and reports accuracy gains over zero-shot baselines on 13 NLP datasets.

  12. From Reddit to Generative AI: Evaluating Large Language Models for Anxiety Support Fine-tuned on Social Media Data

    cs.HC 2025-05 conditional novelty 6.0 of 10

    Fine-tuning GPT-3.5 and Llama 2 on r/Anxiety posts improves readability but raises toxicity and bias while reducing empathy and reflection.

  13. Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs

    cs.CL 2024-12 conditional novelty 6.0 of 10

    The paper proposes confidence and critique metrics for LLM self-correction, finds a trade-off between them under prompting and in-context learning, and introduces a data-format transformation (CCT) that improves both ...

  14. Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying

    cs.AI 2024-12 conditional novelty 6.0 of 10

    CQoT, a pipeline that uses argumentation-theoretic critical questions to check LLM reasoning plans, improves MT-Bench reasoning and math scores by roughly 5% over baseline and CoT prompting.

  15. Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A tree-based, inference-only framework combining execution metrics with LLM self-critique improves code generation correctness and efficiency across models and benchmarks.

  16. Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

    cs.SE 2024-12 conditional novelty 6.0 of 10

    A code-obfuscation benchmark shows LLM code generation pass rates fall sharply when descriptions, code, and dependencies are rewritten to remove training-data familiarity.

  17. SafeWorld: Geo-Diverse Safety Alignment

    cs.CL 2024-12 conditional novelty 6.0 of 10

    This paper introduces a geo-diverse cultural and legal safety benchmark and shows that a DPO-trained 7B model can outperform GPT-4o on it, with caveats about the GPT-4-based evaluation loop.

  18. Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

    cs.AI 2026-05 conditional novelty 5.5 of 10

    Across 140k generations, wrapper-only formatting changes accuracy by over 30× across models, largely via parseability failures, so single-wrapper scores are statistically fragile.

  19. Agent Team Work Zone: An Automated, Persistent Workspace for Long-Lived Claude Code Agent Teams

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A design and open-source implementation of a filesystem layer that persists Claude Code agent-team state to disk, enabling checkpointing, recovery, and cross-session communication.

  20. Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering

    cs.CR 2025-10 conditional novelty 5.0 of 10

    A Bayesian model that groups similar LLM test prompts into clusters gives better predictive scores than a no-clustering baseline but does not prove that it truly corrects prompt dependence.

  21. MATRIX: Multi-Agent simulaTion fRamework for safe Interactions and conteXtual clinical conversational evaluation

    cs.AI 2025-08 conditional novelty 5.0 of 10

    MATRIX combines a structured safety taxonomy, an LLM hazard judge, and a patient simulator to benchmark clinical dialogue agents, claiming expert-level hazard detection and revealing weak emergency handling in current LLMs.

  22. Enterprise Large Language Model Evaluation Benchmark

    cs.AI 2025-06 reject novelty 5.0 of 10

    A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...

  23. Improving LLM Leaderboards with Psychometrical Methodology

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Applying factor analysis to Hugging Face leaderboard benchmarks yields a different, arguably better LLM ranking than the simple average, with an inverted U-shaped relationship between the two.

  24. Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulations

    cs.LG 2025-01 conditional novelty 5.0 of 10

    GPT models asked to make random binary choices show large, version-specific biases; only GPT-4o-Mini came close to a 50/50 split in one-shot tests.

  25. Boosting Short Text Classification with Multi-Source Information Exploration and Dual-Level Contrastive Learning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    MI-DELIGHT boosts short text classification with multi-source graph information and a hierarchy of instance-level and cluster-level contrastive learning.

  26. No-Knowledge Alarms for Misaligned LLMs-as-Judges

    cs.AI 2025-09 conditional novelty 4.0 of 10

    A no-knowledge alarm can prove, without an answer key, that at least one of several LLM judges fails a user-specified per-label accuracy requirement, by showing every possible ground-truth label assignment is infeasible.

  27. Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.

  28. Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation

    cs.AI 2025-02 conditional novelty 4.0 of 10

    A meta-review of about 110 critical studies finds nine systemic weaknesses in AI benchmarking and concludes that benchmarks are receiving disproportionate trust in AI governance.

  29. The Only Way is Ethics: A Guide to Ethical Research with Large Language Models

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A practitioner-focused guide that distills existing AI ethics literature into actionable Do's and Don'ts for each stage of LLM research projects.

  30. Fake News Detection: Comparative Evaluation of BERT-like Models and Large Language Models with Generative AI-Annotated Data

    cs.CL 2024-12 reject novelty 4.0 of 10

    On a GPT-4-plus-human-labeled fake news dataset, fine-tuned BERT and RoBERTa classifiers outperform instruction-tuned 7B LLMs, while LLMs are more robust to text perturbations.

  31. BhashaVerse : Translation Ecosystem for Indian Subcontinent Languages

    cs.CL 2024-12 reject novelty 4.0 of 10

    A claimed 2B-parameter multi-task translation model for 36 Indian languages, built from pivoted and synthetic corpora, evaluated without baselines and with inconsistent reported numbers.

  32. Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

    cs.CV 2024-12 conditional novelty 3.0 of 10

    A LLaVA-style radiology report generator using LoRA fine-tuning and stitched chest X-ray inputs placed fourth in the RRG24 shared task.

  33. Best Practices for Large Language Models in Radiology

    cs.AI 2024-12 conditional novelty 3.0 of 10

    The paper recommends starting LLM use in radiology with prompt optimization and retrieval augmentation, fine-tuning only when needed, and preferring locally hosted open models with human expert evaluation.

  34. Strategic Prompting for Conversational Tasks: A Comparative Analysis of Large Language Models Across Diverse Conversational Tasks

    cs.CL 2024-11 reject novelty 3.0 of 10

    No single open-source LLM among Llama, OPT, Falcon, Alpaca, and MPT performs best across reservation, empathy, counseling, persuasion, and negotiation tasks.

  35. Building Task Bots with Self-learning for Enhanced Adaptability, Extensibility, and Factuality

    cs.CL 2025-08 conditional novelty 2.0 of 10

    A thesis that combines self-learning from dialog logs, schema-guided prompting, and self-aligned factuality to build task bots with minimal human intervention.

Pith tools