REVIEW 11 cited by
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
read the original abstract
In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks (e.g., long-context understanding), and many benchmarks have been proposed. However, we observe that long text generation capabilities are not well investigated. Therefore, we introduce the Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive, in-the-wild, and open-ended benchmark to evaluate LLMs' performance in generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. Besides, we propose Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation method that significantly reduces the time and effort required for human evaluation while maintaining a high correlation with human evaluation. We have conducted extensive experiments across around 30 mainstream LLMs and observed that the current LLMs lack long text generation capabilities. Specifically, first, regardless of whether the instructions include explicit or implicit length constraints, we observe that most LLMs cannot generate text that is longer than 4000 words. Second, we observe that while some LLMs can generate longer text, many issues exist (e.g., severe repetition and quality degradation). Third, to demonstrate the effectiveness of HelloEval, we compare HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge methods, which show that HelloEval has the highest correlation with human evaluation. We release our code in https://github.com/Quehry/HelloBench.
Forward citations
Cited by 11 Pith papers
-
CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
A task-adaptive rubric-selection method using Bayesian measurability and IRT-based greedy assembly that compresses rubric banks while improving agreement and preserving rank fidelity.
-
RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review
A rubric-first LLM pipeline that splits peer review into rubric generation, rubric-conditioned review writing, and final scoring outperforms existing AI reviewers on alignment with human judgments in a 200-paper test.
-
HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing
Tree-of-Writing achieves 0.93 Pearson correlation with human judgments by using a tree-structured workflow to aggregate sub-feature scores, outperforming standard LLM-as-a-judge and overlap metrics on the new HowToBench.
-
FlexStructRAG: Flexible Structure-Aware Multi-Granular Relational Retrieval for RAG
FlexStructRAG jointly constructs knowledge graphs, hypergraphs, and semantic clusters with dynamic partitioning to enable query-adaptive multi-granular retrieval that improves semantic scores over standard RAG baselin...
-
MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems
MemoryBench is a new multi-domain benchmark that simulates ongoing user feedback to evaluate continual learning in LLM systems, finding that state-of-the-art memory methods are ineffective and inefficient.
-
MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems
MemoryBench shows that state-of-the-art LLM memory systems do not reliably learn from simulated user feedback and are often outperformed by naive RAG.
-
DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
DeepResearch Bench supplies 100 expert-crafted PhD-level tasks and two human-aligned evaluation frameworks to measure deep research agents on report quality and citation accuracy.
-
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
LongBench v2 benchmark shows current LLMs underperform humans on deep long-context reasoning tasks, but extended inference-time reasoning enables surpassing the human baseline.
-
FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality
A multi-LLM consensus pipeline turns 14,450 auto-generated candidate rubrics into 2,600 distinguishable gold rubrics that rank 10 financial deep-research systems from 58.58% to 22.23% pass rate.
-
FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality
A benchmark whose 2,600 'gold' rubrics are generated, validated, and applied entirely by LLMs — with no human in the final loop — differentiates 10 financial deep-research systems across a 36-point pass-rate spread.
-
Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
MRRG elicits evaluation criteria from multiple complementary roles to build rubrics that outperform single-role baselines for validating LLM preferences and providing rewards in RLVR.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.