HAMLET introduces a three-level key-fact hierarchy with automated query-focused summarization to evaluate LLM recall and faithfulness in book-length contexts, achieving over 90% agreement with experts.
• Less than 25 MFs → Score 1-2 • 25 - 35 MFs and at least a two-level hierarchy → Score 3-4 • More than 35 MFs and a well-defined multi-level hierarchy → Score 5
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts
HAMLET introduces a three-level key-fact hierarchy with automated query-focused summarization to evaluate LLM recall and faithfulness in book-length contexts, achieving over 90% agreement with experts.