REVIEW 3 cited by
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty and Xiezhi-Interdiscipline, both with 15k questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. We anticipate Xiezhi will help analyze important strengths and shortcomings of LLMs, and the benchmark is released in~\url{https://github.com/MikeGu721/XiezhiBenchmark}.
Forward citations
Cited by 3 Pith papers
-
AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge
AntiLeakBench automatically constructs QA benchmarks from knowledge updated after each model's cutoff, and its experiments suggest that pre-cutoff evaluation overstates LLM ability.
-
dots.llm1 Technical Report
A 14B-active MoE model roughly matches Qwen2.5-72B on a broad benchmark suite while reporting about a 4x reduction in training GPU-hours.
-
LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models
A proposed three-dimensional benchmark for LLM moral reasoning that combines MFQ, WVS, and moral dilemmas, but the reported model scores are not reproducible from the paper.
Discussion (0). Continue with ORCID to comment.