Pith

open record

sign in

arxiv: 2306.05783 · v3 · pith:NRRST6X2 · submitted 2023-06-09 · cs.CL

Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

Zhouhong Gu , Xiaoxuan Zhu , Haoning Ye , Lin Zhang , Jianchen Wang , Yixin Zhu , Sihang Jiang , Zhuozhi Xiong
show 11 more authors
Zihan Li Weijie Wu Qianyu He Rui Xu Wenhao Huang Jingping Liu Zili Wang Shusen Wang Weiguo Zheng Hongwei Feng Yanghua Xiao
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:NRRST6X2record.jsonopen to challenge →

classification cs.CL
keywords xiezhillmsevaluationquestionsbenchmarkdomainholisticknowledge
0
0 comments X
read the original abstract

New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty and Xiezhi-Interdiscipline, both with 15k questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. We anticipate Xiezhi will help analyze important strengths and shortcomings of LLMs, and the benchmark is released in~\url{https://github.com/MikeGu721/XiezhiBenchmark}.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PRIMETIME : Limits of LLMs in Temporal Primitives

    cs.NE 2025-04 unverdicted novelty 7.0

    PRIMETIME generator reveals that LLM datetime parsing and arithmetic primitives are individually unreliable but fully learnable via fine-tuning, enabling frontier-level accuracy on event planning with small LoRA models.