RECORD
LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models
Computation and Language · Submitted 19 Mar 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our findings suggest that LHMKE is a challenging and advanced testbed for Chinese LLMs.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Chinese Large Language Models (LLMs) have recently demonstrated impressive capabilities across various NLP benchmarks and real-world applications. However, the existing benchmarks for comprehensively evaluating these LLMs are still insufficient, particularly in terms of measuring knowledge that LLMs capture. Current datasets collect questions from Chinese examinations across different subjects and educational levels to address this issue. Yet, these benchmarks primarily focus on objective questions such as multiple-choice questions, leading to a lack of diversity in question types. To tackle this problem, we propose LHMKE, a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark in this paper. LHMKE is designed to provide a comprehensive evaluation of the knowledge acquisition capabilities of Chinese LLMs. It encompasses 10,465 questions across 75 tasks covering 30 subjects, ranging from primary school to professional certification exams. Notably, LHMKE includes both objective and subjective questions, offering a more holistic evaluation of the knowledge level of LLMs. We have assessed 11 Chinese LLMs under the zero-shot setting, which aligns with real examinations, and compared their performance across different subjects. We also conduct an in-depth analysis to check whether GPT-4 can automatically score subjective predictions. Our findings suggest that LHMKE is a challenging and advanced testbed for Chinese LLMs.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Chinese Large Language Models (LLMs) have recently demonstrated impressive capabilities across various NLP benchmarks and real-world applications. | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| s1 | However, the existing benchmarks for comprehensively evaluating these LLMs are still insufficient, particularly in terms of measuring knowledge that LLMs capture. | gap 0.98 | 0.06 | 0.02 | asserted 1.00 |
| s2 | Current datasets collect questions from Chinese examinations across different subjects and educational levels to address this issue. | background 0.93 | 0.10 | 0.02 | asserted 1.00 |
| s3 | Yet, these benchmarks primarily focus on objective questions such as multiple-choice questions, leading to a lack of diversity in question types. | gap 0.90 | 0.07 | 0.02 | asserted 1.00 |
| s4 | To tackle this problem, we propose LHMKE, a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark in this paper. | contribution 1.00 | 0.84 | 0.06 | asserted 0.99 |
| s5 | LHMKE is designed to provide a comprehensive evaluation of the knowledge acquisition capabilities of Chinese LLMs. | contribution 0.91 | 0.51 | 0.06 | asserted 1.00 |
| s6 | It encompasses 10,465 questions across 75 tasks covering 30 subjects, ranging from primary school to professional certification exams. | method 0.64 | 0.79 | 0.04 | asserted 1.00 |
| s7 | Notably, LHMKE includes both objective and subjective questions, offering a more holistic evaluation of the knowledge level of LLMs. | contribution 0.79 | 0.72 | 0.07 | asserted 1.00 |
| s8 | We have assessed 11 Chinese LLMs under the zero-shot setting, which aligns with real examinations, and compared their performance across different subjects. | method 0.84 | 0.97 | 0.05 | asserted 0.99 |
| s9 | We also conduct an in-depth analysis to check whether GPT-4 can automatically score subjective predictions. | method 0.80 | 0.94 | 0.03 | asserted 0.85 |
| s10 | Our findings suggest that LHMKE is a challenging and advanced testbed for Chinese LLMs. | interpretation 0.75 | 0.94 | 0.85 | hedged possibility 0.75 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Chinese Large Language Models (LLMs) have recently demonstrated impressive capabilities across various NLP benchmarks and real-world applications. However, the existing benchmarks for comprehensively evaluating these LLMs are still insufficient, particularly in terms of measuring knowledge that LLMs capture. Current datasets collect questions from Chinese examinations across different subjects and educational levels to address this issue. Yet, these benchmarks primarily focus on objective questions such as multiple-choice questions, leading to a lack of diversity in question types. To tackle this problem, we propose LHMKE, a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark in this paper. LHMKE is designed to provide a comprehensive evaluation of the knowledge acquisition capabilities of Chinese LLMs. It encompasses 10,465 questions across 75 tasks covering 30 subjects, ranging from primary school to professional certification exams. Notably, LHMKE includes both objective and subjective questions, offering a more holistic evaluation of the knowledge level of LLMs. We have assessed 11 Chinese LLMs under the zero-shot setting, which aligns with real examinations, and compared their performance across different subjects. We also conduct an in-depth analysis to check whether GPT-4 can automatically score subjective predictions. Our findings suggest that LHMKE is a challenging and advanced testbed for Chinese LLMs.
Human review
How the literature uses it
2 arXiv papers cite this in 3 sentences
Papers citing it · 2 sentences without a confident label
Cites it as background 1 paper
-
The main objective of question answering (QA) [ref] is to provide an accurate answer $a_i$ to the given question $q_i$.
CJEval: A Benchmark for Assessing Large Language Models Using Chinese Junior High School Exam Data · Question Answering · 0.95
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.