Pith. sign in

REVIEW 18 cited by

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.08322 v3 pith:F4423YOS submitted 2023-05-15 cs.CL

Yuzhen Huang , Yuzhuo Bai , Zhihao Zhu , Junlei Zhang , Jinghan Zhang , Tangjun Su , Junteng Liu , Chuancheng Lv
show 5 more authors
Yikai Zhang Jiayi Lei Yao Fu Maosong Sun Junxian He
This is my paper · ORCID
classification cs.CL
keywords c-evalmodelschineseadvancedevaluationfoundationllmsabilities
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Gated amplitude-only FFN interventions improve tool-structured LLM outputs on Qwen models by several points, while direction-changing repairs harm more than they fix.

  2. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.

  3. FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...

  4. Serving Large Language Models on Huawei CloudMatrix384

    cs.DC 2025-06 conditional novelty 6.0 of 10

    CloudMatrix384 with CloudMatrix-Infer reports prefill 6,688 tokens/s per NPU and decode 1,943 tokens/s per NPU on DeepSeek-R1, exceeding published NVIDIA-based efficiencies.

  5. Scalable Complexity Control Facilitates Reasoning Ability of LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Controlling model complexity through smaller initialization rates and stronger weight decay improved LLM benchmark scores and made loss-versus-scale curves descend faster.

  6. DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    DiagnosisArena, a 1,113-case benchmark from top journals, shows state-of-the-art LLMs achieve at most 51% top-1 diagnostic accuracy, far below clinical-level competence.

  7. Enhancing LLMs via High-Knowledge Data Selection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A knowledge-element density and coverage scorer selects pre-training data that improves LLM performance on knowledge-intensive and general understanding benchmarks by 2 to 3 points.

  8. Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A fine-tuned 14B LLM judge, trained with scenario-based prompts and controlled instruction generation, approaches GPT-4's human-agreement performance, and the paper documents why scaling distillation data can fail.

  9. AutoEvoEval: An Automated Framework for Evolving Close-Ended LLM Evaluation Data

    cs.CL 2025-06 conditional novelty 5.0 of 10

    AutoEvoEval applies 22 atomic perturbations and multi-round chains to MCQ benchmarks, causing average accuracy drops of 7.283% and up to 52.932% for long chains.

  10. Enterprise Large Language Model Evaluation Benchmark

    cs.AI 2025-06 reject novelty 5.0 of 10

    A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...

  11. AI Flow: Perspectives, Scenarios, and Approaches

    cs.AI 2025-06 conditional novelty 5.0 of 10

    AI Flow proposes to combine device-edge-cloud deployment, feature-aligned model families, and multi-model collaboration to make large AI models cheaper, faster, and more widely accessible.

  12. A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.

  13. Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A group-constrained Mixture-of-Experts routing rule (MoGE) is proposed and demonstrated in Pangu Pro MoE, a 72B/16B-active sparse LLM that reports faster inference on Ascend NPUs.

  14. OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training

    cs.CL 2025-01 conditional novelty 5.0 of 10

    OpenCSG released four open Chinese LLM training datasets, and 2B-scale tests report improved C-Eval, CMMLU, and Alignbench scores.

  15. Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning

    cs.LG 2025-09 conditional novelty 4.0 of 10

    FAPM prunes task vectors using a combined absolute and relative change magnitude criterion, reducing forgetting on general benchmarks to roughly a third of a percent while keeping downstream accuracy near the fine-tun...

  16. Technical Report of TeleChat2, TeleChat2.5 and T1

    cs.CL 2025-07 conditional novelty 4.0 of 10

    The released T1-115B open-weight model outperforms OpenAI's o1-mini and GPT-4o on MATH500, AlignBench, and IFEval, despite using a standard dense transformer architecture.

  17. ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training

    cs.LG 2025-05 reject novelty 4.0 of 10

    Shared KV caching across the last 25% to 50% of layers, adapted incrementally from pretrained LLaMA weights, is claimed to improve training speed and inference throughput with little accuracy loss.

  18. LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Models

    cs.CL 2025-05 reject novelty 4.0 of 10

    A block-localizing fine-tuning method for gender debiasing is presented, but its stated loss is inconsistent with its reported behavior and the evaluation tables contain duplicate rows.

Pith tools