QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation

Chen Jason Zhang; Di Jiang; Mengze Hong; Wailing Ng

REVIEW 4 major objections 4 minor 48 references

QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark of 17,316 Chinese professional-exam questions finds Qwen2.5-7B outperforming GPT-4o on localized knowledge.

desk verdict The dataset is genuine and useful, but the paper's central claim that Chinese models beat GPT-4o because of localized knowledge is not yet secured by the contamination checks, which omit the headline models and test only surface recall. read the letter →

arxiv 2505.05225 v2 pith:DKLOAFFY submitted 2025-05-08 cs.CL

Mengze Hong , Wailing Ng , Chen Jason Zhang , Di Jiang This is my paper · ORCID

classification cs.CL

keywords ChineseLLMevaluationqualificationexamsverticaldomainbenchmarklocalizedknowledgedatacontaminationcrowdsourcingQwen2.5

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds QualBench, a Chinese question-answering benchmark made from 24 real Chinese professional qualification exams, spanning six vertical domains and containing over 17,000 multiple-choice and true/false questions. The authors use it to test whether Chinese LLMs possess localized professional knowledge that English-centric models lack. Their headline finding is that Chinese models consistently beat non-Chinese models, with the 7-billion-parameter Qwen2.5 model outperforming the much larger GPT-4o. The paper argues this demonstrates the value of localized pretraining knowledge, and reports that even the best model reaches only about 75 percent accuracy, leaving clear room for improvement through retrieval augmentation and continued fine-tuning.

What carries the argument

The central object is QualBench, a dataset of 17,316 multiple-choice and true/false questions extracted via OCR from 24 Chinese professional qualification exams across six domains (production safety, fire safety, civil engineering, economics and finance, oil and gas, banking and insurance), with duplicate removal and expert validation. The mechanism carrying the argument is localization: questions are drawn from Chinese regulatory and professional standards, so answering them correctly requires knowledge of Chinese laws, policies, and industry practices rather than generic reasoning. The evaluation protocol that makes the comparison fair is a one-shot setting with a standardized prompt asking for an answer plus explanation, repeated five times with averaged accuracy and F1.

What would settle it

Build a validated subset from exam papers published after the training cutoff of all evaluated models (e.g., 2025 question sets) and rerun the one-shot evaluation: if Qwen2.5-7B's advantage over GPT-4o shrinks or vanishes on those genuinely unseen questions, the localization conclusion would be an artifact of training-data leakage. A cheaper check is to translate a sample of QualBench questions into English while keeping the same answer options; if English-centric models reach parity on the translated version, the gap is language familiarity rather than domain knowledge.

Watch

Extended reading notes

Core claim

The paper's central claim is that QualBench is the first multi-domain Chinese QA benchmark built specifically for localized assessment of Chinese LLMs, and that on this benchmark Chinese LLMs consistently outperform non-Chinese models regardless of model size: Qwen2.5-7B-instruct achieves 75.26% average accuracy versus 61.61% for GPT-4o and 56.96% for GPT-3.5, while smaller Chinese models like Baichuan2-7B (48.64%) also beat larger non-Chinese models like LLaMA-14B (37.65%). The authors attribute this to native Chinese pretraining knowledge covering localized laws, regulations, and professional standards. They also report that majority-voting crowdsourcing among LLMs degrades performance relative to a single strong model, and that prompt engineering and fine-tuning help non-Chinese models more than Chinese models, reinforcing the localization story. The average accuracy of 53.98% across all models indicates substantial remaining gaps in vertical-domain coverage.

Load-bearing premise

The headline Chinese-versus-non-Chinese comparison assumes the evaluated models have not memorized the QualBench questions during pretraining, so the score gap reflects localized knowledge rather than training-data overlap.

Editorial extensions

If this is right

If QualBench accurately measures localized professional knowledge, Chinese LLMs should be preferred over equally sized or larger non-Chinese models for Chinese vertical-domain applications such as safety compliance, construction management, and financial regulation.
The consistent gap across model sizes implies that pretraining language and regional corpus coverage matter more than raw parameter count for localized expertise tasks.
The finding that LLM crowdsourcing (majority or weighted voting across five models) underperforms a single strong model suggests that aggregation strategies must be redesigned, or model diversity increased, before ensembles help on knowledge-dense benchmarks.
The large gains non-Chinese models show from role prompting and in-context learning indicate that prompt design can partially compensate for missing localized knowledge, but cannot fully close the gap.
The benchmark's low overall average accuracy (53.98%) means current LLMs are not yet reliably deployable in these Chinese professional settings without retrieval augmentation or fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The localization advantage shown on exam questions may extend to other Chinese-specific tasks such as contract review, regulatory compliance chatbots, and safety inspection automation; this is a natural extrapolation the paper does not test.
A stricter contamination check would be to evaluate on post-training-cutoff exam years (e.g., 2025 papers for models whose training ended in 2024); if the Qwen advantage shrinks on genuinely unseen questions, the paper's central contrast would need reinterpretation.
The benchmark could be extended to open-ended questions and image-based diagrams to test whether the localization advantage persists when models cannot rely on option recognition; the paper explicitly limits itself to closed-form questions.
An English-translated version of the same questions would separate knowledge of Chinese domain facts from the ability to read Chinese, a distinction the current design conflates.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The QualBench dataset is real and worth having. 17,316 questions from 24 Chinese qualification exams across six vertical domains is a solid contribution, and the OCR-plus-two-expert validation is a reasonable construction pipeline. The error taxonomy in the appendix is also a nice touch—it gives the benchmark practical value beyond the accuracy numbers. So the resource itself deserves a place in the Chinese LLM evaluation ecosystem.

The soft spots are real, and the stress-test note is right. The contamination checks in Tables 6 and 7 do not carry the weight the paper puts on them. Table 6 omits GPT-4o and the other models in the headline comparison; asking a model to pick the right answer after shuffling letter order does not test whether it has memorized the underlying item, especially if it has seen a paraphrased or translated version. Table 7 runs only on Qwen2.5-7B, and low ROUGE-L shows the model cannot reproduce the exact surface string, not that it fails to recognize the content. That is a load-bearing weakness because the whole localization story depends on the gap not being an artifact of asymmetric contamination. Also, the numbers in Table 6 do not match Table 4 for the same models—Qwen2.5-7B is 74.78% vs 75.26%, GPT-3.5 is 52.24% vs 56.96%—which needs an explanation, and there are no confidence intervals or significance tests anywhere, so we cannot tell if the Qwen-vs-GPT-4o gap is meaningful.

There are two smaller issues. The paper claims "first multi-domain Chinese QA benchmark" but does not cite C-Eval, the most prominent multi-domain Chinese benchmark; the claim needs to be scoped to qualification-exam-based benchmarks. And the LLM crowdsourcing finding rests on a single aggregation setup, so it is more of an observation than a robust result.

Who is this for? Anyone building or evaluating Chinese vertical-domain LLMs, especially for deployment in regulated industries like safety, finance, and banking. The dataset is the contribution, and it is a good one. The performance gap between Chinese and non-Chinese models is interesting but should be treated as provisional until the contamination controls are strengthened.

Recommendation: send this to serious peer review. It deserves referee time, but the authors should be asked to run contamination checks on all headline models, reconcile the table inconsistencies, report variance, and cite C-Eval. If the gap survives better controls, the localization finding becomes much more compelling; if it does not, the dataset still stands on its own.

Referee Report

4 major / 4 minor

Summary. The paper introduces QualBench, a Chinese QA benchmark constructed from 24 Chinese professional qualification examinations spanning six vertical domains, yielding 17,316 multiple-choice and true/false questions. The authors evaluate five Chinese and four non-Chinese LLMs in a one-shot setting, reporting that Qwen2.5-7B achieves the highest accuracy (75.26%) and outperforms GPT-4o. They also report three additional findings: that Chinese LLMs consistently outperform non-Chinese models, that LLM crowdsourcing via voting degrades performance relative to a single strong model, and that prompt engineering and fine-tuning benefit non-Chinese models more than Chinese ones. The paper includes contamination checks, ablations, and a public release of data and code.

Significance. If the benchmark holds up, it provides a valuable and timely resource for vertical-domain evaluation of Chinese LLMs, complementing existing single-domain benchmarks and emphasizing localized knowledge. The data construction process (OCR plus two-expert validation) and the public release are concrete strengths, and the headline result—a 7B Chinese model outperforming GPT-4o on localized professional questions—is an interesting, falsifiable finding. However, the paper's broad claim that Chinese LLMs consistently surpass non-Chinese models is not supported by its own Table 4, and the contamination evidence is incomplete for the models in the headline comparison. The resource itself is worthwhile, but the central generalization needs to be re-scoped and the robustness evidence strengthened.

major comments (4)

[Abstract and §5.1, Table 4] The claim that 'Chinese LLMs consistently surpass non-Chinese models' is contradicted by Table 4. GPT-3.5 (56.96%) and GPT-4o (61.61%) outperform Baichuan2-7B (48.64%), Hunyuan-7B (50.64%), DeepSeek-v2-lite (51.76%), and ChatGLM3-6B (43.62%); only Qwen2.5-7B (75.26%) exceeds GPT-4o. Similarly, the 'regardless of model size' formulation in §5.1 and the Conclusion rests on a single pair in Table 5 (Baichuan2-7B vs LLaMA-14B). Please rephrase the headline contribution to the pattern actually shown, for example: Qwen2.5-7B outperforms GPT-4o on QualBench, and several open Chinese 7B models outperform equivalently sized non-Chinese open models.
[§5.3, Table 6] The original-condition accuracies in Table 6 differ from the corresponding entries in Table 4 (Qwen2.5-7B: 74.78% vs 75.26%; GPT-3.5: 52.24% vs 56.96%) without any statement that a subset or different evaluation protocol was used. Because the answer-shuffling test is intended to assess contamination on the headline benchmark, the original condition must match the main evaluation; otherwise the comparison is uninterpretable. Please state whether Table 6 uses the full test set, report the subset size if not, and rerun the shuffle test under conditions identical to Table 4.
[§5.3, Table 7] The question-completion contamination check is executed on only one model, Qwen2.5-7B, on 1,020 sampled questions, while the headline comparison includes GPT-4o, GPT-3.5, and other models. Contamination is model-specific, and these qualification exam questions are publicly available and predate model training, so a single-model test cannot support the paper's conclusion that 'QualBench has minimal overlap with LLM training data' for all evaluated models. Furthermore, low ROUGE-L on surface completion does not rule out content-level memorization of paraphrased or translated versions. Please provide per-model contamination evidence for each model in the main comparison, or restrict the contamination claim to Qwen2.5-7B and mark the other models' status as unverified.
[§5.2, Tables 4 and 10] The 'LLM crowdsourcing results in performance degradation' finding is reported as a single average over five inference runs, with no confidence intervals or significance tests. Given the temperature=0.5 sampling, the observed gap between the aggregation rows (59.56% and 63.98%) and Qwen2.5-7B (75.26%) is large, but the claim that a single robust LLM is more effective should be supported by variance estimates or a question-level paired test (e.g., McNemar's test). Please add these statistical details or acknowledge that the difference is not tested for significance.

minor comments (4)

[Abstract] The phrase 'The average accuracy of 53.98%' is ambiguous: it appears to be the mean of the five Chinese models' overall accuracies from Table 4, rather than an average over all evaluated models or the best model. Please specify the population over which this average is taken.
[§3.3, Table 3 and Limitations] Production Safety (6,520) and Fire Safety (3,401) together constitute 9,921 of 17,316 questions, or about 57.3% of the dataset. The Limitations section acknowledges the imbalance, but the abstract and results sections should state that the overall accuracy figures are heavily weighted toward these two domains.
[Table 4] The rows labeled 'Aggregation majority voting' and 'weighted majority voting' do not state which models are aggregated or how the weights are assigned. Please add this information to the table caption or to §5.2.
[Acknowledgments] The acknowledgments contain a duplicated word: 'including including P0045948.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: QualBench's benchmark scores are anchored to external Chinese qualification exams, and the paper's central comparisons are empirical rather than derived from its own assumptions.

full rationale

The claimed derivation chain is an empirical benchmark construction and evaluation, not a formal reduction. QualBench questions are drawn from 24 externally published Chinese qualification exams (Table 11), giving an independent criterion for correctness; no score, domain label, or model ranking is defined in terms of the paper's own outputs. The Chinese-versus-non-Chinese performance pattern (Table 4) is an observed result of running held models on those external questions, not a consequence of how the benchmark was assembled. Self-citations (Hong et al., 2025a,b,c; Zhang et al., 2025) appear only in motivational or future-work contexts (e.g., LLM crowdsourcing and RAG discussion) and do not supply the benchmark's correctness labels or rankings. The contamination ablations (Tables 6–7) are methodologically limited—only one model is used in the completion test and the completion task checks surface recall rather than paraphrase-level memorization—but a validity limitation is not circularity: the result is externally falsifiable by running models on the public exam questions, and the paper does not fit parameters to the headline conclusion. Therefore no circular step, self-definitional reduction, or prediction-by-construction is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark paper introduces no free parameters or invented entities; it relies on external exam data. The main upstream assumptions are the validity of qualification exams as a proxy for expertise and the faithfulness of OCR extraction plus expert validation.

assumptions (3)

domain assumption Professional qualification examinations provide a unified and fair framework for evaluating domain expertise.
The entire benchmark design rests on this premise, stated in Sections 1 and 2; if exam scores do not track real job-readiness, the benchmark's practical significance is weakened.
domain assumption The OCR-extracted and human-screened questions faithfully represent the original exam content with correct answer keys.
Section 3.1-3.2 relies on OCR and two domain experts per field for validation; no inter-annotator agreement or error rate is reported, yet question accuracy is load-bearing for all measured scores.
domain assumption The one-shot evaluation protocol with temperature 0.5 and five repeats gives a stable and unbiased accuracy estimate.
Used in Section 4; no confidence intervals are reported for the accuracy/F1 values, and the protocol's sensitivity to prompt format is only partially tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation." pith.science (2026). https://pith.science/paper/DKLOAFFY

@misc{pith2026250505225,
  author       = {Pith},
  title        = {Pith review of: QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DKLOAFFY}},
  note         = {Machine review of arXiv:2505.05225}
}

read the original abstract

The rapid advancement of Chinese LLMs underscores the need for vertical-domain evaluations to ensure reliable applications. However, existing benchmarks often lack domain coverage and provide limited insights into the Chinese working context. Leveraging qualification exams as a unified framework for expertise evaluation, we introduce QualBench, the first multi-domain Chinese QA benchmark dedicated to localized assessment of Chinese LLMs. The dataset includes over 17,000 questions across six vertical domains, drawn from 24 Chinese qualifications to align with national policies and professional standards. Results reveal an interesting pattern of Chinese LLMs consistently surpassing non-Chinese models, with the Qwen2.5 model outperforming the more advanced GPT-4o, emphasizing the value of localized domain knowledge in meeting qualification requirements. The average accuracy of 53.98% reveals the current gaps in domain coverage within model capabilities. Furthermore, we identify performance degradation caused by LLM crowdsourcing, assess data contamination, and illustrate the effectiveness of prompt engineering and model fine-tuning, suggesting opportunities for future improvements through multi-domain RAG and Federated Learning.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 19 canonical work pages

[1]

Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. 2024. A survey on rag with llms. Procedia computer science, 246:3781--3790

work page 2024
[2]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020
[3]

Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2024. Medbench: A large-scale chinese benchmark for evaluating medical large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17709--17717

2024
[4]

Yu, Qiang Yang, and Xing Xie

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. https://doi.org/10.1145/3641289 A survey on evaluation of large language models . ACM Trans. Intell. Syst. Technol., 15(3)

doi:10.1145/3641289 2024
[5]

Wei Chen, Qiushi Wang, Zefei Long, Xianyin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, et al. 2023. Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning. arXiv preprint arXiv:2310.15205

arXiv 2023
[6]

Xiangxiang Cui, Shu Yang, Tianjin Huang, Wanyu Lin, Lijie Hu, and Di Wang. 2025. https://arxiv.org/abs/2506.15617 The compositional architecture of regret in large language models . Preprint, arXiv:2506.15617

work page Pith review arXiv 2025
[7]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

arXiv 2025
[8]

Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. 2023. Fate-llm: A industrial grade federated learning framework for large language models. Symposium on Advances and Open Problems in Large Language Models (LLM@IJCAI'23)

work page 2023

Show all 48 references

[9]

Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. https://doi.org/10.1073/pnas.2305016120 Chatgpt outperforms crowd workers for text-annotation tasks . Proceedings of the National Academy of Sciences, 120(30):e2305016120

2023 doi
[10]

Shahriar Golchin and Mihai Surdeanu. 2025. Data contamination quiz: A tool to detect and estimate contamination in large language models. Transactions of the Association for Computational Linguistics, 13:809--830

2025
[11]

Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. https://doi.org/10.18653/v1/2024.naacl-industry.15 A nno LLM : Making large language models to be better crowdsourced annotators . In Proceedings o...

2024 doi
[12]

Mengze Hong, Wailing Ng, Chen Jason Zhang, Yuanfeng SONG, and Di Jiang. 2025 a . Dial-in LLM : Human-aligned LLM -in-the-loop intent clustering for customer service dialogues. In The 2025 Conference on Empirical Methods in Natural Language Processing

2025
[13]

Mengze Hong, Wailing Ng, Chen Jason Zhang, Yifei Wang, Yuanfeng Song, and Di Jiang. 2025 b . Llm-in-the-loop: Replicating human insight with llms for better machine learning applications. Authorea Preprints

2025
[14]

Mengze Hong, Chen Jason Zhang, Di Jiang, Yuanfeng Song, Lu Wang, Yuanqin He, Zhiyang Su, and Qing Li. 2025 c . https://arxiv.org/abs/2410.12444 Expanding chatbot knowledge in customer service: Context-aware similar question generation using large language models . Preprint, ar...

2025
[15]

Di Jiang, Yuanfeng Song, Yongxin Tong, Xueyang Wu, Weiwei Zhao, Qian Xu, and Qiang Yang. 2019. Federated topic modeling. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1071--1080

2019
[16]

Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270):20230254

2024
[17]

Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIG...

2024
[18]

Haitao Li, You Chen, Qingyao Ai, Yueyue WU, Ruizhe Zhang, and Yiqun LIU. 2024. https://openreview.net/forum?id=8RaxRs5VDf Lexeval: A comprehensive chinese legal benchmark for evaluating large language models . In The Thirty-eight Conference on Neural Information Processing Sys...

2024
[19]

Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. 2023. Domain specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint arXiv:2305.18703

2023 arXiv
[20]

Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu, et al. 2023 a . M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models. arXiv preprint arXiv:2305.10263

2023 arXiv
[21]

Jia Liu, Wei Huang, Tianrui Li, Shenggong Ji, and Junbo Zhang. 2022. Cross-domain knowledge graph chiasmal embedding for multi-domain item-item recommendation. IEEE Transactions on Knowledge and Data Engineering, 35(5):4621--4633

2022
[22]

Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20

2021
[23]

Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Zhu Lei, and Michael Lingzhi Li. 2023 b . https://openreview.net/forum?id=HhcQ0zeqZp Benchmarking large language models on CME xam - a comprehensive chinese medic...

2023
[24]

Qidong Liu, Xiangyu Zhao, Yejing Wang, Zijian Zhang, Howard Zhong, Chong Chen, Xiang Li, Wei Huang, and Feng Tian. 2025. Bridge the domains: Large language models enhanced cross-domain sequential recommendation. In Proceedings of the 48th International ACM SIGIR Conference on ...

2025
[25]

Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki ...

2023 doi
[26]

Kelong Mao, Zheng Liu, Hongjin Qian, Fengran Mo, Chenlong Deng, and Zhicheng Dou. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.41 RAG -studio: Towards in-domain adaptation of retrieval augmented generation through self-alignment . In Findings of the Association for Co...

2024 doi
[27]

Shiwen Ni, Guhong Chen, Shuaimin Li, Xuanang Chen, Siyi Li, Bingli Wang, Qiyao Wang, Xingjian Wang, Yifan Zhang, Liyang Fan, Chengming Li, Ruifeng Xu, Le Sun, and Min Yang. 2025. https://arxiv.org/abs/2508.15361 A survey on large language model benchmarks . Preprint, arXiv:2508.15361

2025 arXiv
[28]

John Oxenham. 2024. Education versus qualifications?: A study of relationships between education, selection for employment and the productivity of labour. Taylor & Francis

2024
[29]

Oscar Sainz, Jon Campos, Iker Garc \'i a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.722 NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark . In Findings of the ...

2023 doi
[30]

Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary Lipton, and J Zico Kolter. 2024. Rethinking llm memorization through the lens of adversarial compression. Advances in Neural Information Processing Systems, 37:56244--56267

2024
[31]

Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172--180

2023
[32]

Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/7a92bcdede88c7afd108072faf5485c8-Paper-Datasets_and_Benchmarks.pdf Planbench: An extensible benchmark for evaluating ...

2023
[33]

Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: Will we run out of data? limits of llm scaling based on human-generated data. In Forty-first International Conference on Machine Learning

2024
[34]

Shuting Wang, Jiongnan Liu, Shiren Song, Jiehan Cheng, Yuqi Fu, Peidong Guo, Kun Fang, Yutao Zhu, and Zhicheng Dou. 2024. Domainrag: A chinese benchmark for evaluating domain-specific retrieval-augmented generation. arXiv preprint arXiv:2406.05654

2024 arXiv
[35]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 a . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

2024 arXiv
[36]

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. https://doi.org/10.1145/3298981 Federated machine learning: Concept and applications . ACM Trans. Intell. Syst. Technol., 10(2)

2019 doi
[37]

Wong, and Di Wang

Shu Yang, Shenzhe Zhu, Zeyu Wu, Keyu Wang, Junchi Yao, Junchao Wu, Lijie Hu, Mengdi Li, Derek F. Wong, and Di Wang. 2025. https://doi.org/10.18653/v1/2025.findings-acl.226 Fraud-r1 : A multi-round benchmark for assessing the robustness of LLM against augmented fraud and phishi...

2025 doi
[38]

Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, et al. 2024 b . Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems, 37:10470--10490

2024
[39]

Fangyuan Zhang, Zhengjun Huang, Yingli Zhou, Qintian Guo, Zhixun Li, Wensheng Luo, Di Jiang, Yixiang Fang, and Xiaofang Zhou. 2025. Erarag: Efficient and incremental retrieval augmented generation for growing corpora. arXiv preprint arXiv:2506.20963

2025 arXiv
[40]

Liwen Zhang, Weige Cai, Zhaowei Liu, Zhi Yang, Wei Dai, Yujie Liao, Qianru Qin, Yifei Li, Xingyu Liu, Zhiqiang Liu, et al. 2023 a . Fineval: A chinese financial domain knowledge evaluation benchmark for large language models. arXiv preprint arXiv:2308.09975

2023 arXiv
[41]

Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023 b . Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474

2023 arXiv
[42]

Guangxiang Zhao, Saier Hu, Xiaoqi Jian, Jinzhu Wu, Yuhan Wu, Change Jia, Lin Sun, and Xiangzheng Zhang. 2025. https://arxiv.org/abs/2502.12459 Large language models badly generalize across option length, problem types, and irrelevant noun replacements . Preprint, arXiv:2502.12459

2025
[43]

Mengjie Zhao, Fei Mi, Yasheng Wang, Minglei Li, Xin Jiang, Qun Liu, and Hinrich Schuetze. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.51 LMT urk: Few-shot learners as crowdsourcing workers in a language-model-as-a-service framework . In Findings of the Association fo...

2022 doi
[44]

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.149 AGIE val: A human-centric benchmark for evaluating foundation models . In Findings of the Association for...

2024 doi
[45]

Jie Zhu, Junhui Li, Yalong Wen, and Lifan Guo. 2024. https://doi.org/10.18653/v1/2024.findings-acl.337 Benchmarking large language models on CFLUE - a C hinese financial language understanding evaluation dataset . In Findings of the Association for Computational Linguistics: A...

2024 doi
[46]

Yi Zong and Xipeng Qiu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.521 GAOKAO - MM : A C hinese human-level benchmark for multimodal models evaluation . In Findings of the Association for Computational Linguistics: ACL 2024, pages 8817--8825, Bangkok, Thailand. Associ...

2024 doi
[47]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[48]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. 2024. A survey on rag with llms. Procedia computer science, 246:3781--3790

work page 2024

[2] [2]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020

[3] [3]

Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2024. Medbench: A large-scale chinese benchmark for evaluating medical large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17709--17717

2024

[4] [4]

Yu, Qiang Yang, and Xing Xie

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. https://doi.org/10.1145/3641289 A survey on evaluation of large language models . ACM Trans. Intell. Syst. Technol., 15(3)

doi:10.1145/3641289 2024

[5] [5]

Wei Chen, Qiushi Wang, Zefei Long, Xianyin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, et al. 2023. Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning. arXiv preprint arXiv:2310.15205

arXiv 2023

[6] [6]

Xiangxiang Cui, Shu Yang, Tianjin Huang, Wanyu Lin, Lijie Hu, and Di Wang. 2025. https://arxiv.org/abs/2506.15617 The compositional architecture of regret in large language models . Preprint, arXiv:2506.15617

work page Pith review arXiv 2025

[7] [7]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

arXiv 2025

[8] [8]

Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. 2023. Fate-llm: A industrial grade federated learning framework for large language models. Symposium on Advances and Open Problems in Large Language Models (LLM@IJCAI'23)

work page 2023