{"id":"695d0fe0-d3b6-4131-8a5c-fec5ca59ab91","arxiv_id":"2505.05225","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"QualBench offers a 17,316-question Chinese evaluation from professional qualification exams, and Chinese models outperform non-Chinese models on it.","lead":"The authors introduce QualBench, a benchmark of over 17,000 questions from 24 Chinese professional qualification exams across six vertical domains. They report that Chinese LLMs, especially Qwen2.5-7B, outperform larger non-Chinese models on this localized test, suggesting that localized domain knowledge matters for vertical-domain AI evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Chinese-vs-GPT-4o finding rests on unverified absence of contamination; Tables 6–7 test only one model and surface recall, not paraphrase-level memorization.","rationale":"The paper's headline contribution has two parts: a new multi-domain dataset and an empirical claim that Chinese LLMs, especially Qwen2.5-7B, outperform stronger non-Chinese models due to localized knowledge. The empirical claim is the more consequential and the more fragile. If the score gap is partly an artifact of training-data overlap, both the 'value of localized domain knowledge' conclusion and the broader recommendation to prefer Chinese models for Chinese vertical domains lose support. The contamination section is the only direct evidence against this threat, and it is methodologically insufficient: answer shuffling is applied to a subset of models without GPT-4o and with inconsistent baselines, and the completion test measures surface recall on a single model rather than membership of semantically equivalent content. A held-out evaluation on questions released after all models' training cutoffs directly tests whether the gap reflects knowledge or memorization. If the gap disappears on such a split, the paper would need to substantially soften its central claim. If it persists, the claim is strongly corroborated. The dataset itself remains a useful resource either way, so a conditional verdict is appropriate rather than rejection. The 'first multi-domain' novelty claim is secondary and weakened by the absence of C-Eval, but the contamination question is more load-bearing because it threatens the main empirical finding.","tokens_in":21428,"tokens_out":5220,"duration_ms":52374,"concrete_test":"Construct a held-out split from qualification-exam papers first published after the training cutoffs of Qwen2.5-7B and GPT-4o (e.g., 2025 papers), preserving the six-domain distribution, and recompute the Table 4 comparisons on this split with 95% confidence intervals from the five inference runs. If the Qwen2.5-7B vs GPT-4o margin on unseen questions is not significantly positive, the contamination concern lands and the localization claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Chinese LLMs outperform non-Chinese models because of localized knowledge presupposes that QualBench questions are not memorized by the evaluated models, at least not asymmetrically. The paper's contamination checks do not establish this. Table 6's answer-shuffling test omits GPT-4o and other headline models, reports baselines that differ from Table 4 (e.g., Qwen2.5-7B 74.78% vs 75.26%; GPT-3.5 52.24% vs 56.96%), and randomizing answer letters does not rule out memorization of paraphrased or translated items. Table 7's question-completion test runs only on Qwen2.5-7B with 1,020 questions; low ROUGE-L shows only that the model cannot reproduce the exact surface string, not that it fails to recognize the underlying content. Because these qualification exam questions are publicly available on Chinese websites and predate model training, asymmetric contamination between Chinese and non-Chinese models is a plausible alternative explanation for the observed gap. Without contamination checks covering all models in the headline comparison or a held-out subset unseen by all models, the localization narrative is not yet secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QualBench, a Chinese QA benchmark constructed from 24 Chinese professional qualification examinations spanning six vertical domains, yielding 17,316 multiple-choice and true/false questions. The authors evaluate five Chinese and four non-Chinese LLMs in a one-shot setting, reporting that Qwen2.5-7B achieves the highest accuracy (75.26%) and outperforms GPT-4o. They also report three additional findings: that Chinese LLMs consistently outperform non-Chinese models, that LLM crowdsourcing via voting degrades performance relative to a single strong model, and that prompt engineering and fine-tuning benefit non-Chinese models more than Chinese ones. The paper includes contamination checks, ablations, and a public release of data and code.","tokens_in":21645,"tokens_out":7127,"duration_ms":67664,"significance":"If the benchmark holds up, it provides a valuable and timely resource for vertical-domain evaluation of Chinese LLMs, complementing existing single-domain benchmarks and emphasizing localized knowledge. The data construction process (OCR plus two-expert validation) and the public release are concrete strengths, and the headline result—a 7B Chinese model outperforming GPT-4o on localized professional questions—is an interesting, falsifiable finding. However, the paper's broad claim that Chinese LLMs consistently surpass non-Chinese models is not supported by its own Table 4, and the contamination evidence is incomplete for the models in the headline comparison. The resource itself is worthwhile, but the central generalization needs to be re-scoped and the robustness evidence strengthened.","major_comments":[{"comment":"The claim that 'Chinese LLMs consistently surpass non-Chinese models' is contradicted by Table 4. GPT-3.5 (56.96%) and GPT-4o (61.61%) outperform Baichuan2-7B (48.64%), Hunyuan-7B (50.64%), DeepSeek-v2-lite (51.76%), and ChatGLM3-6B (43.62%); only Qwen2.5-7B (75.26%) exceeds GPT-4o. Similarly, the 'regardless of model size' formulation in §5.1 and the Conclusion rests on a single pair in Table 5 (Baichuan2-7B vs LLaMA-14B). Please rephrase the headline contribution to the pattern actually shown, for example: Qwen2.5-7B outperforms GPT-4o on QualBench, and several open Chinese 7B models outperform equivalently sized non-Chinese open models.","section":"Abstract and §5.1, Table 4"},{"comment":"The original-condition accuracies in Table 6 differ from the corresponding entries in Table 4 (Qwen2.5-7B: 74.78% vs 75.26%; GPT-3.5: 52.24% vs 56.96%) without any statement that a subset or different evaluation protocol was used. Because the answer-shuffling test is intended to assess contamination on the headline benchmark, the original condition must match the main evaluation; otherwise the comparison is uninterpretable. Please state whether Table 6 uses the full test set, report the subset size if not, and rerun the shuffle test under conditions identical to Table 4.","section":"§5.3, Table 6"},{"comment":"The question-completion contamination check is executed on only one model, Qwen2.5-7B, on 1,020 sampled questions, while the headline comparison includes GPT-4o, GPT-3.5, and other models. Contamination is model-specific, and these qualification exam questions are publicly available and predate model training, so a single-model test cannot support the paper's conclusion that 'QualBench has minimal overlap with LLM training data' for all evaluated models. Furthermore, low ROUGE-L on surface completion does not rule out content-level memorization of paraphrased or translated versions. Please provide per-model contamination evidence for each model in the main comparison, or restrict the contamination claim to Qwen2.5-7B and mark the other models' status as unverified.","section":"§5.3, Table 7"},{"comment":"The 'LLM crowdsourcing results in performance degradation' finding is reported as a single average over five inference runs, with no confidence intervals or significance tests. Given the temperature=0.5 sampling, the observed gap between the aggregation rows (59.56% and 63.98%) and Qwen2.5-7B (75.26%) is large, but the claim that a single robust LLM is more effective should be supported by variance estimates or a question-level paired test (e.g., McNemar's test). Please add these statistical details or acknowledge that the difference is not tested for significance.","section":"§5.2, Tables 4 and 10"}],"minor_comments":[{"comment":"The phrase 'The average accuracy of 53.98%' is ambiguous: it appears to be the mean of the five Chinese models' overall accuracies from Table 4, rather than an average over all evaluated models or the best model. Please specify the population over which this average is taken.","section":"Abstract"},{"comment":"Production Safety (6,520) and Fire Safety (3,401) together constitute 9,921 of 17,316 questions, or about 57.3% of the dataset. The Limitations section acknowledges the imbalance, but the abstract and results sections should state that the overall accuracy figures are heavily weighted toward these two domains.","section":"§3.3, Table 3 and Limitations"},{"comment":"The rows labeled 'Aggregation majority voting' and 'weighted majority voting' do not state which models are aggregated or how the weights are assigned. Please add this information to the table caption or to §5.2.","section":"Table 4"},{"comment":"The acknowledgments contain a duplicated word: 'including including P0045948.'","section":"Acknowledgments"}],"recommendation":"major_revision","confidential_remarks":"The benchmark resource itself is valuable and the dataset construction appears careful, so the paper is worth a round of revision rather than rejection. The main barrier is the overclaimed headline conclusion about Chinese vs non-Chinese models, which conflicts with Table 4, and the incomplete contamination evidence for the models in the headline comparison. The self-citations are used for motivation and future directions, not to define the benchmark scores, so I do not see a circularity concern. The statistical reporting should also be improved, as five repeated runs without any variance measure are insufficient for the crowdsourcing-degradation claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The QualBench dataset is real and worth having. 17,316 questions from 24 Chinese qualification exams across six vertical domains is a solid contribution, and the OCR-plus-two-expert validation is a reasonable construction pipeline. The error taxonomy in the appendix is also a nice touch—it gives the benchmark practical value beyond the accuracy numbers. So the resource itself deserves a place in the Chinese LLM evaluation ecosystem.\n\nThe soft spots are real, and the stress-test note is right. The contamination checks in Tables 6 and 7 do not carry the weight the paper puts on them. Table 6 omits GPT-4o and the other models in the headline comparison; asking a model to pick the right answer after shuffling letter order does not test whether it has memorized the underlying item, especially if it has seen a paraphrased or translated version. Table 7 runs only on Qwen2.5-7B, and low ROUGE-L shows the model cannot reproduce the exact surface string, not that it fails to recognize the content. That is a load-bearing weakness because the whole localization story depends on the gap not being an artifact of asymmetric contamination. Also, the numbers in Table 6 do not match Table 4 for the same models—Qwen2.5-7B is 74.78% vs 75.26%, GPT-3.5 is 52.24% vs 56.96%—which needs an explanation, and there are no confidence intervals or significance tests anywhere, so we cannot tell if the Qwen-vs-GPT-4o gap is meaningful.\n\nThere are two smaller issues. The paper claims \"first multi-domain Chinese QA benchmark\" but does not cite C-Eval, the most prominent multi-domain Chinese benchmark; the claim needs to be scoped to qualification-exam-based benchmarks. And the LLM crowdsourcing finding rests on a single aggregation setup, so it is more of an observation than a robust result.\n\nWho is this for? Anyone building or evaluating Chinese vertical-domain LLMs, especially for deployment in regulated industries like safety, finance, and banking. The dataset is the contribution, and it is a good one. The performance gap between Chinese and non-Chinese models is interesting but should be treated as provisional until the contamination controls are strengthened.\n\nRecommendation: send this to serious peer review. It deserves referee time, but the authors should be asked to run contamination checks on all headline models, reconcile the table inconsistencies, report variance, and cite C-Eval. If the gap survives better controls, the localization finding becomes much more compelling; if it does not, the dataset still stands on its own.","headline":"The dataset is genuine and useful, but the paper's central claim that Chinese models beat GPT-4o because of localized knowledge is not yet secured by the contamination checks, which omit the headline models and test only surface recall.","tokens_in":22160,"tokens_out":2274,"would_cite":true,"duration_ms":22504,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark of 17,316 Chinese professional-exam questions finds Qwen2.5-7B outperforming GPT-4o on localized knowledge.","keywords":["Chinese LLM evaluation","qualification exams","vertical domain benchmark","localized knowledge","data contamination","LLM crowdsourcing","Qwen2.5"],"falsifier":"Build a validated subset from exam papers published after the training cutoff of all evaluated models (e.g., 2025 question sets) and rerun the one-shot evaluation: if Qwen2.5-7B's advantage over GPT-4o shrinks or vanishes on those genuinely unseen questions, the localization conclusion would be an artifact of training-data leakage. A cheaper check is to translate a sample of QualBench questions into English while keeping the same answer options; if English-centric models reach parity on the translated version, the gap is language familiarity rather than domain knowledge.","tokens_in":21242,"feed_emoji":"📝","tokens_out":6399,"duration_ms":53299,"temperature":0.7,"pith_summary":"This paper builds QualBench, a Chinese question-answering benchmark made from 24 real Chinese professional qualification exams, spanning six vertical domains and containing over 17,000 multiple-choice and true/false questions. The authors use it to test whether Chinese LLMs possess localized professional knowledge that English-centric models lack. Their headline finding is that Chinese models consistently beat non-Chinese models, with the 7-billion-parameter Qwen2.5 model outperforming the much larger GPT-4o. The paper argues this demonstrates the value of localized pretraining knowledge, and reports that even the best model reaches only about 75 percent accuracy, leaving clear room for improvement through retrieval augmentation and continued fine-tuning.","feed_headline":"Qwen2.5-7B outscores GPT-4o on Chinese professional exams","feed_subtitle":"A new benchmark built from 24 Chinese qualification exams shows localized knowledge beats model size.","key_machinery":"The central object is QualBench, a dataset of 17,316 multiple-choice and true/false questions extracted via OCR from 24 Chinese professional qualification exams across six domains (production safety, fire safety, civil engineering, economics and finance, oil and gas, banking and insurance), with duplicate removal and expert validation. The mechanism carrying the argument is localization: questions are drawn from Chinese regulatory and professional standards, so answering them correctly requires knowledge of Chinese laws, policies, and industry practices rather than generic reasoning. The evaluation protocol that makes the comparison fair is a one-shot setting with a standardized prompt asking for an answer plus explanation, repeated five times with averaged accuracy and F1.","core_discovery":"The paper's central claim is that QualBench is the first multi-domain Chinese QA benchmark built specifically for localized assessment of Chinese LLMs, and that on this benchmark Chinese LLMs consistently outperform non-Chinese models regardless of model size: Qwen2.5-7B-instruct achieves 75.26% average accuracy versus 61.61% for GPT-4o and 56.96% for GPT-3.5, while smaller Chinese models like Baichuan2-7B (48.64%) also beat larger non-Chinese models like LLaMA-14B (37.65%). The authors attribute this to native Chinese pretraining knowledge covering localized laws, regulations, and professional standards. They also report that majority-voting crowdsourcing among LLMs degrades performance relative to a single strong model, and that prompt engineering and fine-tuning help non-Chinese models more than Chinese models, reinforcing the localization story. The average accuracy of 53.98% across all models indicates substantial remaining gaps in vertical-domain coverage.","pith_inferences":["The localization advantage shown on exam questions may extend to other Chinese-specific tasks such as contract review, regulatory compliance chatbots, and safety inspection automation; this is a natural extrapolation the paper does not test.","A stricter contamination check would be to evaluate on post-training-cutoff exam years (e.g., 2025 papers for models whose training ended in 2024); if the Qwen advantage shrinks on genuinely unseen questions, the paper's central contrast would need reinterpretation.","The benchmark could be extended to open-ended questions and image-based diagrams to test whether the localization advantage persists when models cannot rely on option recognition; the paper explicitly limits itself to closed-form questions.","An English-translated version of the same questions would separate knowledge of Chinese domain facts from the ability to read Chinese, a distinction the current design conflates."],"forward_implications":["If QualBench accurately measures localized professional knowledge, Chinese LLMs should be preferred over equally sized or larger non-Chinese models for Chinese vertical-domain applications such as safety compliance, construction management, and financial regulation.","The consistent gap across model sizes implies that pretraining language and regional corpus coverage matter more than raw parameter count for localized expertise tasks.","The finding that LLM crowdsourcing (majority or weighted voting across five models) underperforms a single strong model suggests that aggregation strategies must be redesigned, or model diversity increased, before ensembles help on knowledge-dense benchmarks.","The large gains non-Chinese models show from role prompting and in-context learning indicate that prompt design can partially compensate for missing localized knowledge, but cannot fully close the gap.","The benchmark's low overall average accuracy (53.98%) means current LLMs are not yet reliably deployable in these Chinese professional settings without retrieval augmentation or fine-tuning."],"supporting_citations":[{"why":"Supplies the Gaokao benchmark that QualBench contrasts against for coverage and localization.","marker":"Zhang et al., 2023b"},{"why":"CFLUE, a single-domain Chinese finance exam benchmark, is the contrast for multi-domain breadth.","marker":"Zhu et al., 2024"},{"why":"MedBench, a single-domain medical exam benchmark, is another comparator for domain coverage.","marker":"Cai et al., 2024"},{"why":"LexEval, a legal qualification benchmark, motivates the localization dimension QualBench adds.","marker":"Li et al., 2024"},{"why":"Provides the rationale for measuring data contamination per benchmark, which QualBench's checks respond to.","marker":"Sainz et al., 2023"},{"why":"Supplies the contamination-detection approach that QualBench's shuffling and completion tests adapt.","marker":"Golchin and Surdeanu, 2025"},{"why":"Technical report for Qwen2.5, the model that achieves QualBench's best results.","marker":"Yang et al., 2024a"},{"why":"Defines DISC-FinLLM, the domain-specific model that QualBench fine-tunes to show dataset value.","marker":"Chen et al., 2023"}],"fun_headline_variants":["China's Qwen2.5-7B beats GPT-4o on local exams","Local knowledge wins: Chinese LLMs top new benchmark","First Chinese qualification exam benchmark exposes 54% gap","Native training beats size: Chinese LLMs dominate new exam test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline Chinese-versus-non-Chinese comparison assumes the evaluated models have not memorized the QualBench questions during pretraining, so the score gap reflects localized knowledge rather than training-data overlap.","fun_headline_variants_meta":{"raw":{"variants":["China's Qwen2.5-7B beats GPT-4o on local exams","Local knowledge wins: Chinese LLMs top new benchmark","First Chinese qualification exam benchmark exposes 54% gap","Native training beats size: Chinese LLMs dominate new exam test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000481,"raw_usage":{"total_tokens":2374,"prompt_tokens":934,"completion_tokens":1440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1367}},"tokens_in":550,"tokens_out":1440,"duration_ms":10483,"temperature":1.0,"reasoning_tokens":1367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:09:15.576129+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a validated subset from exam papers published after the training cutoff of all evaluated models (e.g., 2025 question sets) and rerun the one-shot evaluation: if Qwen2.5-7B's advantage over GPT-4o shrinks or vanishes on those genuinely unseen questions, the localization conclusion would be an artifact of training-data leakage. A cheaper check is to translate a sample of QualBench questions into English while keeping the same answer options; if English-centric models reach parity on the translated version, the gap is language familiarity rather than domain knowledge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the contamination-detection approach that QualBench's shuffling and completion tests adapt."}],"review_version":1}