{"id":"e51f705a-2dec-4511-8fed-5fc6c009995c","arxiv_id":"2504.20609","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors train WenyanGPT, a Classical Chinese LLM, and claim it beats GPT-4o and other models on their WenyanBENCH tasks, though the benchmark shares data sources with the training set.","lead":"WenyanGPT is a large language model fine-tuned for Classical Chinese, trained on a 16GB corpus and 1.85 million instruction pairs. It reports large gains over GPT-4o and other LLMs on its own WenyanBENCH benchmark, but that benchmark draws from the same sources as the training data, so the gains may be inflated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark-source overlap is the load-bearing weakness: WenyanBENCH and the SFT set draw from the same corpora, so the reported F1/BLEU gains may reflect distribution overlap rather than general Classical Chinese skill.","rationale":"After reading the paper, I agree with the reader's weakest_assumption. The benchmark is explicitly drawn from the same sources as the fine-tuning data, so the comparison is not between a model with general Classical Chinese ability and models without it; it is between a model that has seen this corpus distribution and models that have not. The phrase 'undergone duplicate data removal' is the only safeguard, but exact deduplication is insufficient because the benchmark items and training items can overlap at the level of sentences, templates, annotation conventions, and vocabulary. This is the single most load-bearing point because every numeric claim in Tables 6 and 7 depends on WenyanBENCH's validity. I also note additional weaknesses (no error bars, no TongGu baseline, no ablation), but those are secondary. If the proposed overlap test shows low contamination and the margins persist, the concern would be substantially answered; as written, the paper does not provide that evidence. Therefore I see no reason to change the reader's REJECT verdict; the authors can address this with a transparent overlap analysis and an external test set.","tokens_in":13842,"tokens_out":5027,"duration_ms":50988,"concrete_test":"Compute exact and near-duplicate overlap between each WenyanBENCH test input and the released instruction fine-tuning corpus (e.g., MinHash token-set Jaccard ≥0.8, or 13-gram overlap). Report the fraction of test examples with a near-duplicate. If the fraction is non-trivial (e.g., >5%), re-run the Table 6 and Table 7 comparisons on only the non-overlapping subset and check whether WenyanGPT's lead, especially the punctuation and NER margins, persists. If it does not, the central claim is unsupported. A complementary check is to evaluate all models on an independently sourced external test set such as TongGu or GujiBERT tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; §5.2) is that WenyanGPT 'significantly outperforms current advanced LLMs' on Classical Chinese tasks, with the largest margins in punctuation (75.66 vs 59.01 F1 for DeepSeek-V3) and NER (91.16 vs 68.42 F1 for GPT-4o). The comparison is only interpretable if WenyanBENCH is an independent test of Classical Chinese ability. Section 4 states: 'WenyanBench shares the same data sources as the instruction fine-tuning data and has undergone duplicate data removal.' The SFT set (Table 3) is built from Daizhige, evahan, classical-modern, wenyanguji.com, gushiwen.com, chinese-dictionary, and chinese-xinhua; the benchmark tasks (Table 5) come from the same sources. Exact-duplicate removal does not eliminate near-duplicates, format templates, or text fragments from the same underlying corpus. Since WenyanGPT was fine-tuned on ~1.85M examples from these sources, its outputs can exploit in-distribution patterns that the GPT-4o/DeepSeek/other baselines, which never saw this distribution, cannot. The 20-point F1 gaps are therefore plausibly inflated by test-set overlap. No overlap statistics, confidence intervals, or significance tests are provided, and the related-work baseline TongGu—a directly comparable Classical Chinese model—is cited but not evaluated. This is an evaluation-design gap rather than a claim of misconduct; the released resources may still be useful, but the headline empirical claim is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents WenyanGPT, an 8B-parameter LLM for Classical Chinese, built by continued pre-training and instruction fine-tuning on a LLaMA3-8B-Chinese base. The authors contribute a 16GB Classical Chinese pre-training corpus, a 1.85M-example instruction tuning dataset, and WenyanBENCH, a six-task benchmark. They report that WenyanGPT outperforms general-purpose LLMs and one Classical Chinese model on all six tasks, with particularly large margins in NER and punctuation. The paper also describes a pipeline for generating domain-specific instruction data.","tokens_in":14167,"tokens_out":6293,"duration_ms":55994,"significance":"If the empirical claims were sound, this would be a useful contribution: it addresses an under-served language domain, releases a model and datasets, and proposes a generalizable instruction-data construction method. The transparency about data sources and the decision to release the resources are commendable. However, the central evaluation is compromised by the overlap between the benchmark and training data, and by the lack of statistical rigor, so the headline claim of 'significantly outperforms' is not currently supported. The resources themselves may be valuable to the community, but the paper's main empirical conclusion requires a fundamentally more trustworthy evaluation.","major_comments":[{"comment":"The benchmark WenyanBENCH is explicitly stated to share the same data sources as the instruction fine-tuning data, with only 'duplicate data removal' as a safeguard. The fine-tuning set (Table 3) and the benchmark (Table 5) draw from the same corpora (e.g., Daizhige, classical-modern, wenyanguji.com, and for NER a 'Self-built' source). Because WenyanGPT was fine-tuned on 1.85M examples from these sources, its high scores on WenyanBENCH (e.g., NER F1 91.16 vs. 68.42 for GPT-4o) may reflect distribution overlap rather than general Classical Chinese competence. The paper provides no details on how duplicates were detected, no overlap statistics, and no evidence that the advantage persists on a test set from sources not used in training. This is a load-bearing flaw: without an independent test set, the abstract and §5.2 claim of 'significantly outperforms' is not supported.","section":"§4 (Benchmarking), Tables 3 and 5"},{"comment":"All reported results are single-point estimates with no error bars, confidence intervals, or significance tests. Several tasks have small evaluation sets (e.g., part-of-speech tagging has 1,247 examples; Table 5), so the observed gaps (e.g., punctuation F1 75.66 vs. 59.01 for DeepSeek-V3) could be unstable. To substantiate the claim of 'significant' superiority, the paper should report variance across multiple evaluation runs or bootstrap resampling, and perform a significance test such as paired bootstrap or McNemar's test for the categorical tasks.","section":"§5.2, Tables 6 and 7"},{"comment":"The comparison includes only one Classical Chinese-specific model, Xunzi-Qwen1.5-7B-Chat. TongGu (Cao et al., 2024) is described in the related work as a Classical Chinese LLM capable of punctuation, translation, and appreciation, but it is not evaluated. Given that the paper claims to surpass 'current advanced LLMs in various Classical Chinese tasks' and that TongGu is a direct competing model, its absence makes the headline comparison incomplete. The authors should include existing Classical Chinese models as baselines, or justify their exclusion.","section":"§5.1 Baselines"},{"comment":"The evaluation protocol is not described with enough detail to ensure fair comparison. The paper does not specify the prompt format used for each baseline, whether any few-shot examples were provided, how structured outputs (e.g., NER JSON) were parsed and scored, or how BLEU and BERTScore were computed (tokenization, reference selection). These details are necessary for reproducibility and for assessing whether the reported margins could be due to prompt or parsing artifacts rather than model capability.","section":"§4 Evaluation Method and §5.1 Data and Evaluation"}],"minor_comments":[{"comment":"The entry '107,3017' appears to be a formatting error; the intended number is likely '1,073,017' given the stated total of 1,847,922.","section":"Table 3"},{"comment":"The caption says 'Data Sources and Detailed Statistics of Wenyan-Bench' but the table only lists tasks and numbers; either add a source column or revise the caption.","section":"Table 5"},{"comment":"The x-axis label 'characte rs' contains an unwanted space, and the NER subcategory names are not defined in the main text.","section":"Figure 4"},{"comment":"There are several typos and minor language errors: 'outpermforms' in the Conclusion, 'ccessing' in §5.3, 'We attend to explore' (likely 'intend') in the Conclusion, 'the forth year' in §5.3, and inconsistent capitalization of 'WenyanBench'/'WenyanBENCH'.","section":"Throughout"},{"comment":"The abstract contains LaTeX artifacts ('\\footnote') and footnote markers that do not render; these should be removed or resolved.","section":"Abstract"},{"comment":"The BERT citation is given as 'Kenton and Toutanova, 2019' but the actual paper is Devlin et al., 2019; the citation format should be corrected.","section":"References"},{"comment":"In the DeepSeek-V3 output, there is a spurious '/ns' token (an empty field) which appears to be a formatting error.","section":"Table 8"},{"comment":"The radar plot normalizes values to 0-1 but the normalization method is not specified; please clarify how the scaling is performed.","section":"§5.2, Figure 5"}],"recommendation":"reject","confidential_remarks":"The benchmark overlap is acknowledged in the paper itself, which makes the evaluation design problem clear. I would advise the editor that a resubmission with a properly held-out benchmark and rigorous statistical analysis could be viable, as the released resources have potential value. However, the current version's central empirical claim cannot be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper builds a genuinely large open resource for Classical Chinese NLP — a 16GB pre-training corpus, 1.85M instruction examples, a released model, and a six-task benchmark. If you work in this area, the assets alone are worth a look. The second thing: the headline performance claim doesn't hold up as presented, because the benchmark is not independent from the training data.\n\nThe new content here is mostly scale and packaging. Continued pre-training plus SFT on a domain LLM is standard — Xunzi and TongGu, both cited, do the same. What's new is the size of the assembled corpus, the instruction set, and the explicit description of the instruction-generation pipeline, which is clear and reproducible. The authors also publish the data and the model, which is more than many similar papers do.\n\nThe soft spot is the one the stress test points to: Section 4 says WenyanBENCH 'shares the same data sources as the instruction fine-tuning data' with only 'duplicate data removal' as protection. Exact-duplicate removal doesn't eliminate near-duplicates or fragments from the same books and websites. Since the model was fine-tuned on 1.85M examples from exactly those sources, its reported margins — 20+ F1 points on NER, 16 points on punctuation — are very plausibly inflated by distribution overlap. There are no overlap statistics, no confidence intervals, no significance tests. The paper even acknowledges the subjectivity of evaluation in its limitations. That doesn't make the model useless, but it makes the headline claim unsupported.\n\nMissing baselines also hurt: TongGu, a directly comparable Classical Chinese model, is cited but not evaluated. No ablation isolates the contribution of continued pre-training vs. SFT. These are fixable issues.\n\nWho is this for? Researchers in Classical Chinese or historical text processing who want the data and the model, and anyone teaching evaluation design as a cautionary example. The benchmark design flaw is central, but it's not fatal to the resource itself.\n\nMy recommendation: send it to peer review, but expect major revision. The authors should evaluate on an external, independently sourced test set, add basic statistical rigor, and temper the abstract. If they do that, the resources could be genuinely useful.","headline":"Useful open resources for Classical Chinese NLP, but the headline benchmark is contaminated by training data overlap, so the performance claim doesn't stand without an independent test set.","tokens_in":14719,"tokens_out":1712,"would_cite":false,"duration_ms":16243,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WenyanGPT, an 8B model adapted to Classical Chinese by continued pre-training and instruction tuning, reports the best score on all six tasks of the new WenyanBENCH benchmark, beating GPT-4o and DeepSeek-V3.","keywords":["Classical Chinese","large language model","continued pre-training","instruction fine-tuning","benchmark","named entity recognition","punctuation","part-of-speech tagging"],"falsifier":"Construct a held-out Classical Chinese test set from sources that are absent from both the ~16GB pre-training corpus and the ~1.85M instruction set, for example newly published transcriptions of excavated manuscripts, and compare WenyanGPT against GPT-4o and DeepSeek-V3; the central claim fails if the performance margins shrink to noise. A complementary check is to measure n-gram overlap between WenyanBENCH items and the released training data.","tokens_in":13591,"feed_emoji":"📜","tokens_out":8783,"duration_ms":76224,"temperature":0.7,"pith_summary":"This paper tries to establish that a modest open-weight language model can be turned into the best available system for Classical Chinese by domain-specific continued pre-training and instruction tuning. It introduces WenyanGPT, built from LLaMA3-8B-Chinese, and WenyanBENCH, a six-task benchmark covering punctuation, part-of-speech tagging, named entity recognition, translation, word explanation, and reverse dictionary. On WenyanBENCH, WenyanGPT outperforms strong general-purpose models including GPT-4o and DeepSeek-V3 on every task, with the largest gaps in NER and punctuation. The authors also release their pre-training corpus, instruction data, and benchmark so the result can be reproduced and extended.","feed_headline":"WenyanGPT tops GPT-4o on six Classical Chinese tasks","feed_subtitle":"Continual pretraining plus instruction tuning gives an 8B model the edge over much larger rivals on WenyanBENCH.","key_machinery":"The engine of the method is a two-stage training pipeline: continued pre-training of LLaMA3-8B-Chinese on a cleaned ~16GB Classical Chinese corpus, followed by supervised instruction tuning on ~1.85M examples. The instruction data are built by a semi-automatic protocol in which human-written templates are expanded by LLMs, reverse reasoning generates new prompts from existing pairs, and manual plus automatic filtering keeps high-quality samples. WenyanBENCH provides the measuring rod, with six tasks scored by precision, recall, and F1 for understanding tasks and by BLEU and BERT-Score for generation tasks.","core_discovery":"The paper's central claim is that a domain-adapted 8B model can outperform far larger general models on Classical Chinese understanding and generation. WenyanGPT achieves F1 of 91.16 in NER versus 68.42 for GPT-4o, 75.66 in punctuation versus 59.01 for DeepSeek-V3, and 89.1 in part-of-speech tagging versus 81.75 for GPT-4o; in generation, its translation BLEU1 is 0.47 versus 0.41 for GPT-4o, word-explanation BLEU1 is 0.35 versus 0.19 for GPT-4o, and reverse-dictionary BERT-Score F1 is 75.39 versus 71.88 for DeepSeek-V3.","pith_inferences":["Because WenyanBENCH is drawn from the same sources as the instruction-tuning data, the reported margins are best read as evidence of strong domain adaptation rather than proof of general Classical Chinese competence; an independent benchmark built from disjoint sources would be needed to separate the two.","The same data-construction pipeline—manual templates, LLM expansion, reverse reasoning, filtering—could be ported to other historical or low-resource languages, such as Literary Japanese or Medieval Latin, where parallel and tagged corpora are scarce.","The authors' planned move to multimodal input could serve as a discriminating test: if WenyanGPT's gains are mostly memorization of training text, adding image-based manuscript input should not transfer; if they reflect learned syntax and semantics, multimodal extension should succeed."],"forward_implications":["If the results hold, a specialized 8B-parameter model can outperform much larger general-purpose models on Classical Chinese understanding and generation, making domain adaptation a viable path for low-resource historical languages.","WenyanBENCH gives researchers a shared six-task test with fixed metrics, so future Classical Chinese models can be compared directly instead of across inconsistent private datasets.","Releasing the pre-training corpus, instruction data, and benchmark makes the entire pipeline reproducible and lets others test the same recipe on adjacent tasks such as poetry appreciation or ancient-text question answering.","Multitask instruction tuning appears to stabilize fine-grained predictions, with WenyanGPT staying above 80% F1 across NER entity categories, suggesting the model internalizes shared linguistic structure across tasks."],"supporting_citations":[{"why":"Supplies the LLaMA 3 base model family from which LLaMA3-8B-Chinese is derived for continued pre-training.","marker":"(Dubey et al., 2024)"},{"why":"Provides DeepSeek-V3, one of the strongest general baselines whose scores WenyanGPT is compared against on WenyanBENCH.","marker":"(DeepSeek-AI et al., 2025)"},{"why":"Provides GPT-4o, the leading general baseline that WenyanGPT outperforms in NER and POS tagging.","marker":"(Hurst et al., 2024)"},{"why":"Shows that GujiBERT and GujiGPT domain pre-training improves Classical Chinese processing, the strategy WenyanGPT extends.","marker":"(Wang et al., 2023a)"},{"why":"TongGu is a prior Classical Chinese LLM built by two-stage instruction fine-tuning that WenyanGPT aims to surpass with larger data.","marker":"(Cao et al., 2024)"},{"why":"AnchiBERT demonstrates continued pre-training on ancient Chinese corpora, motivating WenyanGPT's pre-training phase.","marker":"(Tian et al., 2020)"}],"fun_headline_variants":["8B WenyanGPT beats GPT-4o on Classical Chinese","Domain-tuned 8B model wins Classical Chinese tasks","WenyanGPT: small model, big Classical Chinese gains","Classical Chinese mastery: 8B beats 100B+ rivals","WenyanGPT outperforms giants on Classical Chinese"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes WenyanBENCH measures genuine Classical Chinese ability, but because the benchmark shares its data sources with the model's instruction-tuning set, the scores may partly reflect overlap with training rather than general skill.","fun_headline_variants_meta":{"raw":{"variants":["8B WenyanGPT beats GPT-4o on Classical Chinese","Domain-tuned 8B model wins Classical Chinese tasks","WenyanGPT: small model, big Classical Chinese gains","Classical Chinese mastery: 8B beats 100B+ rivals","WenyanGPT outperforms giants on Classical Chinese"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1383,"prompt_tokens":861,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":436}},"tokens_in":477,"tokens_out":522,"duration_ms":4694,"temperature":1.0,"reasoning_tokens":436,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:24:00.037440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a held-out Classical Chinese test set from sources that are absent from both the ~16GB pre-training corpus and the ~1.85M instruction set, for example newly published transcriptions of excavated manuscripts, and compare WenyanGPT against GPT-4o and DeepSeek-V3; the central claim fails if the performance margins shrink to noise. A complementary check is to measure n-gram overlap between WenyanBENCH items and the released training data.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TongGu is a prior Classical Chinese LLM built by two-stage instruction fine-tuning that WenyanGPT aims to surpass with larger data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AnchiBERT demonstrates continued pre-training on ancient Chinese corpora, motivating WenyanGPT's pre-training phase."}],"review_version":1}