REVIEW 4 major objections 6 minor 9 references
BaiJia: A Large-Scale Role-Playing Agent Corpus of Chinese Historical Characters
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper introduces BaiJia, a corpus of 19,281 Chinese historical characters, and claims that adding its resumes and dialogues improves all tested LLMs on six evaluation dimensions.
desk verdict A genuinely useful resource corpus for Chinese historical role-playing, but the headline empirical claim depends on an evaluation protocol the paper never actually specifies. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the character resume template: a fixed schema with 15 sub-categories organized into Profile, Career, and Achievement sections (basic information, aliases, social divisions, personal introductions, geographic and wealth information, events, family and other relations, entry and appointment data, institutions, literary writings, poems and essays, and dialogues). The paper also uses a two-step dialogue generation procedure adapted from Character-LLM: GPT-4o-mini first extracts ten scenes per character from the resume, then writes questions and in-character responses for those scenes; the resulting dialogues, combined with resumes, are used for LoRA fine-tuning through the LLaMA-Factory framework. This machinery converts fragmented and heterogeneous historical sources into a uniform training and evaluation format.
What would settle it
Re-score a random sample of the baseline and with-corpus responses using human annotators who are blind to condition and who check every factual claim against the historical sources; if the with-corpus responses no longer systematically beat the baselines on all six dimensions, the central claim fails.
Extended reading notes
Core claim
The core discovery is that low-resource historical role-playing data can be consolidated at scale: BaiJia contains 19,281 characters from the Tang, Song, Yuan, Ming, and Qing dynasties, with roughly 310,000 poems and essays, 190,000 generated dialogue turns, and structured resumes covering profile, relation, career, and achievement information. The authors report that after incorporating this corpus, LLMs ranging from 6B to 236B parameters improved on every one of six evaluation dimensions—Character Consistency, Dialogue Ability, Character Appeal, Emotional Expression and Intellectual Depth, Creativity and Role Depth Expansion, and Cultural and Historical Appropriateness—with average improvements of 4.2% to 23.7% across models. They further report that specialized role-playing LLMs such as Baichuan-NPC and Xingchen perform poorly on historical characters without the corpus, which they attribute to the scarcity of historical data in those models' training distributions.
Load-bearing premise
The claim stands on the assumption that the six-dimension scores in Table 3 came from a reliable, independent judge; the paper never says who or what assigned those scores, how many responses were judged, or how the closed API models were given the corpus condition.
Editorial extensions
If this is right
- Existing role-playing corpora cover only a few dozen to a few hundred anime, novel, and script characters; at 19,281 historical figures, BaiJia changes the scale available for fine-tuning and makes historical role-playing a tractable task.
- Because gains appear in every tested model, the corpus apparently supplies data that current models lack, so adding it should help even as base LLMs grow stronger.
- The largest reported gains are in Character Consistency and Cultural and Historical Appropriateness, so the corpus is most useful where historical fidelity matters most.
- Fine-tuning on the corpus fixes concrete knowledge gaps, such as correctly naming a historical character's most accomplished work instead of hallucinating a title or refusing to answer.
Reading between the lines
- Because the corpus pipeline is structurally language-neutral, applying the same resume template and scene-dialogue generation to other historically documented cultures is a plausible next step, though the paper does not claim this.
- A natural test would separate factual recall from voice imitation, asking whether the measured gains come from knowing the character's biography or from adopting the character's temperament; the paper's evaluation bundles both.
- Since both the training dialogues and evaluation questions were generated with the same model family, part of the improvement could reflect imitation of that generator's style; the paper does not address this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BaiJia, a role-playing agent corpus for 19,281 Chinese historical characters from the Tang, Song, Yuan, Ming, and Qing dynasties. The corpus aggregates structured resume data from CBDB, Wikipedia, and Gushiwen, and adds GPT-4o-mini-generated dialogues for supervised fine-tuning (SFT) and GPT-4o-mini-generated questions for evaluation. The central claim, stated in Section 3.2, is that incorporating BaiJia's resume and dialogue information gives all tested LLMs significant improvements across six evaluation dimensions (Table 3). The authors also report an ablation study, a case study, and release the benchmark and data.
Significance. If the empirical claim is substantiated, BaiJia is a potentially valuable resource: it covers an order of magnitude more characters than existing role-playing datasets such as ChatHaruhi, InCharacter, CharacterEval, and RoleLLM, and it addresses a genuinely low-resource area, namely Chinese historical role-playing. The proposed six-dimension evaluation, including three new dimensions (EI, CR, CHA), is a reasonable starting point for this task. However, the paper's central claim currently rests on an incompletely specified evaluation protocol, and the circularity risk from using GPT-4o-mini in both data generation and evaluation is not addressed. The corpus itself is a useful contribution, but the demonstrated effectiveness is not yet established.
major comments (4)
- [Section 3.1.2 / Table 3] The scoring procedure for every number in Table 3 is absent. The paper defines six metrics and twelve sub-metrics but never states who or what assigned the scores, the prompt or rubric used, how many responses were judged per model and per dimension, whether ratings were blinded, or how the two-decimal scores were aggregated. No variance, confidence intervals, or significance tests are reported. Because the headline claim in Section 3.2 is that 'all kinds of LLMs gain significant improvements,' the lack of any verifiable scoring protocol is load-bearing. The authors should specify the judge (human or LLM), the judgment prompt, sample sizes, inter-annotator agreement if humans were used, and inferential statistics.
- [Sections 2.2, 2.3 / Table 3] There is a serious circularity risk. GPT-4o-mini generates the SFT dialogues (Section 2.2) and the evaluation questions (Section 2.3). If the same model family also scores the outputs, the 'with corpus' condition is favored twice: the fine-tuned models are trained to imitate GPT-4o-mini's dialogue style, and the evaluator may prefer that style over the base models' outputs. The reported improvements in Character Consistency and Cultural & Historical Appropriateness, for example +23.5% for ChatGLM3-6B and +29.5% for Xingchen, could then reflect stylistic self-preference rather than historical fidelity. I am not claiming this is what happened, but the manuscript gives no information to rule it out. A minimal fix is to state the judge model and, ideally, use a judge from a different model family or human annotators with fact-based rubrics tied to the resumes, plus agreement statistics.
- [Sections 3.1.1, 3.2 / Tables 3 and 4] The implementation of the 'with corpus' condition for closed API models is not described. Table 4 marks Xingchen and Baichuan-NPC as closed role-playing LLMs, yet Table 3 reports underlined with-corpus results for them. Since these models cannot be fine-tuned by the authors, the incorporation must be via prompt injection or another mechanism, but no such detail appears. If the resume is appended to the prompt, the improvement is a prompt-engineering effect rather than evidence that the corpus improves model capabilities; moreover, supplying the resume directly gives the model the facts needed for Character Consistency and Cultural & Historical Appropriateness, making gains on those dimensions unsurprising. The authors should explain exactly what was done for each model and ensure the intervention is identical across open and closed models.
- [Section 3.3] The ablation and case study do not compensate for the missing evaluation protocol. Figure 3 is a radar chart with no numerical values and no confidence intervals, and Figure 2 is a single hand-picked example. The case study shows that the fine-tuned model can retrieve a fact from the resume, but it does not validate the scores in Table 3. I would ask the authors to report the numerical ablation values, the number of examples used, and a small set of random examples with success/failure rates judged by humans.
minor comments (6)
- [Section 3.1.1] There is a typo: 'Lama [2]' should be 'Llama [2]'.
- [Section 2.3] The final sentence is awkwardly worded: 'allowing for a more accurate assessment of the model’s ability to acquire and understand character knowledge in the development of role-playing agents of role-playing.' The repeated 'of role-playing' should be revised.
- [Section 3.2] The sentence 'The greatest improvements achieved in the dimensions of Character Consistency (CC) and Culture & Historical Appropriateness (CHA)' lacks an auxiliary verb; it should say 'are achieved.'
- [Table 3] The column 'Avg.Imp↑' is not defined; please state whether it is the simple mean of the six relative improvements or a weighted average.
- [Abstract / Section 2.2] The abstract calls BaiJia 'low-resource data,' but the corpus contains 192,810 dialogue items; clarify that 'low-resource' refers to the scarcity of historical role-playing data, not to the size of the corpus.
- [Section 3.1.1] The text says 'Our LLM BaiJia has been fine-tuned on Qwen2.5-7B,' but Table 3 does not have a separate row for 'BaiJia'; please clarify whether the underlined Qwen2.5-7B row is the BaiJia model.
Circularity Check
No demonstrated circularity: the corpus effect is an empirical comparison, though the evaluation protocol's scorer is unspecified.
full rationale
The paper's derivation chain is not circular in the sense of an equation reducing to itself or a fitted parameter being renamed as a prediction. The corpus resumes are assembled from external sources (CBDB, Wikipedia, Gushiwen; Section 2.1), providing independent grounding for character facts. SFT dialogues and evaluation questions are both generated by GPT-4o-mini from those resumes (Sections 2.2 and 2.3), but the paper never states that GPT-4o-mini also produced the Table 3 scores; no evaluation protocol is given identifying the scorer, the number of responses judged, or whether judgments were blinded. Thus, while there is a methodological risk that an LLM judge could prefer the style of GPT-4o-mini-generated training data, that identity is not established in the manuscript, and the reported gains are not shown to be forced by construction. The central experimental claim is an empirical before/after comparison of models with and without the corpus, not a quantity defined to equal its input. The unspecified scorer and the unstated implementation of the 'with corpus' condition for closed API models are reproducibility and validity gaps, not circular reductions. The paper contains no load-bearing self-citations; cited prior work is used for comparison or for the dialogue-generation recipe, not to justify the central result. I therefore find no demonstrated circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption CBDB, Wikipedia, and Gushiwen provide accurate and sufficiently complete source data for historical resumes.
- domain assumption GPT-4o-mini-generated scenes, dialogues, and questions are historically faithful and do not require human fact-checking.
- domain assumption The six evaluation dimensions (CC, DA, CA, EI, CR, CHA) and their scoring procedure validly measure role-playing quality.
- domain assumption Closed models such as DeepSeekV2.5, Xingchen, and Baichuan-NPC can incorporate the corpus through the same 'with corpus' protocol as open models.
Cite this review
Pith. "Pith review of BaiJia: A Large-Scale Role-Playing Agent Corpus of Chinese Historical Characters." pith.science (2026). https://pith.science/paper/MHCMSCSQ
@misc{pith2026241220024,
author = {Pith},
title = {Pith review of: BaiJia: A Large-Scale Role-Playing Agent Corpus of Chinese Historical Characters},
year = {2026},
howpublished = {\url{https://pith.science/paper/MHCMSCSQ}},
note = {Machine review of arXiv:2412.20024}
}
read the original abstract
We introduce a comprehensive large-scale role-playing agent corpus, termed BaiJia, that comprises various Chinese historical characters. This corpus is noteworthy for being the pioneering compilation of low-resource data that can be utilized in large language models (LLMs) to engage in AI-driven historical role-playing agents. BaiJia addresses the challenges in terms of fragmented historical textual records in different forms and modalities, integrating various characters' information, including their biographical, literary, family relations, historical events, and so on. We conduct extensive experiments to demonstrate the effectiveness of our BaiJia agent corpus in bolstering the role-playing abilities of various foundational LLMs, and promoting the development and assessment of LLMs in the context of historical role-playing tasks. The agent corpus is available at baijia.online.
Figures
Reference graph
Works this paper leans on
-
[1]
DeepSeek-AI. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of- Experts Language Model. arXiv:2405.04434
arXiv 2024
-
[2]
Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 https://arxiv.org/abs/2407.21783
arXiv 2024
-
[3]
Team GLM, Aohan Zeng, Bin Xu, et al . 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv:2406.12793
arXiv 2024
-
[4]
Cheng Li, Ziang Leng, et al. 2023. Chatharuhi: Reviving anime character in reality via large language model. arXiv preprint arXiv:2308.09597 (2023)
arXiv 2023
-
[5]
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A Trainable Agent for Role-Playing. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)
work page 2023
-
[6]
Quan Tu, Shilong Fan, Zihang Tian, et al. 2024. CharacterEval: A Chinese Bench- mark for Role-Playing Conversational Agent Evaluation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics . 11836–11850
work page 2024
-
[7]
Noah Wang, Z.y. Peng, Haoran Que, et al. 2024. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024 . 14743–14777
work page 2024
-
[8]
Xintao Wang, Yunze Xiao, et al. 2024. Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics . 1840–1873
work page 2024
Show all 9 references
-
[9]
Yaowei Zheng, Richong Zhang, Junhao Zhang, et al . 2024. LlamaFactory: Uni- fied Efficient Fine-Tuning of 100+ Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.