{"id":"eaaf1ae2-e9fb-4442-a4bd-f6f21f20f0c2","arxiv_id":"2412.20024","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 19,281-character corpus of Chinese historical resumes and dialogues is claimed to improve LLM role-playing, with an evaluation whose scoring method is not disclosed.","lead":"This paper presents BaiJia, a dataset of 19,281 Chinese historical figures with biographical resumes, GPT-generated dialogues, and evaluation questions for teaching AI to role-play as those figures. It reports that fine-tuning several open language models on this data improves their scores on six role-playing dimensions, though the evaluation details are not fully described.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's gains rest on an unspecified evaluation protocol; if GPT-4o-mini both generated the training data and judged the responses, the reported improvements may reflect stylistic self-preference rather than historical role-playing.","rationale":"The reader's conditional verdict centers on the evaluation protocol and the GPT-4o-mini generator/evaluator overlap, and I find this is indeed the most load-bearing weakness. The paper's contribution is a corpus, and the empirical demonstration is that the corpus improves role-playing; Table 3 is the only quantitative evidence. Because the scorer is unnamed, the experiment is not reproducible from the text, and the generator/evaluator overlap creates a concrete mechanism by which the observed gains could be inflated. The closed-API 'with corpus' protocol is also missing, but this is secondary; the primary issue is that the scores themselves are unattributed. The paper does have genuine independent support: the corpus is large and sourced from public databases, and the case study in Fig. 2 shows a specific factual improvement that a human can verify. That is real evidence for the corpus's factual utility, but it does not establish the quantitative claim in Table 3. I therefore keep the verdict CONDITIONAL; the condition is releasing and independently auditing the evaluation protocol. I agree with the reader's identification of the weakest assumption rather than proposing a different one.","tokens_in":5977,"tokens_out":2996,"duration_ms":29867,"concrete_test":"Release the full evaluation harness (scorer model/prompt, response sample, and raw scores) and independently re-score a fixed sample of 200 responses per model condition (or all available) with two Chinese-speaking historians and a strong judge model not used in data generation, e.g., GPT-4o or Claude 3.5 Sonnet. Compute with-corpus deltas and inter-annotator agreement; if the improvements in Table 3 shrink to noise or reverse under independent scoring, the central claim fails. If they persist across human raters and an independent judge, the circularity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central experimental claim in Section 3.2 is that injecting BaiJia data improves all LLMs across six dimensions (Table 3). This claim depends entirely on scores that the paper never attributes. Section 3.1.2 lists dimensions and sub-dimensions but does not state who or what assigned the Table 3 scores, how many responses were judged, whether judgments were blinded, or whether any human verified historical fidelity. In parallel, Section 2.2 generates all SFT dialogues with GPT-4o-mini and Section 2.3 generates all evaluation questions with GPT-4o-mini. If the same model family also scored the outputs, the 'with corpus' condition is advantaged twice: the model was fine-tuned to imitate GPT-4o-mini's dialogue style, and the evaluator may prefer that style over the unmodified base models' outputs. This is a real circularity risk, not merely a missing detail, because the largest gains (+23.5% CC, +21.7% CHA for ChatGLM3-6B; +29.5% CC for Xingchen) are exactly in dimensions where an LLM judge's stylistic preferences are most likely to express themselves. A second under-specification compounds the first: Table 4 lists DeepSeekV2.5, Xingchen, and Baichuan-NPC as closed API models, but no section explains how the 'with corpus' condition was implemented for them. If it was resume injection at inference time, then the comparison mixes prompt engineering with the claimed corpus effect; if it was fine-tuning, that requires access the authors do not describe. Either way, the headline result is not independently checkable from the paper text.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BaiJia, a role-playing agent corpus for 19,281 Chinese historical characters from the Tang, Song, Yuan, Ming, and Qing dynasties. The corpus aggregates structured resume data from CBDB, Wikipedia, and Gushiwen, and adds GPT-4o-mini-generated dialogues for supervised fine-tuning (SFT) and GPT-4o-mini-generated questions for evaluation. The central claim, stated in Section 3.2, is that incorporating BaiJia's resume and dialogue information gives all tested LLMs significant improvements across six evaluation dimensions (Table 3). The authors also report an ablation study, a case study, and release the benchmark and data.","tokens_in":6231,"tokens_out":5855,"duration_ms":55700,"significance":"If the empirical claim is substantiated, BaiJia is a potentially valuable resource: it covers an order of magnitude more characters than existing role-playing datasets such as ChatHaruhi, InCharacter, CharacterEval, and RoleLLM, and it addresses a genuinely low-resource area, namely Chinese historical role-playing. The proposed six-dimension evaluation, including three new dimensions (EI, CR, CHA), is a reasonable starting point for this task. However, the paper's central claim currently rests on an incompletely specified evaluation protocol, and the circularity risk from using GPT-4o-mini in both data generation and evaluation is not addressed. The corpus itself is a useful contribution, but the demonstrated effectiveness is not yet established.","major_comments":[{"comment":"The scoring procedure for every number in Table 3 is absent. The paper defines six metrics and twelve sub-metrics but never states who or what assigned the scores, the prompt or rubric used, how many responses were judged per model and per dimension, whether ratings were blinded, or how the two-decimal scores were aggregated. No variance, confidence intervals, or significance tests are reported. Because the headline claim in Section 3.2 is that 'all kinds of LLMs gain significant improvements,' the lack of any verifiable scoring protocol is load-bearing. The authors should specify the judge (human or LLM), the judgment prompt, sample sizes, inter-annotator agreement if humans were used, and inferential statistics.","section":"Section 3.1.2 / Table 3"},{"comment":"There is a serious circularity risk. GPT-4o-mini generates the SFT dialogues (Section 2.2) and the evaluation questions (Section 2.3). If the same model family also scores the outputs, the 'with corpus' condition is favored twice: the fine-tuned models are trained to imitate GPT-4o-mini's dialogue style, and the evaluator may prefer that style over the base models' outputs. The reported improvements in Character Consistency and Cultural & Historical Appropriateness, for example +23.5% for ChatGLM3-6B and +29.5% for Xingchen, could then reflect stylistic self-preference rather than historical fidelity. I am not claiming this is what happened, but the manuscript gives no information to rule it out. A minimal fix is to state the judge model and, ideally, use a judge from a different model family or human annotators with fact-based rubrics tied to the resumes, plus agreement statistics.","section":"Sections 2.2, 2.3 / Table 3"},{"comment":"The implementation of the 'with corpus' condition for closed API models is not described. Table 4 marks Xingchen and Baichuan-NPC as closed role-playing LLMs, yet Table 3 reports underlined with-corpus results for them. Since these models cannot be fine-tuned by the authors, the incorporation must be via prompt injection or another mechanism, but no such detail appears. If the resume is appended to the prompt, the improvement is a prompt-engineering effect rather than evidence that the corpus improves model capabilities; moreover, supplying the resume directly gives the model the facts needed for Character Consistency and Cultural & Historical Appropriateness, making gains on those dimensions unsurprising. The authors should explain exactly what was done for each model and ensure the intervention is identical across open and closed models.","section":"Sections 3.1.1, 3.2 / Tables 3 and 4"},{"comment":"The ablation and case study do not compensate for the missing evaluation protocol. Figure 3 is a radar chart with no numerical values and no confidence intervals, and Figure 2 is a single hand-picked example. The case study shows that the fine-tuned model can retrieve a fact from the resume, but it does not validate the scores in Table 3. I would ask the authors to report the numerical ablation values, the number of examples used, and a small set of random examples with success/failure rates judged by humans.","section":"Section 3.3"}],"minor_comments":[{"comment":"There is a typo: 'Lama [2]' should be 'Llama [2]'.","section":"Section 3.1.1"},{"comment":"The final sentence is awkwardly worded: 'allowing for a more accurate assessment of the model’s ability to acquire and understand character knowledge in the development of role-playing agents of role-playing.' The repeated 'of role-playing' should be revised.","section":"Section 2.3"},{"comment":"The sentence 'The greatest improvements achieved in the dimensions of Character Consistency (CC) and Culture & Historical Appropriateness (CHA)' lacks an auxiliary verb; it should say 'are achieved.'","section":"Section 3.2"},{"comment":"The column 'Avg.Imp↑' is not defined; please state whether it is the simple mean of the six relative improvements or a weighted average.","section":"Table 3"},{"comment":"The abstract calls BaiJia 'low-resource data,' but the corpus contains 192,810 dialogue items; clarify that 'low-resource' refers to the scarcity of historical role-playing data, not to the size of the corpus.","section":"Abstract / Section 2.2"},{"comment":"The text says 'Our LLM BaiJia has been fine-tuned on Qwen2.5-7B,' but Table 3 does not have a separate row for 'BaiJia'; please clarify whether the underlined Qwen2.5-7B row is the BaiJia model.","section":"Section 3.1.1"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the missing evaluation protocol. If the judge turns out to be GPT-4o-mini, the circularity concern could be decisive for the paper's empirical claim. I would advise the editor to require the authors to disclose the judge, the scoring prompt, and the sample sizes, and to rerun the evaluation with either human annotators or an independent judge model with fact-based rubrics. The corpus itself appears potentially valuable, so the paper is repairable, but the current version does not support the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick version: BaiJia is a genuinely new resource — the first large-scale corpus of Chinese historical characters for role-playing, 19,281 characters across five dynasties, with structured resumes, generated dialogues, and an evaluation benchmark. That is the real contribution, and it looks useful for the community. But the headline empirical claim — that injecting this corpus improves all LLMs — rests on an evaluation protocol that the paper never actually describes. That needs to be fixed before the numbers can be trusted.\n\nWhat's good: the resume template is thoughtful, with completion rates per sub-category, and the choice to draw on CBDB, Wikipedia, and Gushiwen is sensible. The scale alone, compared to the 32–100 characters in prior work, makes this the first resource that could support SFT at scale for historical role-playing. The construction pipeline is adapted from Character-LLM and RoleLLM, which is fine — the novelty is the resource, not the method.\n\nThe serious problem is Table 3. The six dimensions are defined, but there is no statement of who assigned the scores, how many responses were judged, whether humans were involved, or any variance. Given that both the training dialogues and the evaluation questions are generated by GPT-4o-mini, if the judge is also GPT-family, the results could be rewarding stylistic imitation rather than historical fidelity. The stress-test flags this correctly, and I don't think the paper preempts it. Also, Table 4 lists closed API models — DeepSeekV2.5, Xingchen, Baichuan-NPC — but there is no explanation of how the 'with corpus' condition was implemented for them. Resume injection at inference time is a different intervention than fine-tuning, and the paper conflates them. These are not minor details; they bear directly on the central claim. The ablation and case study are suggestive but only cover one model.\n\nOne smaller point: the word 'significant' is used without any statistical test. The percentage improvements are large, so this is probably a descriptive statement that would survive a real test, but it should be phrased less formally.\n\nWho is this for? Researchers working on LLM role-playing, especially Chinese historical or cultural characters, and anyone building character corpora. The resource itself deserves a serious referee. The empirical section needs major revision — release the evaluation code, specify the judge, include human validation, and explain the closed API protocol.\n\nRecommendation: send to peer review. It's a desk-accept candidate for the corpus, but not for the current experimental write-up.","headline":"A genuinely useful resource corpus for Chinese historical role-playing, but the headline empirical claim depends on an evaluation protocol the paper never actually specifies.","tokens_in":6825,"tokens_out":1945,"would_cite":false,"duration_ms":20097,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces BaiJia, a corpus of 19,281 Chinese historical characters, and claims that adding its resumes and dialogues improves all tested LLMs on six evaluation dimensions.","keywords":["Chinese Historical Characters","Role-Playing Agent","Large Language Models","Low-Resource Data Corpus","LoRA Fine-Tuning","Historical Role-Playing Evaluation","Chinese Dynasties","Dialogue Generation"],"falsifier":"Re-score a random sample of the baseline and with-corpus responses using human annotators who are blind to condition and who check every factual claim against the historical sources; if the with-corpus responses no longer systematically beat the baselines on all six dimensions, the central claim fails.","tokens_in":5729,"feed_emoji":"🎭","tokens_out":6125,"duration_ms":57092,"temperature":0.7,"pith_summary":"BaiJia is a role-playing agent corpus built from the fragmented historical record: it assembles resumes for 19,281 Chinese historical characters from the Tang through Qing dynasties, combining biographical details, family relations, careers, literary works, and events. The paper's central claim is that large language models, after being fine-tuned on these resumes plus generated scene dialogues, become noticeably better at speaking as those historical characters than the same models without the corpus. Across eight open and closed LLMs, every one improved on all six evaluation dimensions, with the largest gains in character consistency and cultural-historical appropriateness. A sympathetic reader would care because prior role-playing corpora cover only dozens to a hundred fictional, anime, or novel characters, leaving the far larger space of historically documented humans untouched.","feed_headline":"19,281 Chinese historical figures become an LLM role-play corpus","feed_subtitle":"Adding resumes and dialogues lifts role-playing consistency and cultural accuracy across six dimensions.","key_machinery":"The load-bearing machinery is the character resume template: a fixed schema with 15 sub-categories organized into Profile, Career, and Achievement sections (basic information, aliases, social divisions, personal introductions, geographic and wealth information, events, family and other relations, entry and appointment data, institutions, literary writings, poems and essays, and dialogues). The paper also uses a two-step dialogue generation procedure adapted from Character-LLM: GPT-4o-mini first extracts ten scenes per character from the resume, then writes questions and in-character responses for those scenes; the resulting dialogues, combined with resumes, are used for LoRA fine-tuning through the LLaMA-Factory framework. This machinery converts fragmented and heterogeneous historical sources into a uniform training and evaluation format.","core_discovery":"The core discovery is that low-resource historical role-playing data can be consolidated at scale: BaiJia contains 19,281 characters from the Tang, Song, Yuan, Ming, and Qing dynasties, with roughly 310,000 poems and essays, 190,000 generated dialogue turns, and structured resumes covering profile, relation, career, and achievement information. The authors report that after incorporating this corpus, LLMs ranging from 6B to 236B parameters improved on every one of six evaluation dimensions—Character Consistency, Dialogue Ability, Character Appeal, Emotional Expression and Intellectual Depth, Creativity and Role Depth Expansion, and Cultural and Historical Appropriateness—with average improvements of 4.2% to 23.7% across models. They further report that specialized role-playing LLMs such as Baichuan-NPC and Xingchen perform poorly on historical characters without the corpus, which they attribute to the scarcity of historical data in those models' training distributions.","pith_inferences":["Because the corpus pipeline is structurally language-neutral, applying the same resume template and scene-dialogue generation to other historically documented cultures is a plausible next step, though the paper does not claim this.","A natural test would separate factual recall from voice imitation, asking whether the measured gains come from knowing the character's biography or from adopting the character's temperament; the paper's evaluation bundles both.","Since both the training dialogues and evaluation questions were generated with the same model family, part of the improvement could reflect imitation of that generator's style; the paper does not address this."],"forward_implications":["Existing role-playing corpora cover only a few dozen to a few hundred anime, novel, and script characters; at 19,281 historical figures, BaiJia changes the scale available for fine-tuning and makes historical role-playing a tractable task.","Because gains appear in every tested model, the corpus apparently supplies data that current models lack, so adding it should help even as base LLMs grow stronger.","The largest reported gains are in Character Consistency and Cultural and Historical Appropriateness, so the corpus is most useful where historical fidelity matters most.","Fine-tuning on the corpus fixes concrete knowledge gaps, such as correctly naming a historical character's most accomplished work instead of hallucinating a title or refusing to answer."],"supporting_citations":[{"why":"Describes the DeepSeek-V2 model used as one of the eight baseline LLMs in Table 3.","marker":"[1]"},{"why":"Describes the Llama 3 model family used for the Llama-3.1-8B and Llama-3.1-70B baselines.","marker":"[2]"},{"why":"Describes the ChatGLM family used for the ChatGLM3-6B baseline.","marker":"[3]"},{"why":"Supplies a role-playing benchmark and the CC/DA/CA evaluation dimensions that BaiJia's evaluation extends.","marker":"[4]"},{"why":"Supplies the two-step scene-extraction and dialogue-generation procedure that BaiJia adapts for historical characters.","marker":"[5]"},{"why":"Supplies a Chinese role-playing benchmark and evaluation dimensions used as baselines.","marker":"[6]"},{"why":"Supplies role-playing benchmarking and elicitation methods, including evaluation dimensions used for comparison.","marker":"[7]"},{"why":"Supplies personality-fidelity evaluation methods that inform the CC, DA, and CA dimensions.","marker":"[8]"},{"why":"Supplies the LLaMA-Factory framework used for LoRA fine-tuning of all trainable models.","marker":"[9]"}],"fun_headline_variants":["19,281 Chinese historical figures power role-playing LLMs","BaiJia corpus: 19K historical figures boost LLM role-play","Chinese history corpus lifts LLM role-playing by up to 23.7%","BaiJia: 310K poems, 190K dialogues for historical AI personas"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the assumption that the six-dimension scores in Table 3 came from a reliable, independent judge; the paper never says who or what assigned those scores, how many responses were judged, or how the closed API models were given the corpus condition.","fun_headline_variants_meta":{"raw":{"variants":["19,281 Chinese historical figures power role-playing LLMs","BaiJia corpus: 19K historical figures boost LLM role-play","Chinese history corpus lifts LLM role-playing by up to 23.7%","BaiJia: 310K poems, 190K dialogues for historical AI personas"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2733,"prompt_tokens":862,"completion_tokens":1871,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1787}},"tokens_in":478,"tokens_out":1871,"duration_ms":14098,"temperature":1.0,"reasoning_tokens":1787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:38:28.364408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score a random sample of the baseline and with-corpus responses using human annotators who are blind to condition and who check every factual claim against the historical sources; if the with-corpus responses no longer systematically beat the baselines on all six dimensions, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the two-step scene-extraction and dialogue-generation procedure that BaiJia adapts for historical characters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a Chinese role-playing benchmark and evaluation dimensions used as baselines."},{"cited_title":"Peng, Haoran Que, et al","cited_arxiv_id":null,"evidence_quote":"Supplies role-playing benchmarking and elicitation methods, including evaluation dimensions used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies personality-fidelity evaluation methods that inform the CC, DA, and CA dimensions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLaMA-Factory framework used for LoRA fine-tuning of all trainable models."}],"review_version":1}