{"id":"78cf979d-3574-4fd0-ac04-b9dbd6297957","arxiv_id":"2507.16799","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A training-free, three-stage pipeline that decouples personality, memory, and linguistic style improves LLM role-playing fidelity in human evaluations.","lead":"This paper presents a training-free AI pipeline that separates a character's personality, memory, and speaking style when generating role-playing dialogue. Human evaluations suggest it produces more faithful and stylistically consistent dialogue than prompt-only or fine-tuned alternatives across six literary characters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The decoupling claim is untested: Stage 1 'styleless' responses are never shown to be style-free, and the appendix concedes base-model priors may drive quality, so the three-stage benefit may be a two-stage content-plus-rewrite effect.","rationale":"The paper is a coherent training-free pipeline with real assets: released code, an adversarial detection study showing final TTM output is harder to distinguish from canonical character quotes than a weaker base model, and human and LLM evaluations where TTM often ranks first. Those support the claim that the full system produces character-consistent dialogue. They do not support the load-bearing attribution claim that the explicit decoupling is the mechanism. The reader's weakest_assumption identifies the right soft spot: if Stage 1 is not style-free, the pipeline is effectively content generation plus style rewriting, and the claimed decoupling is a design choice rather than an empirically grounded decomposition. The paper's own appendix strengthens this worry by conceding that base-model priors improve the styleless phase; it also concedes that system-prompt handling is model-dependent and can hurt performance. The absence of any ablation that removes or reorders stages, together with the one-sample-per-method-per-character evaluation with 17 raters and no significance testing, means that the headline comparison in Table 1 cannot be attributed to the decoupling. This is a missing-evidence problem rather than an internal contradiction, so the right disposition is to keep the reader's CONDITIONAL verdict and require the ablations before accepting the decoupling claim.","tokens_in":24033,"tokens_out":5290,"duration_ms":59425,"concrete_test":"Run a same-base-model ablation on the six evaluation characters. Condition A: full TTM. Condition B: one single prompt containing the same extracted personality profile, memory snippets, and style definition with no staged rewriting. Condition C: TTM with the style-rewrite step removed, so the final output is the memory-checked response. Condition D: TTM with the memory stage removed. Generate at least 5 independent dialogue samples per condition, score with the paper's C.P., A.K., and Q.C. rubric, and report paired significance testing. If A does not beat B, or A does not beat C on C.P., the decoupling claim fails. Separately, ask independent judges to classify Stage-1 outputs as target-character-styled versus neutral; high rates of target style in Stage 1 would invalidate the 'styleless' premise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 'Method' rests on the premise that personality and memory can be separated from linguistic style, so that a 'styleless response' can be generated first and style added later. No measurement establishes that Stage 1 output is style-neutral. In fact, Stage 1 uses the character's personality, background, and RAG-retrieved chunks, and the paper concedes this retrieved content 'may still contain stylistically marked expressions'; the optional style-removal step 'may change the original meaning of the sentence' and is used 'with caution.' The appendix ('The Influences of Base Model') further concedes that pretraining knowledge of the target character 'can significantly enhance the quality of the model's responses during the styleless phase,' leading to an overall improvement. If the base model already produces character-typical wording from priors, the 'styleless response' is not style-free, and the claimed decoupling is not what the pipeline implements. No ablation removes the style-rewrite stage, removes the memory stage, or compares against a single-prompt combined-feature baseline on the same base model, so Table 1's gains over Qwen-3-32B could come from RAG, longer responses, or extra LLM calls rather than from decoupling. The central causal attribution is therefore unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Test-Time-Matching (TTM), a training-free role-playing framework that decouples a character's personality, memory, and linguistic style through a three-stage generation pipeline: first, a styleless response is generated from personality and background; second, retrieval-augmented generation (RAG) provides memory-checked details; third, progressive matching rewrites the response to match the target character's extracted linguistic style. The method is evaluated against ChatHaruhi, CoSER, GPT-4o, Gemini-2.5-pro, and Qwen-3 baselines using both LLM-as-Judge and human assessments over three Chinese and three English characters. The reported results show TTM (built on Qwen-3-32B) receiving the highest scores in most dimensions, and the paper claims that explicitly decoupling personality, memory, and linguistic style is what produces this fidelity.","tokens_in":24290,"tokens_out":3240,"duration_ms":35940,"significance":"The paper's core idea — that linguistic style can be cleanly separated from cognitive traits such as personality and memory, enabling modular and controllable role-playing — is interesting and practically relevant. The proposed framework is fully automatic, requires no fine-tuning, and the authors release code, which supports reproducibility. The three-stage pipeline is a novel organizational principle for test-time role-playing, and the inclusion of both general-participant and linguistics-expert human evaluation is a strength relative to LLM-only evaluations. However, the central causal claim that decoupling itself drives the observed gains is not currently supported: the evaluation lacks ablations isolating the stages, the style-neutrality of the first-stage response is never measured, and the human study has a small rater pool with one dialogue per condition and no significance testing. The paper's appendices explicitly concede that base-model pretraining knowledge may drive much of the styleless-phase quality and that style removal is risky, further weakening the decoupling attribution.","major_comments":[{"comment":"The central claim that the three-stage decoupling produces the performance gains is not supported because TTM is only compared as a complete pipeline against baselines. No ablation removes the style-rewrite stage, removes the memory stage, or compares against a single-prompt combined-feature baseline on the same base model. The improvements over Qwen-3-32B could come from RAG, longer responses, or additional LLM calls rather than from decoupling. Please add stage-wise ablations and report per-condition scores on the same base model.","section":"Experiments, Table 1"},{"comment":"The premise that the first-stage response is style-free is never measured. The paper concedes that RAG-retrieved content 'may still contain stylistically marked expressions,' that the optional style-removal step 'may change the original meaning of the sentence,' and that it is used 'with caution.' The appendix section 'The Influences of Base Model' further states that pretraining knowledge of the target character 'can significantly enhance the quality of the model's responses during the styleless phase.' Without a quantitative or human-judged measure of style neutrality in the first-stage output, the pipeline may be better described as content generation followed by style transfer, and the claimed benefit of decoupling is not established.","section":"Method, 'Styleless Response and Memory-checked Response Generation'"},{"comment":"The human evaluation comprises 17 raters and one dialogue per character per method, with no standard deviations, confidence intervals, or significance tests reported. The abstract and the Experiments section use the word 'significantly,' but the data as presented do not support statistical significance. Additionally, for Quality of Conversation among linguistics experts, TTM ranks below Gemini-2.5-pro and Qwen-3-235B-A22B, a result the paper attributes to response length; this should be analyzed rather than only described. Please provide per-item scores, inter-rater agreement, and appropriate statistical tests.","section":"Experiments, 'Human Assessment'"},{"comment":"The adversarial detection study is a useful idea, but the reported comparison (TTM identified 15.48% of the time vs. Qwen-2.5-32B-Instruct identified 24.80% of the time) is presented without any measure of uncertainty or a significance test. Because the base model in this study is Qwen-2.5-32B-Instruct rather than the Qwen-3-32B used in the main experiments, this result cannot directly support the main claim about the primary system. Clarify whether the observed difference is statistically significant and discuss how it transfers to the main base model.","section":"Experiments, 'Linguistic Style'"}],"minor_comments":[{"comment":"The label 'Styless Response' in Figure 3 appears to be a typo; it should read 'Styleless Response' consistently with the text and Figure 1.","section":"Figure 3"},{"comment":"The table footnote notes that Gemini-2.5-pro employs online grounding, which is a confound in a method comparison. Please state whether any other method had internet or non-text access, and discuss the implications for the comparison.","section":"Table 1"},{"comment":"The first sentence of the 'More Discussions' subsection, 'Though TTM reaches the best performance when againsting with other methods,' contains a typo and overstates the expert Q.C. result; please revise the wording and qualify the claim.","section":"Appendix, 'More Discussions'"},{"comment":"In the Dumbledore sample, the user turn contains 'Professer Dumbledore'; if this is an intentional transcription of the user's speech, mark it as such, otherwise correct the spelling.","section":"Sample dialogues in Appendix"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the authors have been transparent about limitations, which is appreciated. The recommendation of major revision is driven solely by the missing ablation and measurement evidence for the central decoupling claim, not by any concern about novelty or citation practices. The code release and the human evaluation efforts are real strengths; the authors should be encouraged to add stage-wise ablations and statistical reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: TTM is a training-free role-playing framework that splits generation into three stages — styleless content, memory-checking via RAG, then style rewrite. That specific combination is new relative to ChatHaruhi, CoSER, StyleChat, and CAT-LLM, and the paper ships code plus a fairly detailed pipeline description. The strong version of the claim, that decoupling is what causes the fidelity gain, is not supported by the experiments.\n\nWhat the paper does well: the three-stage architecture is coherent and modular; extracting personality, background, and style into explicit plain-text profiles is genuinely useful for controllability; progressive matching and hybrid retrieval are reasonable design choices. The authors are also transparent — the Limitations and Appendix sections openly concede that the base model's pretraining knowledge can drive quality, that the system prompt may hurt some models, that retrieval can cause information overload and knowledge misuse, and that the style-removal step can change meaning. That honesty is to their credit.\n\nThe soft spots are real and central. The decoupling claim is never isolated. There are no ablations that remove the style stage, remove memory, or compare against a single-prompt baseline on the same base model. The 'styleless response' is asserted but never shown to be style-free; retrieved chunks can carry style, and the optional style-removal step is used 'with caution' because it may alter meaning. So the measured gain over Qwen-3-32B could come from RAG, longer responses, or extra LLM calls rather than from decoupling. The human evaluation is also thin: 17 raters, one dialogue per character per method, no significance tests. The LLM-as-judge result aligns with the human result on TTM's favor, which is something, but the judge is the same kind of model that does the extraction and rewriting, so there is a mild self-referential flavor. The style identification study is a useful complement, though it uses a weaker base model and the difference is not dramatic.\n\nWho is this for: researchers doing test-time adaptation for role-playing or modular style control. They will get value from the pipeline description, but should treat the decoupling claim as a hypothesis, not a demonstrated result. The paper deserves a serious referee because the idea is novel and the missing experiments are concrete and addressable.\n\nRecommendation: send to peer review, but require ablations that isolate each stage, a single-prompt baseline on the same base model, and at least error bars or significance testing. With those, this could become a solid contribution.","headline":"The decoupling claim is the paper's load-bearing wall and it is not tested; the pipeline itself is a solid training-free contribution that deserves review, but the causal story needs ablations before it can be believed.","tokens_in":24873,"tokens_out":1980,"would_cite":false,"duration_ms":21843,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Test-Time-Matching claims that role-playing fidelity improves when a character's personality, memory, and linguistic style are decoupled and generated in three separate stages.","keywords":["role-playing language agents","test-time scaling","context engineering","linguistic style transfer","personality decoupling","memory retrieval","style consistency","LLM evaluation"],"falsifier":"If an ablation that generates the final response in one combined prompt, or that applies style transfer before memory-checking, achieved equal or better ratings on persona consistency and knowledge accuracy across the same six characters, then the decoupling claim would fail; a more direct test would be to have expert readers identify the target character from the first-stage 'styleless' responses alone—if they can do so reliably, the styleless stage is not style-free.","tokens_in":23797,"feed_emoji":"🎭","tokens_out":3919,"duration_ms":37645,"temperature":0.7,"pith_summary":"The paper proposes Test-Time-Matching (TTM), a training-free framework for role-playing language agents that deliberately separates three components of a character—personality, memory, and linguistic style—and generates dialogue in three stages: a styleless response driven by personality and background, a memory-checked revision using retrieval-augmented generation, and a final style transfer via progressive matching. The claim is that this decoupling, rather than blending all features in one prompt, is what produces higher-fidelity role-playing. Human raters and an LLM judge scored TTM above prompt-only and fine-tuned baselines on persona consistency, knowledge accuracy, and conversation quality. If right, it means style transfer should be a separate final step after content and memory are settled, and characters can be recombined modularly without retraining.","feed_headline":"Three-stage pipeline beats fine-tuning for role-playing agents","feed_subtitle":"TTM separates what a character says from how they say it, earning top ratings from humans and LLM judges.","key_machinery":"The load-bearing mechanism is the three-stage pipeline with explicit decoupling: first, a personality and background prompt generates a styleless response; second, that response is rewritten into retrieval keywords for a graph-based retrieval-augmented database, producing a memory-checked response; third, linguistic style, represented as style preferences plus common word categories and retrieved similar utterances, is applied through progressive matching, rewriting sentence by sentence with hybrid retrieval from the character's historical dialogue. The named procedure, Test-Time-Matching, is what carries the argument: by keeping style out of the early stages, the paper claims cognitive content and factual grounding are settled before verbal mannerisms are added.","core_discovery":"On its own terms, the paper's central discovery is that an LLM can play a character more faithfully when the character is decomposed into cognitive tendency (personality plus memory) and linguistic style, generated separately, and then recombined. TTM first builds a structured profile—personality traits, background facts, a graph-based memory database, and style features such as preferred word classes and tone—automatically from raw text. At generation time it produces a styleless response, retrieves knowledge to correct and enrich it, and only then applies the character's style using retrieved similar utterances and sentence-by-sentence progressive matching. In the paper's evaluations TTM on Qwen-3-32B scores highest among all methods on persona consistency, accuracy of knowledge, and conversation quality in both LLM-as-judge and general-participant ratings.","pith_inferences":["If the decoupling is robust, role-playing becomes modular: the same personality and memory profile could be paired with any style profile, letting users audition or swap mannerisms without touching the character's knowledge or disposition; the paper demonstrates combinations but does not stress-test this trade-off.","The main confound in the evaluation is the base model's pretraining familiarity with famous characters, so a natural extension is to test TTM on obscure or synthetic characters to isolate what the pipeline adds beyond memorized persona.","Because TTM is training-free and spends extra test-time compute, it suggests a general recipe—generate content, ground memory, then restyle—that could transfer to other generation tasks where tone should be decoupled from content, such as summarization or translation."],"forward_implications":["A training-free system can surpass fine-tuned role-players by spending additional compute and context engineering at inference time.","Personality, memory, and style can be swapped independently, allowing controlled customization and recombination across characters.","Applying style as a separate final step improves style consistency: TTM sentences were identified as the target character's with probability 15.48%, versus 24.80% for the base model.","The three-stage pipeline is modular, so memory-checking or style generation can be removed or reordered without retraining.","Test-time methods inherit base model knowledge, so overall performance depends on model capability and pretraining familiarity with the character."],"supporting_citations":[{"why":"ChatHaruhi is the prompt-based RPLA baseline whose character library and comparison setup TTM builds on and is evaluated against.","marker":"Li et al. 2023a"},{"why":"CoSER is the fine-tuning-based baseline that TTM must beat to show training-free decoupling is competitive.","marker":"Wang et al. 2025"},{"why":"CAT-LLM supplies the text style definition approach that TTM adapts for representing linguistic preferences and common words.","marker":"Tao et al. 2024"},{"why":"Conversation style transfer through style removal and few-shot learning is the basis for TTM's optional style removal step.","marker":"Roy et al. 2023"},{"why":"Query rewriting for retrieval-augmented generation is the technique TTM uses to turn the styleless response into retrieval keywords.","marker":"Ma et al. 2023"},{"why":"Graph-based RAG is used as one of the frameworks for constructing the detailed memory database.","marker":"Zhou et al. 2025"},{"why":"LightRAG is used alongside graph-based RAG to build the memory database for long-term character memory.","marker":"Guo et al. 2024"},{"why":"MMRole supplies the LLM-as-Judge prompt that TTM adapts with minor changes for automated evaluation.","marker":"Dai et al. 2025"}],"fun_headline_variants":["Decouple persona and style for better role-play","TTM: test-time matching boosts role-play fidelity","Training-free role-play: separate then recombine","Beyond fine-tuning: split and style-match at test time","Personality and style decoupled, then matched"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's load-bearing premise is that personality and memory can be cleanly separated from linguistic style, so a genuinely styleless response can be generated first and style added later; the paper itself notes that removing style may change the original meaning of a sentence.","fun_headline_variants_meta":{"raw":{"variants":["Decouple persona and style for better role-play","TTM: test-time matching boosts role-play fidelity","Training-free role-play: separate then recombine","Beyond fine-tuning: split and style-match at test time","Personality and style decoupled, then matched"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000339,"raw_usage":{"total_tokens":1851,"prompt_tokens":901,"completion_tokens":950,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":875}},"tokens_in":517,"tokens_out":950,"duration_ms":10201,"temperature":1.0,"reasoning_tokens":875,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:01:17.133147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If an ablation that generates the final response in one combined prompt, or that applies style transfer before memory-checking, achieved equal or better ratings on persona consistency and knowledge accuracy across the same six characters, then the decoupling claim would fail; a more direct test would be to have expert readers identify the target character from the first-stage 'styleless' responses alone—if they can do so reliably, the styleless stage is not style-free.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CoSER is the fine-tuning-based baseline that TTM must beat to show training-free decoupling is competitive."},{"cited_title":"CAT-LLM: Style-enhanced Large Language Models with Text Style Definition for Chinese Article-style Transfer","cited_arxiv_id":"2401.05707","evidence_quote":"CAT-LLM supplies the text style definition approach that TTM adapts for representing linguistic preferences and common words."},{"cited_title":"Conversation Style Transfer using Few-Shot Learning","cited_arxiv_id":"2302.08362","evidence_quote":"Conversation style transfer through style removal and few-shot learning is the basis for TTM's optional style removal step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MMRole supplies the LLM-as-Judge prompt that TTM adapts with minor changes for automated evaluation."}],"review_version":1}