{"id":"f0ea5b1d-79ea-462b-8037-7305924f2056","arxiv_id":"2505.07705","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Representing role-play personas as executable if-then-else logic with semantic condition checks improves consistency and lets 1B-parameter models approach the role-play quality of 8B text-prompted models.","lead":"The paper compiles character profiles into executable Python functions that decide a character's next action from scene conditions, then tests these codified profiles on a new Fandom-derived benchmark of 83 characters and 5,141 scenes. A generalist should read it because it proposes a concrete way to make role-playing AI more consistent, updatable, and controllable, especially when running on small local models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Fandom Benchmark's guiding questions are generated with access to the reference and never audited; if they leak, the reported codified advantage may reflect answer-copying rather than character logic.","rationale":"I agree with the reader that the benchmark's construct validity is the weakest load-bearing premise. The method has independent support: the paper releases code, reports a manual codification audit (Table 5), and a human preference study favoring codified responses, which makes a straightforward rejection inappropriate. But the main quantitative claim rests on 5,141 LLM-generated questions and gpt-4.1 as both codifier and judge, and none of the controls provided rule out leakage or style bias. A leakage audit and a generic-question rerun would settle this. This does not move the verdict: CONDITIONAL remains appropriate.","tokens_in":28205,"tokens_out":12996,"duration_ms":131835,"concrete_test":"Publish the full set of 5,141 guiding questions and have two annotators blind to the reference label whether each question contains the reference action or a near-paraphrase; then rerun Table 3 on a random 200-scene sample using a generic question ('What does {character} do next?') and a judge from a different model family or human annotators. If leakage is frequent, or if the codified-vs-textual gap shrinks to noise under generic questions, the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Table 3 comparison (68.38 vs 65.98 for main characters) is valid only if the Fandom Benchmark's guiding questions constrain the response without revealing the reference action. Section 4 asserts this property, and Figure 3 says questions are 'generated to steer predictions to be relevant to the reference,' but the paper reports no leakage audit, no release of the 5,141 questions, and no control condition using generic questions. The example in Figure 17 shows a question that restates substantial scene content ('his decision to arrest Janos Slynt and appoint Bronn...'), and it is unknown how many of the 5,141 questions similarly echo the reference or its key entities. If leakage is common, both conditions receive the answer in the prompt, and the 2.4-point advantage could come from codified profiles making it easier for llama-3.1-8b to copy the leaked answer, not from better character logic. A related uncontrolled channel is scoring style: gpt-4.1 both writes the codified profiles and judges NLI, so any preference for concise, triggered-statement-grounded responses inflates the same comparison; the human preference check covers only 5 scenes per character and is reported as aggregate bars without full protocols.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Codified Profiles, a representation of role-play character logic as executable parse_by_scene functions that emit triggered statements conditioned on scene checks, and claims this improves persistence, updatability, and controllable randomness relative to textual profiles. The authors build an 83-character, 5,141-scene Fandom benchmark with NLI-based scoring, compare codified profiles against textual profiles and Codified RAG under llama-3.1-8b, test profile evolution along storylines, and show that smaller models benefit from codification. They also analyze segmentation and codification fidelity, and include a distilled condition checker to lower deployment cost.","tokens_in":28369,"tokens_out":7999,"duration_ms":74675,"significance":"If the benchmark is valid, the paper offers a practical and falsifiable mechanism for making role-play behavior more deterministic and updatable, and it demonstrates that reasoning can be partially offloaded to smaller models. The authors provide a public code/dataset link, a human validation of the NLI metric on 415 cases, and a manual audit of codification quality, which are genuine strengths. The leakage and statistical-significance concerns below are the main obstacles to accepting the headline claims.","major_comments":[{"comment":"The benchmark construction in §4 generates each guiding question with access to the reference action, and the paper provides no audit for leakage. The text states that questions are 'designed to constrain the response without providing clues toward the correct answer', but the caption of Figure 3 says the questions are 'generated to steer predictions to be relevant to the reference', and the Tyrion example in Figure 17 asks 'How does Tyrion respond to Cersei's confrontation about his decision to arrest Janos Slynt and appoint Bronn as Commander of the City Watch?', which restates the core of the reference. If even a fraction of the 5,141 questions behave this way, the NLI scores in Table 3 can be inflated by answer-copying rather than by character logic, and the comparison between codified and textual profiles becomes uninterpretable. Please release the questions, have humans rate leakage on a random sample, and add a control condition with generic or entity-removed questions.","section":"§4, Fandom Benchmark; Fig. 17"},{"comment":"Table 3 reports only artifact-level means, without confidence intervals, per-character standard deviations, or significance tests. The headline difference on main characters is 68.38 vs 65.98 (2.40 points) and on minor characters 69.87 vs 65.70 (4.17 points); given that scene counts per character range from about 48 to 214 and that Table 6 already shows large per-character variation (e.g., Sansa 81.25% vs Bran 38.89% win rates), these averages may not be statistically robust. Report bootstrap CIs or per-character paired tests, and show the full per-character distribution for the central codified-vs-textual comparison.","section":"§5.3, Table 3"},{"comment":"The claim that '1B + Code + Distill' approaches the 8B + Text baseline is not supported by the reported numbers. The distilled condition checker achieves only 70.53% agreement with gpt-4.1 on the held-out 10% of 415 scenes (about 42 examples), and the main-character NLI improves from 58.43 (1B+Code) to 60.21 (1B+Code+Distill), still 5.77 points below 8B+Text (65.98). The margin is within the likely noise given the small held-out set and the absence of error bars in Table 4. Please report the checker's per-class agreement, confidence intervals for Table 4, and either a larger distillation set or a threshold analysis for when the distilled checker becomes beneficial.","section":"§5.4, Table 4"},{"comment":"The manual codification audit in Table 5 shows that recall falls to 81.3% for if-depth 4 and 'Both' falls to 75.0%, based on only 16 cases. Because the paper's persistence claim depends on codified profiles executing 'complete' character logic, a one-in-five miss rate on the most complex segments is not a minor caveat; it may directly cap the measured advantage on logic-heavy characters. Please report error categories, evaluate Table 3 with imperfectly codified segments removed or flagged, and state whether the main results are robust when the 16 depth-4 cases are excluded.","section":"§6.2, Table 5"},{"comment":"The same model (gpt-4.1) both performs codification and serves as the NLI judge, and the human NLI validation in Appendix C is a reliability check of the judge in general, not a check against judge bias between the codified and textual conditions. Since the codified responses are generated under conditions that gpt-4.1 itself wrote, the 2.4-point gap could partly reflect stylistic compatibility with the judge. A judge-swap experiment with a different LLM scorer, or a blinded human re-scoring on a random subsample of Table 3, would establish that the advantage is not an artifact of judge preference.","section":"§5.1, §5.3, Appendix C"}],"minor_comments":[{"comment":"Please correct the typos 'Codifed Profiles' and 'scholastic response' to 'Codified Profiles' and 'stochastic response'.","section":"§3.3, Figure 12 caption"},{"comment":"Report the number of human judges, their inter-annotator agreement, and the exact tie-handling protocol; the current aggregate bars lack error bars and the reader cannot tell whether the 26–38 win/tie/lose counts are per-character or per-scene.","section":"Figure 5 and surrounding text"},{"comment":"Specify the exact model variants (llama-3.2-1b, llama-3.2-3b, llama-3.1-8b) in the axis labels and define the 'forward pass count' used in Figure 4, since the efficiency comparison depends on this unit.","section":"Figure 8 and Figure 4"},{"comment":"The Jotaro/Hol Horse case study is a single hand-built scene; state explicitly that it is illustrative and should not be interpreted as quantitative evidence, and consider moving it to the appendix.","section":"§6.3"},{"comment":"The contribution list says the dataset 'will be open-sourced' while the abstract describes it as introduced; clarify the current release status and provide the exact license and access instructions.","section":"Abstract and Contributions"}],"recommendation":"major_revision","confidential_remarks":"The core idea is attractive and the experiments are extensive, but the benchmark's guiding-question construction is the linchpin. If the authors can release the 5,141 questions and demonstrate no leakage (e.g., via human ratings or a generic-question control), the contribution is likely to be solid; without that audit, the central comparison in Table 3 is not interpretable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real new idea—encoding character logic as executable parse_by_scene functions with semantic check_condition calls—and the empirical case is plausible, but it rests on a benchmark whose guiding questions may leak the reference action. I would send it to review, but I'd want the leakage question settled first.\n\nWhat is new: the paper does not just prompt or retrieve profiles; it compiles them into control flow. That changes the interface between profile and role-player, and it makes profile updating and randomness control concrete. The small-model result (1B + code + distilled checker approaching 8B + text) is practically interesting, and the randomness case studies show that temperature alone does not give controllable probabilities. The manual codification evaluation, with precision and recall broken down by if-depth, is the right kind of sanity check and shows the method is mostly faithful.\n\nSoft spots. First, the Fandom benchmark's guiding questions are generated with access to the reference action and never audited. Section 4 says they are designed to steer predictions to be relevant to the reference; that is exactly the channel where leakage could enter. If a question echoes the reference or its key entities, both arms receive the answer in the prompt, and the 68.38 vs. 65.98 gain could reflect how easily a model copies the leaked answer under each profile format rather than better character logic. The paper needs a leakage audit, a generic-question control, and release of the 5,141 questions. Second, the headline numbers are point averages with no confidence intervals or significance tests. Several comparisons are in the 2–4 point range, so variance matters. Third, gpt-4.1 both codifies the profiles and judges NLI; that is a real confound, and the human preference check covers only 5 scenes per character and is reported as aggregate bars. The codification recall drop to 81.3% at depth 4 is disclosed, and it means the measured benefit is probably a lower bound rather than an upper bound.\n\nI do not think any of these sink the paper. The direction is sound, the baselines are sensible, and the authors are honest about scope. But the central quantitative claim needs the leakage audit and significance reporting before I would trust the 2.4-point average.\n\nWho it is for: anyone building persona-consistent agents or local role-play systems. It deserves a serious referee. I would ask for a revised version that adds the leakage control, reports confidence intervals, and releases the benchmark questions and codified code with a commit hash.","headline":"Codified Profiles is a genuinely new idea for role-play—profiles as executable parse_by_scene functions—with plausible but not yet airtight evidence; the main open question is whether the Fandom benchmark's guiding questions leak the answer.","tokens_in":28942,"tokens_out":2683,"would_cite":true,"duration_ms":28578,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Precompiling character profiles into executable if-then logic outperforms free-text prompts for role-playing consistency, updatability, and controlled randomness.","keywords":["codified profiles","role-playing","character logic","executable functions","condition checking","natural language inference","controllable randomness","small language models"],"falsifier":"Have human annotators audit a random sample of benchmark scenes to check whether the guiding question can be answered from the ground-truth sentence alone, or whether scene truncation removes context the character logic depends on; if a substantial share leak the reference or lose decisive context, the NLI gap between codified and textual profiles is an artifact of benchmark construction.","tokens_in":27923,"feed_emoji":"🎭","tokens_out":6724,"duration_ms":55288,"temperature":0.7,"pith_summary":"The paper claims that a character's behavioral logic can be written down as executable functions—parse_by_scene(scene) returning triggered_statements, with semantic checks like check_condition(scene, question)—and that this codified form makes role-playing more consistent, more updateable, and more controllable than appending free-text profiles to prompts. On a new 83-character, 5,141-scene benchmark built from fan wiki narrative summaries, the authors report average NLI scores of 68.38 for codified profiles versus 65.98 for original textual profiles on main characters, and larger gaps on minor characters. They also report that evolving the codified profile episode-by-episode yields bigger gains than evolving a textual profile, and that explicit random.choice logic produces precise stochastic behavior that temperature sampling cannot. The practical upshot, if the claim holds, is that a 1B-parameter model with a codified profile and a distilled condition checker approaches the role-playing quality of an 8B model prompted with text, which matters for local and low-resource deployment.","feed_headline":"Character logic as code beats text prompts in role-play","feed_subtitle":"Precompiled profiles score 68.38 vs 65.98 on main characters and let 1B models rival 8B text-prompted ones.","key_machinery":"The load-bearing object is the codified profile: a set of executable functions parse_by_scene(scene) that return triggered_statements, a list of character-consistent assertions, by running explicit if-then-else control flow over semantic condition checks. The helper check_condition(scene, question) queries the role-playing LLM with a natural-language question and reads its logits over \"yes\", \"no\", and \"unknown\" to return True, False, or None, so the code can test scene properties that symbolic matching cannot. This machinery offloads most of the character-reasoning burden from open-ended generation to localized classification and control flow, which is what lets smaller models perform competitively and what makes behavioral rules traceable and patchable.","core_discovery":"The central discovery is that role-playing consistency can be manufactured rather than inferred: convert each paragraph of a character profile into a Python function parse_by_scene(scene) whose control flow decides which triggered_statements apply to the current scene. Semantic calls to check_condition(scene, question) let the function ask an LLM yes/no/unknown questions such as \"Is the character in danger?\" and branch on the answer. Because the logic is explicit, it can be inspected, revised along the storyline, and given precise randomness with constructs such as random.choice([...]) and random.random() < p. The paper's evidence is a set of comparisons on its Fandom Benchmark, where codified profiles outperform original textual profiles and a retrieval-style Codified RAG baseline in NLI scores and in human/LLM preference, and where a 1B model using a codified profile plus a distilled 0.1B condition checker reaches 60.21 average NLI on main characters versus 65.98 for an 8B model with a textual profile.","pith_inferences":["If codification works by decomposing long profile text into localized yes/no decisions, the same pattern might improve other instruction-following tasks that require persistent rules, such as tool-use policies or safety guardrails, by making each decision auditable.","The character-level win-rate pattern suggests a testable routing rule: codify profiles for logic-heavy, strategic characters and keep textual prompting for emotionally expressive characters; an automatic classifier that picks the format per character might beat either method alone.","The distilled condition checker result implies that much of role-playing can be reduced to binary scene classification; a natural extension would be to train the condition checker on a larger corpus and measure whether small models then close the residual gap to frontier models entirely.","Nothing in the mechanism is fiction-specific; one could codify a customer-service persona or an AI agent's decision policy the same way, with the same promises of persistence and updatability."],"forward_implications":["Character profiles become inspectable artifacts: a behavioral contradiction in a role-play output can be traced to a specific code block and fixed by editing that block, rather than by rewriting prose and hoping the model complies.","Precompiling logic into conditions cuts inference cost, because the role-playing LLM no longer has to reason over the full profile text at every response; the paper measures this efficiency gain in reduced forward passes.","Small models become viable role-play engines: a 1B model with a codified profile and distilled condition checker approaches an 8B text-prompted model, and beats it on some minor-character subsets.","Storyline evolution can be handled as a code update loop, and the paper shows codified evolving profiles (average 69.28 NLI) outperform evolving textual profiles (65.60) across eight Game-of-Thrones protagonists.","Precise stochastic personality traits, such as equal-probability emotion, low-probability humor, and fair rock-paper-scissors choices, can be encoded with explicit probability statements rather than left to temperature, and Best@K coverage improves as a result."],"supporting_citations":[{"why":"Supplies the NLI entailment scoring that turns predicted versus reference actions into 100/50/0 scores.","marker":"Bowman et al., 2015"},{"why":"Establishes profile-based role-playing agents and persona evaluation that the codified-profile method builds on and compares against.","marker":"Shao et al., 2023"},{"why":"Provides the faithfulness/NLI evaluation approach and grounding concerns that motivate offloading reasoning into structured logic.","marker":"Peng & Shang, 2024"},{"why":"Dense passage retrieval is used as the basis for the Codified RAG baseline that retrieves relevant profile segments.","marker":"Karpukhin et al., 2020"},{"why":"Defines the retrieval-augmented generation paradigm behind the Codified RAG baseline.","marker":"Lewis et al., 2020"},{"why":"Chain-of-thought prompting is the reasoning baseline whose efficiency trade-off is compared against codified reasoning.","marker":"Wei et al., 2022"},{"why":"Provides the model family used as the role-playing LLM across 1B, 3B, and 8B scales.","marker":"Touvron et al., 2023"},{"why":"DeBERTa-v3 is the architecture for the distilled 0.1B condition checker.","marker":"He et al., 2021"},{"why":"Documents LLMs' biased probability sampling, which motivates code-level controllable randomness.","marker":"Gu et al., 2025"}],"fun_headline_variants":["Executable character logic beats text prompts in role-play","Precompiled role-play logic lets 1B models rival 8B","Codified character logic: persistent, updatable, random","1B model matches 8B role-play with codified logic","Precompiled profiles: better role-play, smaller models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Fandom Benchmark must faithfully measure character logic: the LLM-extracted scenes, ground-truth actions, and guiding questions have to be accurate and free of leakage, because every comparison is scored against that reference.","fun_headline_variants_meta":{"raw":{"variants":["Executable character logic beats text prompts in role-play","Precompiled role-play logic lets 1B models rival 8B","Codified character logic: persistent, updatable, random","1B model matches 8B role-play with codified logic","Precompiled profiles: better role-play, smaller models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000913,"raw_usage":{"total_tokens":3978,"prompt_tokens":1061,"completion_tokens":2917,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":2831}},"tokens_in":677,"tokens_out":2917,"duration_ms":19291,"temperature":1.0,"reasoning_tokens":2831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:09:49.395948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators audit a random sample of benchmark scenes to check whether the guiding question can be answered from the ground-truth sentence alone, or whether scene truncation removes context the character logic depends on; if a substantial share leak the reference or lose decisive context, the NLI gap between codified and textual profiles is an artifact of benchmark construction.","supporting_citations":[{"cited_title":"and Shang, J","cited_arxiv_id":null,"evidence_quote":"Provides the faithfulness/NLI evaluation approach and grounding concerns that motivate offloading reasoning into structured logic."},{"cited_title":"u ttler, H., Lewis, M., Yih, W., Rockt \\","cited_arxiv_id":null,"evidence_quote":"Defines the retrieval-augmented generation paradigm behind the Codified RAG baseline."},{"cited_title":"H., Le, Q","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting is the reasoning baseline whose efficiency trade-off is compared against codified reasoning."}],"review_version":1}