{"id":"266deca9-63ed-4891-bcdf-0d889f914f5e","arxiv_id":"2506.19352","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"New atomic-level metrics (ACCatom, ICatom, RCatom) reveal sentence-level personality drift in persona-assigned LLMs that whole-response scores overlook.","lead":"This paper proposes a sentence-by-sentence evaluation framework for checking whether LLMs stay in character, with three metrics that measure alignment, inner consistency, and repeatability. It may help builders of chatbots and role-play systems detect subtle personality drift that whole-response scores miss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scorer validation only confirms rank-order agreement on a selected sample; the three-level bins that ACCatom depends on are never validated against human absolute judgments, so the claimed OOC-detection advantage may rest on miscalibrated thresholds.","rationale":"The reader's weakest assumption—that GPT-4o's per-sentence scores are accurate and unbiased and that the hand-chosen bins correctly map scores to persona levels—matches my reading. I sharpen it by pointing out that the human validation is rank-order only, on pairs selected for differing LLM scores, which can inflate apparent agreement; it never tests the absolute bin classification that ACCatom is built on. This is the foundation for all three metrics, so if it fails, the central claim of 'detecting persona inconsistencies prior methods overlook' collapses. I do not propose a harder verdict because the paper is transparent about its limitations, releases code, and the metric framework is coherent; the concern is about missing evidence, not a demonstrated error. The concrete test is a single, decisive check: human absolute-label agreement on a stratified sample including boundary cases. If that test passes, the framework's core is much stronger; if it fails, the conditional acceptance should be tightened. I therefore agree with the reader's CONDITIONAL verdict and recommend no change to it.","tokens_in":35838,"tokens_out":6618,"duration_ms":78300,"concrete_test":"Run a new human evaluation in which annotators assign an absolute three-level label (low/neutral/high, or a 1–5 score) to a random, model-stratified sample of the atomic sentences actually used in the experiments, oversampling sentences whose GPT-4o scores lie within 0.3 of a bin boundary. Compute Fleiss' κ and exact-match agreement between human labels and the paper's three-bin mapping of GPT-4o scores. If κ < 0.6 or exact-match < 70%, the bin thresholds are not validated and the reported ACCatom/ICatom/RCatom values, and the claimed advantage over ACC/RC, are called into question.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that ACCatom and ICatom reveal OOC behavior that response-level ACC and RC miss—rests entirely on GPT-4o assigning each sentence a personality score in [1,5] and on mapping those scores to three persona levels via the equal-width bins [1,2.33), [2.33,3.67), [3.67,5]. The human validation (Section 4.3) tests only whether humans can rank pairs of sentences by the target trait when the pairs were deliberately chosen to have differing LLM scores; it does not test whether the absolute bin labels (low/neutral/high) agree with human judgments. A Kendall's τ of 0.67–0.76 on a selected, rank-only task cannot certify the absolute bin boundaries that define ACCatom, and it cannot rule out systematic scorer bias (e.g., toward middle scores, or toward GPT-4o's own stylistic patterns). The paper's own Limitations section acknowledges LLM-scoring bias and granularity limits, but the missing validation is precisely the link between sentence scores and the three-class fidelity judgment. If the bins are miscalibrated—for example, if the true high/neutral boundary is closer to 3.5 than 3.67, or if the scorer compresses scores toward 3—then ACCatom values, ICatom distributions, and the 'glitch' cases in Figure 2 may be artifacts of the thresholds rather than genuine OOC signals. This is the load-bearing assumption because every proposed metric is a function of these binned scores; the case studies and model rankings inherit any error.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an atomic-level evaluation framework for persona fidelity in open-ended generation. Instead of assigning a single personality score to an entire response, the framework splits responses into sentences (atomic units), scores each sentence with GPT-4o on a 1–5 personality scale, and derives three metrics: ACCatom (sentence-level alignment with the assigned persona), ICatom (internal consistency of the score distribution within a response), and RCatom (distributional consistency across repeated generations). The metrics are evaluated on 12 LLMs, 15 personality personas, and 3 generation tasks, and the paper claims that the atomic-level metrics reveal out-of-character sentences that response-level accuracy and consistency metrics miss. The authors include a human validation of the sentence-scoring step, correlation analyses between new and previous metrics, and several case studies.","tokens_in":36124,"tokens_out":5068,"duration_ms":56742,"significance":"If the central claim holds, the framework is a useful and practical addition to persona-fidelity evaluation: it is substantially finer-grained than response-level scoring, the three proposed metrics are clearly and simply defined, the experimental matrix is broad (12 models, 3 tasks, 15 personas), and the code is released. The human validation of the sentence-level rank ordering (Kendall's tau 0.67–0.76, Fleiss' kappa 0.74–0.96) is a genuine strength, as is the explicit comparison with prior response-level metrics. However, the headline contribution—that ACCatom and ICatom detect persona inconsistencies that prior methods overlook—is currently supported mainly by correlation patterns and hand-picked case studies rather than by a quantitative benchmark against human response-level judgments, and the absolute bin thresholds on which ACCatom depends are not directly validated. These gaps are fixable and should be addressed before the claim can be accepted at face value.","major_comments":[{"comment":"The human validation in Section 4.3 establishes only that GPT-4o's sentence-level scores rank-order sentences correctly on a selected sample (Kendall's tau 0.67–0.76), but ACCatom and the binning in Section 4.2 require absolute agreement at the boundaries [1, 2.33), [2.33, 3.67), [3.67, 5]. A rank-ordering of pairs deliberately chosen to have differing LLM scores does not validate these thresholds; if the scorer compresses scores toward 3 or if the true high/neutral boundary is not 3.67, then ACCatom values, the ICatom interpretation, and the 'glitch' cases in Figure 2 all inherit the error. Please add a human study that labels sentences directly as low/neutral/high, or report calibrated thresholds, so that the mapping from scores to the three persona levels is not an unvalidated free parameter.","section":"Section 4.2–4.3, Eq. (1)"},{"comment":"The central claim—that ACCatom and ICatom reveal OOC behavior that response-level ACC and RC miss—is supported by correlation tables (Table 4) and by a small number of hand-picked examples (Table 1 and Table 17), not by a quantitative comparison against human response-level judgments. Selected case studies illustrate the phenomenon but do not establish detection accuracy. Please report a benchmark in which humans mark off-persona sentences in full responses and then compare ACCatom against response-level ACC in terms of precision, recall, or agreement; this would directly test the load-bearing assertion that atomic-level evaluation spots inconsistencies that response-level scoring overlooks.","section":"Section 5.1 and Section 5.2"},{"comment":"The scoring pipeline discards sentences with no personality signal (score 9) and computes all metrics only on valid sentences. Table 6 shows valid rates as low as 35% for Davinci-002 and around 50–60% for base LLaMA and Mistral models, so for these models the metrics describe a filtered subset of the generated text. If the discarded sentences contain off-persona or inconsistent content, ACCatom and ICatom will overstate fidelity; if they are mostly neutral filler, the effective sample for OOC detection is small. Please report sensitivity analyses that treat invalid sentences either as misaligned or as neutral, and discuss how this filtering affects the model-level conclusions in Table 7.","section":"Section 4.2 and Table 6"},{"comment":"RCatom averages the Earth Mover's Distance over count histograms of sentence scores without normalizing by the number of atomic units in each generation. Because EMD between histograms with different total mass scales with the number of sentences, longer responses can yield lower RCatom for reasons unrelated to persona consistency. The length-correlation analysis in Section 6 aggregates over tasks and reports only overall coefficients; it does not rule out a length confound specifically within RCatom. Please normalize the distributions (for example, use empirical probability distributions over the score bins) or report per-task, length-stratified results for RCatom.","section":"Section 3, definition of RCatom"}],"minor_comments":[{"comment":"The notation I(s_i = s_target) should be I(s_i ∈ s_target), since s_target is defined as a score range, not a single score.","section":"Section 3, Eq. (1)"},{"comment":"The Limitations state that 'over 80% of sentences in our generation contain identifiable personality signals, as shown in Table 6', but Table 6 reports valid rates of 35–38% for Davinci-002 and roughly 50–60% for LLaMA-3-8B and Mistral-7B in several tasks; this statement should be corrected or qualified.","section":"Limitations, Table 6"},{"comment":"The shaded region in Figure 2 is described as 'highlighting instances that do not follow the correlation', but the criterion for shading is not stated and the points are not labeled by model or persona; please specify how the shaded points were selected.","section":"Figure 2"},{"comment":"The sentence 'This result coincides with their lower valid generation rates' has an ambiguous antecedent; it should be clarified that the lower valid rates belong to the base models, not the tuned models that outperform them.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The framework is a reasonable incremental contribution and the authors have done a thorough job on the evaluation matrix and on releasing code. The main risk is that the headline OOC-detection claim rests on unvalidated absolute bin thresholds and on selected case studies rather than on a human-grounded detection benchmark. If the authors add the bin-calibration study and a quantitative comparison against human response-level judgments, the paper would be much stronger. I would also encourage them to consider a scorer other than GPT-4o for at least a subset of the experiments, since GPT-4o is both the evaluator and one of the evaluated models; this is a bias risk even though it is not a circularity in the derivation of the metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about arXiv:2506.19352. First, it's a serious attempt to measure persona fidelity at sentence granularity, something previous work sidestepped by scoring whole responses. Second, the headline claim—that it catches OOC behavior prior metrics miss—rests on a validation gap that should be fixed before the claim is taken as established.\n\nThe new part is real: three metrics, ACC_atom, IC_atom, RC_atom, adapted from FActScore's atomic decomposition. Definitions are clear, and the experiments are substantial: 12 LLMs, 3 tasks, 15 personas, 30 runs each. The human validation of the sentence judge is genuine—250 pairs, six annotators, Kendall's tau 0.67–0.76—so the judge can rank sentences by trait level.\n\nNow the soft spots, in proportion. The validation only covers ranking, not the absolute bin boundaries that ACC_atom depends on. The equal-width bins [1,2.33), [2.33,3.67), [3.67,5] are arbitrary, and no sensitivity analysis is done. If the judge compresses scores or the true high/neutral boundary sits elsewhere, ACC_atom values and the 'glitch' case studies inherit the error. That's a load-bearing assumption, and the paper should test it. A second issue: unscorable sentences are filtered out, and for some models—Davinci-002, for instance—over half the sentences are dropped. That could bias the metrics, and the paper doesn't quantify the impact. The case-study evidence for the 'glitches' is suggestive, not a benchmark against human response-level judgment. There's no error bar anywhere, either.\n\nNone of this sinks the paper. The limitations section is honest about the scorer bias and segmentation choices, and the framework is reproducible—code and prompts are on GitHub. The central idea stands; the evidence for the OOC-detection advantage is just incomplete.\n\nWho's it for? Anyone building or debugging role-playing agents, social simulation, or persona-conditioned generation. It's a useful methodological reference, and I'd cite it.\n\nPeer review: yes, this deserves a serious referee. The concerns are addressable with a threshold-sensitivity analysis, a small human absolute-judgment study, and some error bars. That's revision material, not a rejection.","headline":"A genuinely fine-grained persona-fidelity framework whose OOC-detection claim outruns its validation; worth reading and revising, not rejecting.","tokens_in":36693,"tokens_out":2960,"would_cite":true,"duration_ms":29750,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that persona fidelity must be measured sentence by sentence, because whole-response scores can look fine while individual sentences drift out of character.","keywords":["persona fidelity","out-of-character behavior","atomic-level evaluation","large language models","Big Five personality","open-ended generation","consistency metrics","LLM evaluation"],"falsifier":"Have human raters score every generated sentence on the same 1-5 scale for one task and model set, then compare model rankings by ACCatom with rankings based on human sentence scores; a reversal would refute the claim that GPT-4o-based atomic metrics capture persona fidelity.","tokens_in":35620,"feed_emoji":"🎭","tokens_out":6461,"duration_ms":62108,"temperature":0.7,"pith_summary":"The paper argues that standard persona-fidelity evaluation, which assigns one overall personality score to an entire model response, can miss real out-of-character behavior: a generation can average out to the right persona while individual sentences drift toward the opposite trait. To fix this, the authors propose measuring fidelity on atomic units — here, single sentences — and aggregating them into three metrics: atomic-level accuracy (ACCatom), internal consistency within a response (ICatom), and retest consistency across repeated generations (RCatom). Using GPT-4o as a sentence-level scorer, they validate the scores against human judgments and apply the metrics to 12 LLMs across 15 Big Five personality personas and three open-ended generation tasks. Their central claim is that the atomic-level view exposes sentence-level persona glitches that response-level ACC and RC miss, and that the three metrics capture distinct, weakly correlated failure modes. If right, this offers a more precise diagnostic for when a persona-assigned model is actually out of character in the outputs users see.","feed_headline":"Atomic scoring catches LLM persona slips response scores miss","feed_subtitle":"New sentence-level metrics reveal out-of-character lines that whole-response averages hide.","key_machinery":"An atomic unit is the smallest textual segment that conveys persona-relevant information; following the FActScore paradigm, the framework splits each generation into sentences (the chosen grain size), has GPT-4o assign each sentence a score from 1 to 5 on the assigned personality dimension, filters out sentences with no personality signal, and then computes three aggregates: ACCatom is the fraction of sentences whose score falls in the target persona range ([1,2.33), [2.33,3.67), or [3.67,5]); ICatom is 1 minus the normalized standard deviation of the sentence scores within a response; and RCatom is 1 minus the normalized Earth Mover's Distance between sentence-score distributions of different generations for the same prompt. The machinery's work is to turn a single response-level judgment into a distribution over sentences, which is what lets the authors see the persona glitches that average scores hide.","core_discovery":"On the paper's own terms, the central discovery is that persona fidelity is not a property of a whole response but of the atomic units inside it. When generations are split into sentences and each sentence is scored on the target personality dimension, many responses that pass a response-level accuracy check (the mean score falls in the right range) nevertheless contain sentences that contradict the assigned persona; the proposed ACCatom quantifies this directly, while ICatom measures how much persona expression fluctuates within one response and RCatom measures how stable the atom-level score distribution is across repeated generations. The authors show that ACCatom and RCatom correlate highly with their response-level counterparts, so they are not measuring something unrelated, but they expose disagreements — e.g., high ACC with low ACCatom — that prior metrics collapse. They further report that ICatom is only weakly correlated with either accuracy or reproducibility, arguing that accuracy, internal coherence, and retest stability are separate dimensions that need separate metrics.","pith_inferences":["A natural test not run in the paper: re-segment at clause or phrase level; if intra-sentence persona contradictions are common in long outputs, the sentence-level numbers still understate out-of-character behavior.","Since the metric definitions do not reference any specific trait, the framework should transfer to value, political, or demographic personas; the observed socially-desirable-trait bias suggests those transfers will surface alignment-shaped biases too.","If the scorer were swapped for a different LLM, model rankings might shift; an ensemble of independent scorers or a small human audit on each model family would make the reported differences more robust.","The strong performance of high-level, socially desirable personas hints that RLHF does not just teach helpfulness and honesty, it also subtly canalizes personality expression, which persona designers should treat as a constraint rather than assume neutral persona assignment is a blank slate."],"forward_implications":["A model can earn a high response-level accuracy score while a large fraction of its sentences contradict the assigned persona; the paper shows this concretely for low-level personas where ACC and ACCatom diverge.","Structured tasks with explicit personality vocabulary (questionnaire interviews) yield higher ACCatom and ICatom than essays or social media posts, implying task design changes how reliably models can stay in character.","Neutral personas and socially undesirable personas are the hardest to maintain, which the authors attribute to model alignment favoring socially valued traits.","Because ICatom correlates only weakly with ACCatom and RCatom, a persona-faithful model must be checked on at least three axes: does it hit the persona, stay stable inside a response, and reproduce the same distribution across runs.","Instruction-tuned and RLHF-trained models beat their base counterparts on all atomic metrics, so fine-tuning appears to improve persona fidelity at the sentence level, not just the response level."],"supporting_citations":[{"why":"Supplies the atomic unit decomposition paradigm (FActScore) that the evaluation framework adapts to persona scoring.","marker":"Min et al., 2023"},{"why":"Defines the response-level ACC and RC baseline metrics the paper compares against and provides the scoring instruction style.","marker":"Wang et al., 2024"},{"why":"Provides the Earth Mover's Distance used to compare score distributions across repeated generations.","marker":"Rubner et al., 1998"},{"why":"Supplies the test-retest reliability conceptual basis for RCatom.","marker":"Guttman, 1945"},{"why":"Provides the Big Five personality taxonomy and the IPIP-BFFM questionnaire items used to build the personas.","marker":"Goldberg, 1992"},{"why":"Provides the essay-writing task prompts used as one of the three open-ended generation settings.","marker":"Kwantes et al., 2016"},{"why":"Establishes that personality traits are expressed in social media language, motivating the social media post task.","marker":"Park et al., 2015"},{"why":"Demonstrates that personality personas influence LLM essay writing, which the paper extends to atomic-level measurement.","marker":"Jiang et al., 2024a"}],"fun_headline_variants":["Sentence-level scores expose hidden LLM persona breaks","Atomic metrics catch persona slips response means hide","LLM persona check now zooms to sentence level","New atom-level metrics reveal LLM out-of-character lines","Persona fidelity: whole-response scores miss the slips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement chain depends on GPT-4o's per-sentence personality scores being accurate and unbiased across all model families, and on the chosen score boundaries matching the intended persona levels; if either is off, the reported ACCatom and ICatom values misstate fidelity.","fun_headline_variants_meta":{"raw":{"variants":["Sentence-level scores expose hidden LLM persona breaks","Atomic metrics catch persona slips response means hide","LLM persona check now zooms to sentence level","New atom-level metrics reveal LLM out-of-character lines","Persona fidelity: whole-response scores miss the slips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1303,"prompt_tokens":899,"completion_tokens":404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":330}},"tokens_in":515,"tokens_out":404,"duration_ms":4953,"temperature":1.0,"reasoning_tokens":330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:49.043748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human raters score every generated sentence on the same 1-5 scale for one task and model set, then compare model rankings by ACCatom with rankings based on human sentence scores; a reversal would refute the claim that GPT-4o-based atomic metrics capture persona fidelity.","supporting_citations":[],"review_version":1}