pith. sign in

arxiv: 2603.25620 · v4 · pith:VVBG2CSPnew · submitted 2026-03-26 · 💻 cs.CL

PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency

Minseo Kim , Sujeong Im , Junseong Choi , Junhee Lee , Chaeeun Shim , Hwajung Hong , Edward Choi This is my paper

Pith reviewed 2026-05-21 10:41 UTC · model grok-4.3

classification 💻 cs.CL
keywords persona agentsconsistency evaluationmulti-turn questioningLLM evaluationinterrogation frameworkhuman baseline
0
0 comments X

The pith

Persona agents often contradict themselves and evade questions when subjected to chained multi-turn interrogation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces PICon as a way to test whether LLM-based persona agents keep their responses consistent over extended interactions. It borrows the interrogation idea that repeated, logical follow-up questions can uncover fabrications and applies it to three consistency checks: no internal contradictions, alignment with real facts, and stability when questions are repeated. Experiments with several persona agent groups and 63 human participants show that even agents previously described as highly consistent fall short of the human level and produce contradictions or evasive replies. This matters because persona agents are now used as stand-ins for people in research, training, and services. The framework supplies both a method and evidence that current systems need improvement before they can reliably replace humans.

Core claim

PICon probes persona agents through logically chained multi-turn questioning and evaluates consistency along three dimensions: internal consistency (freedom from self-contradiction), external consistency (alignment with real-world facts), and retest consistency (stability under repetition). When applied to seven groups of persona agents and 63 real humans, even systems previously reported as highly consistent fail to meet the human baseline, revealing contradictions and evasive responses under chained questioning.

What carries the argument

The PICon framework, which uses systematic multi-turn chained questioning drawn from interrogation principles to expose inconsistencies across internal, external, and retest dimensions.

If this is right

  • Persona agents cannot yet serve as reliable substitutes for human participants without first passing multi-turn consistency tests.
  • Single-turn evaluations miss many inconsistencies that only appear under chained questioning.
  • Developers can apply the three-dimensional checks to diagnose and fix specific failure modes in their agents.
  • Human performance on the same interrogation tasks sets a concrete target for improving LLM consistency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same chained-questioning approach could be adapted to test consistency in other LLM applications such as long-form assistants or role-playing chatbots.
  • Training procedures might be modified to reward maintenance of coherence across extended conversation histories rather than single responses.
  • This method opens a route for comparing consistency across different model sizes or fine-tuning strategies in future experiments.

Load-bearing premise

That the human interrogation principle of exposing fabrications through systematic follow-up questions transfers directly to LLM persona agents without needing AI-specific adjustments for hallucination or training artifacts.

What would settle it

A persona agent that sustains full consistency, with no contradictions, factual deviations, or evasive replies, across dozens of logically chained follow-up questions on the same topic would challenge the reported gap with human performance.

read the original abstract

Large language model (LLM)-based persona agents are rapidly being adopted as scalable proxies for human participants across diverse domains. Yet there is no systematic method for verifying whether a persona agent's responses remain free of contradictions and factual inaccuracies throughout an interaction. A principle from interrogation methodology offers a lens: no matter how elaborate a fabricated identity, systematic interrogation will expose its contradictions. We apply this principle to propose PICon, an evaluation framework that probes persona agents through logically chained multi-turn questioning. PICon evaluates consistency along three core dimensions: internal consistency (freedom from self-contradiction), external consistency (alignment with real-world facts), and retest consistency (stability under repetition). Evaluating seven groups of persona agents alongside 63 real human participants, we find that even systems previously reported as highly consistent fail to meet the human baseline across all three dimensions, revealing contradictions and evasive responses under chained questioning. This work provides both a conceptual foundation and a practical methodology for evaluating persona agents before trusting them as substitutes for human participants. We provide the source code and an interactive demo at: https://kaist-edlab.github.io/picon/

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes PICon, a multi-turn interrogation framework for evaluating consistency in LLM-based persona agents. Drawing on interrogation principles, it probes agents via logically chained questions across three dimensions: internal consistency (freedom from self-contradiction), external consistency (alignment with real-world facts), and retest consistency (stability under repetition). The central empirical claim is that seven groups of persona agents, including those previously reported as highly consistent, fail to meet the performance of 63 human participants on all three dimensions and exhibit contradictions and evasive responses.

Significance. If the comparative result holds under equivalent conditions, the work is significant for the growing use of persona agents as scalable human proxies in research. It supplies a practical, principle-based methodology with falsifiable checks rather than ad-hoc consistency metrics, and the release of source code plus an interactive demo directly supports reproducibility and adoption.

major comments (2)
  1. [Human Evaluation / Participant Instructions] Human participant protocol: The manuscript does not state whether the 63 human participants received instructions to maintain a specific assigned persona profile across the chained interrogation questions (as required of the agent groups). If humans were simply asked to answer as themselves, the baseline measures ordinary human response stability rather than the harder task of sustaining a coherent fabricated identity; this mismatch would make the headline gap between agents and humans non-diagnostic of the intended claim.
  2. [Evaluation Methodology] Question generation and controls: The abstract and evaluation sections supply no details on how the logically chained questions were generated, what statistical controls were applied, or the exact construction of the human baseline. These omissions leave the support for the claim that agents 'fail to meet the human baseline across all three dimensions' difficult to verify.
minor comments (2)
  1. [Experimental Setup] Clarify the composition of the 'seven groups of persona agents' (specific models, prompting strategies, or fine-tuning) in the main text or a table for reproducibility.
  2. [Code and Demo] The provided code link and demo are a strength; ensure the released materials include the exact question templates and scoring rubrics used for the three consistency dimensions.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. The comments highlight important areas for improving methodological transparency, which we will address through targeted revisions. Below we respond point-by-point to the major comments.

read point-by-point responses

  1. Referee: [Human Evaluation / Participant Instructions] Human participant protocol: The manuscript does not state whether the 63 human participants received instructions to maintain a specific assigned persona profile across the chained interrogation questions (as required of the agent groups). If humans were simply asked to answer as themselves, the baseline measures ordinary human response stability rather than the harder task of sustaining a coherent fabricated identity; this mismatch would make the headline gap between agents and humans non-diagnostic of the intended claim.

    Authors: We agree that the manuscript lacks an explicit description of the instructions given to the 63 human participants, which creates ambiguity about the nature of the baseline. We will revise the Human Evaluation section to provide the full participant instructions and protocol. The humans were asked to respond naturally as themselves to establish a baseline of ordinary human consistency under interrogation; we will add explicit discussion of this design choice and its implications for interpreting the gap with persona agents, including whether the comparison tests fabricated-identity maintenance or natural response stability. This will allow readers to assess the diagnostic strength of the results. revision: yes

  2. Referee: [Evaluation Methodology] Question generation and controls: The abstract and evaluation sections supply no details on how the logically chained questions were generated, what statistical controls were applied, or the exact construction of the human baseline. These omissions leave the support for the claim that agents 'fail to meet the human baseline across all three dimensions' difficult to verify.

    Authors: We acknowledge that the current manuscript provides insufficient detail on question generation, controls, and baseline construction, which limits independent verification. We will substantially expand the Evaluation Methodology section (and add an appendix if needed) to describe: (1) the process for generating logically chained questions, including the interrogation principles, chaining logic, and any manual or automated steps; (2) the statistical controls employed, such as randomization, balancing across dimensions, and any checks for question validity; and (3) the precise construction of the human baseline, including recruitment criteria, task presentation, and how responses were scored. We will also cross-reference specific components of the released source code to facilitate verification of the implementation. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical evaluation against external human baseline

full rationale

The paper introduces PICon as a multi-turn interrogation framework drawing on an external principle from interrogation methodology, then applies it to measure internal, external, and retest consistency in persona agents versus 63 human participants. No equations, fitted parameters, or self-citations are used to derive the core results; the reported gaps are direct empirical outcomes from the evaluation protocol. The human baseline functions as an independent reference rather than a quantity defined by the framework itself, so the derivation chain remains self-contained and non-circular.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Only the abstract is available, so the ledger is necessarily incomplete; the central claim rests on the unstated premise that chained questioning reliably surfaces inconsistencies in LLMs in the same manner as in human interrogation.

axioms (1)
  • domain assumption Systematic multi-turn questioning will expose contradictions in persona agents analogously to human interrogation methodology.
    Invoked in the abstract when applying the interrogation principle to LLM agents.

pith-pipeline@v0.9.0 · 5747 in / 1129 out tokens · 32105 ms · 2026-05-21T10:41:18.714492+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.