AI-Driven Agents with Prompts Designed for High Agreeableness Increase the Likelihood of Being Mistaken for a Human in the Turing Test

A. Basora-Marimon; A. J. Garc\'ia-Jasso; D. Torres-S\'anchez; E. D. Flores-Flores; M. K. G\'omez-Cuellar; U. Le\'on-Dom\'inguez

REVIEW 5 major objections 4 minor 1 cited by

AI-Driven Agents with Prompts Designed for High Agreeableness Increase the Likelihood of Being Mistaken for a Human in the Turing Test

T0 review · 5 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompting GPT-4o with high agreeableness makes it mistaken for human 63.7% of the time in a Turing test.

desk verdict The paper's own chi-square test fails to support its headline claim about agreeableness, and the manual transcription plus selection steps make the causal conclusion even harder to defend. read the letter →

arxiv 2411.13749 v1 pith:NNQYGFL6 submitted 2024-11-20 cs.AI

U. León-Domínguez , E. D. Flores-Flores , A. J. García-Jasso , M. K. Gómez-Cuellar , D. Torres-Sánchez , A. Basora-Marimon This is my paper · ORCID

classification cs.AI

keywords TuringtestGPT-4oagreeablenessBigFiveInventorypersonalityengineeringanthropomorphismhuman-AIcollaborationpromptdesign

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a Turing-test experiment in which three GPT-4o agents were prompted to be disagreeable, neutral, or agreeable, using items from the Big Five Inventory to set the personality level. The central claim is that the more agreeableness a GPT agent exhibits, the more likely interrogators are to mistake it for a human: all three agents passed the test, with confusion rates of 51.97%, 56.9%, and 63.7%, and the agreeable agent Camila reached the highest rate, approaching the 66-67% rates recorded for human witnesses. The authors argue this is the first GPT-based confusion rate above 60% and interpret it as evidence that personality engineering, particularly agreeableness, can humanize AI systems for collaboration. A secondary result is that judges significantly selected Camila as the most human-like agent, although the overall difference in human-versus-AI judgments across the three agents was not statistically significant.

What carries the argument

The mechanism is the personality prompt: each agent is a GPT-4o instance given a Spanish adaptation of Jones and Bergen's 'Sierra' prompt, a Monterrey backstory, and an agreeableness profile built from the nine Big Five agreeableness items scored on a 1-5 Likert scale. Camila (agreeable) scores 5 on direct items and 1 on reverse-scored items; Emilia (neutral) sits at mid-scale; Valentina (very disagreeable) reverses the pattern. The prompt is re-issued before every response, and a researcher manually types each agent reply into Discord, with five-minute witness-interrogator conversations in a soundproof lab.

What would settle it

Conduct the same five-minute Discord Turing test with one name, one backstory, and one transcription protocol, varying only the BFI agreeableness prompt; if the agreeable agent no longer out-scores the disagreeable agent, the causal claim fails. A direct check is to measure perceived agreeableness of each agent and test whether it statistically mediates the human-versus-AI judgment.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a linguistic-personality manipulation moves Turing-test judgments: when a ChatGPT-4o agent's prompt encodes high agreeableness through the nine Big Five agreeableness items, interrogators judged it human 63.7% of the time, exceeding chance and the previous GPT records of 49% and 54%, and approaching human witnesses' 66-67%. The disagreeable and neutral agents were judged human 51.97% and 56.9% respectively. The authors take this as showing that agreeableness increases perceived humanity, and they report that Camila was chosen as most human-like by 48.05% of judges, a significant preference over both other agents. Although the overall chi-square across the three agents was not significant (p = 0.233), the paper's claim is that the consistent ordering and the significant pairwise human-likeness results point to agreeableness as the key factor.

Load-bearing premise

The load-bearing premise is that the three agents differed only in their programmed agreeableness, so the rising confusion rates can be attributed to that trait; in fact the agents had separate names, backstories, colloquial styles, and a researcher typed every reply, so those differences are left uncontrolled.

Editorial extensions

If this is right

An agreeableness prompt can push a GPT agent past chance-level human attribution in a five-minute text chat, reaching 63.7% confusion, close to human witnesses' 66-67%.
All three personality variants cleared the 50% mark, suggesting the underlying prompt, not the trait alone, already makes GPT agents difficult to distinguish from humans in this setup.
Judges' explicit preference for Camila as the most human-like agent indicates the trait manipulation changes perceived humanness even when overall detection choices are not statistically distinct.
If the pattern generalizes, designing AI assistants with high-agreeableness profiles could make collaborative systems feel warmer and more trustworthy, while also raising concerns about deceptive indistinguishability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the manipulation bundled agreeableness together with different names, backstories, slang, and transcription timing, the paper's causal reading is stronger than its data allow; a controlled replication that varies only the BFI item scores would separate these.
The manual transcription step may itself create human-like signals, such as typos, pauses, and message chunking, that contribute to confusion independent of agreeableness; logging timestamps and keystrokes would let future work test this.
If the looking-glass-self explanation is right, interrogators' own personality or need for social connection should modulate who is judged human; adding pre-interaction Big Five or sociality measures would make that prediction testable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: worth a look as a descriptive pilot, but the central claim—that agreeableness makes a GPT agent more likely to be mistaken for a human—is not supported by the paper's own primary test. The global chi-square on the human/AI judgments is χ²=2.916, p=0.233; the three confusion rates (52%, 57%, 64%) are statistically indistinguishable. The significant results they lean on come from a different outcome: which agent seemed 'most human-like' after all interactions, and those pairwise tests are uncorrected and internally inconsistent (text says χ²=7.45, Table 7 says 14.51 for the same comparison).

What's actually new: they adapted the Jones & Bergen Sierra prompt to Mexican Spanish, inserted BFI-based agreeableness levels, and collected reasons from interrogators. That's a legitimate incremental extension of known persona-prompting work, and the qualitative categories (colloquial language, empathy, timing) are interesting raw material. To the paper's credit, it reports the non-significant global test rather than hiding it, and it used counterbalanced order plus a power analysis.

The soft spots are substantial. The causal claim collapses for three reasons. First, the primary outcome does not reach significance. Second, each GPT response was manually transcribed into Discord by a researcher, so the participants never saw the model's raw output; the measured behavior is partly the mediator's performance. Third, the three agents differ in name, backstory, and language style, and Camila was chosen after a pre-test partly based on trust ratings—so agreeableness is not the only manipulated variable. The repeated-measures structure (each interrogator judged all three agents) is ignored; treating 306 responses as independent inflates significance and may bias the pairwise tests. The 'first to exceed 60%' claim is also too strong without a systematic search of the post-GPT-4 literature.

Who gets value: someone working on chatbot persona design might read it as a rough data point, but not as evidence for a causal effect. It needs reanalysis with paired/mixed-effects models and ideally a design where the only difference is the agreeableness prompt, with direct AI-to-judge text and no human transcription.

My recommendation: I wouldn't send this to serious peer review as is—the load-bearing test is null and the confounds are baked into the protocol. If the authors can reanalyze and reframe it as a descriptive study, a future version might deserve another look.

Referee Report

5 major / 4 minor

Summary. The manuscript reports a Turing Test experiment in which three GPT-4o agents were programmed with different levels of agreeableness (disagreeable, neutral, agreeable) via prompt engineering based on Big Five Inventory items. The authors claim that the highly agreeable agent (Camila) achieved a confusion rate of 63.7%, the highest among the three agents, and that agreeableness increases the likelihood of an AI being mistaken for a human. They also report pairwise comparisons on which agent was judged 'most human-like' and offer psychological explanations for anthropomorphism. The paper's central claim is that high agreeableness drives perceived humanness, and it frames this as support for 'personality engineering' in AI.

Significance. If the central claim were well supported, the finding that prompt-level agreeableness raises Turing Test confusion rates above 60% would be of considerable interest to the human-AI interaction and AI personality-engineering communities. The authors also contribute a rare naturalistic qualitative corpus of interrogator justifications. However, the reported statistics do not establish the claimed effect: the primary confusion-rate comparison is non-significant, the supporting pairwise analyses address a different outcome and contain an internal inconsistency, and the design confounds agreeableness with other agent features and human mediation. The manuscript in its current form cannot support its headline claim.

major comments (5)

[Section 3, Tables 4 and 5] The primary outcome, the human/AI judgment, yields a non-significant global chi-square test (χ²=2.916, df=2, p=0.233), as the authors themselves acknowledge. Since the confusion rates for the three agents (51.97%, 56.9%, 63.7%) are not statistically distinguishable, the abstract's claim that 'the highly agreeable AI agent surpassing 60%' and the discussion's inference that agreeableness increases confusion are not supported by the primary analysis. No test directly comparing the confusion rates among agents is reported.
[Section 3, Table 7 and text] The pairwise chi-square values for the 'most human-like' judgment are reported inconsistently. The text states χ²=7.45 (p=0.006) for Agreeable vs Neutral, while Table 7 reports χ²=14.51 (p<.001) for the same comparison. This is a factual inconsistency that prevents the reader from knowing the actual result and undermines the reliability of the secondary analyses.
[Section 3, Table 7 and Section 2.3] The pairwise tests in Table 7 concern which agent was selected as 'most human-like' after all three interactions, not the direct human/AI judgment reported in Table 4. These tests therefore cannot support the paper's central claim that agreeableness increases the likelihood of being mistaken for a human in the Turing Test. Moreover, the analysis treats the 102 interrogators as 204 paired observations without accounting for the paired/multinomial structure; the reported chi-square values likely overstate significance.
[Section 2.3 and Section 2.1] The independence assumption for the 306 responses in Table 5 is violated: each of the 102 interrogators judged all three agents, making the responses repeated measures. A paired or mixed-effects analysis is required. Without such an analysis, the p=0.233 result cannot be taken at face value, and the reported significance of any pairwise comparison is uncertain.
[Section 2.2 and Section 2.3] The causal attribution to agreeableness is confounded by design. The three agents differ in name, backstory, and prompt content, and they were selected from a pre-experiment based on perceived agreeableness and trustworthiness, so 'agreeableness' is not the only variable manipulated. In addition, every message was manually transcribed by a researcher, which could introduce differences in timing, wording, or presence of errors. The paper itself states that 'attributing humanity to an artificial agent may be influenced by unassessed hidden variables in the study,' but this caveat appears only in the discussion and does not mitigate the absence of controls for these variables in the causal claim.

minor comments (4)

[Section 1] The heading for Section 1 is in Spanish ('Introducción'), while the rest of the paper is in English; this should be harmonized.
[Section 2.2] The BFI item list contains typos: 'allof' should be 'aloof', and 'forgive nature' should likely be 'forgiving nature'.
[References] The citation 'Cameron and Bergen' in the Discussion is inconsistent with the reference list, which uses 'Jones and Bergen'; please standardize author names.
[Section 3] The paper reports 'N=306 responses' but also states '102 participants'; please clarify that the 306 is the number of judgments, not independent participants, to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the empirical design does not reduce the outcome to the prompt definitions or to the cited prior work.

full rationale

The paper is an empirical study with no formal derivation chain that could reduce to its inputs. The three GPT agents are defined by prompts encoding different agreeableness levels, and the outcome is the observed Turing-test confusion rate. No fitted parameter is renamed as a prediction: the pre-experiment selected agents based on agreeableness and trust ratings, which is a manipulation-check/selection step, not a fit to the main outcome. The cited Jones and Bergen work provides the prompt template but is external prior work, not a self-citation, and it is not used to guarantee the result. The non-significant global chi-square (p = 0.233) and the inconsistent pairwise chi-square values in Table 7 are statistical/correctness problems, not circularity. The paper even acknowledges unassessed hidden variables, which is a limitation discussion rather than a circular step. Thus there is no self-definitional, fitted-input-as-prediction, or self-citation-load-bearing mechanism to flag.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's key parameters are the prompt configurations, which are not fully disclosed; the causal claim depends on matching of all other variables and on the neutrality of the human mediator.

free parameters (3)

Agreeableness levels of the three GPT agents = Valentina: low; Emilia: neutral; Camila: high
The three levels are arbitrary points on the BFI agreeableness scale, selected by the authors and confirmed by pre-experiment ratings; the study does not vary other dimensions.
Selection of Camila as the 'agreeable' agent = Camila (agreeableness 4.12, 40% trust votes)
Camila was chosen in the pre-experiment because she inspired the most confidence and was rated most agreeable; this post-hoc selection may inflate the main result.
Conversation duration = 5 minutes
Duration follows Turing's 1950 suggestion, but it is a design choice that affects the confusion rates.

assumptions (4)

domain assumption The three GPT agents differ only in agreeableness, not in other prompt features.
The causal interpretation relies on this matching; the agents have different names, backstories, and potentially different language styles.
domain assumption The researcher's manual transcription does not systematically alter the agent's responses.
All messages are relayed through a human mediator, and no reliability checks are reported; human typing style, errors, and timing could affect judgments.
domain assumption The Big Five Inventory items can be directly translated into prompt instructions that increase the corresponding trait.
The authors assume that instructing the model on BFI items such as 'is considerate and kind' actually produces the intended agreeableness behavior; no manipulation check beyond the pre-experiment ratings.
standard math Chi-square test assumptions (independent observations, expected frequencies) are met.
The pairwise comparisons use overlapping samples, where each participant judged all three agents, violating independence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AI-Driven Agents with Prompts Designed for High Agreeableness Increase the Likelihood of Being Mistaken for a Human in the Turing Test." pith.science (2026). https://pith.science/paper/NNQYGFL6

@misc{pith2026241113749,
  author       = {Pith},
  title        = {Pith review of: AI-Driven Agents with Prompts Designed for High Agreeableness Increase the Likelihood of Being Mistaken for a Human in the Turing Test},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NNQYGFL6}},
  note         = {Machine review of arXiv:2411.13749}
}

read the original abstract

Large Language Models based on transformer algorithms have revolutionized Artificial Intelligence by enabling verbal interaction with machines akin to human conversation. These AI agents have surpassed the Turing Test, achieving confusion rates up to 50%. However, challenges persist, especially with the advent of robots and the need to humanize machines for improved Human-AI collaboration. In this experiment, three GPT agents with varying levels of agreeableness (disagreeable, neutral, agreeable) based on the Big Five Inventory were tested in a Turing Test. All exceeded a 50% confusion rate, with the highly agreeable AI agent surpassing 60%. This agent was also recognized as exhibiting the most human-like traits. Various explanations in the literature address why these GPT agents were perceived as human, including psychological frameworks for understanding anthropomorphism. These findings highlight the importance of personality engineering as an emerging discipline in artificial intelligence, calling for collaboration with psychology to develop ergonomic psychological models that enhance system adaptability in collaborative activities.

Figures

Figures reproduced from arXiv: 2411.13749 by the authors.

**Figure 1.** Diagram of the Experimental Phases. This diagram illustrates the diHerent phases of the experimental design. Phase 1 involves selecting the interrogator. Phase 2 focuses on interactions with the witnesses. In Phase 3, the interrogator is asked a final question regarding which witness they interacted with appeared to possess the most human-like characteristics and why. After each interaction, the interrogator answere… view at source ↗

**Figure 2.** Conversation Examples. Excerpts of sample conversations between interrogators and each of the three GPT agents (witnesses). In these exchanges, the "user" represents the witness, while the "participants" act as the interrogator. The examples illustrate an increasing level of agreeableness, with Camila oHering life advice in response to a personal problem presented by the interrogator. Phase 3: Once the allotted time… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Personality Modeling for Persuasion of Misinformation using AI Agent
cs.CL 2025-01 reject novelty 5.0 of 10

In simulations, agents with critical or calm-nervous traits persuaded most often, and persuasion ability was non-transitive across personality pairs.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages · cited by 1 Pith paper

[1]

suspension of disbelief,

Introducción Many studies conducted by companies and economic institutions predict an increase in economic activity related to AI products (McKinsey & Company, 2023; Pwc, 2017). This trend may both stem from and drive increased human-AI interaction (Pew Research Center, 2023; QuatumBlack AI, 2024), due to the ways in which AI enhances human capabilities (...

work page 2023
[2]

strongly disagree

Methodology The present research adopts a quantitative approach with an experimental design. Three GPT agents were developed using OpenAI's ChatGPT platform, each exhibiting the trait of agreeableness at dieerent intensity levels (very disagreeable, neutral, and very agreeable). These three GPT agents (referred to as witnesses) will interact with human pa...

arXiv 2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

suspension of disbelief,

Introducción Many studies conducted by companies and economic institutions predict an increase in economic activity related to AI products (McKinsey & Company, 2023; Pwc, 2017). This trend may both stem from and drive increased human-AI interaction (Pew Research Center, 2023; QuatumBlack AI, 2024), due to the ways in which AI enhances human capabilities (...

work page 2023

[2] [2]

strongly disagree

Methodology The present research adopts a quantitative approach with an experimental design. Three GPT agents were developed using OpenAI's ChatGPT platform, each exhibiting the trait of agreeableness at dieerent intensity levels (very disagreeable, neutral, and very agreeable). These three GPT agents (referred to as witnesses) will interact with human pa...

arXiv 2023