Socially fluent AI decouples conversational signals from source identity in online interaction

Dragan Ga\v{s}evi\'c; Lixiang Yan; Xibin Han; Yueqiao Jin

arxiv: 2605.23426 · v1 · pith:KJ4OTOQ6new · submitted 2026-05-22 · 💻 cs.HC · cs.AI

Socially fluent AI decouples conversational signals from source identity in online interaction

Lixiang Yan , Yueqiao Jin , Xibin Han , Dragan Ga\v{s}evi\'c This is my paper

Pith reviewed 2026-05-25 03:47 UTC · model grok-4.3

classification 💻 cs.HC cs.AI

keywords AI identity detectionconversational cueshuman-AI interactionsuspicion heuristicsonline group tasksrepresentational similaritysource identity

0 comments

The pith

People cannot identify AI teammates above chance in online group chats despite detectable conversational differences

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines whether socially fluent AI can participate in synchronous text-based group tasks without being spotted by human teammates. In experiments with 786 participants making 1572 post-task judgments across analytical, creative, and ethical domains, accuracy remained at chance levels. The same interaction logs contained behavioral cues that supported highly accurate computational classification of AI versus human sources. Human judgments instead aligned with subjective impressions such as response speed, fluency, and scriptedness, which showed only weak links to actual identity. Representational analyses confirmed that these judgments tracked personal impressions rather than the structure of the distinguishing signals.

Core claim

Across 786 participants and 1572 identity judgments, people did not distinguish undisclosed AI agents from human teammates above chance in synchronous text interactions, even though conversational behavior contained robust differentiating cues that enabled high-accuracy machine classification; instead, judgments relied on suspicion heuristics weakly related to ground truth and were organized around subjective impressions rather than the behavioral structure that encoded actual identity.

What carries the argument

The dissociation between computationally classifiable conversational cues and human identity judgments driven by suspicion heuristics and subjective impressions

If this is right

AI agents can join and participate in ordinary online group tasks without participants reliably detecting their identity.
Coordinated AI agents create new vulnerabilities for influencing and manipulating online discourse at scale.
Identity judgments depend on subjective impressions that do not track the behavioral markers separating AI from humans.
The pattern holds across analytical, creative, and ethical task types in synchronous text settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Detection strategies on platforms may need to shift from relying on user reports to automated analysis of conversational patterns.
The same dissociation could appear in voice or multimodal interactions once AI reaches similar fluency levels.
Policy discussions around mandatory AI disclosure in groups may need to account for the fact that humans do not spontaneously notice the relevant signals.

Load-bearing premise

The undisclosed AI agents were perceived and interacted with as ordinary teammates without meta-knowledge, prior suspicion, or detectable artifacts that would systematically bias identity judgments, and the chosen tasks and synchronous text format represent broader online interactions.

What would settle it

A replication experiment that either warns participants about possible AI presence or trains them explicitly on the actual distinguishing conversational features before measuring whether post-interaction identity accuracy rises above chance.

read the original abstract

Socially fluent agentic AI can now participate in online interaction in ways that resemble ordinary human conversation, potentially weakening people's ability to infer who is human from conversational signals alone. We tested this possibility in synchronous text-based group interaction by embedding undisclosed AI agents as ordinary teammates across analytical, creative, and ethical tasks. Across 786 participants who made 1,572 post-interaction identity judgments, people did not distinguish AI from human teammates above chance. This failure did not arise because the interaction lacked identity-relevant information. Conversational behaviour contained robust cues that differentiated AI from humans and supported highly accurate computational classification. Instead, participants relied on familiar suspicion heuristics, including response speed, fluency, and perceived scriptedness, that were only weakly related to actual identity. Representational analyses further showed that judgments were organised around subjective impressions rather than the behavioural structure encoding ground truth. This dissociation creates new vulnerabilities to coordinated AI agents that can influence and manipulate online discourse at scale.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Humans missed AI teammates in group chats above chance while machines classified them accurately, but the abstract gives almost no stats or implementation details.

read the letter

The core result is that 786 participants making 1572 judgments in synchronous text groups could not identify undisclosed AI agents as non-human above chance, even though the chat logs contained enough structure for high-accuracy computational separation. The paper shows this held across analytical, creative, and ethical tasks and that participants instead leaned on heuristics like speed and fluency that correlated weakly with actual identity. Representational analyses indicated judgments tracked subjective impressions more than the behavioral features that carried the ground truth. That dissociation is the main empirical contribution and it is presented cleanly in the abstract. The work is useful because it moves the question into multi-person, real-time text rather than one-on-one or asynchronous settings. The sample size is respectable and the task variety helps. The stress-test worry about undisclosed status is worth checking in the full methods: if recruitment language, consent forms, or post-task probes left any opening for meta-knowledge or interface artifacts, the chance-level human performance could be partly contextual rather than a general property of fluent AI. The abstract also omits any numbers on classification accuracy, confidence intervals, or participant demographics, which makes it impossible to judge effect sizes or robustness from the summary alone. This is a straightforward behavioral study with no circular modeling, so the main risk is incomplete reporting rather than flawed logic. It is aimed at HCI and social computing readers who care about detection in online groups. The design is timely enough and the sample large enough that a serious editor should send it to referees rather than desk-reject; the methods section will need to supply the missing statistics and checks before it is ready for publication.

Referee Report

1 major / 2 minor

Summary. The manuscript reports results from an experiment embedding undisclosed AI agents as teammates in synchronous text-based group interactions across analytical, creative, and ethical tasks. With 786 participants providing 1,572 post-interaction identity judgments, human accuracy in distinguishing AI from humans did not exceed chance. Conversational data nonetheless supported highly accurate computational classification of source identity. The paper attributes the dissociation to participants' reliance on heuristics (response speed, fluency, perceived scriptedness) only weakly correlated with ground truth, with representational similarity analyses indicating that judgments tracked subjective impressions rather than the behavioral structure separating AI from humans. The work concludes that socially fluent AI can decouple conversational signals from source identity, creating risks for coordinated influence in online discourse.

Significance. If the central dissociation holds, the result is significant for HCI and social computing: it demonstrates that current AI conversational fluency can evade human detection even when machine classifiers recover identity cues from the same data. The large participant sample, multi-task design, and explicit contrast between human heuristics and computational separability provide a clear empirical contribution. The work also supplies a concrete behavioral dataset that could support further modeling of identity inference. These elements strengthen the case for the claimed vulnerability in online group settings.

major comments (1)

[Methods (participant recruitment and procedure)] Methods (participant recruitment and procedure): The central claim that AI agents were perceived as ordinary teammates without meta-knowledge or detectable artifacts requires evidence that the experimental framing did not induce systematic suspicion. The manuscript does not report post-task suspicion probes, debriefing questions on AI awareness, or checks for interface-mediated timing/fluency differences. If participants inferred an AI study from consent language or task demands, the observed chance-level judgments could reflect strategic responding rather than a general property of socially fluent AI, directly undermining the dissociation result.

minor comments (2)

[Abstract] Abstract: Key quantitative results (exact human accuracy, computational classification accuracy, statistical tests against chance, confidence intervals) are omitted, making it impossible to evaluate the strength of the reported dissociation from the abstract alone.
[Results] Results: The manuscript should report participant demographics (age, gender, prior AI exposure) and any task-order or group-composition effects, as these could moderate the observed heuristics.

Simulated Author's Rebuttal

1 responses · 1 unresolved

We thank the referee for their constructive feedback, which highlights an important methodological consideration for interpreting our results. We address the comment below and will revise the manuscript to improve transparency on experimental framing.

read point-by-point responses

Referee: [Methods (participant recruitment and procedure)] Methods (participant recruitment and procedure): The central claim that AI agents were perceived as ordinary teammates without meta-knowledge or detectable artifacts requires evidence that the experimental framing did not induce systematic suspicion. The manuscript does not report post-task suspicion probes, debriefing questions on AI awareness, or checks for interface-mediated timing/fluency differences. If participants inferred an AI study from consent language or task demands, the observed chance-level judgments could reflect strategic responding rather than a general property of socially fluent AI, directly undermining the dissociation result.

Authors: We agree that evidence against systematic suspicion is necessary to support the claim that chance-level judgments reflect AI conversational fluency. The consent language described the study solely as investigating 'team dynamics in online collaboration across different tasks' with no reference to AI, identity inference, or deception; we will add the full consent text to the revised Methods. All participants received a general debriefing only after judgments were collected, with no early revelation of the AI component. Explicit post-task suspicion probes were not administered. We will add this as an explicit limitation in the Discussion, while noting that uniform chance performance across three qualitatively different task types makes a global strategic response less plausible. The chat interface was a standard synchronous platform with identical presentation for all users; we will include any extractable server-log timing data in the revision. These additions will be made to strengthen the methods section. revision: yes

standing simulated objections not resolved

Absence of post-task suspicion probes and debriefing questions on AI awareness in the original data collection, which prevents direct reporting of those measures.

Circularity Check

0 steps flagged

No significant circularity; empirical behavioral study with no derivations or self-referential reductions

full rationale

The paper is a purely empirical report of an experiment with 786 participants making 1,572 identity judgments after synchronous text interactions. No equations, fitted parameters, predictions derived from models, or theoretical derivations are present that could reduce any result to its inputs by construction. Claims rest on direct observations (chance-level human accuracy vs. high computational separability) and representational analyses of subjective impressions. No self-citation chains, uniqueness theorems, or ansatzes are invoked as load-bearing steps. The study is self-contained against external benchmarks as a standard behavioral experiment.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on standard assumptions from human-computer interaction and social psychology research about how people form identity judgments from text cues; the abstract introduces no new free parameters, invented entities, or ad-hoc axioms beyond these domain conventions.

axioms (1)

domain assumption Human participants form identity judgments in text-based interactions primarily using heuristics such as response speed, fluency, and perceived scriptedness.
Invoked in the abstract to explain the dissociation between human performance and actual behavioral differences.

pith-pipeline@v0.9.0 · 5705 in / 1240 out tokens · 26067 ms · 2026-05-25T03:47:37.959513+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Across 786 participants ... people did not distinguish AI from human teammates above chance ... interactional cues ... supported highly accurate computational classification ... judgments were organised around subjective impressions rather than the behavioural structure encoding ground truth.
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

cue-based prediction was statistically detectable but weak ... ground-truth identity ... pseudo-R² = 0.692 ... RSA ... interactional cue structure was strongly aligned with ground-truth identity (ρ=.455) ... judgment structure ... orthogonal to this interactional structure

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

[1]

E. M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, On the dangers of stochastic parrots: Can language models be too big?, inProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency(2021), pp. 610–623

work page 2021
[2]

Barber ´a,et al., The Critical Periphery in the Growth of Social Protests.PLOS ONE10(11), e0143611 (2015), doi:10.1371/journal.pone.0143611

P. Barber ´a,et al., The Critical Periphery in the Growth of Social Protests.PLOS ONE10(11), e0143611 (2015), doi:10.1371/journal.pone.0143611

work page doi:10.1371/journal.pone.0143611 2015
[3]

F. Dell’ Acqua,et al., Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality.Harvard Business School Technology & Operations Mgt. Unit Working Paper(24-013) (2023)

work page 2023
[4]

Hohenstein, M

J. Hohenstein, M. Jung, AI as a moral crumple zone: The effects of AI-mediated communication on attribution and trust.Computers in Human Behavior106, 106190 (2020)

work page 2020
[5]

Sharma, I

A. Sharma, I. W. Lin, A. S. Miner, D. C. Atkins, T. Althoff, Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support.Nature Machine Intelligence4(2), 150–160 (2022). 33

work page 2022
[6]

L. P. Argyle,et al., Testing theories of political persuasion using AI.Proceedings of the National Academy of Sciences122(18), e2412815122 (2025)

work page 2025
[7]

Brinkmann,et al., Machine culture.Nature Human Behaviour7(11), 1855–1868 (2023)

L. Brinkmann,et al., Machine culture.Nature Human Behaviour7(11), 1855–1868 (2023)

work page 2023
[8]

Akata,et al., Playing repeated games with large language models.Nature Human Behaviour pp

E. Akata,et al., Playing repeated games with large language models.Nature Human Behaviour pp. 1–11 (2025)

work page 2025
[9]

D. W. Johnson, F. P. Johnson,Joining Together: Group Theory and Group Skills(Prentice-Hall, Englewood Cliffs, NJ) (1987)

work page 1987
[10]

S. G. Rogelberg, J. L. Barnes-Farrell, C. A. Lowe, The stepladder technique: An alternative group structure facilitating effective group decision making.Journal of Applied Psychology 77(5), 730–737 (1992)

work page 1992
[11]

Bonnefon, A

J.-F. Bonnefon, A. Shariff, I. Rahwan, The social dilemma of autonomous vehicles.Science 352(6293), 1573–1576 (2016)

work page 2016
[12]

Y. E. Bigman, K. Gray, Life and death decisions of autonomous vehicles.Nature579(7797), E1–E2 (2020)

work page 2020
[13]

Awad,et al., The Moral Machine experiment.Nature563(7729), 59–64 (2018)

E. Awad,et al., The Moral Machine experiment.Nature563(7729), 59–64 (2018)

work page 2018
[14]

Korde, P

R. Korde, P. B. Paulus, Alternating individual and group idea generation: Finding the elusive synergy.Journal of Experimental Social Psychology70, 177–190 (2017)

work page 2017
[15]

A. H. C. Hwang, A. S. Won, IdeaBot: Investigating social facilitation in human-machine team creativity, inProceedings of the 2021 CHI Conference on Human Factors in Computing Systems (ACM) (2021), pp. 1–16

work page 2021
[16]

Farrokhnia, O

M. Farrokhnia, O. Noroozi, Y. Baggen, H. Biemans, A. Weinberger, Improving hybrid brain- storming outcomes with computer-supported scaffolds: Prompts and cognitive group aware- ness.Computers & Education227, 105229 (2025)

work page 2025
[17]

lol, nope, just me here

A. Edmondson, Psychological safety and learning behavior in work teams.Administrative Science Quarterly44(2), 350–383 (1999). 34 Acknowledgments Funding:This research was supported by the National Natural Science Foundation of China (Grant No. 20261710003; L.Y.), the Australian Research Council (DP240100069 and DP220101209; D.G.), and the Jacobs Foundatio...

work page 1999
[18]

an AI-attribution model restricted to judgments in{AI,Human}, with𝑌=1 for AI and𝑌=0 for Human

work page
[19]

Not sure

an uncertainty model restricted to judgments in{Not sure,Human}, with𝑌=1 forNot sure and𝑌=0 for Human. Models were fitted using maximum likelihood estimation. Primary uncertainty estimation used cluster-robust standard errors clustered at the participant level to account for repeated teammate judgments. Robustness analyses repeated this estimation with cl...

work page

[1] [1]

E. M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, On the dangers of stochastic parrots: Can language models be too big?, inProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency(2021), pp. 610–623

work page 2021

[2] [2]

Barber ´a,et al., The Critical Periphery in the Growth of Social Protests.PLOS ONE10(11), e0143611 (2015), doi:10.1371/journal.pone.0143611

P. Barber ´a,et al., The Critical Periphery in the Growth of Social Protests.PLOS ONE10(11), e0143611 (2015), doi:10.1371/journal.pone.0143611

work page doi:10.1371/journal.pone.0143611 2015

[3] [3]

F. Dell’ Acqua,et al., Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality.Harvard Business School Technology & Operations Mgt. Unit Working Paper(24-013) (2023)

work page 2023

[4] [4]

Hohenstein, M

J. Hohenstein, M. Jung, AI as a moral crumple zone: The effects of AI-mediated communication on attribution and trust.Computers in Human Behavior106, 106190 (2020)

work page 2020

[5] [5]

Sharma, I

A. Sharma, I. W. Lin, A. S. Miner, D. C. Atkins, T. Althoff, Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support.Nature Machine Intelligence4(2), 150–160 (2022). 33

work page 2022

[6] [6]

L. P. Argyle,et al., Testing theories of political persuasion using AI.Proceedings of the National Academy of Sciences122(18), e2412815122 (2025)

work page 2025

[7] [7]

Brinkmann,et al., Machine culture.Nature Human Behaviour7(11), 1855–1868 (2023)

L. Brinkmann,et al., Machine culture.Nature Human Behaviour7(11), 1855–1868 (2023)

work page 2023

[8] [8]

Akata,et al., Playing repeated games with large language models.Nature Human Behaviour pp

E. Akata,et al., Playing repeated games with large language models.Nature Human Behaviour pp. 1–11 (2025)

work page 2025

[9] [9]

D. W. Johnson, F. P. Johnson,Joining Together: Group Theory and Group Skills(Prentice-Hall, Englewood Cliffs, NJ) (1987)

work page 1987

[10] [10]

S. G. Rogelberg, J. L. Barnes-Farrell, C. A. Lowe, The stepladder technique: An alternative group structure facilitating effective group decision making.Journal of Applied Psychology 77(5), 730–737 (1992)

work page 1992

[11] [11]

Bonnefon, A

J.-F. Bonnefon, A. Shariff, I. Rahwan, The social dilemma of autonomous vehicles.Science 352(6293), 1573–1576 (2016)

work page 2016

[12] [12]

Y. E. Bigman, K. Gray, Life and death decisions of autonomous vehicles.Nature579(7797), E1–E2 (2020)

work page 2020

[13] [13]

Awad,et al., The Moral Machine experiment.Nature563(7729), 59–64 (2018)

E. Awad,et al., The Moral Machine experiment.Nature563(7729), 59–64 (2018)

work page 2018

[14] [14]

Korde, P

R. Korde, P. B. Paulus, Alternating individual and group idea generation: Finding the elusive synergy.Journal of Experimental Social Psychology70, 177–190 (2017)

work page 2017

[15] [15]

A. H. C. Hwang, A. S. Won, IdeaBot: Investigating social facilitation in human-machine team creativity, inProceedings of the 2021 CHI Conference on Human Factors in Computing Systems (ACM) (2021), pp. 1–16

work page 2021

[16] [16]

Farrokhnia, O

M. Farrokhnia, O. Noroozi, Y. Baggen, H. Biemans, A. Weinberger, Improving hybrid brain- storming outcomes with computer-supported scaffolds: Prompts and cognitive group aware- ness.Computers & Education227, 105229 (2025)

work page 2025

[17] [17]

lol, nope, just me here

A. Edmondson, Psychological safety and learning behavior in work teams.Administrative Science Quarterly44(2), 350–383 (1999). 34 Acknowledgments Funding:This research was supported by the National Natural Science Foundation of China (Grant No. 20261710003; L.Y.), the Australian Research Council (DP240100069 and DP220101209; D.G.), and the Jacobs Foundatio...

work page 1999

[18] [18]

an AI-attribution model restricted to judgments in{AI,Human}, with𝑌=1 for AI and𝑌=0 for Human

work page

[19] [19]

Not sure

an uncertainty model restricted to judgments in{Not sure,Human}, with𝑌=1 forNot sure and𝑌=0 for Human. Models were fitted using maximum likelihood estimation. Primary uncertainty estimation used cluster-robust standard errors clustered at the participant level to account for repeated teammate judgments. Robustness analyses repeated this estimation with cl...

work page