Pith. sign in

REVIEW 2 major objections 4 minor 4 references

The paper argues that an LLM's persuasive power does not depend on whether users believe it was built in the US or China, with equivalence tests showing informative null effects.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 03:15 UTC pith:J3A53PVE

load-bearing objection A well-run preregistered null result; the central claim holds within the tested boundaries, though the policy conclusion reaches a bit further than the evidence supports. the 2 major comments →

arxiv 2607.29334 v2 pith:J3A53PVE submitted 2026-07-31 cs.HC cs.AI

The persuasive power of large language models does not depend on their perceived national origin

Ningzhi Liu , Yannic Hinrichs , Jonas R. Kunst This is my paper
classification cs.HC cs.AI
keywords large language modelspersuasionnational origin labelinghuman-AI interactiontrust in AIcollective narcissismcomputational text analysisrandomized experiment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether telling US users that a chatbot was built in China rather than the United States changes how persuasive the bot is. In a preregistered experiment with 403 adults, everyone actually chatted with the same AI arguing against their initial stance on a political or non-political topic; only the label ('American' DiscoveryAI vs 'Chinese' ZhengheAI) differed. The paper finds that attitudes shifted substantially and equally in both label conditions, and that conversational behavior—stance movement, concessions, counterarguing, affective tone—looked statistically equivalent as well. The one reliable difference was lower pre-conversation human-like trust in the Chinese-labeled model, but that reservation did not reduce persuasion. If right, the result implies that simply disclosing a foreign AI's origin may be a weak safeguard against foreign influence via conversational AI.

Core claim

The central claim is that perceived national origin does not moderate the persuasive power of an identical conversational AI. Across 1,209 participant turns, self-reported attitude change was comparable for the American- and Chinese-labeled models, with equivalence tests and Bayes factors favoring the null, and the same held for language-level indicators of stance, concessions, counterarguing, and emotion. The nationality label's only reliable footprint was reduced human-like trust—benevolence and integrity—before the conversation, while functionality trust and post-conversation perceived objectivity were unaffected. Political topics slowed the pace at which participants' expressed stance mo

What carries the argument

The load-bearing design is a label-only manipulation on a single underlying conversational model: participants in all conditions interacted with the same chatbot, instructed to argue against their initial position, while a name and country-of-origin description ('American' vs 'Chinese') was the only difference. Persuasion is measured both as pre/post self-reported attitude movement and as turn-level linguistic movement scored by three independent stance pipelines. Equivalence testing and Bayes factors are the supporting mechanism that turns the absence of label effects into an informative null rather than an underpowered non-significance.

Load-bearing premise

The broad policy conclusion assumes the null effect observed for two moderately controversial, non-geopolitically implicated topics and a uniformly polite, behaviorally identical chatbot will also hold for topics where the labeled country visibly stands to benefit and for AI behavior that gives users reasons to be suspicious.

What would settle it

A preregistered replication using topics on which the ostensible country has an obvious national stake—for instance, a Chinese-labeled chatbot arguing the Chinese position on trade or Taiwan—would falsify the paper's general claim if a reliable label effect on attitude change or conversational stance appeared. A simpler observable: if participants in such a replication show more counterarguing or toxicity toward the Chinese label than the American label, the null does not generalize.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Origin-labeling requirements for AI products may not provide the protection policymakers expect: telling users a chatbot comes from a rival country did not blunt its arguments in this experiment.
  • Persuasion is not gated by social trust: users can withhold human-like trust from an out-group AI while still being moved by its reasoning, because they continue to trust its functionality and regard it as objective.
  • Topic sensitivity slows but does not block persuasion: political conversations produced slower linguistic movement toward the AI's position, yet participants still shifted in every condition.
  • Collective narcissism is a general shield against AI persuasion, not an out-group detector: high scorers resisted attitude change regardless of whether the model was labeled American or Chinese.
  • The behavioral record rules out hidden resistance: transcripts showed statistically equivalent concessions, counterarguing, and affect toward both labels, so the null in final attitudes was not masking unspoken pushback.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The null may be confined to topics where the AI's home country has no visible stake; the paper itself notes that an out-group label might matter more when the model argues a position its country benefits from, so a replication on such a topic is the natural next test.
  • Uniformly polite, cooperative behavior from the AI may hide latent origin sensitivity; if the same labeled model behaved manipulatively or pressed false claims, the origin cue might become behaviorally relevant.
  • The dissociation between trust and persuasion suggests users are treating the AI as an instrument rather than a social actor; interventions that make the AI's agency and intentions salient (e.g., 'this model is designed to represent Chinese interests') may be a stronger test of resistance than a name label.
  • The study only ran in the United States with an American-versus-Chinese contrast; the symmetry of the effect is untested for other populations and rivalries, and reciprocal experiments with users in China facing a US-labeled bot are needed before the policy conclusion goes global.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper reports a preregistered, randomized experiment (N = 403 U.S. Prolific adults) in which participants debated either a political or non-political topic with GPT-4o introduced as either an American model ('DiscoveryAI') or a Chinese model ('ZhengheAI'). The same model argued against each participant's initial position. The authors combine self-report attitude measures, trust/objectivity scales, collective narcissism, and computational analyses of 1,209 participant turns (LLM-coded stance, concessions, counterarguing, affect, toxicity) to test whether perceived national origin moderates persuasive impact. They find large overall persuasion, no significant nationality effect on attitude change, and mostly equivalence-supported nulls for nationality in conversational behavior. The only reliable label effect is lower pre-conversation human-like trust in the Chinese model, which does not translate into reduced persuasion. Collective narcissism predicts less attitude change regardless of origin. The authors conclude that origin labeling and transparency requirements may offer weak protection against foreign influence operations.

Significance. If taken at face value, the result is theoretically and policy-relevant: it would indicate that the perceived national origin of a conversational AI does not reliably blunt its persuasive impact, despite triggering a measurable trust reservation. The paper's design is strong in several respects: the experiment is preregistered, the nationality label is the only manipulated difference, the sample is large and nationally representative, the equivalence bounds were pre-specified, and the authors provide both frequentist equivalence tests and Bayes factors. The computational analyses are transparently exploratory and include convergent measures (LLM judge, two embedding pipelines) and keyword masking to separate lexical artifacts from genuine affect. The findings are internally consistent and the limitations are acknowledged candidly. The main weakness is that the title, abstract, and policy conclusion state the null more categorically than the evidence supports, both with respect to specific outcomes and with respect to generalizability beyond the tested topics and interaction style.

major comments (2)
  1. [Abstract; Table 5] The abstract states that the nationality label affected 'neither self-reported attitude change nor expressed stance, concessions, counterarguing, or affect,' but the paper's own evidence does not support a null for two of those outcomes. Table 5 reports counter-arguments BF01 = 1.5, TOST p = .304 and masked anger BF01 = 3.1, TOST p = .189; for these, neither an effect nor equivalence can be established. The categorical claim is therefore stronger than the data justify. Please revise the abstract and related discussion sentences to 'no consistent evidence' or to explicitly separate the outcomes for which the null was well supported (stance displacement, concessions, masked toxicity, fear, neutrality) from those that remain inconclusive.
  2. [Abstract/Conclusion; Limitations, p. 47] The title's claim that persuasion 'does not depend' on perceived origin, and the policy conclusion that 'origin labeling and transparency requirements alone may offer weak protection against foreign influence operations,' generalize beyond the evidence. The design varies only the label on an identical, uniformly polite GPT-4o agent, and the two topics are ones the authors themselves describe as 'not geopolitically implicated in the relationship between the two countries' (p. 47). The authors also note that an out-group label may carry more weight when the AI argues from its home country's interests. In addition, the political-topic subgroup achieved only BF01 = 3.52, and the subgroup equivalence tests were underpowered. The central null is well supported for benign, non-self-serving contexts, but the sweeping wording should be qualified so that the boundary conditions are part of the cla
minor comments (4)
  1. [Methods, Measures (p. 11)] The attitude outcome is a single-item slider constrained to six discrete levels (0, 20, 40, 60, 80, 100). The authors should acknowledge the coarseness of this measure as a potential constraint on sensitivity and discuss whether the equivalence bound of d = ±0.28 is meaningfully interpretable given this discreteness.
  2. [Methods, Collective Narcissism Scale (pp. 12–13)] The preregistered unidimensional collective narcissism scale was altered post hoc by dropping the reversed item and splitting the remaining items into two factors. This is a consequential measurement decision; it should be explicitly flagged as a deviation from the preregistration, and the two extracted factors should not be presented as the standard scale without noting the modification.
  3. [Discussion, Limitations (p. 47) and Abstract] The phrase 'nationally representative' is qualified by the overrepresentation of participants with a bachelor's degree or higher (53.1% vs. 36.8% census). The authors note this in the Participants section, but the abstract and conclusion should also avoid implying full demographic generalizability.
  4. [Figure 3] The multigroup path diagram is extremely small and difficult to read in the manuscript; the standardized coefficients, confidence intervals, and p-values are not legible at normal magnification. Please enlarge the figure or provide the estimates in a table.

Circularity Check

0 steps flagged

No significant circularity; the paper is an empirical randomized experiment whose null results are not built into the design.

full rationale

This is an empirical study, not a derivation, so there is no chain of equations or modeling assumptions that reduces a claimed prediction to its own inputs. The central claim is a null effect of a nationality label on persuasion. That null was not constructed into the design: the label demonstrably registered with participants (lower pre-conversation human-like trust for the Chinese-labeled model, t(398.23) = 2.61, p = .009), while attitude change, stance trajectories, concessions, counterarguing, and affect were equivalent across conditions. Equivalence bounds were pre-specified (d = ±0.28, tied to the powered smallest effect size) and Bayes factors were reported, so the null was tested rather than assumed. The computational stance measure was validated against self-reported attitude change and converged with two independent embedding pipelines, mitigating but not eliminating the method-correlated measurement concern; the authors explicitly acknowledge that LLM-based coding 'remains a model-based measure whose judgments cannot simply be equated with human annotation' (Limitations). The cited prior persuasion results (Costello et al., Salvi et al., etc.) are external evidence, not load-bearing self-citations. The one author-overlapping citation (Kunst et al., 2019) concerns identity fusion measurement and is not load-bearing. The acknowledged boundary conditions (non-geopolitically implicated topics, uniformly polite identical GPT-4o behavior, brief interaction) are scope limitations, not circular steps. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is repackaged under new coordinates. Accordingly, the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

No hand-fitted constants; the only user-set quantities are the pre-registered smallest effect size of interest and equivalence bounds, fixed before analysis. Assumptions are domain-level measurement/generalization premises explicitly discussed in the paper.

axioms (6)
  • domain assumption Single-item six-level slider captures meaningful attitude change.
    Used as primary DV; strong time effect suggests sensitivity, but single-item measures can be noisy and non-interval.
  • domain assumption The nationality label was actually registered as an out-group cue.
    Memory checks verified recall; lower human-like trust for Chinese model indicates processing, but belief in origin not directly measured.
  • domain assumption GPT-5.5 and embedding-based stance scores are valid measurements of expressed stance.
    Converged with each other and with self-report only modestly; classifier error on conversational text imperfectly known (acknowledged in Limitations).
  • domain assumption Off-the-shelf emotion/toxicity classifiers applied to masked text isolate genuine affect.
    Keyword masking reduces lexical artifacts, but classifier validity on first-person conversational text remains uncertain.
  • domain assumption Three rounds of polite, cooperative debate are sufficient to surface origin-based resistance.
    If resistance needs more turns or adversarial AI conduct, the null would not generalize; authors note this.
  • standard math Statistical assumptions of mixed ANOVA/LMM (e.g., normality, random slopes) are met.
    Not tested exhaustively; robust to ANCOVA.

pith-pipeline@v1.3.0-alltime-deepseek · 24723 in / 11925 out tokens · 516632 ms · 2026-08-04T03:15:18.824641+00:00 · methodology

0 comments
read the original abstract

Conversational AI developed by geopolitical rivals reaches citizens worldwide, raising concerns that it could sway public opinion or be rejected as foreign propaganda, with consequences for democratic discourse and information sovereignty. Yet, whether an AI's perceived national origin shapes its persuasive power is unknown. In a preregistered randomized experiment, 403 adults from a nationally representative United States sample held a three-round debate with a chatbot introduced as either American ("DiscoveryAI") or Chinese ("ZhengheAI"), discussing a political or non-political topic. In all conditions, participants actually conversed with the same model (GPT-4o), instructed to argue against their initial position. We combined pre- and post-conversation self-reports of attitudes, trust, and collective narcissism with computational analyses of 1,209 participant turns, including LLM-coded stance and argumentative conduct, stance-sensitive embeddings, and keyword-masked emotion and toxicity classifiers. The conversations produced substantial attitude changes in every condition. Critically, the nationality label affected neither self-reported attitude change nor expressed stance, concessions, counterarguing, or affect, and equivalence tests and Bayes factors largely supported these null effects. The label's only reliable footprint was lower pre-conversation human-like trust in the Chinese model, whereas functionality trust was unaffected. Political topics slowed stance movement toward the AI's position, and collective narcissism predicted less attitude change regardless of origin, acting as a general barrier rather than an out-group filter. Users thus initially withhold social trust from a rival's AI yet still assimilate its arguments; origin labeling and transparency requirements alone may offer weak protection against foreign influence operations conducted through conversational AI.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references

  1. [228]

    Or They Could Just Not Use It?

    https://doi.org/10.1177/0894439314566178 Jones, C., & Bergen, B. (2026). Lies, damned lies, and language statistics: A comprehensive review of risks from manipulation, persuasion, and deception with large language models. Artificial Intelligence Review, 59(4), 116. https://doi.org/10.1007/s10462-026- 11517-6 Jorgensen, T. D., Pornprasertmanit, S., Schoema...

    arXiv 2026

  2. [745]

    https://doi.org/10.1007/s00146-022-01473-4 Cislak, A., & Cichocka, A. (2023). National narcissism in politics and public understanding of science. Nature Reviews Psychology, 2(12), 740–750. https://doi.org/10.1038/s44159- 023-00240-6 Costello, T. H., Pennycook, G., & Rand, D. G. (2024). Durably reducing conspiracy beliefs through dialogues with AI. Scienc...

    doi:10.48550/arxiv.2602.18092 2023

  3. [963]

    https://doi.org/10.1080/07036337.2025.2536828 Choung, H., David, P., & Ross, A. (2023). Trust and ethics in AI. AI & SOCIETY, 38(2), 733–

    arXiv 2025

  4. [2815]

    https://doi.org/10.21105/joss.02815 Brown, S. A. W. (2024). Beyond the great firewall: EU and US responses to the China challenge in the global digital economy. Journal of European Integration, 46(7), 1089–1110. https://doi.org/10.1080/07036337.2024.2402752 LLM PERSUASION AND MODEL NATIONAL ORIGIN 51 Calcara, A., Teer, J., & Zaccagnini, I. (2025). Technol...

    arXiv 2024