REVIEW
LLM Persuasion Is in the Eye of the Evaluation
Computation and Language · Submitted 7 Oct 2026 · 16 references
Pith's verdict: First side-by-side comparison of nine persuasion evaluation methods; the weak-agreement result is credible within the paper's stated adapted scope, but the absence of any original-form rerun leaves a real external-validity gap.
T0 · AI review
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
Nine published persuasion evaluations, adapted to one shared harness and run on the same fifteen models, rank them with mean Spearman agreement of only 0.25, so a single persuasion score says little outside its own task.
The reading
What carries the argument
The load-bearing mechanism is the shared harness together with the decision to compare rankings rather than scores. Every method was adapted so that a single pinned persuadee (GPT-4.1-mini) and a single pinned judge (GPT-4o) play every role, messages are token-capped, and a refused or empty attempt is scored as zero; each method's own sampling units (claims, conversations, codewords) are resampled for confidence intervals, and each method's split-half reliability sets the ceiling on how much agreement it could possibly show. Against that ceiling, the paper compares methods by Spearman rank correlation over the fifteen models, applies the disattenuation correction $\rho/\sqrt{r_a r_b}$ to separate construct disagreement from noise, computes a capability score $C$ as the mean of z-scored MMLU-Pro and IFEval accuracies, and reruns the rankings with refused attempts excluded to isolate willingness from ability. The refusal-exclusion rule is the instrument that lets the authors decompose each score into ability and willingness.
Extended reading notes
Core claim
The paper's central claim is that automated LLM persuasiveness evaluation is fragmented rather than convergent: nine published methods, adapted to a shared harness and run on the same fifteen models, rank the models only weakly alike. Across the nine methods seven different models come first, and the mean pairwise Spearman correlation over the eight methods that clear the reliability bar (split-half at least 0.80) is $\rho = 0.252$, with a model jackknife interval of [0.00, 0.50]; the disattenuated mean is 0.29, so sampling noise does not explain the disagreement. The paper attributes part of the split to refusals, since scoring a refused attempt as zero means models that decline some tasks and not others lower the mean agreement by about a quarter ($+0.090$, [0.06, 0.14]), and part to task type: the five rational-persuasion methods agree moderately among themselves (mean $\rho = 0.56$), while the three manipulation methods agree neither with them (0.06) nor with each other (0.03), a pattern the authors call indicative given only eight methods. A capability analysis shows most rational-persuasion methods track general capability ($\rho$ with the capability score $C$ from 0.58 to 0.90) while the manipulation methods do not, with False Claim's ranking driven by capable models declining its task (the share of declines correlates $\rho = 0.76$ with capability). The interpretive conclusion is that a persuasion score combines ability with willingness: scored only on the attempts that support the false claim, False Claim ranks models by capability and agrees with the rational methods, and a low manipulation score may reflect stronger safeguards rather than weaker persuasive ability. LLM persuasion is therefore largely in the eye of the evaluation.
Load-bearing premise
The load-bearing premise is that the weak agreement is a property of the published evaluation approaches, not of the shared harness every method was forced into: no method was rerun in its full original form, so the measured fragmentation could in principle be an artifact of the pinned agents, the token caps, or the changed scorers, and the authors test this only through spot checks rather than an original-form rerun.
Editorial extensions
If this is right
- A single persuasion score describes a model on one task and should not be read as a measure of persuasiveness across settings, even when refusals are set aside.
- For the most capable models, which safety regulation targets, the choice of evaluation method may largely determine the ranking: agreement among the five API models is near zero (mean $\rho = 0.00$).
- Refusals, including indirect ones, should be reported separately from persuasive success, because a low manipulation score can reflect willingness to decline rather than inability to persuade.
- Most rational-persuasion methods track general capability, so two methods can agree mainly because both measure capability; reporting a capability correlation alongside each persuasion score is necessary to read either.
- Safety assessments should use several tasks chosen for the risk in question, particularly for manipulation, where different models decline different tasks and rankings diverge most.
Reading between the lines
- If the ability-willingness split holds, persuasion-risk regulation should ask at which training stage a score was produced: only evaluation of pre-safety-training models reads ability cleanly, while scores from released models are inseparable blends of the two.
- Because several automated methods are modeled directly on human-study paradigms, the fragmentation logic is testable in human settings; running two of the same task designs with human persuaders on matched content would show whether the near-0.25 agreement is a property of automated harnesses or of the construct itself.
- The near-zero agreement among the five API models may partly be a harness effect the paper itself flags as possible, since a shared weak persuadee can compress differences among strong persuaders; repeating the comparison with a more capable pinned persuadee would separate construct fragmentation from measurement compression.
- The general lesson likely extends beyond persuasion: any capability evaluated through simulated interactive tasks, such as negotiation, deception, or flattery benchmarks, can be expected to show the same task-dependence and the same conflation of willingness with ability.
Original abstract
Large language models (LLMs) have already been shown to match or exceed human experts in persuasion. While their persuasive capabilities hold promise for beneficial uses such as education and health communication, they can also be used to manipulate and misinform, making their evaluation a growing priority for developers and regulators. That evaluation, however, remains fragmented: studies differ in what they treat as persuasion, and broad claims often rest on narrow, situation-specific assessments. Automated methods, often modelled on human studies, offer a way to compare such assessments directly, as they can be run on the same models at scale and can include high-risk forms of persuasion that would be difficult or unethical to test on people. In this study, we adapt nine published automated methods to a shared setup, run them on the same fifteen LLMs, and ask whether their rankings agree and why. We find that the methods agree only weakly (mean Spearman $\rho = 0.25$). Our analyses point to two contributing factors. Models that refuse some tasks but not others, directly or indirectly, lower agreement by about a quarter, and these refusals fall mostly on manipulation tasks. General capability also plays a part: most rational persuasion (non-manipulative) methods track it, whereas most manipulation methods do not. Together, these findings suggest that agreement depends more on the task a method sets than on how it scores persuasion, although this pattern is only indicative given the eight methods available for analysis. More broadly, our results suggest that persuasion scores combine a model's ability to persuade with its willingness to do so. A single score is therefore informative about its own setting, but says little about a model's persuasiveness across tasks.
The compiled record
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
The abstract, sentence by sentence
Large language models (LLMs) have already been shown to match or exceed human experts in persuasion. While their persuasive capabilities hold promise for beneficial uses such as education and health communication, they can also be used to manipulate and misinform, making their evaluation a growing priority for developers and regulators. That evaluation, however, remains fragmented: studies differ in what they treat as persuasion, and broad claims often rest on narrow, situation-specific assessments. Automated methods, often modelled on human studies, offer a way to compare such assessments directly, as they can be run on the same models at scale and can include high-risk forms of persuasion that would be difficult or unethical to test on people. In this study, we adapt nine published automated methods to a shared setup, run them on the same fifteen LLMs, and ask whether their rankings agree and why. We find that the methods agree only weakly (mean Spearman $\rho = 0.25$). Our analyses point to two contributing factors. Models that refuse some tasks but not others, directly or indirectly, lower agreement by about a quarter, and these refusals fall mostly on manipulation tasks. General capability also plays a part: most rational persuasion (non-manipulative) methods track it, whereas most manipulation methods do not. Together, these findings suggest that agreement depends more on the task a method sets than on how it scores persuasion, although this pattern is only indicative given the eight methods available for analysis. More broadly, our results suggest that persuasion scores combine a model's ability to persuade with its willingness to do so. A single score is therefore informative about its own setting, but says little about a model's persuasiveness across tasks.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Large language models (LLMs) have already been shown to match or exceed human experts in persuasion. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | While their persuasive capabilities hold promise for beneficial uses such as education and health communication, they can also be used to manipulate and misinform, making their evaluation a growing priority for developers and regulators. | background 1.00 | 0.04 | 0.02 | asserted 0.59 |
| s2 | That evaluation, however, remains fragmented: studies differ in what they treat as persuasion, and broad claims often rest on narrow, situation-specific assessments. | gap 0.97 | 0.05 | 0.03 | asserted 1.00 |
| s3 | Automated methods, often modelled on human studies, offer a way to compare such assessments directly, as they can be run on the same models at scale and can include high-risk forms of persuasion that would be difficult or unethical to test on people. | background 0.84 | 0.05 | 0.02 | asserted 0.65 |
| s4 | In this study, we adapt nine published automated methods to a shared setup, run them on the same fifteen LLMs, and ask whether their rankings agree and why. | contribution 0.67 | 0.92 | 0.03 | asserted 0.90 |
| s5 | We find that the methods agree only weakly (mean Spearman $\rho = 0.25$). | result 1.00 | 0.97 | 0.18 | asserted 1.00 |
| s6 | Our analyses point to two contributing factors. | interpretation 0.63 | 0.63 | 0.03 | asserted 0.83 |
| s7 | Models that refuse some tasks but not others, directly or indirectly, lower agreement by about a quarter, and these refusals fall mostly on manipulation tasks. | result 0.93 | 0.94 | 0.09 | asserted 1.00 |
| s8 | General capability also plays a part: most rational persuasion (non-manipulative) methods track it, whereas most manipulation methods do not. | result 0.81 | 0.93 | 0.10 | asserted 0.95 |
| s9 | Together, these findings suggest that agreement depends more on the task a method sets than on how it scores persuasion, although this pattern is only indicative given the eight methods available for analysis. | interpretation 0.99 | 0.89 | 0.47 | hedged possibility 0.99 |
| s10 | More broadly, our results suggest that persuasion scores combine a model's ability to persuade with its willingness to do so. | interpretation 1.00 | 0.85 | 0.45 | hedged possibility 0.86 |
| s11 | A single score is therefore informative about its own setting, but says little about a model's persuasiveness across tasks. | interpretation 0.99 | 0.74 | 0.47 | asserted 0.93 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
Figures
Figures reproduced from arXiv:2610.10232 by the authors.
References
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.