{"id":"f64cf3e3-32fa-4dd8-9cf8-3ad065079561","arxiv_id":"2506.06800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An adaptive preference-optimization method helps LLM persuaders choose among 11 psychological strategies, improving persuasion success on counterfactual facts while preserving general capability.","lead":"Large language models can be prompted to use psychological tactics such as authority or repetition to persuade other AIs to accept false facts, but no single tactic works in every situation. This paper trains models with preference optimization to choose tactics adaptively, and reports higher success on a counterfactual benchmark without lowering MMLU scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.3's Eq. 2 is not a valid DPO loss as written: it omits the generated message, the policy model, and the reference model, so the central training mechanism is unverified until the released code is checked.","rationale":"The paper's central empirical claim rests on the validity of the preference-optimization step. Eq. 2 is the only formal statement of that step, and it is internally inconsistent: it conditions on neither the message nor the policy nor a reference model. This is more load-bearing than the absence of significance tests or the GPT-4o-based strategy labels, because if the objective is mis-specified the mechanism may be doing something entirely different (e.g., exposure to successful examples via supervised fine-tuning). The code repository is the natural arbiter: the manuscript says code is available, so inspecting the actual loss is a concrete, low-cost check. If the code implements canonical DPO, the concern reduces to a notation error and the verdict should remain conditional pending the emendation; if it implements Eq. 2 literally, the central mechanism would be unverified and the appropriate verdict would be unverdictable without a corrected loss definition. I agree with the reader's identification of this as the weakest assumption.","tokens_in":20389,"tokens_out":5546,"duration_ms":57115,"concrete_test":"Check the training code at https://github.com/KalinaEine/PsychologicalPersuasion and print the exact loss used for the 3,000-sample DPO step. Verify (a) that log-likelihoods are computed over persuasive messages m (or strategy-conditioned outputs) given (s,r,o∗), and (b) that a reference model π_ref is included as in the canonical DPO objective. If both hold, re-run the LLaMA-3.1-8B-Instruct training with the canonical loss and compare the 'No Strategy' row of Table 3; if the gain disappears or the loss expression matches Eq. 2 literally, the claimed adaptive-selection mechanism is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (2) in Section 3.3 defines L_DPO = -E log σ(β (log p_π+(o∗|s,r) / p_π−(o∗|s,r))). This is not the direct preference optimization objective. The canonical DPO loss (Rafailov et al., NeurIPS 2023, Eq. 5) is over preferred and dispreferred responses y_w, y_l given prompt x, and includes both the learned policy π_θ and a frozen reference policy π_ref: -E log σ(β (log π_θ(y_w|x)/π_ref(y_w|x) - log π_θ(y_l|x)/π_ref(y_l|x))). In Eq. 2, the arguments are p_π+(o∗|s,r) and p_π−(o∗|s,r): probabilities of the counterfactual object conditioned only on subject and relation. The persuasive message m generated by the persuader does not appear, the policy being trained does not appear, and the reference model does not appear. Taken literally, the loss is not over strategy-specific persuasive responses at all, so Table 3's before/after PSR gains cannot be attributed to DPO-based strategy selection. The surrounding text says 'all messages are generated in-house by Mp', suggesting the authors intended to optimize message likelihoods, but that intent is not captured in Eq. 2. The released repository may implement the correct loss; if so the paper's formal description is inaccurate but the empirical claim could survive. If the implementation literally matches Eq. 2, the Section 4.4 claim that adaptive DPO training improves persuasion is unsupported by the reported training objective.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether LLMs can autonomously persuade and resist persuasion on counterfactual edits from the COUNTERFACT dataset. It first measures baseline dual capabilities of four LLMs, then introduces eleven psychological persuasion strategies and shows that strategy effectiveness varies by model and by semantic domain, and finally fine-tunes three open-source models with a DPO-style objective on strategy-specific preference pairs to learn adaptive strategy selection. The main claims are that DPO-trained models improve persuasion success rates on held-out data, choose a more diverse set of strategies without explicit instructions, and maintain general capabilities as measured by MMLU. The empirical narrative is clear, but the formal training objective in Eq. (2) is not the standard DPO loss as written, and the headline gains lack confidence intervals or significance tests.","tokens_in":20696,"tokens_out":5735,"duration_ms":61876,"significance":"If validated, the adaptive framework is a useful contribution: it turns persuasion outcomes into preference pairs, reports systematic comparisons across four LLMs and four semantic domains, and includes an MMLU sanity check. The paper also provides a useful negative result that no static psychological strategy dominates across all contexts. The positive claim that 3,000 training examples suffice to improve autonomous strategy selection is plausible, but it currently rests on unvalidated GPT-4o annotations, internally inconsistent percentages in Table 4, and a training-objective equation that does not match standard DPO. These issues need to be resolved before the central empirical claims can be accepted.","major_comments":[{"comment":"The DPO objective as written compares log p_{π+}(o*|s,r) with log p_{π−}(o*|s,r). This is not the direct preference optimization loss: it does not contain the persuader message m, the trained policy's likelihood, or a frozen reference policy, all of which appear in the canonical DPO loss over preferred and dispreferred completions. Since Section 4.4's before/after PSR differences are the paper's central evidence, Eq. (2) must be corrected to the actual objective used in the released code. If the implementation literally matches Eq. (2), then the reported gains cannot be attributed to DPO-based strategy selection.","section":"§3.3, Eq. (2)"},{"comment":"The claim that adaptive training \"significantly enhances\" success rates is not supported by any confidence intervals, significance tests, or multiple-seed variance. The table contains several sizable decreases (Qwen Authority Effect −7.29, Qwen Argument Quantity −4.95, Falcon Authority Effect −4.74), and many increases are under two percentage points; without error bars it is impossible to tell which differences exceed noise. Please report bootstrap intervals or multiple training runs.","section":"§4.4, Table 3"},{"comment":"Several percentages in Table 4 are internally inconsistent: LLaMA Authority Effect \"After\" is reported as 131 (68.26%) and LLaMA Fluency Effect as 59 (30.75%), yet the column totals to roughly 100% only if these are read as 6.83% and 3.07%; Falcon Repetition Effect 5 (0.03%) and Scarcity Effect 4 (0.02%) show similar decimal errors. Since the post-training diversification claim is one of the two main adaptive-training results, these numbers must be corrected and the aggregate statistics recomputed.","section":"Table 4"},{"comment":"The semantic-domain partition in Table 2 and the strategy-usage analysis in Table 4 rely entirely on GPT-4o annotations with no validation. The prompt in Appendix C asks for a single strategy label, but no agreement rate, error analysis, or human-annotated sample is reported. Noisy labels could create the appearance of domain-dependent strategy effectiveness or of post-training diversification; please report annotation agreement on a sample and provide representative annotated examples.","section":"Tables 2 and 4"}],"minor_comments":[{"comment":"The terminology is inconsistent: \"Flattery Effect\" and \"Flattery Trap\" are used interchangeably, and \"Repetitive Effect\" appears where \"Repetition Effect\" is intended; please unify the strategy names throughout.","section":"§4.3"},{"comment":"Section 3.3 defines k strategy pairs but Section 4.1.2 states that five pairs are sampled per training instance; please define k and clarify whether pairs are sampled with or without replacement.","section":"§3.3 and §4.1.2"},{"comment":"The domain counts (844 + 516 + 393 + 161 = 1,914) do not sum to the stated 1,919 evaluation samples; please explain the discrepancy or correct the counts.","section":"Table 2"},{"comment":"The GPT-4o columns list \"Before\" values but no \"After\" values; since GPT-4o is not fine-tuned, this should be stated explicitly or the empty columns should be removed.","section":"Table 4"},{"comment":"The caption says the four LLMs play in a \"4 × 4 adversarial game,\" but Figure 2 appears to plot aggregate persuasion and resistance values; consider clarifying the relationship between Figure 2 and the pairwise heatmap in Figure 3.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the empirical direction is likely salvageable. The most important request is to reconcile Eq. (2) with the released code and to add uncertainty quantification for Table 3; without these, the central adaptive-training claim is not verifiable. I would not reject solely on the Eq. (2) discrepancy if the released code implements the correct DPO loss, but the current text is misleading and must be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper does something genuinely new. Instead of just showing that LLMs can be persuasive, it builds an explicit toolbox of eleven psychological strategies, measures them across four models and semantic domains, and then trains models to pick the strategy adaptively via DPO. The empirical finding that no single strategy wins everywhere, and that some strategies (Flattery Trap) reliably fail while others (Fluency Effect) reliably help, is a useful descriptive result. The held-out evaluation, the diversity analysis in Table 4, and the MMLU sanity check are the right kinds of evidence.\n\nNow the soft spots, in descending order of severity.\n\nFirst, Eq. 2 in Section 3.3 is not the DPO objective. As written, it compares pπ+(o*|s,r) with pπ-(o*|s,r): no generated message m, no policy being trained, no reference model. Canonical DPO compares the probabilities of preferred/dispreferred responses under the learned policy relative to a frozen reference. The surrounding text says all messages are generated in-house, so the authors clearly intended something like π_θ(m|s,r,o*) vs π_ref(m|s,r,o*), but the equation doesn't say that. This matters because the Table 3 gains are attributed to DPO-based strategy selection. The released code should settle it; if the implementation is canonical DPO, then the formal description is just sloppy and the empirical claim can survive. If the implementation literally matches Eq. 2, the central mechanism is unsupported.\n\nSecond, no uncertainty estimates anywhere. The deltas in Table 3 are small—Qwen drops 7.29 points on Authority Effect and 4.95 on Argument Quantity, while LLaMA gains up to 12.67 on Flattery Trap. Without confidence intervals or significance tests, it is hard to distinguish real effects from noise. That is a fixable reporting gap.\n\nThird, GPT-4o is used as the annotator for both domain labels and strategy labels without validation. The domain taxonomy is coarse, and the strategy categorization in Table 4 is the entire basis for the \"more diverse strategy selection\" claim. A small human-annotated check would make this credible.\n\nFourth, there is no control that separates preference optimization from simply being exposed to successful responses during training. A baseline like fine-tuning on the winning strategy's messages would clarify the mechanism.\n\nThe limitations section is honest, and nobody is trying to hide these gaps. The central idea is plausible and the empirical scope is solid enough that a serious referee should see it. My recommendation: send it to review, but ask the authors to fix the DPO equation (or point to the code), add uncertainty estimates, and validate the GPT-4o annotations. If the code does what the text says it does, this is a decent contribution.","headline":"Worth reading and worth reviewing: real empirical work on adaptive strategy selection, but the DPO equation as written isn't the DPO objective, so the released code or a corrected formal description is needed before the central claim is fully supported.","tokens_in":21278,"tokens_out":3015,"would_cite":true,"duration_ms":30783,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a preference-optimization training loop over strategy-specific persuasive responses lets LLMs autonomously choose contextually effective psychological tactics, raising success rates on held-out counterfactuals after…","keywords":["psychological persuasion","large language models","direct preference optimization","counterfactual belief editing","persuasion success rate","epistemic resistance","strategy selection"],"falsifier":"Train the same three models with a corrected DPO objective that includes the persuasive message and a frozen reference model; if the Table 3 improvements over the unguided baseline disappear, the claimed mechanism of learning strategy preferences from persuasion outcomes is not what drives the gains. A second check is to shuffle the success/failure labels and see whether persuasion success still rises, which would indicate the effect is not preference-driven.","tokens_in":20142,"feed_emoji":"🧠","tokens_out":6196,"duration_ms":65702,"temperature":0.7,"pith_summary":"The paper asks whether large language models can not only produce persuasive rhetoric but also choose which psychological tactic fits a given counterfactual claim. It first shows that unguided persuader LLMs lean on repetitive, weak strategies, and that explicitly prompting one of eleven psychological strategies helps but that no single strategy wins across contexts. The central proposal is an adaptive training loop: for each counterfactual, the model's attempts under different strategies are scored by whether the listener accepts the alternate object, and direct preference optimization (DPO) is used to make the persuader favor strategies that worked. With 3,000 training examples, the fine-tuned models outperform their originals under almost every strategy instruction and shift toward a more diverse, context-dependent mix of strategies, while MMLU scores stay essentially flat. The paper concludes that adaptive strategy selection is a viable route to stronger and more flexible machine persuasion.","feed_headline":"3,000 examples teach LLMs to choose the best persuasion tactic","feed_subtitle":"Context-dependent strategy selection beats any single psychological trick and leaves general knowledge intact.","key_machinery":"The load-bearing mechanism is an adaptive preference-optimization loop built on direct preference optimization (DPO), a training objective that increases the probability of preferred responses and decreases dispreferred ones. Here the 'preferences' are not human labels but outcomes: for each counterfactual $(s, r, o^*)$, the persuader generates messages under several strategies, and the listener's acceptance or rejection turns each message into a win or loss. Successful and failed strategy-conditioned responses become the positive and negative examples in the DPO objective, with five such pairs sampled per training instance, and instances with no mix of success and failure are discarded. The authors claim this teaches the persuader to autonomously favor strategies that work in a given semantic context. The eleven-strategy toolbox (fluency, framing, authority, scarcity, repetition, conformity, flattery, information isolation, argument quantity, argument quality, affective forecasting) supplies the action space the model learns to choose from.","core_discovery":"On its own terms, the paper's discovery is that persuasion success in LLMs is a context-sensitive function of rhetorical strategy, and that this function can be learned. The authors define persuasion as updating a listener model's belief from a true object $o$ to a counterfactual object $o^*$ for a factual triple $(s, r, o)$. They assemble a toolbox of eleven psychologically grounded strategies and prompt LLMs to adopt each one; across four LLMs and four semantic domains (person, geography, culture, life), they find large but inconsistent gains, so no static strategy dominates. The paper then fine-tunes three open-source LLMs with DPO, constructing preference pairs from strategy-specific responses that did or did not win over the listener. After one LoRA epoch on 3,000 samples, the trained persuaders beat their untrained versions under nearly all strategy instructions, pick a broader and more effective strategy mix when left unguided, and retain their MMLU performance.","pith_inferences":["Beyond the paper: the same outcome-pair loop could be reversed to train listeners to resist persuasion, using failed persuasion attempts as negative examples and yielding an epistemic-resistance policy.","Beyond the paper: the measured strategy diversity depends on GPT-4o's classification of which strategy a message uses; a human-annotated or open-label variant could change Table 4.","Beyond the paper: if the adaptive mechanism holds up, it is dual-use, since the same 3,000-example recipe could be aimed at misinformation campaigns; safety work should explore contrastive training that keeps listeners rigid."],"forward_implications":["If the central claim is correct, explicit strategy prompting is not the ceiling: a model trained on its own persuasion outcomes can exceed every fixed strategy, including the best static one.","Adaptively trained persuaders remain within noise on MMLU, so the gains are not bought by a collapse of general reasoning.","Because the training signal comes from in-model interactions, the same pipeline can be applied to any open-weight model without external preference labels.","The learned behavior transfers to unguided generation, producing a more diversified set of strategies on held-out counterfactuals.","The result suggests that persuasion capability is better modeled as a per-context policy than as a fixed skill."],"supporting_citations":[{"why":"Supplies the COUNTER FACT dataset of counterfactual triples used for both training and held-out evaluation.","marker":"[22]"},{"why":"Supplies the DPO objective that the adaptive strategy-selection training is built on.","marker":"[24]"},{"why":"Supplies the 3,000-train / 1,919-evaluation split procedure.","marker":"[37]"},{"why":"MMLU benchmark used to verify that general capabilities are preserved after adaptive training.","marker":"[11]"},{"why":"LLaMA-3.1-8B-Instruct, one of the three open models fine-tuned in the adaptive framework.","marker":"[9]"},{"why":"Qwen-2.5-7B-Instruct, one of the three open models fine-tuned in the adaptive framework.","marker":"[43]"},{"why":"Falcon-3-7B-Instruct, one of the three open models fine-tuned in the adaptive framework.","marker":"[30]"}],"fun_headline_variants":["LLMs learn context-specific persuasion tactics via DPO","No one-size-fits-all persuasion: LLMs adapt strategies","Adaptive LLM persuasion outperforms fixed rhetorical tricks","Training LLMs to choose persuasion strategy per context","DPO helps LLMs pick optimal persuasion moves"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the printed DPO objective is a valid preference optimization over persuasive strategies; the equation as written compares probabilities of the counterfactual object given only subject and relation, omitting the persuasive message and any reference model, so the training signal it describes may not be the strategy-selection signal the paper relies on.","fun_headline_variants_meta":{"raw":{"variants":["LLMs learn context-specific persuasion tactics via DPO","No one-size-fits-all persuasion: LLMs adapt strategies","Adaptive LLM persuasion outperforms fixed rhetorical tricks","Training LLMs to choose persuasion strategy per context","DPO helps LLMs pick optimal persuasion moves"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1300,"prompt_tokens":955,"completion_tokens":345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":269}},"tokens_in":571,"tokens_out":345,"duration_ms":4012,"temperature":1.0,"reasoning_tokens":269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:49:35.231981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same three models with a corrected DPO objective that includes the persuasive message and a frozen reference model; if the Table 3 improvements over the unguided baseline disappear, the claimed mechanism of learning strategy preferences from persuasion outcomes is not what drives the gains. A second check is to shuffle the success/failure labels and see whether persuasion success still rises, which would indicate the effect is not preference-driven.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the COUNTER FACT dataset of counterfactual triples used for both training and held-out evaluation."},{"cited_title":"Rafailov, A","cited_arxiv_id":null,"evidence_quote":"Supplies the DPO objective that the adaptive strategy-selection training is built on."},{"cited_title":"Hendrycks, C","cited_arxiv_id":null,"evidence_quote":"MMLU benchmark used to verify that general capabilities are preserved after adaptive training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Falcon-3-7B-Instruct, one of the three open models fine-tuned in the adaptive framework."}],"review_version":1}