{"id":"3fac7bd5-38c1-4a07-9f32-f4e179881dad","arxiv_id":"2506.07211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a 25-participant Werewolf-style game, all roles used an LLM chatbot strategically, as a sword for disinformation and a shield against it.","lead":"In a five-player online forum game modeled on Werewolf, 25 participants used a chatbot while playing Disinformer, Moderator, or User roles. The study finds that LLMs served as strategic advisors for both spreading and countering disinformation, not just as content generators.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Strategic-advisor claim depends on broad, unreliability-checked coding of 'Chatbot as strategist' (Table 4, §4.5); with only five Disinformers, one participant's repeated prompts could drive the headline finding.","rationale":"The reader's weakest assumption was external generalizability from the Werewolf game to real-world disinformation actors. That is a legitimate concern, and Section 6 explicitly acknowledges it. However, the more immediately load-bearing assumption is internal: the distinctiveness of the paper's central claim rests on the 'Chatbot as strategist' coding category. Even if the game setting is accepted as a valid simulation, the claim that LLMs serve as strategic advisors beyond content generation depends on the reliability and distribution of that qualitative code. The paper gives rich logged data that could settle this, but no inter-rater reliability or per-participant breakdown is reported. This is not an accusation of fraud or bias; it is a call for a standard transparency check in qualitative HCI research. If the re-coding test passes, the conditional verdict remains appropriate and the main residual worry is external validity. If the test fails, the central novelty is unverified. Either way, the check is cheap because the chatbot transcripts are already in hand.","tokens_in":31424,"tokens_out":3153,"duration_ms":41570,"concrete_test":"Re-code the complete logged chatbot transcripts for all five Disinformers, and for completeness the Moderator and User prompts coded as 'Chatbot as strategist,' into mutually exclusive categories: (S) planning actions to conceal identity, shift suspicion, or coordinate future steps; and (C) drafting or refining content, gathering facts, or summarizing. Use two independent coders blind to the paper's theme list and report per-participant category counts plus Cohen's kappa. If kappa is below 0.60, or if one Disinformer accounts for more than half of all 'strategist' prompts, the §5.1 strategic-advisor generalization is not supported by the current data.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in §1 and §5.1 is that LLMs act as strategic advisors, not mere content generators or detectors. The main quantitative support for this novelty is Table 4's 'Chatbot as strategist' category, defined as 'Use chatbot for ideas on how to play the role,' with n=27 (27%) of Disinformer chatbot interactions. This definition is broad enough to include ordinary content-generation queries, such as asking how to phrase a post or how to argue a point. The paper reports no codebook, no category boundary rules, and no inter-rater reliability for the reflexive thematic analysis in §4.1. Because there are only five Disinformers, the aggregate 27% could be dominated by one participant: G5D is quoted in §5.1.1 as giving extensive role-playing prompts and explicitly treating the chatbot as a strategist throughout the game. If the 'strategy' coding is subjective or driven by a single participant, the distinction between 'strategic advisor' and 'content tool' collapses, and the paper's central novelty reduces to the already-known capability of LLMs to help draft text. The strategic-advisor claim is thus load-bearing on a coding procedure that is not demonstrated to be reproducible, and the stated limitation in Section 6 does not address this internal evidentiary question.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a qualitative empirical study (n=25, five five-player groups) in which participants played a custom Werewolf-inspired online forum game with a built-in, uncensored LLM chatbot. Disinformers, Moderators, and Users each pursued role-specific goals, and the authors collected forum interactions, chatbot prompts and responses, think-aloud protocols, surveys, and interviews. Using reflexive thematic analysis, they identify role-specific chatbot use cases and argue that LLMs serve not only as content generators and verifiers but also as strategic advisors that help both malicious and defensive actors plan role-appropriate actions. The paper further describes how group dynamics moderate LLM influence, reports obstacles to effective LLM use, and draws implications for platform design and policy.","tokens_in":31656,"tokens_out":5941,"duration_ms":71413,"significance":"If the central claim holds, the paper broadens the current understanding of LLM-assisted disinformation beyond content generation and detection to include strategic coaching of actors on both sides. The study is valuable for its direct observational design: real-world disinformers are difficult to study, and the game environment allows the authors to capture intentions and adaptive strategies, triangulated from forum logs, chatbot transcripts, think-alouds, interviews, and surveys. The case summaries and participant quotes give the paper concrete texture. The authors are also appropriately candid about the game's external validity in Section 6. However, the paper's headline finding about the chatbot as a 'strategic advisor' rests on a thematic category whose coding is not demonstrated to be reproducible, and the effectiveness claims are not systematically tied to game outcomes. These are load-bearing gaps for the stated contributions.","major_comments":[{"comment":"The paper's central novelty—that LLMs serve as strategic advisors, not merely content generators or verifiers—rests on the code 'Chatbot as strategist' (n=27, 27% of Disinformer interactions). The Table 4 definition ('Use chatbot for ideas on how to play the role') is broad enough to include asking for phrasing, counterarguments, or role-relevant content, and no codebook, category-boundary rules, or inter-rater reliability check (or a documented audit trail) are reported for the reflexive thematic analysis. Because the paper presents these counts as quantitative support for the claim, coding consistency is not merely a methodological nicety. Moreover, with only five Disinformers, one participant (G5D) may account for a large share of the 27 'strategist' interactions; G5D is repeatedly quoted as treating the chatbot as a strategist, and Figure 5 does not provide per-participant counts. The manuscript therefore does not rule out the possibility that the aggregate 27% is driven by a single participant's extended role-playing prompts. Please report per-participant counts for the categories in Table 4, provide a more operationalized coding scheme with boundary examples, and either add an inter-coder reliability check or justify why the quantitative percentages can stand without one; if the reliability evidence cannot be supplied, the 'strategic advisor' claim should be reframed as a hypothesis rather than a demonstrated finding.","section":"§4.1, Table 4, Figure 5, §5.1.1"},{"comment":"The paper claims to uncover 'varying efficacy' of LLMs depending on role and strategy, and §5.3 asserts that successful Disinformers exercised greater control over chatbot outputs and 'performed better, receiving fewer suspicion votes (G4D, G3D, G5D)'. However, no analysis systematically links chatbot-use categories to objective game outcomes such as detection, suspicion votes, or stance shifts. The group summaries in Tables 3 and 5 show that outcomes were heavily influenced by confounds (e.g., G4U2's disruptive behavior, G5U1's emotional posts), and Section 6 itself reports only minute stance shifts. With n=5, the success pattern in §5.3 is post hoc and not quantified. Please either provide a systematic within-case comparison that connects chatbot-use patterns to outcome measures (for example, per-round suspicion votes alongside chatbot interaction counts and qualitative evidence of causal influence) or explicitly downgrade the effectiveness claims to exploratory observations suitable for hypothesis generation.","section":"§4.5, §5.3"}],"minor_comments":[{"comment":"The first sentence of Section 6 contains a typo: 'a inherent limitation' should be 'an inherent limitation'.","section":"§6"},{"comment":"In the Disinformer row for 'Mitigate risk of detection', the text reads 'as as deceptive'; the duplicated 'as' should be removed.","section":"Table 4"},{"comment":"The sentence 'Following up with with \"what is this year\"' contains a doubled 'with'; please correct it.","section":"§5.2.2"},{"comment":"The bar charts would be easier to evaluate if the exact counts per group were printed on the bars or reproduced in an appendix table, since the aggregate percentages currently conceal the per-participant distribution that is critical to the 'strategist' claim.","section":"Figures 5–7"},{"comment":"The 'sword and shield' metaphor is not introduced until Section 5; a brief anticipatory mention in the introduction would help readers map the central metaphor onto the paper's structure.","section":"§1"},{"comment":"The phrase 'the game has five phases in each round' could be clarified by stating the total duration (2 hours 30 minutes) and the per-phase durations in Figure 1, since the figure does not include numeric durations in the text.","section":"§3.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope well. The main risk is overclaiming from a small qualitative dataset: the strategic-advisor finding is intriguing but currently depends on a coding category whose reliability and per-participant distribution are not shown. I would encourage the editor to treat the requested reliability/audit evidence and the outcome-based effectiveness analysis as condition for acceptance. The authors' existing limitations section is a positive signal, but it does not address the internal evidentiary question about the coding of chatbot use."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper earns its central claim, but it overreaches in the quantitative framing. The genuinely new thing is the observation that participants used the LLM as a strategy coach—planning how to avoid suspicion, how to cast blame, how to draw out a deceiver—rather than only as a text generator or fact-checker. That is a real empirical finding, and I have not seen it demonstrated with human players in a disinformation-adjacent setting before.\n\nWhat it does well: the Werewolf-style forum game is a sensible way to elicit intentions and adaptive tactics without pretending to recruit actual disinformers. The authors triangulate forum logs, think-aloud, observations, and interviews, and the quotes are specific enough to make the strategic-advisor pattern concrete. G5D's long prompt about blending in is the kind of evidence that grounds the claim. The paper is also candid about the main external limitation in Section 6: non-professional actors and checklist-driven behavior may not generalize to real-world disinformation.\n\nSoft spots: the internal coding evidence is weaker than the claim. The reflexive thematic analysis has no codebook, no category boundary rules, and no inter-rater reliability. The Table 4 definition of 'chatbot as strategist'—'use chatbot for ideas on how to play the role'—is broad enough to include ordinary drafting or argument-generation prompts. With only five Disinformers, the aggregate 27% figure could be heavily influenced by G5D's extensive prompting; the authors should report per-participant counts and show that 'strategy' is distinct from 'content creation.' The stress-test note is right about that gap, though it overstates the consequence: strategic use is also visible among Moderators and Users, so the central claim does not collapse even if the Disinformer percentage is inflated. Effectiveness claims are descriptive rather than measured, and the longer design/policy discussion is speculative, but it is clearly framed as implications rather than results. The citation pattern looks fine; the one self-citation is peripheral.\n\nBottom line: an exploratory qualitative paper with a real contribution and addressable weaknesses. A serious referee should engage with it. I would bring it to a reading group and would cite it if I were writing about human-LLM interaction in disinformation contexts.","headline":"A small Werewolf study with a real finding—participants use LLMs as strategy coaches, not just content tools—though the coding needs more rigor before the percentages carry weight.","tokens_in":32199,"tokens_out":3628,"would_cite":true,"duration_ms":44633,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs act as strategy coaches for deceivers and defenders in a Werewolf-style forum game.","keywords":["Disinformation","Influence","Manipulation Strategies","Chatbot Uses","Large Language Model","Communication Game","Werewolf","LLM-Assisted Disinformation"],"falsifier":"If a similar game (or an analysis of real-world influence-operation logs from platform takedowns) showed no strategy-seeking chatbot prompts—no requests for advice on deceiving, deflecting suspicion, or coordinating—the strategic-advisory claim would be exposed as an artifact of the role-play and checklist design.","tokens_in":31259,"feed_emoji":"🐺","tokens_out":5877,"duration_ms":62331,"temperature":0.7,"pith_summary":"This paper argues that large language models are used in disinformation settings not only to generate false content or verify claims, but also as strategic advisors. In a controlled Werewolf-inspired forum game with 25 participants, Disinformers, Moderators, and Users all turned to a built-in chatbot for advice on how to play their roles: how to deceive without detection, how to shed suspicion, and how to identify the liar. The authors read this as evidence that LLM assistance to disinformation is broader than content production and detection, extending into planning and coordination for both malicious and virtuous actors. They also find that the chatbot's influence depends on group dynamics, with critical engagement exposing LLM-generated disinformation and distraction or rapport-building letting it pass.","feed_headline":"LLMs act as strategy coaches for deceivers and defenders","feed_subtitle":"In a Werewolf-style forum game, players used chatbots to plan deception, dodge suspicion, and hunt the liar.","key_machinery":"The central mechanism is a Werewolf-inspired communication game played on a custom online forum platform. Five players per group—one Disinformer, one Moderator, three Users—discussed locally relevant topics with fixed agendas, while all had continuous access to an uncensored open-source chatbot. The game's asymmetric-information structure and deception goals make hidden intents observable; forum posts, votes, reports, chatbot prompts and responses, think-aloud commentary, and post-game interviews were then analysed with reflexive thematic analysis, using an influence guide adapted from established disinformation and persuasion techniques to support participants. This setup lets the authors directly observe the strategies users employ with LLM assistance in a controlled but dynamic setting.","core_discovery":"The study's central discovery is that LLMs act as sword and shield simultaneously: they serve as strategic advisors for every role in a disinformation ecosystem, not merely as informational tools. Disinformers most frequently used the chatbot as a strategist (27% of their interactions), asking how to push an agenda while staying hidden, how to frame false claims so they survive fact-checking, and how to shift blame; Moderators and Users used it to verify claims, identify suspicious content, and seek guidance on drawing out the Disinformer or clearing their own name. The paper's contribution is the mapping of these use cases across roles, showing that LLM-based assistance includes coaching on deception, concealment, and detection, and that its effectiveness is mediated by social dynamics such as group scrutiny, emotional tone, and rapport-building.","pith_inferences":["If strategic-advisory use generalizes beyond the lab, safety evaluations of LLMs should test not only refusal to generate disinformation but also refusal to coach deception plans under role-play.","The game's small, intimate setting maps more naturally to local online communities than to mass-scale influence campaigns; the strategic-advisory role may matter most where trust and rapport are built through sustained interaction.","A testable extension would compare Disinformers who sought strategic advice against those who only generated content, to isolate whether coaching actually improves concealment or whether it is the Disinformer's editing and social skill that matters."],"forward_implications":["Disinformation detection should monitor not only AI-generated content but also strategy-seeking interactions, since both malicious and defensive actors use LLMs for planning.","Groups that critically engage with chatbot output are more likely to catch LLM-generated disinformation, suggesting that platform designs promoting scrutiny can blunt the sword.","The effectiveness gap between uncensored open-source models and guardrailed commercial models may create a differential where defenders have access to better tools than typical malicious actors.","Platform design should foster appropriate reliance on LLMs, balancing verification tools with transparency and preserving space for subjective opinion."],"supporting_citations":[{"why":"Supplies the reflexive thematic analysis method used to derive the paper's central themes from forum logs, think-aloud, and interviews.","marker":"[11]"},{"why":"Provides the disinformation techniques adapted into the influence guide given to all players, shaping observed strategies.","marker":"[99]"},{"why":"Provides the persuasion principles that the influence guide adapts, informing the strategies players could deploy.","marker":"[98]"},{"why":"Grounds the Werewolf game mechanics and rationale for using social deduction to study communication and hidden roles.","marker":"[130]"},{"why":"Provides the critical incident technique used in post-game interviews to probe specific chatbot-related behaviours.","marker":"[27]"},{"why":"Documents the synthetic dataset behind the uncensored chatbot model, relevant to knowledge-cutoff and hallucination limitations.","marker":"[113]"},{"why":"Supplies the base model whose capabilities and reasoning limits the study discusses for open-source LLMs.","marker":"[52]"},{"why":"Frames the prior assumption that LLM use in disinformation is mainly informational, which the strategic-advisory finding extends.","marker":"[18]"},{"why":"Describes the double-edged-sword view of LLMs in disinformation that the study's sword-and-shield finding operationalizes.","marker":"[105]"}],"fun_headline_variants":["LLMs coach both deceivers and defenders in disinformation","In Werewolf-style game, LLMs help liars and truth-seekers alike","LLMs act as dual-use tools: strategy for deception and detection","Study maps how LLMs aid disinformers, moderators, and users"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study assumes that a role-playing game with non-professional participants, assigned checklists, and a chatbot is a trustworthy stand-in for how real-world disinformation actors and defenders actually behave.","fun_headline_variants_meta":{"raw":{"variants":["LLMs coach both deceivers and defenders in disinformation","In Werewolf-style game, LLMs help liars and truth-seekers alike","LLMs act as dual-use tools: strategy for deception and detection","Study maps how LLMs aid disinformers, moderators, and users"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000287,"raw_usage":{"total_tokens":1648,"prompt_tokens":869,"completion_tokens":779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":700}},"tokens_in":485,"tokens_out":779,"duration_ms":7469,"temperature":1.0,"reasoning_tokens":700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:38:46.761333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a similar game (or an analysis of real-world influence-operation logs from platform takedowns) showed no strategy-seeking chatbot prompts—no requests for advice on deceiving, deflecting suspicion, or coordinating—the strategic-advisory claim would be exposed as an artifact of the role-play and checklist design.","supporting_citations":[{"cited_title":"2023.OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants","cited_arxiv_id":null,"evidence_quote":"Documents the synthetic dataset behind the uncensored chatbot model, relevant to knowledge-cutoff and hallucination limitations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the double-edged-sword view of LLMs in disinformation that the study's sword-and-shield finding operationalizes."}],"review_version":1}