Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Non-invasive probes after every utterance turn Mafia into a live instrument for measuring LLM agents' private social beliefs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 10:14 UTC pith:DBYV2PCG

load-bearing objection Real open instrument for interactive machine ToM; the non-invasive contract is only fully clean for stateless APIs, but the tooling and scoped DeepSeek demo are solid enough for a demos track. the 2 major comments →

arxiv 2607.10645 v2 pith:DBYV2PCG submitted 2026-07-12 cs.CL cs.MAcs.SI

MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games

Ilia Karpov This is my paper
classification cs.CL cs.MAcs.SI
keywords machine Theory of Mindsocial deduction gamesLLM multi-agent systemsbelief probingcalibrationcounterfactual replayMafiaself-report faithfulness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Public behaviour by language-model agents in social deduction games reveals almost nothing about the beliefs that produced it: a correct vote may be a guess, and a successful lie leaves no record of what the liar actually thought. This paper presents MafiaScope, an open testbed that freezes the game after every public message, privately asks every living agent a configurable battery of structured questions about roles, suspicions and second-order attitudes, then discards the answers so they never re-enter any agent's context. Because the engine already knows the hidden roles, every answer can be scored automatically, producing hundreds of time-stamped belief snapshots per game. An interactive viewer renders those trajectories from any agent's point of view, charts accuracy and calibration along the timeline, and forks any recorded step for counterfactual replay. A thirty-two-game case study shows that stated confidence is poorly calibrated, that agents systematically over-predict being suspected, and that probed rankings track subsequent votes—measurements outcome-only evaluation cannot see.

Core claim

The paper shows that privately interrogating every LLM agent after every public utterance in Mafia—on a throw-away copy of its message list whose answers never re-enter game context—and scoring the answers against engine ground truth yields dense, self-scoring belief trajectories that expose first-order accuracy, second-order consistency, calibration error and vote coupling invisible to win rates or vote accuracy alone.

What carries the argument

Non-invasive, time-resolved belief probing: after each public utterance the engine pauses play, questions every alive agent on a disposable context copy, never writes the answers back, and auto-scores first-order role claims against known roles and second-order claims against the targets' simultaneous first-order reports.

Load-bearing premise

That an agent's private probe answers faithfully track the decision-relevant beliefs that drive its public moves, rather than unfaithful verbalizations left untested by the throw-away context.

What would settle it

If pre-vote probe rankings fail to predict the agent's subsequent vote above chance while post-vote rankings match it almost perfectly, or if re-probing the identical information state under nonzero temperature yields uncorrelated answers, the claim that probes measure usable beliefs fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Belief flips can be localized to a single utterance and their causal weight estimated by forking the game before and after that utterance.
  • Calibration error and second-order over-prediction become first-class, automatically scored metrics for multi-agent social reasoning.
  • The released engine, viewer and 200-plus-game corpus let researchers compare models under identical information asymmetry.
  • Probe reports that track votes supply an operational check that self-reports are not idle text.
  • Second-order suspicion over-prediction becomes a measurable machine counterpart of the human spotlight effect.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same out-of-band probe design could be dropped into other hidden-role or multi-party negotiation settings to audit whether agents maintain coherent models of partners over time.
  • Exporting the belief stream as a temporal multigraph invites training of graph models that predict the next belief revision from dialogue alone.
  • If probe answers remain stable under temperature variation, the method could serve as a continuous internal monitor for deployed multi-agent systems rather than only a post-hoc testbed.
  • Counterfactual replay of pivotal utterances could later support automated search for language that reliably flips group beliefs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. MafiaScope is a systems demonstration that turns the social-deduction game Mafia into a measurement instrument for machine Theory of Mind. After every public utterance the engine privately probes every alive agent with a configurable YAML set of structured questions; answers never re-enter any agent’s persistent context and are auto-scored against engine-known roles (first-order) or contemporaneous peer reports (second-order). The release comprises the game engine, probe DSL, interactive D3 visualizer (impersonate mode, calibration/metrics panels, fork-and-replay), and a 200+ game corpus. A 32-game DeepSeek case study (13 815 parsed answers) reports belief trajectories (F1), ECE 0.168 (F2), 1.53× over-prediction of being suspected (F3), vote-coupling rates (F4), and a 30-fork reroll-only attribution walk-through (F5/§6).

Significance. If the non-invasive contract holds, the work supplies a missing measurement layer for multi-agent LLM research: dense, time-resolved, automatically scored belief trajectories that outcome-only evaluations (win rates, vote accuracy) cannot see. Strengths that raise the contribution above a pure demo include the open MIT release of engine + viewer + pinned corpus, the explicit second-order scoring design, the cluster-bootstrap CIs, the wording ablation for F3, and the end-to-end counterfactual replay workflow. These are concrete, reusable artefacts for the ToM and multi-agent communities rather than one-off numbers about a single model.

major comments (2)
  1. §4 states that for stateless API backends the probed game is distributed identically to an unprobed one “up to provider-side effects we cannot observe,” and that for local backends batch scheduling may still differ. The case-study corpus is DeepSeek API, yet the released engine and 200+ corpus explicitly support local HF/vLLM/Ollama and heterogeneous line-ups. No side-by-side comparison of win rates, vote sequences, or public-utterance statistics between probed and unprobed runs is reported for either backend class. Because the central claim is that the instrument measures the unprobed process, a distributional check (or an explicit scope restriction of the guarantee) is load-bearing and should be added or the claim narrowed.
  2. §9 and the F3 discussion correctly flag that probe answers are self-reports that may be unfaithful and that second-order scores are consistency measures, not ToM against engine ground truth. Test-retest reliability under nonzero temperature is unquantified. Given that F1–F4 treat the reports as operational belief measurements, at least a small repeated-probe reliability estimate (or a clearer operational caveat in the abstract/contributions) is needed so readers can judge how much weight the trajectories can carry.
minor comments (5)
  1. Table 2 / Appendix B: the round-3 cell is tiny (n=77 targets) and the paper already notes survivorship and probe-chaining confounds; a one-sentence reminder in the main-text F1 paragraph would keep the trajectory claim proportionate.
  2. §6 / Appendix E: the 30-fork experiment is correctly labelled illustrative (Fisher p≈0.52, N=5 per arm). The abstract’s phrasing “walks the counterfactual replay workflow end to end” is accurate; ensure the same caution appears wherever the point estimates are quoted.
  3. Table 1 capability comparison is useful; a short note on how the parenthesized partial marks were assigned (paper vs. repository inspection) would improve reproducibility of the table.
  4. Figure 4 and the ECE binning (width 20, conf 100 folded) are clear; stating the bin edges once in the main text (they appear only in Appendix C) would help readers who skip the appendix.
  5. Minor polish: arXiv id and submission track are given; ensure the live demo URL and screencast remain stable or are archived with the camera-ready version.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose belief scores rest on engine ground truth and contemporaneous self-reports, not on fitted parameters or self-citation chains.

full rationale

MafiaScope is a measurement-instrument and case-study paper, not a first-principles derivation. First-order role beliefs are scored against roles the engine already knows; second-order social-map predictions are graded against other agents’ same-step first-order reports (an operational consistency measure the paper itself labels as such). Calibration (ECE, Brier), over-prediction ratios, vote-coupling rates, and trajectory statistics are computed from these scored logs; none is obtained by fitting a free parameter and then re-predicting a quantity forced by that fit. The non-invasive guarantee is an engineering claim about throw-away context copies, not a mathematical uniqueness theorem imported from the authors’ prior work. Self-references to the instrument’s own design (probe DSL, viewer, replay) describe the system under evaluation; they do not force the empirical findings F1–F5. No ansatz is smuggled in via citation, no known empirical pattern is merely renamed, and no load-bearing uniqueness result is self-cited. The honest finding is therefore score 0: the derivation chain is self-contained against external engine ground truth and does not reduce predictions to inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

Load-bearing content is mostly engineering and measurement design, not free-parameter physics. The central claim rests on the non-invasive probe contract, the game engine’s ground-truth roles, operational treatment of self-reports, and a few scoring thresholds and budgets chosen for the case study. No new physical entities; invented pieces are software instruments whose independent handle is the released code and logs.

free parameters (4)
  • social-map attitude confidence threshold (canonical 50)
    Maps role-assessment confidence into suspects/trusts/neutral for F3; Table 3 shows agreement vs majority baseline flips with threshold 30/50/70.
  • probe token budget (960 in revised corpus; 400 legacy)
    Chosen configuration that eliminates mid-JSON truncation; F5 shows measurement quality depends on this budget.
  • ECE fixed-width bin size (width 20; conf 100 folded into top bin)
    Binning choice enters the reported ECE 0.168 and calibration narrative (F2 / Appendix C).
  • replay arm size N=5 rerolls per PRE/POST utterance
    Hand-chosen sample size for the 30-fork workflow demo; underpins the non-significant pivotality contrasts.
axioms (4)
  • domain assumption Probe Q&A runs on a throw-away copy of the agent message list and is never written back into any game-facing LLM call.
    §4 non-invasive guarantee; stated as checkable for stateless APIs, with residual provider-side and local batch-scheduling limits.
  • domain assumption The engine’s hidden role assignment is correct ground truth for scoring first-order role beliefs.
    Core self-scoring premise in abstract and §1; standard for a simulated Mafia engine.
  • ad hoc to paper Probe answers are treated as operational belief reports with checkable predictive structure, not as philosophical belief states.
    §2 and §7; F4 (vote coupling) is offered as partial behavioral validation; faithfulness remains open (§9).
  • domain assumption Classic Mafia loop (night kill/doctor save, day discussion, public vote) with fixed role multiset induces the information asymmetry needed for ToM measurement.
    §3 game engine; standard game rules assumed throughout.
invented entities (3)
  • MafiaScope non-invasive time-resolved probe engine (YAML DSL with conditional triggers, chaining, budgets) independent evidence
    purpose: Produce dense, out-of-band, auto-scored belief logs without contaminating play.
    Central software artifact; independent handle is the released engine and JSONL logs, not an external physical prediction.
  • Impersonate-mode second-order gap view (believed vs actual same-step reports) independent evidence
    purpose: Make second-order error and deception success visually and numerically inspectable.
    Viewer centrepiece (§5); evidence is the shipped SPA and scored panel headers.
  • Reroll-only counterfactual fork-and-replay attribution workflow independent evidence
    purpose: Turn correlational belief flips into PRE/POST outcome distributions for candidate utterances.
    §6; currently reroll-only without intervention editing; independent evidence is logged forks, not causal identification theorems.

pith-pipeline@v1.1.0-grok45 · 19357 in / 3477 out tokens · 56217 ms · 2026-07-14T10:14:34.179449+00:00 · methodology

0 comments
read the original abstract

An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind. It distinguishes whether an agent lost because it misread the game or because it failed to act on a correct assessment, a distinction that is invisible from outcomes and dialogue transcripts alone. After every public utterance, each agent privately answers structured probe questions whose responses never re-enter the game and are scored against the ground truth known to the engine. An interactive visualizer replays games from the perspective of an individual agent's beliefs, displays timeline-aligned accuracy and calibration, and supports counterfactual replay from any recorded step. In a case study across two model families comprising tens of thousands of parsed probe responses, we find that stated confidence is poorly calibrated, agents overestimate how often they are suspected by a factor of 1.5, and single-vote counterfactual replays rarely change game outcomes: outcome flips occur primarily when the agent had already formed a correct belief state, whereas decisions made under an incorrect model of the world remain largely unchanged under resampling. The engine, visualizer, recorded games, and counterfactual replay corpus are released under an open-source licence. Code: https://github.com/karpovilia/mafiascope. Live demo: https://karpovilia.github.io/mafiascope/. Screencast: https://vimeo.com/1208920221.

Figures

Figures reproduced from arXiv: 2607.10645 by Ilia Karpov.

Figure 1
Figure 1. Figure 1: The viewer on an English demo game: ground [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Impersonate mode: the game as Mafia agent [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The in-viewer metrics panel, aligned with the timeline: per-agent first-order accuracy, crowd Mafia [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The calibration view for a single game: stated [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The viewer’s branch dialog: any timeline step [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game

    cs.CL 2026-07 conditional novelty 6.0

    Frontier LLMs win Secret Hitler matches and can deceive, but most fail to keep a consistent false persona as evidence accumulates, with DRR often falling below 50%.