{"id":"acd057c0-6dd4-4fb2-b04e-b429ef31f5c4","arxiv_id":"2607.26836","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"HalluProp infers per-agent and system-level hallucination risk in multi-agent LLMs before interaction via role–query misalignment, topology-aware propagation, and differentiable Noisy-OR aggregation.","lead":"HalluProp predicts which agents in an LLM multi-agent system are likely to hallucinate before any of them speak, using role–query mismatch and communication topology. It offers a fast upstream screen that can prune or re-prompt risky agents and cut cascading failures without waiting for a full run.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Ground-truth labels are post-generation answer-match outcomes, so high AUROC may largely reflect role–query predictability of which agents fail rather than true pre-hoc foresight of cascading hallucination dynamics.","rationale":"The reader correctly flags that static role–query misalignment plus degree features under Noisy-OR independence may not track failures driven by decoding stochasticity or multi-round drift. The sharper load-bearing issue is label construction: yi is answer-match after generation/interaction, and roles are intentionally noisy, so the model can succeed as a role-quality ranker. That does not falsify a useful signal, but it weakens the specific claim that the method infers cascading hallucination risk before agents speak. Keeping CONDITIONAL is right; the concrete matched-role control would decide whether the contribution is truly pre-hoc cascade inference or mainly supervised detection of bad prompts. Artifacts and error bars remain necessary but secondary to this validity check.","tokens_in":18401,"tokens_out":601,"duration_ms":15470,"concrete_test":"Hold out a matched subset where every agent receives an identical clean, domain-aligned role (no noisy/mismatched prompts) and only topology and query vary; retrain/evaluate AUROC, Hit@1, and the pruning/refinement accuracy lift on that subset alone. If AUROC falls near chance or intervention gains vanish while the original mixed-role setting stays high, the headline localization claim is driven by role quality detection, not pre-hoc cascade risk.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that HalluProp localizes agents that will hallucinate before any inter-agent interaction (AUROC 84.6%, intervention parity with GUARDIAN). Labels yi are defined by comparing each agent’s generated output to the reference answer after three interaction rounds (Experimental Setup). Training uses y_node (pre-communication) and y_graph (post-communication) under the same answer-match rule (Eqs. 10–11). Thus the supervised target is ultimately “will this agent’s final answer be wrong,” not an independent measure of hallucination propensity or infection. Because roles are deliberately clean vs. noisy/mismatched (Role Construction), a frozen encoder + competence/susceptibility gate (Eqs. 2–5) can achieve high AUROC simply by ranking domain-mismatched or perturbed roles as high-risk—without needing the Noisy-OR multi-hop story (Eqs. 8–9) to be causally correct. Ablation “w/o Inter-Agent Propagation” still leaves 66.9 AUROC, consistent with a large intrinsic/role signal. If most predictive power is static role–query mismatch rather than pre-hoc inference of interaction-induced cascades, the “before agents speak / cascading failure” framing and the claim of complementary upstream screening overstate what the numbers establish.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes HalluProp, a pre-hoc framework that estimates agent-level and system-level hallucination risk in LLM multi-agent systems before any inter-agent interaction. Intrinsic risk is obtained from query-conditioned competence/susceptibility decomposition of role and query embeddings (Eqs. 2–5); inter-agent propagation risk combines semantic compatibility with degree-based topology factors (Eqs. 6–7); these are aggregated by K-step differentiable Noisy-OR fixed-point iteration on the communication graph (Eqs. 8–9). Two-stage training uses node-level then graph-level binary cross-entropy (Eqs. 10–11). On MMLU, MATH, and MedQA with four LLM backbones, the method reports average AUROC 84.6%, Hit@1 83.9%, sub-second diagnosis (>65× vs post-hoc baselines), and post-intervention accuracy comparable to GUARDIAN (pruning 63.6%, refinement 67.6%). Ablations attribute large drops to removing intrinsic, topology, or propagation modules.","tokens_in":18838,"tokens_out":1636,"duration_ms":52295,"significance":"If the pre-hoc localization claim holds under realistic role and topology variation, the work offers a practically useful upstream screening layer for MAS reliability: risk scores can guide pruning or role refinement before costly multi-round execution, with clear efficiency gains over post-hoc detectors. The combination of a structured competence–susceptibility prior, topology-aware edge risks, and end-to-end differentiable Noisy-OR aggregation is a coherent modeling contribution relative to purely response-based or trajectory-based MAS failure attribution. Strengths include multi-backbone/multi-dataset evaluation, intervention utility against four post-hoc baselines, scalability and canonical-topology checks, and ablations that quantify module contributions. The appendix’s monotone-convergence and gradient analysis of Noisy-OR is a welcome formalization. The main significance risk is that predictive power may largely track static role–query mismatch (especially under deliberately noisy/mismatched roles) rather than true foresight of interaction-induced cascades; resolving that distinction would determine whether the result is a strong cascade-risk tool or primarily a role-fitness","major_comments":[{"comment":"Experimental Setup and Eqs. 10–11: Ground-truth labels yi are defined by matching each agent’s output to the reference answer after three interaction rounds, and both y_node and y_graph are described under the same answer-match rule. How y_node (“before communication”) is obtained is not operationally specified—whether via isolated single-agent runs, a separate generation pass, or another proxy. Without a precise protocol, it is unclear that stage-1 supervision is truly pre-interaction, and the central “before agents speak” claim cannot be audited. Please define the exact generation and labeling procedure for y_node vs y_graph, including whether isolated runs use the same decoding settings and whether labels can leak multi-round context.","section":"§4 Experimental Setup; Eqs. (10)–(11)"},{"comment":"Role Construction (Appendix) deliberately injects domain-mismatched, underspecified, and adversarially perturbed roles. Under that design, high AUROC (Table 1) and a still-substantial 66.9 AUROC with propagation disabled (Table 4, “w/o Inter-Agent Propagation”) are consistent with ranking static role–query misfit rather than inferring cascading infection dynamics. The load-bearing narrative in the abstract and §1–3 (propagation-aware pre-hoc cascade diagnosis) therefore needs a control that holds: (i) all roles clean/well-specified, and/or (ii) fixed roles with only topology or query varied. Report AUROC/Hit and the intrinsic-vs-propagation ablation under those controls; if gains collapse when roles are clean, the cascade framing should be narrowed to role-risk screening plus a secondary graph aggregator.","section":"Appendix Role Construction; Table 4; Abstract"},{"comment":"Table 1–2 and Figure 3 report point estimates only (AUROC, Hit/Prec/Rec, post-intervention accuracy) with no standard errors, confidence intervals, or repeated seeds over topology/role sampling. Given randomized topologies (25–100% edge probability), role perturbations, and temperature 0.7 decoding for labels, variance is first-order for the average 84.6% AUROC and the pruning≈GUARDIAN parity claim (63.6% vs 63.5%). Please add multi-seed or bootstrap intervals and, where intervention gaps are small, a simple significance check so that “comparable to SOTA post-hoc” is statistically supported.","section":"Tables 1–2; Figure 3"},{"comment":"Eqs. 8–9 and Appendix “Causal Independence”: system risk hi depends on neighbor hj under conditional independence of intrinsic and infection causes. Labels remain final answer-match after multi-round dialogue, so hi is trained as a correlate of eventual wrongness, not of an observed infection event or intermediate adoption of a neighbor’s error. To support the propagation mechanism as more than a flexible graph smoother, provide at least one diagnostic: e.g., correlation of learned tj→i with measured error adoption along edges, or accuracy of predicting who fails only after a known upstream hallucinator is introduced. Absent that, tone down causal language (“infects,” “cascading”) to predictive risk aggregation.","section":"§3 System-Level Risk Inference; Eqs. (8)–(9); Appendix"}],"minor_comments":[{"comment":"Figure 2 and Eq. (9): notation mixes h_i^{k+1}, h^{(k)}, and final h; a single consistent superscript convention would help.","section":"Figure 2; Eq. (9)"},{"comment":"Table 4 caption says “Average AUROC” 84.8 while Table 1 average is 84.6; clarify whether the ablation subset or backbone differs.","section":"Table 4"},{"comment":"§4 MAS Configuration: “three rounds” of interaction is fixed; a brief sensitivity note on rounds would clarify label stability for post-hoc baselines and y_graph.","section":"§4 MAS Configuration"},{"comment":"Related Work is adequate but could more sharply separate single-agent uncertainty/consistency detectors from MAS topology methods when motivating pre-hoc vs post-hoc.","section":"§5"},{"comment":"Minor typography/spacing issues appear in the abstract and early sections (e.g., missing spaces in “newsystemicrisk”, “cascading hallucinationsetting”); a pass for PDF text extraction artifacts would improve readability.","section":"Abstract; §1"}],"recommendation":"major_revision","confidential_remarks":"The skeptic’s concern about static role–query predictability is the main reason I chose major_revision rather than minor_revision: the experimental recipe (noisy/mismatched roles + answer-match labels) can inflate AUROC without validating the cascade story. If the authors add clean-role controls, clear y_node protocol, and error bars, this could become a solid systems/reliability paper; if those controls fail, the contribution is still publishable but should be reframed. Venue fit for cs.CR is acceptable given the agent-communication security framing, though the core is reliability/ML rather than classical security."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a usable upstream risk scorer for multi-agent LLM setups, not a deep theory of cascading hallucination. The move that matters is diagnosing before any agent speaks—role/query misalignment plus a light topology-modulated propagation term, folded with differentiable Noisy-OR—and they back it with multi-backbone tables, interventions, and a real latency win.\n\nWhat is actually new is the pre-interaction framing plus a concrete end-to-end graph estimator (competence/susceptibility split, degree-gated semantic edges, fixed-point Noisy-OR). Post-hoc debate/inspector/SelfCheck/GUARDIAN work is cited fairly and contrasted on timing. Empirically they deliver average AUROC ~84.6%, Hit@1 often >80%, pruning accuracy roughly matching GUARDIAN (~63.6 vs 63.5), refinement higher (~67.6), and sub-second diagnosis versus full multi-round runs. Ablations move the needle in the right direction (intrinsic, topology, propagation each hurt when removed). Appendix Noisy-OR notes (independence, monotone fixed point, gradient gating) are standard but clean enough.\n\nSoft spots, in proportion. The stress-test lands partially: ground truth is answer-match after three rounds, and roles are deliberately clean vs noisy/mismatched, so a frozen encoder ranking domain-mismatched prompts already gets you far. “w/o propagation” still ~66.9 AUROC; propagation and multi-step K help, but they are not the whole story. So the “before agents speak / cascading failure” rhetoric slightly outruns what the labels isolate. Also no error bars, no released code/data in the manuscript, free parameters as usual for this genre, and MedQA is weaker than MMLU/MATH. None of that sinks the main engineering claim that pre-hoc scores can guide cheap prune/refine interventions.\n\nWho it is for: people building or securing MAS who need a fast firewall before rollout, not pure hallucination theorists. Math is ordinary supervised + Noisy-OR, citations look appropriate, data pattern is typical empirical agent work. I would send it to referees; it deserves a serious read and revision pressure on labels, variance, and how much signal is static role mismatch versus interaction dynamics. Engage if you care about MAS reliability tooling; skim if you only want causal accounts of cascade.","headline":"Solid pre-hoc MAS risk screen with real speed and localization numbers; the cascade story is only partly load-bearing because role–query mismatch already predicts a lot of the label.","tokens_in":19442,"tokens_out":590,"would_cite":true,"duration_ms":21700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Hallucination risk in multi-agent LLM systems can be scored before any agent speaks, by combining role–query misalignment with how errors would travel on the communication graph.","keywords":["multi-agent systems","hallucination detection","pre-hoc risk inference","Noisy-OR","propagation modeling","LLM agents","communication topology"],"falsifier":"Hold roles, query, and topology fixed, run the multi-agent system, and check whether agents HalluProp ranked highest-risk before any message was sent are systematically not the ones whose outputs disagree with the reference answer—especially when failures come from decoding noise or multi-round drift rather than role–query mismatch.","tokens_in":19289,"feed_emoji":"🛡️","tokens_out":931,"duration_ms":22146,"temperature":0.7,"pith_summary":"When several language-model agents talk to each other, a single wrong answer can be copied and amplified until the whole team fails. Today’s defenses usually wait until answers already exist, which is too late once the damage has spread. This paper argues you can instead estimate each agent’s failure risk and the system-wide cascade risk before any message is sent. HalluProp scores an agent’s intrinsic risk from fine-grained mismatch between its role prompt and the user query, scores how likely a bad answer would jump along each communication link from role semantics plus simple topology features, then folds both into a differentiable Noisy-OR over the graph. On three benchmarks and four model backbones it localizes agents that later hallucinate with average AUROC 84.6%, runs in under a second (over 65× faster than post-hoc baselines), and, when used to prune or re-prompt the riskiest agent, yields post-intervention accuracy on par with or better than strong after-the-fact methods. A sympathetic reader cares because upstream screening can reshape roles and edges before unsafe content ever circulates.","feed_headline":"Score which agents will hallucinate before they speak","feed_subtitle":"Role–query mismatch plus graph Noisy-OR yields 84.6% AUROC and 65× faster screening than post-hoc checks","key_machinery":"Differentiable Noisy-OR fixed-point inference on the agent communication graph: each agent’s final risk is one minus the product of its intrinsic safety and the safety of every incoming link, iterated a few steps from the intrinsic prior so both node risks and edge transmission probabilities are learned end-to-end.","core_discovery":"HalluProp shows that pre-execution, graph-level hallucination risk in multi-agent systems is predictable from two sources alone: intrinsic risk from query-conditioned competence versus susceptibility in each agent’s role embedding, and inter-agent propagation risk from semantic influence plus degree-based topology, combined by iterative differentiable Noisy-OR. That diagnosis localizes faulty agents at average AUROC 84.6% and supports interventions whose accuracy matches post-hoc baselines without observing any generation.","pith_inferences":["If pre-hoc scores are reliable enough, role libraries and topology templates could be optimized offline for low cascade risk before deployment.","The competence–susceptibility split suggests adversarial or underspecified system prompts are a primary controllable lever, not only model choice.","Combining this upstream filter with a cheap post-hoc check on only the top-ranked agents could cut total verification cost while covering both static and drift-driven failures."],"forward_implications":["MAS designers can rank and reassign or silence high-risk agents before the first round of dialogue.","Upstream screening can sit as a sub-second firewall in front of multi-round agent workflows without replacing runtime monitors.","Refining a risky agent’s prompt (making it cautious) can beat simply cutting it out of the graph while preserving collaboration.","The same Noisy-OR graph diagnosis applies across random, star, chain, and hierarchical topologies and across closed- and open-source backbones."],"fun_headline_variants":["Predict which agents will hallucinate before they speak","Pre-hoc scores flag faulty agents before any interaction","Spot cascading hallucination risk before agents communicate","Role mismatch plus graph Noisy-OR localizes failures early","Upstream screening finds hallucination-prone agents pre-execution"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Whether an agent will later output a wrong answer is largely readable from static role–query text misalignment and simple graph degrees, under the assumption that intrinsic failure and neighbor infections act as independent causes.","fun_headline_variants_meta":{"raw":{"variants":["Predict which agents will hallucinate before they speak","Pre-hoc scores flag faulty agents before any interaction","Spot cascading hallucination risk before agents communicate","Role mismatch plus graph Noisy-OR localizes failures early","Upstream screening finds hallucination-prone agents pre-execution"]},"model":"grok-4.5","effort":"low","cost_usd":0.003789,"raw_usage":{"total_tokens":1252,"prompt_tokens":822,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":37888000,"prompt_tokens_details":{"text_tokens":822,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":373,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":822,"tokens_out":57,"duration_ms":6100,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T19:37:47.778799+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Hold roles, query, and topology fixed, run the multi-agent system, and check whether agents HalluProp ranked highest-risk before any message was sent are systematically not the ones whose outputs disagree with the reference answer—especially when failures come from decoding noise or multi-round drift rather than role–query mismatch.","supporting_citations":[],"review_version":1}