{"id":"e8b781dc-1651-4ac3-adc6-a644b367d422","arxiv_id":"2507.04105","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Randomized smoothing with adaptive sampling is claimed to give probabilistic robustness guarantees for LLM-driven multi-agent consensus, with simulations showing a 90.24% reduction in deviation from ideal consensus.","lead":"This paper applies randomized smoothing, a statistical certification method from neural network robustness, to LLM-driven multi-agent consensus, claiming probabilistic safety guarantees against malicious agents and hallucinations. It reports simulations in which a Gaussian-noise filtering defense reduces deviation from ideal consensus by 90.24% in a ten-agent setting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 certifies the argmax of a smoothed region classifier, but the deployed update (Eq. 8) is a trimmed mean of continuous outputs; no argument connects them, so the claimed probabilistic guarantee does not cover the algorithm actually run.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap: the certificate in Theorem 1 concerns the stability of a most-likely decision region, while the deployed estimator is a trimmed mean of continuous outputs. This is not a cosmetic mismatch. Randomized smoothing certificates are intrinsically tied to the decision rule being smoothed; for an argmax classifier, the Neyman-Pearson argument controls the probability that the top class changes. For a trimmed mean, the value of the estimator can change even when the most likely region does not, because outputs within a region may have arbitrary magnitude. Since the paper's claimed guarantee is about agent decisions and consensus, and the actual decision function is Eq. 8, the missing implication is fatal to the theoretical claim as stated. The paper also asserts Propositions 1 and 2 without proof, and those propositions are necessary to move from per-agent stability to network-wide attenuation; their absence compounds the problem. The empirical evidence is a single trajectory set with no error bars, no code, and no baselines, and the numeric inconsistency in the average deviation (0.1323 in Figure 2 vs 0.1251 in the text) undermines confidence in the reported 90.24% improvement. No independent supporting evidence, such as a machine-checked proof, reproducible code, or a parameter-free derivation, is present. Under the reviewing rule that in-manuscript limitations must be weighed, the absence of any statement acknowledging these gaps is itself a concern. For these reasons, the reader's REJECT verdict is appropriate, and my independent read does not change it.","tokens_in":11373,"tokens_out":4601,"duration_ms":54145,"concrete_test":"Construct a minimal instance of the deployed pipeline: take X=[0,1], a deterministic base function Φ(z)=0.1 if z<0 and Φ(z)=0.9 if z≥0 (or any two-cluster map), set σ=1, choose z0 with p_A≈0.9 and p_B≈0.1 so r≈0.9σ, and pick a perturbation δ with ||δ||<r. Compute the smoothed argmax region and the trimmed mean (Eq. 8) for z0 and z0+δ over repeated sample sets of size m (e.g., m=20, trim 10%). If the trimmed mean shifts by more than a consensus tolerance (e.g., 0.05) while the smoothed argmax region is unchanged, then Theorem 1 does not imply stability of the deployed estimator. Repeating this test across random seeds and several two-cluster/continuous functions would settle whether the guaranteed object transfers to the actual algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is a probabilistic safety guarantee for LLM-driven MAS. Section 3.2.1 partitions the decision space into regions R1...Rk, and Theorem 1 (Eq. 22) certifies that the most likely region is unchanged under perturbations δ with ||δ||_2 < r = σ/2(Ψ^{-1}(p_A) - Ψ^{-1}(p_B)). The guarantee is a restatement of Cohen et al.'s randomized-smoothing bound for a smoothed argmax decision rule. However, the deployed estimator, Eq. 8, is a trimmed mean of Monte Carlo samples of the LLM output—not an argmax over regions. The paper never specifies how R1...Rk are defined for continuous scalar states, never shows that the trimmed mean is a function of the most likely region, and never proves any bound on the trimmed-mean output under input perturbation. Region stability is compatible with large value shifts inside the winning region, so consensus can fail even while the region certificate holds. Propositions 1 and 2, which are needed to extend local certificates to network-wide attenuation and malicious-agent tolerance, are asserted without proof. In particular, Eq. 23's attenuation factor (1 - Ψ(r_i/σ)) is not derived from any formal definition of 'probability of perturbation classification and filtration,' and no argument shows that local region stability compounds multiplicatively along the network. The empirical section provides a single run with no error bars, no code, no baselines, and inconsistent numerical values (0.1323 vs 0.1251 for the same quantity). Thus the headline guarantee is supported neither by the theory for the actual estimator nor by reproducible experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes randomized smoothing as a defense for LLM-driven multi-agent systems (MAS) in safety-critical settings such as aerospace. It introduces a two-level smoothing procedure (smoothed neighbor queries and a trimmed-mean Monte Carlo estimator of each agent's LLM output), an adaptive sampling strategy to control query cost, and claims Theorem 1 (a local decision-stability certificate) based on Cohen et al.'s randomized-smoothing bound. It then extends the local certificate to network-wide claims in Propositions 1 and 2 (perturbation attenuation and malicious-agent tolerance) and presents simulations with GPT-3.5-Turbo, including a 3D UAV formation scenario. The central claim is that the framework provides probabilistic safety guarantees against adversarial information and hallucinations.","tokens_in":11748,"tokens_out":6111,"duration_ms":67648,"significance":"The topic is timely and relevant: applying randomized smoothing to black-box LLM-driven consensus is a plausible direction, and the adaptive sampling heuristic is a sensible practical concern. If the advertised guarantees actually applied to the deployed algorithm, the paper would be a useful contribution to safe LLM-based MAS. However, as written, the theoretical certificate does not cover the algorithm that is executed, the two network-level propositions are asserted without proof, and the empirical evaluation is a single anecdotal run with inconsistent reported numbers. The paper's positive elements are the clear threat model and the honest identification of the difficulty of formal verification for stochastic LLMs, but these do not compensate for the absence of a load-bearing connection between the theory and the system.","major_comments":[{"comment":"The certificate in Theorem 1 concerns the stability of the most likely decision region of the smoothed base classifier, i.e., an argmax over regions R_1,...,R_k as defined in Eqs. (11)-(12), but the deployed estimator in Eq. (8) is a trimmed mean of continuous LLM outputs. The paper never defines the regions for a continuous state space, never shows that the trimmed mean is a function of the most likely region, and never proves any bound on the trimmed-mean estimator under input perturbation. Region stability permits arbitrarily large value shifts inside the winning region, so consensus can fail even while the region certificate holds. Thus the claimed probabilistic guarantee does not apply to the algorithm that is actually run.","section":"Section 3.2.1, Theorem 1 (Eq. 22); Section 3.1.1 (Eq. 8)"},{"comment":"Proposition 1 is asserted without derivation. The factor (1 - Psi(r_i/sigma)) is not derived from any formal definition of \"probability of perturbation classification and filtration,\" and no argument establishes that a local certification radius translates into a multiplicative attenuation of the output perturbation at each propagation hop. Consequently, the exponential-decay claim in Eq. (24) is unsupported and cannot be used to justify network-wide safety.","section":"Section 3.3.3, Eqs. (23)-(24)"},{"comment":"Proposition 2 is stated as a bare proportionality relation without a theorem statement, without a derived constant, and without a specification of the tolerance criterion (e.g., failure probability or final consensus error). In its current form, the claim |A_mal|/|A| proportional to r_min/||delta_mal||_max cannot be verified or used, so the asserted malicious-agent tolerance is not established.","section":"Section 3.3.4, Eq. (25)"},{"comment":"The empirical evaluation consists of a single run with no stochastic repetitions, error bars, confidence intervals, random seed specifications, or implementation details such as prompts, temperature, trimming fraction, or the values of c, tau, m_1, and m_max. In addition, the text reports an average normal-agent deviation of 0.1251 while the Figure 2(a) caption reports 0.1323 for the same quantity. Without statistical evidence, the reported 90.24% improvement does not validate the framework.","section":"Section 4.3, Figure 2"},{"comment":"To instantiate the certificate, the framework requires a lower bound p_A on the probability of the most likely decision region and an upper bound p_B on the second-most-likely region, obtained from finite Monte Carlo samples. The adaptive sampling strategy in Eqs. (9)-(10) estimates output variance, not region probabilities, and the paper never specifies a confidence-aware procedure for obtaining p_A and p_B. Without such a certification procedure, Theorem 1 cannot be applied to any concrete LLM-based agent.","section":"Section 3.2.1, Eqs. (11)-(12)"}],"minor_comments":[{"comment":"The phrase \"a adaptive sampling strategy\" should read \"an adaptive sampling strategy.\"","section":"Section 3.1.2"},{"comment":"The variance formula has malformed summation notation; the sample mean should be defined explicitly and the brackets typeset correctly.","section":"Section 3.1.2, Eq. (9)"},{"comment":"The simulation parameters c, tau, m_1, m_max, and the trim fraction used in the trimmed mean are not specified, so the adaptive sampling behavior cannot be reproduced.","section":"Section 4.1"},{"comment":"Reference [12] contains a typo in the arXiv identifier (\"Arxiv:2310,20151\" should presumably be \"arXiv:2310.20151\").","section":"References"},{"comment":"The claim that the trimmed mean converges to the \"true intended output\" by the law of large numbers is imprecise: trimming changes the estimator, and the \"true intended output\" is not formally defined.","section":"Section 3.3.1"}],"recommendation":"reject","confidential_remarks":"The core difficulty is that Theorem 1 is a restatement of Cohen et al.'s randomized-smoothing bound for a region classifier, while the deployed algorithm uses a trimmed-mean estimator; this mismatch means the paper's central probabilistic guarantee does not apply to the system it describes. The two global propositions are unsupported, and the empirical section is not reproducible from the information given. These are load-bearing issues rather than presentation problems, and addressing them would require substantial new theory and a proper experimental study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nTwo things you should know about arXiv:2507.04105. First, it is a decent idea: randomized smoothing, which certifies robustness of black-box functions against bounded perturbations, is a natural fit for LLM-driven multi-agent consensus, and the paper packages it as a defense against malicious agents and hallucinations. Second, the paper's own headline guarantee does not cover the algorithm it runs. Theorem 1 certifies stability of the most-likely decision region under input perturbation, but the deployed update (Eq. 8) is a trimmed mean of samples of the LLM output—a continuous value, not an argmax over regions. The paper never defines the regions for continuous scalar states, never shows trimmed-mean stability follows from region stability, and never bounds the trimmed-mean output under perturbation. A region can stay the same while the value inside it shifts a lot, so consensus can break even when the certificate holds. That gap is load-bearing.\n\nWhat is genuinely new: the framing of randomized smoothing as an inter-agent verification mechanism (repeated querying of neighbors with noise), the two-stage adaptive sampling heuristic to control query count, and the application to an aerospace formation scenario. The adaptive sampling is engineering, not analyzed, but it is a reasonable way to trade compute for robustness.\n\nThe soft spots are the theory and the experiments. Propositions 1 and 2, which claim perturbation attenuation and malicious tolerance, are asserted without proof. Eq. 23's factor (1 − Ψ(r/σ)) is not derived from any formal definition of 'probability of perturbation classification and filtration.' The simulations are a single run with no error bars, no code, no baselines beyond the three scenarios, and the numbers are inconsistent: the text says average deviation without defense is 0.1251 while the figure caption says 0.1323. That is sloppy.\n\nThe related work is fine—Cohen et al. is cited, and the paper acknowledges the proof is based on it. There is no circularity or invented entities.\n\nWho is this for? Researchers in LLM safety and MAS who want a black-box, sampling-based way to reason about perturbation robustness in consensus. They should read the framework, but they should not rely on the certificate as stated.\n\nMy recommendation: send it to peer review—the topic is timely and the gap is fixable in principle—but do not accept it as is. The authors need to either change the theory to certify the trimmed-mean estimator, or change the deployed estimator to match the certified argmax rule. Until then, the central claim is unsupported.\n\nBest,\n[Your name]","headline":"Applies randomized smoothing to LLM-based MAS consensus, but the certified object (argmax region) is not the deployed estimator (trimmed mean), so the headline guarantee is unsupported.","tokens_in":12261,"tokens_out":1861,"would_cite":false,"duration_ms":18650,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Injecting Gaussian noise into each LLM agent's inputs yields a probabilistic certificate that its decision region holds against bounded neighbor misinformation, with perturbations damped as they propagate through the network.","keywords":["multi-agent systems","large language models","randomized smoothing","certified robustness","consensus seeking","adversarial robustness","hallucination mitigation","safe planning"],"falsifier":"Take one LLM agent, estimate $p_A$ and $p_B$ by Monte Carlo sampling to compute $r$, then feed it neighbor inputs with $\\|\\delta\\|_2$ just below $r$ and observe whether the trimmed-mean output from Eq. (8) or the subsequent consensus trajectory changes by more than the noise floor; if the output shifts measurably while the most likely region does not flip, the deployed estimator is not covered by the certificate. A positive control would be an input with $\\|\\delta\\|_2$ above $r$, where the region is predicted to flip.","tokens_in":11170,"feed_emoji":"🛡️","tokens_out":8735,"duration_ms":89793,"temperature":0.7,"pith_summary":"The paper tries to establish that randomized smoothing, the standard technique of feeding Gaussian-perturbed inputs to a model and averaging, can be adapted to LLM-driven multi-agent systems to give each agent a probabilistic guarantee about its own decisions. The target is consensus-seeking: agents share numerical states, an LLM updates each agent's state from its own and its neighbors' states, and a malicious neighbor or a hallucinated output can push the group off course. The central claim is Theorem 1: if neighbors' manipulations are bounded in Euclidean norm by a radius computed from the smoothed probabilities of the agent's top two decision regions, the agent's most likely decision region cannot change. If the radius bound holds, misinformation should not merely be detected but mathematically contained, and a separate proposition says perturbations shrink as they propagate, so a single bad source cannot cascade. A reader should care because LLM-based multi-agent coordination is being proposed for safety-critical aerospace settings where hard guarantees are otherwise unavailable.","feed_headline":"Randomized smoothing blocks misinformation in LLM agent teams","feed_subtitle":"Neighbor lies within a certified radius cannot flip an agent's decision; noise-damping cuts consensus error by 90%.","key_machinery":"The load-bearing object is the smoothed decision function $S_i(z) = \\mathbb{E}_{\\epsilon\\sim\\mathcal{N}(0,\\sigma^2 I)}[\\Phi_i(z+\\epsilon)]$, approximated in practice by a Monte Carlo trimmed mean over $m$ perturbed LLM queries. Around it sit three pieces: the decision-region partition of the output space, the pair $(p_A, p_B)$ of top-two region probabilities, and the certified radius $r = \\frac{\\sigma}{2}(\\Psi^{-1}(p_A)-\\Psi^{-1}(p_B))$ from Theorem 1. The proof mechanism is the Neyman-Pearson lemma, which bounds how much a mean shift $\\delta$ can change the probability mass of any decision region; the inequality $\\Psi(\\Psi^{-1}(p_A)-\\|\\delta\\|_2/\\sigma) > \\Psi(\\Psi^{-1}(p_B)+\\|\\delta\\|_2/\\sigma)$ is what converts the radius into a stability certificate. Supporting machinery includes the two-stage adaptive sampler, which estimates output variance with $m_1$ samples and then allocates $m_2(i) = \\min(\\lceil c V_i / \\tau \\rceil, m_{\\max})$ additional queries, and the product formula $\\|\\delta_{\\mathrm{out},k}\\| \\approx \\|\\delta_1\\| \\prod_i (1-\\Psi(r_i/\\sigma))$ that formalizes network-level attenuation. The paper also states a proportionality claim $|A_{\\mathrm{mal}}|/|A| \\propto r_{\\min}/\\|\\delta_{\\mathrm{mal}}\\|_{\\max}$ for how many malicious agents the system tolerates.","core_discovery":"The core discovery is a transfer of the classical randomized-smoothing certification argument from image classifiers to the consensus dynamics of LLM agents. The paper partitions each agent's output space into decision regions $R_1,\\dots,R_k$ and defines $p_A$ and $p_B$ as the smoothed probabilities of the most likely and second-most-likely regions under Gaussian noise $\\mathcal{N}(0,\\sigma^2 I)$. Theorem 1 states that any neighbor perturbation with $\\|\\delta\\|_2 < \\frac{\\sigma}{2}(\\Psi^{-1}(p_A)-\\Psi^{-1}(p_B))$ leaves the agent's decision region unchanged, with the proof running the Neyman-Pearson lemma on binary hypotheses for each region. Proposition 1 extends this locally: an output perturbation entering an agent is multiplied by a factor $(1-\\Psi(r/\\sigma))<1$, so along a chain of $k$ agents the effective perturbation is bounded by the product of these factors and decays exponentially. The paper reports simulations with an LLM-driven ring network in which the defense cuts normal agents' average deviation from baseline by 90.24% and preserves UAV formation in a 3D aerospace scenario.","pith_inferences":["The guarantee binds the decision region, not the numerical state; the trimmed mean could drift inside the certified region, so the consensus guarantee is likely weaker than the reported 90.24% improvement suggests.","Proposition 1's product formula assumes each agent attenuates independently; correlated misinformation or shared LLM context could break the exponential-decay picture, and measuring per-hop attenuation on a real chain of agents would settle it.","A direct continuous analogue, certifying the trimmed-mean output rather than region membership, would connect randomized smoothing to conventional consensus-convergence proofs and is a natural next step.","The defense might also filter covert collusion signals, since steganographic coordination likely appears as statistical outliers across repeated perturbed queries; that is a testable application outside the paper's stated scope."],"forward_implications":["Each agent can compute a certified radius from sample counts alone, so operators know exactly how wrong a neighbor can be before that agent's decision region is threatened.","If Proposition 1 holds, bounded misinformation decays exponentially along communication paths, so a single malicious source or hallucinated output cannot cascade into system-wide failure.","Because the defense only queries the LLM, it works against closed black-box API models with no weight access.","The adaptive sampling rule lets applications trade query budget for certified robustness, which matters for latency-limited aerospace missions.","The 3D UAV formation simulation indicates the defense preserves formation integrity against the same attacks that break the undefended system."],"supporting_citations":[{"why":"Supplies the randomized smoothing certification bound that Theorem 1 adapts from classification to decision regions.","marker":"[37]"},{"why":"Provides the Neyman-Pearson lemma used to bound probability shifts under a Gaussian mean shift in Theorem 1's proof.","marker":"[38]"},{"why":"Defines the LLM-driven consensus-seeking setup and state update equation that the paper extends with defenses.","marker":"[12]"},{"why":"Motivates applying noise-based smoothing to LLMs for robustness, which the paper generalizes to multi-agent settings.","marker":"[24]"}],"fun_headline_variants":["Randomized smoothing certifies LLM agent teams against lies","Noise-damped consensus gives LLM agents certified safety","Smoothing guarantees LLM agent decisions resist adversarial noise","Black-box smoothing certifies agent consensus against poison","Statistical smoothing blocks adversarial drift in LLM teams"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The certificate is about the most likely decision region staying stable, while the deployed estimator is a trimmed mean of continuous outputs, and the paper never shows that region stability forces the trimmed-mean update to be stable.","fun_headline_variants_meta":{"raw":{"variants":["Randomized smoothing certifies LLM agent teams against lies","Noise-damped consensus gives LLM agents certified safety","Smoothing guarantees LLM agent decisions resist adversarial noise","Black-box smoothing certifies agent consensus against poison","Statistical smoothing blocks adversarial drift in LLM teams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00063,"raw_usage":{"total_tokens":2883,"prompt_tokens":892,"completion_tokens":1991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1924}},"tokens_in":508,"tokens_out":1991,"duration_ms":13805,"temperature":1.0,"reasoning_tokens":1924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:55:18.226699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one LLM agent, estimate $p_A$ and $p_B$ by Monte Carlo sampling to compute $r$, then feed it neighbor inputs with $\\|\\delta\\|_2$ just below $r$ and observe whether the trimmed-mean output from Eq. (8) or the subsequent consensus trajectory changes by more than the noise floor; if the output shifts measurably while the most likely region does not flip, the deployed estimator is not covered by the certificate. A positive control would be an input with $\\|\\delta\\|_2$ above $r$, where the region is predicted to flip.","supporting_citations":[{"cited_title":"Cohen, E","cited_arxiv_id":null,"evidence_quote":"Supplies the randomized smoothing certification bound that Theorem 1 adapts from classification to decision regions."},{"cited_title":"Neyman, E","cited_arxiv_id":null,"evidence_quote":"Provides the Neyman-Pearson lemma used to bound probability shifts under a Gaussian mean shift in Theorem 1's proof."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the LLM-driven consensus-seeking setup and state update equation that the paper extends with defenses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates applying noise-based smoothing to LLMs for robustness, which the paper generalizes to multi-agent settings."}],"review_version":1}