{"id":"31f2c3e6-63b9-4cbc-9f09-e04ff887e3d2","arxiv_id":"2605.29910","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Agora's domain-aware multi-agent LLM framework detects 15 previously unknown safety-violating logic bugs across Raft, EPaxos, HotStuff, and BullShark implementations where prior LLM agents found none.","lead":"Agora introduces a multi-agent LLM system that uses specialized agents to explore protocol states and generate attack scenarios for finding logic bugs in consensus protocols. A smart generalist might read it because these protocols secure blockchains and distributed systems where undetected bugs can lead to data loss or financial damage.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No independent reproduction or formal check confirms the 15 bugs violate safety properties rather than LLM artifacts","rationale":"The reader's weakest assumption already isolates the exact point at which the central claim could fail (genuine violations vs. LLM artifacts). The abstract supplies no counter-evidence to that assumption, so the load-bearing risk remains exactly where identified and the UNVERDICTED verdict is appropriate.","tokens_in":1677,"tokens_out":327,"duration_ms":12016,"concrete_test":"Take the first reported bug for Raft (or whichever protocol has the most detailed scenario in the paper); extract the exact sequence of messages and state transitions claimed to violate safety, then replay it in the canonical etcd/raft implementation or a TLA+ model; if the safety property is not violated under that trace, the corresponding bug report is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (15 previously unknown protocol-level logic bugs violating safety properties) requires that each reported violation is a genuine, reproducible breach of an invariant such as linearizability or leader uniqueness. The abstract states that validation occurs via the same multi-agent LLM framework through iterative refinement, with no mention of (a) execution against the original protocol binaries under the synthesized scenarios, (b) model-checking against a TLA+ or similar spec, or (c) manual audit by protocol experts. Because the discovery and validation agents share the same LLM backbone, any systematic hallucination about state transitions or invariant definitions would be self-reinforcing and would not be caught by the described pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents Agora, a domain-aware multi-agent LLM framework that uses specialized agents for hypothesis-driven testing, attack scenario synthesis with domain constraints, and iterative refinement to detect protocol-level logic bugs in consensus implementations. It evaluates the approach on four protocols (Raft, EPaxos, HotStuff, BullShark) using four LLMs, claiming discovery of 15 previously unknown bugs that violate safety properties (e.g., linearizability or leader uniqueness), while baseline LLM agents detect none.","tokens_in":1848,"tokens_out":512,"duration_ms":12233,"significance":"If the reported bugs are confirmed as genuine violations through independent means, the work would demonstrate a practical advance in automated detection of deep, state-dependent logic errors in production consensus protocols, where single-function analysis falls short. The explicit separation of agent roles and use of domain-specific constraints is a strength worth building upon.","major_comments":[{"comment":"Evaluation section (likely §4 or §5): the claim that the 15 bugs 'violate safety properties' rests on validation performed entirely within the same multi-agent LLM pipeline via iterative refinement, with no reported execution of synthesized scenarios against the original protocol binaries, no model-checking against TLA+ or equivalent formal specs, and no independent expert audit. This is load-bearing for the central empirical result, as shared LLM backbone risks self-reinforcing hallucinations about state transitions or invariants.","section":"Evaluation section"},{"comment":"§4 (or equivalent results section), comparison with baselines: the statement that 'existing LLM-based agents fail to detect any such protocol-level logic bugs' requires explicit description of the baseline prompts, agent configurations, and state-exploration budgets used; without these, the performance gap cannot be attributed to Agora's design rather than weaker baseline setups.","section":"Results/comparison section"}],"minor_comments":[{"comment":"Abstract and introduction: the four protocols are named but the specific implementations (e.g., version, language, or repository) should be cited with links or commit hashes for reproducibility.","section":"Abstract/Introduction"},{"comment":"Figure or table presenting the 15 bugs: each bug entry should include the violated invariant, the protocol, the LLM used, and whether the scenario was reproducible outside the LLM framework.","section":"Results table/figure"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback, which highlights important aspects of empirical rigor in our evaluation. We address each major comment below and indicate the corresponding revisions.","responses":[{"response":"We acknowledge that the primary validation mechanism is the iterative refinement process within the multi-agent framework, which uses domain constraints and cross-agent checks to confirm violations of safety properties such as linearizability. The manuscript does not report direct execution of attack scenarios on the original protocol binaries or formal model checking. We agree this is a substantive limitation for confirming the bugs as genuine. We will revise the evaluation section to explicitly state this scope, add any manual reproduction steps performed by the authors outside the LLM pipeline, and include a dedicated limitations paragraph discussing the risk of LLM-induced hallucinations along with plans for future external validation.","revision_made":"partial","referee_comment":"[Evaluation section] Evaluation section (likely §4 or §5): the claim that the 15 bugs 'violate safety properties' rests on validation performed entirely within the same multi-agent LLM pipeline via iterative refinement, with no reported execution of synthesized scenarios against the original protocol binaries, no model-checking against TLA+ or equivalent formal specs, and no independent expert audit. This is load-bearing for the central empirical result, as shared LLM backbone risks self-reinforcing hallucinations about state transitions or invariants."},{"response":"We will expand the results section (and associated appendix) to include the precise baseline prompts, agent role configurations, and state-exploration budgets used for the comparison experiments. This will allow readers to assess that the baselines were given comparable resources and that the performance difference stems from Agora's domain-aware role separation rather than under-configured controls.","revision_made":"yes","referee_comment":"[Results/comparison section] §4 (or equivalent results section), comparison with baselines: the statement that 'existing LLM-based agents fail to detect any such protocol-level logic bugs' requires explicit description of the baseline prompts, agent configurations, and state-exploration budgets used; without these, the performance gap cannot be attributed to Agora's design rather than weaker baseline setups."}],"tokens_in":1353,"tokens_out":458,"duration_ms":18831,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core claim is that a multi-agent setup with hypothesis-driven testing and domain constraints can surface protocol-level logic bugs in Raft, EPaxos, HotStuff, and BullShark that single LLM agents miss. The paper shows this by running the same four LLMs under both the new framework and baseline approaches.\n\nThe explicit agent roles and the use of protocol invariants as constraints are the concrete addition. Earlier LLM work on code tends to stay at the function or local level; this one tries to reason about global safety properties across execution stages. That distinction is real and worth testing.\n\nThe main weakness is the validation step. The abstract describes confirmation through iterative refinement by the same agents, with no reference to running the synthesized scenarios on the original binaries, checking against a TLA+ spec, or having protocol experts review the traces. When discovery and validation share the LLM backbone, systematic errors in state modeling can reinforce themselves.\n\nThe headline number of 15 bugs therefore rests on internal consistency rather than independent reproduction. The paper does not report false-positive rates or how many candidate scenarios were discarded.\n\nThis is for people working on automated verification of distributed systems who already follow LLM-agent work. A reader who wants to know whether the method actually catches real bugs will need the full methods section and any released artifacts before treating the count as settled.\n\nIt should go to peer review. The idea is narrow enough to evaluate quickly, and referees can press on the validation pipeline and ask for reproduction details.","headline":"Agora applies role-separated LLM agents to hunt state-dependent bugs in consensus protocols and reports 15 new ones, but the validation stays inside the LLM loop with no external confirmation.","tokens_in":2356,"tokens_out":383,"would_cite":false,"duration_ms":15893,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Agora's multi-agent LLM framework detects 15 protocol-level logic bugs in consensus implementations that single-agent approaches miss.","keywords":["bug detection","consensus protocols","LLM agents","multi-agent systems","protocol verification","safety properties","distributed systems","Raft"],"falsifier":"Manual inspection or formal proof showing that none of the 15 reported bugs actually violate the stated safety properties, or that the multi-agent method finds zero bugs on a fifth independent consensus implementation.","tokens_in":2575,"feed_emoji":"🔍","tokens_out":623,"duration_ms":20818,"temperature":0.7,"pith_summary":"Consensus protocols are central to distributed systems but contain subtle logic bugs that break safety across multiple states. Current LLM agents analyze code in isolation and miss these global, state-dependent violations. Agora deploys specialized agents that collaborate on hypothesis generation, state-space exploration, attack synthesis under domain constraints, and iterative validation. On Raft, EPaxos, HotStuff, and BullShark, using four LLMs, the system finds 15 previously unknown safety bugs while baseline agents find none. The results indicate that explicit role separation and domain knowledge are required to surface deep protocol errors.","feed_headline":"Multi-agent LLMs find 15 safety bugs in consensus protocols","feed_subtitle":"Role-separated agents explore global states and synthesize attacks that single LLM agents overlook in Raft, EPaxos, HotStuff and BullShark.","key_machinery":"Domain-aware multi-agent framework with role-separated agents for collaborative state exploration, attack synthesis, and validation that reasons about global protocol invariants.","core_discovery":"Agora discovers 15 previously unknown protocol-level logic bugs that violate safety properties in four consensus implementations, while existing LLM-based agents fail to detect any such protocol-level logic bugs. The framework achieves this through a domain-aware multi-agent setup that integrates hypothesis-driven testing, state-space exploration, attack synthesis with protocol constraints, and iterative refinement.","pith_inferences":["The same role-separation pattern could extend to verifying other state-machine-based distributed systems such as databases or ledgers.","Combining the agents with model checking tools might reduce false positives from LLM-generated scenarios.","Ongoing use of such agents during protocol updates could catch regressions before deployment in live systems."],"forward_implications":["Domain-aware multi-agent collaboration is required to detect deep protocol-level logic bugs.","Existing single-agent LLM approaches cannot surface safety violations involving complex state interactions.","The method applies across multiple protocols (Raft, EPaxos, HotStuff, BullShark) and multiple LLMs.","Hypothesis-driven testing combined with domain constraints enables systematic verification beyond local code analysis."],"fun_headline_variants":["Agora finds 15 logic bugs in Raft EPaxos HotStuff BullShark","Multi-agent LLMs detect 15 protocol-level bugs in consensus systems","Domain-aware agents identify 15 safety violations in four protocols","15 unknown bugs violating safety detected across consensus protocols"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The 15 reported bugs are genuine safety violations in the protocols rather than LLM hallucinations or incomplete explorations, and the four tested protocols represent typical production consensus code.","fun_headline_variants_meta":{"raw":{"variants":["Agora finds 15 logic bugs in Raft EPaxos HotStuff BullShark","Multi-agent LLMs detect 15 protocol-level bugs in consensus systems","Domain-aware agents identify 15 safety violations in four protocols","15 unknown bugs violating safety detected across consensus protocols"]},"model":"grok-4.3","cost_usd":0.006894,"raw_usage":{"total_tokens":3174,"prompt_tokens":618,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":68937000,"prompt_tokens_details":{"text_tokens":618,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2486,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":618,"tokens_out":70,"duration_ms":18043,"temperature":1.0,"reasoning_tokens":2486,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T06:26:29.715546+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Manual inspection or formal proof showing that none of the 15 reported bugs actually violate the stated safety properties, or that the multi-agent method finds zero bugs on a fifth independent consensus implementation.","supporting_citations":[],"review_version":1}