Pith. sign in

REVIEW 2 major objections 2 minor 85 references

Agora's multi-agent LLM framework detects 15 protocol-level logic bugs in consensus implementations that single-agent approaches miss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 06:26 UTC pith:OKCR5MVG

load-bearing objection Agora applies role-separated LLM agents to hunt state-dependent bugs in consensus protocols and reports 15 new ones, but the validation stays inside the LLM loop with no external confirmation. the 2 major comments →

arxiv 2605.29910 v1 pith:OKCR5MVG submitted 2026-05-28 cs.SE cs.AI

Agora: Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents

Xiang Liu , Sa Song , Zhaowei Zhang , Huiying Lan , Jason Zeng , Ming Wu , Michael Heinrich , Yong Sun
show 1 more author
Ceyao Zhang
This is my paper
classification cs.SE cs.AI
keywords bug detectionconsensus protocolsLLM agentsmulti-agent systemsprotocol verificationsafety propertiesdistributed systemsRaft
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Consensus protocols are central to distributed systems but contain subtle logic bugs that break safety across multiple states. Current LLM agents analyze code in isolation and miss these global, state-dependent violations. Agora deploys specialized agents that collaborate on hypothesis generation, state-space exploration, attack synthesis under domain constraints, and iterative validation. On Raft, EPaxos, HotStuff, and BullShark, using four LLMs, the system finds 15 previously unknown safety bugs while baseline agents find none. The results indicate that explicit role separation and domain knowledge are required to surface deep protocol errors.

Core claim

Agora discovers 15 previously unknown protocol-level logic bugs that violate safety properties in four consensus implementations, while existing LLM-based agents fail to detect any such protocol-level logic bugs. The framework achieves this through a domain-aware multi-agent setup that integrates hypothesis-driven testing, state-space exploration, attack synthesis with protocol constraints, and iterative refinement.

What carries the argument

Domain-aware multi-agent framework with role-separated agents for collaborative state exploration, attack synthesis, and validation that reasons about global protocol invariants.

Load-bearing premise

The 15 reported bugs are genuine safety violations in the protocols rather than LLM hallucinations or incomplete explorations, and the four tested protocols represent typical production consensus code.

What would settle it

Manual inspection or formal proof showing that none of the 15 reported bugs actually violate the stated safety properties, or that the multi-agent method finds zero bugs on a fifth independent consensus implementation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Domain-aware multi-agent collaboration is required to detect deep protocol-level logic bugs.
  • Existing single-agent LLM approaches cannot surface safety violations involving complex state interactions.
  • The method applies across multiple protocols (Raft, EPaxos, HotStuff, BullShark) and multiple LLMs.
  • Hypothesis-driven testing combined with domain constraints enables systematic verification beyond local code analysis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same role-separation pattern could extend to verifying other state-machine-based distributed systems such as databases or ledgers.
  • Combining the agents with model checking tools might reduce false positives from LLM-generated scenarios.
  • Ongoing use of such agents during protocol updates could catch regressions before deployment in live systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript presents Agora, a domain-aware multi-agent LLM framework that uses specialized agents for hypothesis-driven testing, attack scenario synthesis with domain constraints, and iterative refinement to detect protocol-level logic bugs in consensus implementations. It evaluates the approach on four protocols (Raft, EPaxos, HotStuff, BullShark) using four LLMs, claiming discovery of 15 previously unknown bugs that violate safety properties (e.g., linearizability or leader uniqueness), while baseline LLM agents detect none.

Significance. If the reported bugs are confirmed as genuine violations through independent means, the work would demonstrate a practical advance in automated detection of deep, state-dependent logic errors in production consensus protocols, where single-function analysis falls short. The explicit separation of agent roles and use of domain-specific constraints is a strength worth building upon.

major comments (2)
  1. [Evaluation section] Evaluation section (likely §4 or §5): the claim that the 15 bugs 'violate safety properties' rests on validation performed entirely within the same multi-agent LLM pipeline via iterative refinement, with no reported execution of synthesized scenarios against the original protocol binaries, no model-checking against TLA+ or equivalent formal specs, and no independent expert audit. This is load-bearing for the central empirical result, as shared LLM backbone risks self-reinforcing hallucinations about state transitions or invariants.
  2. [Results/comparison section] §4 (or equivalent results section), comparison with baselines: the statement that 'existing LLM-based agents fail to detect any such protocol-level logic bugs' requires explicit description of the baseline prompts, agent configurations, and state-exploration budgets used; without these, the performance gap cannot be attributed to Agora's design rather than weaker baseline setups.
minor comments (2)
  1. [Abstract/Introduction] Abstract and introduction: the four protocols are named but the specific implementations (e.g., version, language, or repository) should be cited with links or commit hashes for reproducibility.
  2. [Results table/figure] Figure or table presenting the 15 bugs: each bug entry should include the violated invariant, the protocol, the LLM used, and whether the scenario was reproducible outside the LLM framework.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback, which highlights important aspects of empirical rigor in our evaluation. We address each major comment below and indicate the corresponding revisions.

read point-by-point responses

  1. Referee: [Evaluation section] Evaluation section (likely §4 or §5): the claim that the 15 bugs 'violate safety properties' rests on validation performed entirely within the same multi-agent LLM pipeline via iterative refinement, with no reported execution of synthesized scenarios against the original protocol binaries, no model-checking against TLA+ or equivalent formal specs, and no independent expert audit. This is load-bearing for the central empirical result, as shared LLM backbone risks self-reinforcing hallucinations about state transitions or invariants.

    Authors: We acknowledge that the primary validation mechanism is the iterative refinement process within the multi-agent framework, which uses domain constraints and cross-agent checks to confirm violations of safety properties such as linearizability. The manuscript does not report direct execution of attack scenarios on the original protocol binaries or formal model checking. We agree this is a substantive limitation for confirming the bugs as genuine. We will revise the evaluation section to explicitly state this scope, add any manual reproduction steps performed by the authors outside the LLM pipeline, and include a dedicated limitations paragraph discussing the risk of LLM-induced hallucinations along with plans for future external validation. revision: partial

  2. Referee: [Results/comparison section] §4 (or equivalent results section), comparison with baselines: the statement that 'existing LLM-based agents fail to detect any such protocol-level logic bugs' requires explicit description of the baseline prompts, agent configurations, and state-exploration budgets used; without these, the performance gap cannot be attributed to Agora's design rather than weaker baseline setups.

    Authors: We will expand the results section (and associated appendix) to include the precise baseline prompts, agent role configurations, and state-exploration budgets used for the comparison experiments. This will allow readers to assess that the baselines were given comparable resources and that the performance difference stems from Agora's domain-aware role separation rather than under-configured controls. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical comparison against external baselines

full rationale

The paper reports an empirical evaluation of a multi-agent LLM framework on four named consensus protocol implementations, claiming discovery of 15 bugs where prior LLM agents find none. No equations, fitted parameters, self-definitional relations, or load-bearing self-citations are present in the provided text that would reduce any claimed result to its inputs by construction. The central result is framed as a direct comparison to independently described existing agents on fixed protocols, satisfying the condition for a self-contained empirical finding.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The central claim rests on the assumption that LLM agents can be guided by domain constraints to explore protocol state spaces exhaustively enough to surface real safety bugs, plus standard distributed-systems assumptions about safety properties; no free parameters or invented physical entities are described.

axioms (2)
  • domain assumption Consensus protocols have well-defined safety properties that can be violated by implementation logic bugs involving multi-stage state transitions.
    Invoked in the abstract when stating that discovered bugs violate safety properties.
  • ad hoc to paper Specialized LLM agents can collaboratively synthesize and validate attack scenarios using domain-specific constraints.
    Core premise of the Agora framework described in the abstract.
invented entities (1)
  • Hypothesis-driven testing agents and iterative refinement agents within Agora no independent evidence
    purpose: To explore protocol state spaces and validate findings beyond single-function analysis
    New agents introduced by the framework; no independent evidence outside the paper is provided in the abstract.

pith-pipeline@v0.9.1-grok · 5720 in / 1486 out tokens · 21391 ms · 2026-06-29T06:26:29.715546+00:00 · methodology

0 comments
read the original abstract

Consensus protocols form the backbone of distributed systems and blockchains, where implementation bugs can cause data corruption and financial losses. While LLM-based approaches show promise in code analysis, they struggle with deep protocol-level logic bugs involving complex state-dependent behaviors across multiple execution stages. We present Agora, a domain-aware multi-agent framework that integrates hypothesis-driven testing with LLM capabilities for systematic protocol verification. Agora employs specialized agents that collaboratively explore protocol state spaces, synthesize attack scenarios using domain-specific constraints, and validate findings through iterative refinement. This explicit role separation enables reasoning about global protocol invariants beyond single-function code analysis. We evaluate Agora on four consensus implementations (Raft, EPaxos, HotStuff, BullShark) using four state-of-the-art LLMs. Agora discovers 15 previously unknown protocol-level logic bugs that violate safety properties, while existing LLM-based agents fail to detect any such protocol-level logic bugs. Our results demonstrate that domain-aware multi-agent collaboration is essential for detecting deep logic bugs in complex protocols.

Figures

Figures reproduced from arXiv: 2605.29910 by Ceyao Zhang, Huiying Lan, Jason Zeng, Michael Heinrich, Ming Wu, Sa Song, Xiang Liu, Yong Sun, Zhaowei Zhang.

Figure 1
Figure 1. Figure 1: Left is the baseline overview. Middle shows the input and result comparison where baselines could only detect low-level implementation bugs, but Agora could detect critical protocol-level logic bugs. Right is the overview of Agora, which consists of three agents: Orchestrator agent, Strategy agent and TestGen agent. integrate domain knowledge of distributed consensus with multi-agent systems (Yang et al., … view at source ↗
Figure 2
Figure 2. Figure 2: The overview of Agora workflow, which consists of three agents and twelve steps/subtasks (following the time order, from Step 1 to Step 12 ). Orchestrator agent is responsible for bug exploitation, coordinating the agents, maintaining global state, and sending and receiving messages among agents; Strategy agent is responsible for generating attack scenarios based on the analysis results and the information… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of discovered 15 protocol-level logic bugs of Agora by category. the required state information from Orchestrator. Similarly, TestGen obtains the attack scenarios through Orchestrator. As a result, Strategy and TestGen do not need to manage contextual information and can focus solely on their respec￾tive tasks, while Orchestrator maintains the system-wide state and effectively manages long-hor… view at source ↗
Figure 4
Figure 4. Figure 4: The left (a) shows the total token consumption in the ablation studies. The right (b) shows the average token consumption for each discovered bug in the ablation studies, where “cross” indicates that no bugs are discovered. fail to discover any bugs, Agora is the most efficient, discov￾ering one critical logic bug with an average of 5.32M tokens (approximately $40). Identifying such protocol-level logic bu… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

85 extracted references · 5 canonical work pages · 3 internal anchors

  1. [1]

    Qwen3 Technical Report

    Accessed: 2026-01-29. Anthropic. Claude sonnet 4.5 system card. Technical report, Anthropic, 2025. URL https://www.anthropic. com/claude-sonnet-4-5-system-card . Fo- cuses on autonomous agents and tool-use capabilities. Bai, J., Chu, Y ., Duan, Z., Fan, K., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    arXiv preprint arXiv:2401.03428

    URL https://api.semanticscholar. org/CorpusID:261048935. Cheng, Y ., Zhang, C., Zhang, Z., Meng, X., Hong, S., Li, W., Wang, Z., Wang, Z., Yin, F., Zhao, J., et al. Ex- ploring large language model based intelligent agents: Definitions, methods, and prospects.arXiv preprint arXiv:2401.03428, 2024. Clark, J., Donaldson, A. F., Wickerson, J., and Rigger, M....

    work page arXiv 2024

  3. [3]

    org/CorpusID:269262064

    URL https://api.semanticscholar. org/CorpusID:269262064. Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., et al. Spanner: Google’s globally dis- tributed database.ACM Transactions on Computer Sys- tems (TOCS), 31(3):1–22, 2013. Danezis, G., Kokoris-Kogias, E., Sonnino, A., a...

    2013

  4. [4]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    URL https://ai.google.dev/static/ gemini_3_report.pdf. Includes details on Deep Think reasoning modes. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J.-M., Zhang, R., Xu, R., Zhu, Q., Ma, S., and et.al., P. W. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.ArXiv, abs/2501.12948, 2025. Ding, Y ., Fu, Y ., Ibrahim, O.,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    CodeBERT: A Pre-Trained Model for Programming and Natural Languages

    URL https://api.semanticscholar. org/CorpusID:268724273. 9 Agora : Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents efficient. epaxos: Go implementation of the egalitarian paxos consensus protocol. https://github.com/ efficient/epaxos, 2026. GitHub repository (ac- cessed 2026-01-09). etcd-io. raft: Raft library for m...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  6. [6]

    org/CorpusID:273877854

    URL https://api.semanticscholar. org/CorpusID:273877854. Giridharan, N., Kokoris-Kogias, L., Sonnino, A., and Spiegelman, A. Bullshark: Dag bft protocols made prac- tical.Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022. Guo, T., Chen, X., Wang, Y ., Chang, R., Pei, S., Chawla, N., Wiest, O., and Zhang, X. Large ...

    2022

  7. [7]

    org/CorpusID:267412980

    URL https://api.semanticscholar. org/CorpusID:267412980. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y ., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., et al. Metagpt: Meta programming for a multi-agent collabora- tive framework. InThe twelfth international conference on learning representations, 2023. Hu, M., Zhou, Y ., Fan, W., Nie, Y ., X...

    2023

  8. [8]

    org/CorpusID:279071173

    URL https://api.semanticscholar. org/CorpusID:279071173. Hunt, P., Konar, M., Junqueira, F. P., and Reed, B. Zookeeper: Wait-free coordination for internet-scale sys- tems. In2010 USENIX annual technical conference, pp. 11–11, 2010. Kaniewski, S., Schmidt, F., Enzweiler, M., Menth, M., and Heer, T. A systematic literature review on detecting software vuln...

    work page arXiv 2010

  9. [9]

    org/CorpusID:246426909

    URL https://api.semanticscholar. org/CorpusID:246426909. Pearce, H., Tan, B., and Dolan-Gavitt, B. Asleep at the keyboard? assessing the security of github copilot’s code contributions. InProceedings of the 2022 IEEE Sympo- sium on Security and Privacy (S&P), 2022. Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y ., Li, J., Yang, C., Chen, W., Su, Y ., Cong,...

    2022

  10. [10]

    org/CorpusID:270257715

    URL https://api.semanticscholar. org/CorpusID:270257715. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Man- ning, C. D., and Finn, C. Direct preference op- timization: Your language model is secretly a re- ward model.Neural Information Processing Systems,

  11. [11]

    org/CorpusID:258959321

    URL https://api.semanticscholar. org/CorpusID:258959321. relab. hotstuff: A framework for experimenting with the hotstuff consensus protocol. https://github.com/ relab/hotstuff, 2026. GitHub repository (accessed 2026-01-09). Sheng, Z., Chen, Z., Gu, S., Huang, H., Gu, G., and Huang, J. Llms in software security: A survey of vulnerability detection techniq...

    2026

  12. [12]

    org/CorpusID:276258857

    URL https://api.semanticscholar. org/CorpusID:276258857. Sun, Y ., Wu, D., Xue, Y ., Liu, H., Wang, H., Xu, Z., Xie, X., and Liu, Y . Gptscan: Detecting logic vulner- abilities in smart contracts by combining gpt with pro- gram analysis.IEEE/ACM 46th International Confer- ence on Software Engineering (ICSE), pp. 2048–2060,

    2048

  13. [13]

    org/CorpusID:260682978

    URL https://api.semanticscholar. org/CorpusID:260682978. Taft, R., Sharif, I., Matei, A., VanBenschoten, N., Lewis, J., Grieger, T., Niemi, K., Woods, A., Birzin, A., Poss, R., et al. Cockroachdb: The resilient geo-distributed sql database. InProceedings of the 2020 ACM SIGMOD in- ternational conference on management of data, pp. 1493– 1509, 2020. Tan, C....

    2020

  14. [14]

    Since VerifyQuorumCert(highQC) fails, the entire proposal is rejected

    Rejection:Honest nodes attempt to verify the AggregateQC. Since VerifyQuorumCert(highQC) fails, the entire proposal is rejected. 5.Stall:As the leader cannot form a valid proposal that honest nodes accept, the consensus process halts. Impact:Complete blockchain liveness failure. A single malicious actor can perpetually halt the network’s progress. Fix & M...

  15. [15]

    There is no ”half-dead” or Byzantine behavior

    Binary State:Nodes are either ”Correct” (following the protocol) or ”Failed” (stopped). There is no ”half-dead” or Byzantine behavior. 2.Honest Participation:All participating nodes are assumed to act in good faith to reach a common consensus

  16. [16]

    Note:Testing for CFT protocols should exhaustively simulate unstable infrastructure and edge-case logic rather than adversarial intent

    Recovery Consistency:The system must maintain safety and eventually regain liveness after network partitions heal or nodes recover. Note:Testing for CFT protocols should exhaustively simulate unstable infrastructure and edge-case logic rather than adversarial intent. E. Memory Details The memory contains repository knowledge and bug patterns, with a small...

    2026

  17. [17]

    2.Protocol Collapse: •Recovery Failure:Doubtful instances remain unresolved, leading to a permanent Liveness hang

    Mangled Communication:The leader broadcasts a message intended as TryPreAccept, but the payload is cast from aPreAcceptstructure. 2.Protocol Collapse: •Recovery Failure:Doubtful instances remain unresolved, leading to a permanent Liveness hang. • State Machine Divergence:Peers may accept inconsistent dependency graphs based on outdated pa data. Corrective...

  18. [18]

    A target honest node has bLock = Genesis

    State:Honest nodes hold a high QC for B3 (Chain: G→B 1 →B 2 →B 3). A target honest node has bLock = Genesis

  19. [19]

    Since QCB1.V iew(1)> bLock.V iew(0) , the node votes, formingQC ′ 4

    Step 1 (View 4):A Byzantine leader proposes B′ 4 extending B1. Since QCB1.V iew(1)> bLock.V iew(0) , the node votes, formingQC ′ 4

  20. [20]

    Step 2 (Views 5–6):Consecutive Byzantine leaders extend the fork to B′ 5 and B′

  21. [21]

    The node’s bLock is updated toB ′ 4, facilitating the fork’s progression

  22. [22]

    Conse- quently,B ′ 4 is committed, bypassing the correctB 2 →B 3 sequence

    Step 3 (View 7):Upon receiving B′ 7, the CommitRule detects a valid 3-chain (B′ 4 →B ′ 5 →B ′ 6). Conse- quently,B ′ 4 is committed, bypassing the correctB 2 →B 3 sequence. Impact & Security AnalysisThis attack is critical as it leads to aSafety Violation. Unlike transient liveness issues, this results in permanent state pollution and ledger divergence ac...

  23. [23]

    R0 and R1 accept it with deps={}

    Fast Path Execution: R0 proposes Cmd A (Key=”x”). R0 and R1 accept it with deps={} . R0 reaches a Fast Quorum, commits, and executesCmd A locally (x= 1). 19 Agora : Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents 2.Leader Crash:R 0 crashes immediately before broadcasting theCommitmessage to others

  24. [24]

    R2 and R1 accept Cmd B

    Conflict Injection: R2 proposes Cmd B (conflicting with Cmd A). R2 and R1 accept Cmd B. Crucially, R1 now seesCmd B in its log

  25. [25]

    Failing to observe a Fast Quorum, it executes startPhase1

    Vulnerability Trigger: R1 initiates recovery for Cmd A. Failing to observe a Fast Quorum, it executes startPhase1. The updateAttributes function sees Cmd B and re-calculates Cmd A’s dependencies as deps={Cmd B}

  26. [26]

    However, the failed leaderR 0 has already executedAwith deps={}

    State Divergence:The cluster eventually commits Cmd A with deps={Cmd B}, implying an execution order ofB→A. However, the failed leaderR 0 has already executedAwith deps={}. Impact & Security AnalysisThis bug represents aLinearizability Violation. Upon recovery, R0 will find its local execution history contradicts the globally committed serialized order. I...

  27. [27]

    The Deep Fork:A Byzantine leader ignores the entire history and proposes B′ 6 extending directly from Genesis(Parent:Genesis, QC:QC G). 20 Agora : Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents 3.Validation Failure: •Liveness:If usingQC B1,QC B1.V iew(1)> bLock.V iew(0), condition passes. •Safety:Extends(B ′ 6, Gen...

  28. [28]

    ) that competes with the legitimate history, potentially leading to a massive state roll-back if the fork gains quorum

    Outcome:Honest nodes vote for B′ 6, enabling the Byzantine leader to build an alternative chain (G→B ′ 6 → B′ 7 . . .) that competes with the legitimate history, potentially leading to a massive state roll-back if the fork gains quorum. Impact & Security AnalysisThis bug exposes the system toTotal History Erasure. Unlike standard forks that deviate by a f...

  29. [29]

    The Liveness condition passes, and the honest node mistakenly votes for a fork that ignoresB 3

    Validation: VoteRule sees QCB1.V iew(1)> bLock.V iew(0) . The Liveness condition passes, and the honest node mistakenly votes for a fork that ignoresB 3. 21 Agora : Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents VerificationReproduced via: go test -v -run TestByzantineLeaderIgnoresHighQC ./protocol/consensus/ BUG D...

  30. [30]

    Nodes R0, R1, R2 accept it (Fast Quorum)

    Fast Path Execution: R0 proposes Cmd A (Key="k", Val=999 ). Nodes R0, R1, R2 accept it (Fast Quorum). 2.Partial Commit:R 0 sends aCommitmessageonlytoR 2.R 2 marks the command asCommitted. 3.Compound Failure: •R 0 crashes permanently. •R 1 crashes and reboots (Amnesia), wiping its log. •R 2 is partitioned from the rest of the cluster. 4.Flawed Recovery:R 3...

  31. [31]

    R3 incorrectly concludes the instance was never used and proposesNO-OP

    Voiding the Instance: R1, R3, and R4 all report no record. R3 incorrectly concludes the instance was never used and proposesNO-OP

  32. [32]

    When the partition heals, R2 has already executed Cmd A, causing a permanent state divergence

    State Divergence: R1, R3, R4 commit and execute NO-OP. When the partition heals, R2 has already executed Cmd A, causing a permanent state divergence. 22 Agora : Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents Impact & Risk Assessment •Outcome:Violation of the ”Stability” property and Linearizability. •Feasibility:Hi...

  33. [33]

    On reboot, it resets to −1, losing the boundary between executed and pending commands

    Volatile Progress Tracking:The ExecedUpTo array is stored purely in memory. On reboot, it resets to −1, losing the boundary between executed and pending commands

  34. [34]

    Status Volatility:The transition to EXECUTED (epaxos-exec.go:128) is never recorded via recordInstanceMetadata

  35. [35]

    Attack Scenario (Double Execution)

    Asynchronous Gap:Since execution happens asynchronously after the COMMITTED phase, the metadata update logic (which usually triggers on commit) fails to capture the final ”executed” lifecycle of the instance. Attack Scenario (Double Execution)

  36. [36]

    State x= 2 , ExecedUpTo[0] = 1

    Initial State: R0 commits and executes (0,0) :INC(x) , (0,1) :INC(x) . State x= 2 , ExecedUpTo[0] = 1. 2.Crash & Amnesia:R 0 crashes.ExecedUpTois wiped

  37. [37]

    Recovery: R0 reboots, restores instances (0,0) and (0,1) as COMMITTED from disk, but its memory-based cursor is−1

  38. [38]

    5.Divergence:Final statex= 4, while non-crashing replicas remain atx= 2

    Re-execution:The executeCommands loop re-scans from index 0 ( ExecedU pT o+ 1 ) and applies the commands again. 5.Divergence:Final statex= 4, while non-crashing replicas remain atx= 2. Impact & Security Analysis •Outcome:Violation ofExactly-Once Semantics. •Risk Level: Criticalfor financial ledgers or non-idempotent state transitions. •Integrity:Causes ir...

  39. [39]

    Cmd B is executed before Cmd A

    Execution Inversion:The surviving nodes apply tie-breaking ( R0.id < R 1.id). Cmd B is executed before Cmd A. Final state:x= 1. Impact & Risk Assessment • Outcome: Linearizability Violation. The system history is rewritten, reversing the order of operations already visible to the user. •Risk Level: Highin high-concurrency environments with non-persistent ...

  40. [40]

    4.Replay:The attacker sends thisfakeQCto an honest node

    Tampering:The attacker constructs a fakeQC with the original valid signature for Block A but an inflated View: v + 1000. 4.Replay:The attacker sends thisfakeQCto an honest node

  41. [41]

    QC view mismatch

    State Corruption:Because Block A.View() exists and is valid, the node updates its highQC to fakeQC. The node now erroneously believes the highest certified view is v + 1000 , causing it to reject legitimate messages from the actual current view. Impact & Security Analysis •Liveness Violation:Disrupts View Synchronization, locking honest nodes out of conse...

  42. [42]

    Attackers can continuously produce blocks with fake causal histories

    Unbounded Growth:There is no hard limit on the number of entries in the missing blocks collection. Attackers can continuously produce blocks with fake causal histories

  43. [43]

    Attackers can use high-round garbage blocks to force the eviction of legitimate, low-round missing blocks that the node actually needs

    Future-Flooding Attack:A naive ”lowest round first” eviction strategy can be bypassed. Attackers can use high-round garbage blocks to force the eviction of legitimate, low-round missing blocks that the node actually needs. Attack Scenario (Future-Flooding)

  44. [44]

    Normal State:An honest node is at Round 10 and is missing a legitimate parent block P (from Round 10)

  45. [45]

    Flooding:A Byzantine attacker floods the network with blocks referencing thousands of fake parent blocks at Round 1000

  46. [46]

    Malicious Eviction:If a capacity limit exists but uses a simple round-based eviction, the node will discard the legitimateRound 10entry to make room for the ”newer”Round 1000entries

  47. [47]

    Impact & Security Analysis •Vulnerability Type: CWE-770: Allocation of Resources Without Limits

    Liveness Failure:The honest node can never complete the causal history for Round 10 , stalling its consensus progress. Impact & Security Analysis •Vulnerability Type: CWE-770: Allocation of Resources Without Limits. • Attack Cost:Extremely low. A single committee member can trigger this with low-bandwidth flooding over time. •Consequence:System-wide crash...

  48. [48]

    Analyze confirmed bugs discovered by the Strategy Agent and TestGen Agent

  49. [49]

    Generate comprehensive bug reports with root cause analysis

  50. [50]

    Extract reusable vulnerability patterns for testing other protocol implementations

  51. [51]

    Focus on providing insights that help developers understand and fix the vulnerabilities

    Track resource usage (tokens, cost) for each bug discovery Be technical, precise, and actionable. Focus on providing insights that help developers understand and fix the vulnerabilities. User: As the Orchestrator Agent, analyze the following confirmed vulnerability and generate a comprehensive report. ## Confirmed Bug Details The Strategy Agent generated ...

    2000

  52. [52]

    **Bug Report **: A comprehensive analysis including: - Root cause of the vulnerability - Potential security impact - Recommended fixes

  53. [53]

    **Bug Pattern **: A reusable pattern for testing similar vulnerabilities in other consensus protocol implementations. ## Output Format 27 Agora : Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents <bug_report> [Your detailed bug report with root cause analysis, impact assessment, and \\ fix recommendations] </bug_repor...

  54. [54]

    **Understand the Protocol Type ** - What is the fault model? (CFT vs BFT) - What are the core safety/liveness assumptions? - What are the key invariants that must hold? 28 Agora : Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents

  55. [55]

    **Identify Attack Vectors ** Think about common vulnerability categories: - Race conditions in concurrent message handling - State persistence and recovery bugs - View change / leader election edge cases - Quorum intersection violations - Timeout and timing-related issues - Message ordering and delivery assumptions - Equivocation and double-voting scenari...

  56. [56]

    **Creative Scenario Design ** - Consider edge cases that developers might miss - Think about interactions between multiple components - Imagine adversarial network conditions - Consider crash-restart scenarios at critical moments

  57. [57]

    **Design Verification ** - How do we determine if the attack succeeded? - What specific assertions can capture this issue? ## IMPORTANT: Avoid Unrealistic Scenarios Before proposing any attack scenario, verify it is REALISTIC:

  58. [58]

    Can the trigger condition actually happen in a real deployment?

  59. [59]

    Are there upstream checks that prevent the scenario?

  60. [60]

    CrashInducedLockAmnesiaLeadingToConflictingCommits

    Is the fault model (CFT vs BFT) correctly applied? Examples of UNREALISTIC scenarios to avoid: - Testing CFT protocol with Byzantine behavior (CFT assumes honest nodes) - Assuming attacker can bypass all input validation - Assuming network delivers messages in impossible orders - Creating test states that cannot exist in normal operation **A scenario is o...

  61. [61]

    [What initial state is needed]

  62. [62]

    [...] 29 Agora : Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents Expected Bug Behavior: [What will happen if there’s a bug] Correct Behavior: [How the protocol should respond] Assertions:

  63. [63]

    [How to verify the attack succeeded]

  64. [64]

    Use your knowledge of distributed systems and consensus protocols to identify potential vulnerabilities

    [...] </attack_scenario> User: ## Task Devise a **cunning, creative, ORIGINAL ** attack scenario for {protocol_name}. Use your knowledge of distributed systems and consensus protocols to identify potential vulnerabilities. Think about: - What could go wrong in the implementation? - What edge cases might developers miss? - How could an adversary (or faulty...

  65. [65]

    **Build on the Discovery **: Use the confirmed bug as a foundation

  66. [66]

    **Think Creatively **: The bug reveals a blind spot - where else might it exist?

  67. [67]

    **Avoid Repetition **: Generate genuinely NEW scenarios, not minor variations

  68. [68]

    This reveals an assumption that

    **Stay Realistic **: All scenarios must be achievable in real deployments ## Output Format <thinking> [Analyzing the Confirmed Bug] ...The root cause of this bug is... ...This reveals an assumption that... [Identifying Related Attack Vectors] ...Based on this, I should explore... ...A similar vulnerability might exist in... [Designing New Scenario] 30 Ago...

  69. [69]

    [Required conditions]

  70. [70]

    [...] Expected Bug Behavior: [What happens if this variant exists] Correct Behavior: [Expected correct protocol response] Assertions:

  71. [71]

    [Verification method]

  72. [72]

    TestGen Prompt TestGen Agent System: You are a test code generation expert, skilled at transforming attack scenarios into executable test code

    [...] </attack_scenario> I.3. TestGen Prompt TestGen Agent System: You are a test code generation expert, skilled at transforming attack scenarios into executable test code. ## Your Goal Transform the attack scenario generated by Strategy Agent into:

  73. [73]

    **Repository-style compliant ** test code

  74. [74]

    **Executable, verifiable ** test cases

  75. [75]

    ", usage_example=

    **Clear assertions ** to determine if the attack succeeded ## Available Tools for Repository Knowledge You have these powerful tools to leverage accumulated knowledge: ### ‘repo_knowledge(compact=True)‘ Get cached knowledge about the repository. Use ‘compact=True‘ (default) for LLM-optimized summary. - Contains: test structure, coding style, helper functi...

  76. [76]

    **Analyze Repository Structure ** - Call ‘repo_knowledge()‘ to get test structure, coding style, and helper functions - Look for helpers with ‘signature‘ and ‘usage_example‘ - use them directly - Check ‘lessons_learned‘ for common pitfalls to avoid - Find similar test files as reference if needed 31 Agora : Toward Autonomous Bug Detection in Production-Le...

  77. [77]

    **Generate Test Code ** - Follow the repository’s coding style exactly - Use helper functions with their documented signatures - Use correct package/module declarations - Import necessary dependencies

  78. [78]

    **Add Assertions ** - Detect expected vulnerability behavior - If protocol is correct, test should PASS - If there’s a bug, test should FAIL

  79. [79]

    What went wrong (be specific)

    **Execute Test ** - Write the test file - Run the test command - Collect output results ## Code Quality Requirements - **Correct package name **: Read from existing files, don’t guess - **Correct imports **: Use modules that actually exist in the repository - **Follow style **: Mimic how other tests in the repository are written - **Clear comments **: Exp...

  80. [80]

    Review the repo_knowledge above (especially helper signatures and lessons learned)

Showing first 80 references.