REVIEW 2 major objections 6 minor 46 references
The fault-tolerant cluster-sending problem
T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper formalizes cluster-sending, proves linear lower bounds for it, and gives protocols that meet them.
desk verdict A solid formalization of a useful cluster-sending primitive with mostly sound lower bounds and matching protocols; the crash lower bound misses one case, but it is easily patched. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Bijective sending is the load-bearing mechanism: choose equal-sized subsets $S_1 \subseteq C_1$ and $S_2 \subseteq C_2$, fix a bijection between them, and have each selected sender send the agreed value to its matched receiver. Because the sets are large enough that at least one non-faulty sender is paired with a non-faulty receiver, that receiver broadcasts the value inside $C_2$, giving receipt and confirmation. Partitioned bijective sending generalizes this to unequal clusters by cutting the larger cluster into $c$-partitions and running independent bijections for enough partitions; the lower-bound proofs use counting arguments over worst-case faulty senders and receivers. The identity that carries the argument is the relation between the required number of non-faulty sender-receiver pairs ($f_{C_2}+1$ for cluster signing, $f_{C_1}+1$ for replica signing) and the chosen subset sizes.
What would settle it
Run the worst-case crash pattern behind Theorem 4.1 for concrete small clusters, e.g. $n_{C_1}=3$, $f_{C_1}=1$, $n_{C_2}=2$, $f_{C_2}=0$; the claimed bound is $\sigma_1=2$ messages. A protocol that reliably delivers the value with a single message in that configuration would refute the lower bound, so systematically searching all one-message sender-receiver choices under that crash pattern is a direct test.
Extended reading notes
Core claim
The paper's central claim is that the cost of reliably sending a message between two Byzantine clusters is fundamentally linear, not quadratic. Under crash failures, any correct protocol must exchange at least $\sigma_i$ messages, where $\sigma_i$ counts the smallest number of sender-receiver pairs that survive the worst crash pattern; under Byzantine failures with replica signing, any correct protocol must exchange at least $\tau_i$ certificates signed by distinct sender replicas. These bounds hold for asynchronous reliable communication and do not depend on delivery timing. The paper further claims that the bijective-sending protocols BS-bcs, BS-brs, and the partitioned variants SPBS and RPBS attain these bounds under the stated resilience conditions ($n_C > 3f_C$ with cluster signing or crash/omission failures, $n_C > 4f_C$ with replica signing), giving optimal communication complexity.
Load-bearing premise
The protocols assume replicas inside a cluster can reach agreement with an off-the-shelf consensus protocol and that messages sent by non-faulty replicas eventually arrive; if intra-cluster consensus is unavailable or the network can drop messages indefinitely, the correctness proofs and linear lower bounds no longer apply.
Editorial extensions
If this is right
- No cluster-sending protocol can beat roughly $\max(n_{C_1}, n_{C_2})$ messages or certificates, so the linear floor is now a design target.
- Sharded Byzantine systems can send values between clusters with only linear wide-area messages instead of quadratic all-to-all communication.
- With cluster signing, message size stays $O(\|v\|)$; with replica signing, certificates can be emulated at message size $O(\|v\| + f_{C_1})$ or optimized when certificates have constant size.
- The protocols are one-way and leaderless, so they remain correct under dynamically changing faulty sets and do not require an acknowledgement phase in the reliable network model.
Reading between the lines
- A fair end-to-end comparison in practice must add the local consensus cost inside each cluster, which the lower bounds deliberately exclude; for very small clusters that cost could dominate the linear cross-cluster count.
- The linear bound suggests cross-shard transactions in sharded Byzantine databases can be built from this primitive; the protocols here would provide the inter-shard send step.
- One testable extension is to benchmark the partitioned protocols on highly unequal cluster sizes, where certificate emulation's $O(\|v\| + f_{C_1})$ message size may make broadcast-based baselines competitive despite higher message counts.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the cluster-sending problem, in which a value v is sent from cluster C1 to cluster C2 so that all non-faulty replicas of C2 receive v, v is sent only if C1 agreed, and C1 can confirm receipt. After formalizing the model with crash/omission/Byzantine failures, replica signing, and cluster signing, the paper gives broadcast-based protocols (RB-bcs, RB-brs), proves lower bounds on message complexity for crash failures (Theorem 4.1) and on certificate complexity for Byzantine failures with replica signing (Theorem 4.2), and then presents bijective-sending protocols (BS, SPBS, RPBS) that it claims meet these lower bounds and are therefore optimal (Theorem 6.6). The protocols exploit the observation that linear communication between clusters can replace quadratic all-to-all broadcasting.
Significance. If the optimality claims are fully established, the paper identifies a clean primitive for sharded Byzantine fault-tolerant systems and shows that cluster-to-cluster communication can be linear in cluster size, with matching lower bounds under two failure models. The paper's contributions are concrete and falsifiable, and the protocol constructions (bijective and partitioned bijective sending) are elegant and likely reusable. The main caveat is that one of the two cases of the crash lower bound is not proved in the text, and the Byzantine lower bound's second case is only sketched; because these cases are needed for the optimality of the RPBS protocols, the contribution is not yet fully supported.
major comments (2)
- [Section 4, Theorem 4.1] The theorem states a lower bound for both orderings of nC1 and nC2 via the index i for the larger cluster, but the proof begins "We assume i = 1, j = 2, and nC1 ≥ nC2" and never treats i = 2, j = 1. This is not a symmetry argument: C1 is always the sender and must confirm receipt, and the bound counts messages from C1 to C2, so interchanging the cluster names changes the problem. The missing case is exactly the regime used by RPBS-(σ2,bcs) in Theorem 6.6(1). Please give a full proof that any protocol using at most σ2−1 messages has a failure pattern in which no message from a non-faulty C1 replica reaches a non-faulty C2 replica.
- [Section 4, Theorem 4.2(2)] The proof of the nC2 ≥ nC1 case is only sketched. The step "Via a contradiction argument similar to the one used in the proof of Theorem 4.1, one can show..." needs to be written out, since the quantities involved (nfC1 − fC1, fC2 + 1, 2fC1) are not the same as in the nC1 ≥ nC2 case. More importantly, the concluding indistinguishability argument is incomplete: after partitioning CR into CR,1 and CR,2, each signed by at most fC1 distinct replicas, it is not shown that the receiving cluster cannot tell the two executions apart; in particular the number of certificates per value and the protocol's acceptance rule are not addressed. This case underpins the optimality claim for RPBS-(τ2,brs) in Theorem 6.6(2).
minor comments (6)
- [Section 2] The sentence "We extend the notations f(·), nf(·), n(·), f(·), and nf(·) to arbitrary sets of replicas" duplicates f(·) and omits n(·); the notation list should be corrected.
- [Section 5, Proposition 5.3] In the proof, "nf S1 ≥ f(C1) + 1" should be "nf S1 ≥ fC1 + 1", since f(C1) is a set and the count is meant.
- [Section 4, Theorem 4.2] The proof uses C both for a cluster and for the set of certificates sent in a run ("messages C, each message representing a single certificate"), which makes the argument harder to follow; a different name for the certificate set would help.
- [Section 6, Propositions 6.4 and 6.5] The letter P is used both for the chosen subset of a cluster and for a generic element of its partition ("For each P ∈ partition(P)"), and the same overloading occurs for z(P); renaming the partition elements would improve readability.
- [Theorem 6.6(2)] The phrase "Byzantine failures and replica sending" should read "Byzantine failures and replica signing" to match the terminology used throughout the paper.
- [References] Reference [22] appears as "33nd International Symposium on Distributed Computing"; the ordinal should be "33rd".
Circularity Check
No circularity: the cluster-sending lower bounds are proved independently of the proposed protocols, whose optimality is then shown by matching those bounds.
full rationale
The paper's central derivation chain is not circular. The cluster-sending problem is defined in Section 2 (Definition 2.1), and the lower bounds in Theorems 4.1 and 4.2 are proved by adversary/pigeonhole arguments from that definition, not from the protocols. The protocols BS, SPBS, and RPBS are then constructed in Sections 5-6 and proved correct; Theorem 6.6 claims optimality only because the protocol message counts are shown to equal the independently derived lower-bound values sigma_1, sigma_2, tau_1, and tau_2. No parameter is fitted to data, no 'prediction' is a renamed input, and no load-bearing uniqueness theorem is imported from the authors' prior work. The only self-citation, the brief announcement [22], is not load-bearing. One non-circular defect should be noted: the proof of Theorem 4.1 begins 'We assume i=1, j=2, and nC1>=nC2' and does not return to the nC2>=nC1 case, which is the case needed for the RPBS optimality claims in Proposition 6.5 and Theorem 6.6. That is an omitted proof case, not a circular reduction: the missing case would require an independent argument, not an appeal to the paper's own conclusion. Similarly, Theorem 4.2's second case says 'Via a contradiction argument similar to the one used in the proof of Theorem 4.1' without giving the full argument; again this is a proof-completeness gap rather than circularity. Because the derivation of the lower bounds is not equivalent to the protocols' construction, the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Asynchronous reliable communication: all messages sent by non-faulty replicas eventually arrive.
- domain assumption Replica signing and cluster signing are non-forgeable, and faulty replicas cannot impersonate non-faulty replicas.
- domain assumption Local communication within a cluster is free and unbounded, while global communication is costly.
- domain assumption Each cluster can reach agreement on a value using an off-the-shelf consensus protocol when necessary, requiring nC > 2fC for the sending cluster.
Cite this review
Pith. "Pith review of The fault-tolerant cluster-sending problem." pith.science (2026). https://pith.science/paper/EWD4KFGE
@misc{pith2026190801455,
author = {Pith},
title = {Pith review of: The fault-tolerant cluster-sending problem},
year = {2026},
howpublished = {\url{https://pith.science/paper/EWD4KFGE}},
note = {Machine review of arXiv:1908.01455}
}
read the original abstract
The development of fault-tolerant distributed systems that can tolerate Byzantine behavior has traditionally been focused on consensus protocols, which support fully-replicated designs. For the development of more sophisticated high-performance Byzantine distributed systems, more specialized fault-tolerant communication primitives are necessary, however. In this paper, we identify an essential communication primitive and study it in depth. In specifics, we formalize the cluster-sending problem, the problem of sending a message from one Byzantine cluster to another Byzantine cluster in a reliable manner. We not only formalize this fundamental problem, but also establish lower bounds on the complexity of this problem under crash failures and Byzantine failures. Furthermore, we develop practical cluster-sending protocols that meet these lower bounds and, hence, have optimal complexity. As such, our work provides a strong foundation for the further exploration of novel designs that address challenges encountered in fault-tolerant distributed systems.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Steward: Scaling byzantine fault-tolerant replication to wide area networks
Yair Amir, Claudiu Danilov, Danny Dolev, Jonathan Kirsch, John Lane, Cristina Nita- Rotaru, Josh Olsen, and David Zage. Steward: Scaling byzantine fault-tolerant replication to wide area networks. IEEE Transactions on Dependable and Secure Computing, 7(1):80–93,
-
[2]
Blockchain for development: Emerging opportunities for mobile, iden- tity and aid, 2017
GSM Association. Blockchain for development: Emerging opportunities for mobile, iden- tity and aid, 2017. URL: https://www.gsma.com/mobilefordevelopment/wp-content/ uploads/2017/12/Blockchain-for-Development.pdf. 12
work page 2017
-
[3]
Pierre-Louis Aublin, Rachid Guerraoui, Nikola Kneˇ zevi´ c, Vivien Qu´ ema, and Marko Vukoli´ c. The next 700 bft protocols. ACM Transactions on Computer Systems , 32(4):12:1–12:45,
-
[4]
RBFT: Redundant byzantine fault tolerance
Pierre-Louis Aublin, Sonia Ben Mokhtar, and Vivien Qu´ ema. RBFT: Redundant byzantine fault tolerance. In 2013 IEEE 33rd International Conference on Distributed Computing Systems, pages 297–306. IEEE, 2013. doi:10.1109/ICDCS.2013.53
-
[5]
Christian Berger and Hans P. Reiser. Scaling byzantine consensus: A broad analysis. In Proceedings of the 2Nd Workshop on Scalable and Resilient Infrastructures for Distributed Ledgers, SERIAL’18, pages 13–18. ACM, 2018. doi:10.1145/3284764.3284767
arXiv 2018
-
[6]
Blockchain in Europe: Closing the strategy gap
Burkhard Blechschmidt. Blockchain in Europe: Closing the strategy gap. Technical report, Cognizant Consulting, 2018. URL: https://www.cognizant.com/whitepapers/ blockchain-in-europe-closing-the-strategy-gap-codex3320.pdf
work page 2018
-
[7]
CAP twelve years later: How the “rules” have changed
Eric Brewer. CAP twelve years later: How the “rules” have changed. Computer, 45(2):23–29,
-
[8]
Eric A. Brewer. Towards robust distributed systems (abstract). In Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing , pages 7–7. ACM, 2000. doi:10.1145/343477.343502
arXiv 2000
Show all 46 references
-
[9]
The impact of blockchain technology on finance: A catalyst for change
Michael Casey, Jonah Crane, Gary Gensler, Simon Johnson, and Neha Narula. The impact of blockchain technology on finance: A catalyst for change. Technical report, International Center for Monetary and Banking Studies, 2018. URL: https://www.cimb.ch/uploads/ 1/1/5/4/115414161/ge...
2018
-
[10]
Practical byzantine fault tolerance
Miguel Castro and Barbara Liskov. Practical byzantine fault tolerance. In Proceedings of the Third Symposium on Operating Systems Design and Implementation , pages 173–186. USENIX Association, 1999
1999
-
[11]
Practical byzantine fault tolerance and proactive recovery
Miguel Castro and Barbara Liskov. Practical byzantine fault tolerance and proactive recovery. ACM Transactions on Computer Systems , 20(4):398–461, 2002. doi:10.1145/ 571637.571640
2002
-
[12]
Major collection of the fall auction season to be recorded with blockchain technology, 2018
Christie’s. Major collection of the fall auction season to be recorded with blockchain technology, 2018. URL: https://www.christies.com/presscenter/pdf/9160/RELEASE_ ChristiesxArtoryxEbsworth_9160_1.pdf
2018
-
[13]
Making byzantine fault tolerant systems tolerate byzantine faults
Allen Clement, Edmund Wong, Lorenzo Alvisi, Mike Dahlin, and Mirco Marchetti. Making byzantine fault tolerant systems tolerate byzantine faults. In Proceedings of the 6th USENIX Symposium on Networked Systems Design and Implementation , pages 153–168. USENIX Association, 2009
2009
-
[14]
Blockchain and GDPR: How blockchain could address five areas associated with gdpr compliance
Cindy Compert, Maurizio Luinetti, and Bertrand Portier. Blockchain and GDPR: How blockchain could address five areas associated with gdpr compliance. Technical report, IBM Security, 2018. URL: https://public.dhe.ibm.com/common/ssi/ecm/61/ en/61014461usen/security-ibm-security-s...
2018
-
[15]
Dolev and H
D. Dolev and H. Strong. Authenticated algorithms for byzantine agreement. SIAM Journal on Computing, 12(4):656–666, 1983. doi:10.1137/0212045
1983 doi
-
[16]
Blockchain for agriculture and food: Findings from the pilot study
Lan Ge, Christopher Brewster, Jacco Spek, Anton Smeenk, and Jan Top. Blockchain for agriculture and food: Findings from the pilot study. Technical report, Wageningen Univer- sity, 2017. URL: https://www.wur.nl/nl/Publicatie-details.htm?publicationId= publication-way-353330323634
2017
-
[17]
Algorand: Scaling byzantine agreements for cryptocurrencies
Yossi Gilad, Rotem Hemo, Silvio Micali, Georgios Vlachos, and Nickolai Zeldovich. Algorand: Scaling byzantine agreements for cryptocurrencies. In Proceedings of the 26th Symposium on Operating Systems Principles, pages 51–68. ACM, 2017. doi:10.1145/3132747.3132757
2017
-
[18]
Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services
Seth Gilbert and Nancy Lynch. Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News, 33(2):51–59, 2002. doi:10.1145/ 564585.564601. 13
2002
-
[19]
Gordon and Christian Catalini
William J. Gordon and Christian Catalini. Blockchain technology for healthcare: Facilitating the transition to patient-driven interoperability.Computational and Structural Biotechnology Journal, 16:224–230, 2018. doi:10.1016/j.csbj.2018.06.003
2018 doi
-
[20]
Brief announcement: revisiting consensus protocols through wait-free parallelization
Suyash Gupta, Jelle Hellings, and Mohammad Sadoghi. Brief announcement: revisiting consensus protocols through wait-free parallelization. In 33nd International Symposium on Distributed Computing, 2019
2019
-
[21]
Blockchain Transaction Processing, pages 1–11
Suyash Gupta and Mohammad Sadoghi. Blockchain Transaction Processing, pages 1–11. Springer International Publishing, 2018. doi:10.1007/978-3-319-63962-8_333-1
2018 doi
-
[22]
Brief announcement: the fault-tolerant cluster- sending problem
Jelle Hellings and Mohammad Sadoghi. Brief announcement: the fault-tolerant cluster- sending problem. In 33nd International Symposium on Distributed Computing , 2019
2019
-
[23]
The promise of blockchain
Matt Higginson, Johannes-Tobias Lorenz, Bjrn Mnstermann, and Peter Braad Ole- sen. The promise of blockchain. Technical report, McKinsey&Company, 2017. URL: https://www.mckinsey.com/industries/financial-services/our-insights/the- promise-of-blockchain
2017
-
[24]
Kamel Boulos, James T
Maged N. Kamel Boulos, James T. Wilson, and Kevin A. Clauson. Geospatial blockchain: promises, challenges, and scenarios in health and healthcare. International Journal of Health Geographics, 17(1):1211–1220, 2018. doi:10.1186/s12942-018-0144-x
2018 doi
-
[25]
CheapBFT: Resource-efficient byzantine fault tolerance
R¨ udiger Kapitza, Johannes Behl, Christian Cachin, Tobias Distler, Simon Kuhnle, Seyed Vahid Mohammadi, Wolfgang Schr¨ oder-Preikschat, and Klaus Stengel. CheapBFT: Resource-efficient byzantine fault tolerance. In Proceedings of the 7th ACM European Confer- ence on Computer Sys...
2012
-
[26]
Zyzzyva: Speculative byzantine fault tolerance
Ramakrishna Kotla, Lorenzo Alvisi, Mike Dahlin, Allen Clement, and Edmund Wong. Zyzzyva: Speculative byzantine fault tolerance. In Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles , pages 45–58. ACM, 2007. doi: 10.1145/1294261.1294267
2007
-
[27]
Zyzzyva: Speculative byzantine fault tolerance
Ramakrishna Kotla, Lorenzo Alvisi, Mike Dahlin, Allen Clement, and Edmund Wong. Zyzzyva: Speculative byzantine fault tolerance. ACM Transactions on Computer Systems , 27(4):7:1–7:39, 2009. doi:10.1145/1658357.1658358
2009
-
[28]
The implementation of reliable distributed multiprocess systems
Leslie Lamport. The implementation of reliable distributed multiprocess systems. Computer Networks (1976), 2(2):95–114, 1978. doi:10.1016/0376-5075(78)90045-4
1976 doi
-
[29]
Paxos made simple
Leslie Lamport. Paxos made simple. ACM SIGACT News, Distributed Computing Column 5, 32(4):51–58, 2001. doi:10.1145/568425.568433
2001
-
[30]
Karame, and N
Jian Liu, Wenting Li, Ghassan O. Karame, and N. Asokan. Scalable byzantine consensus via hardware-assisted secret sharing. IEEE Transactions on Computers , 68(1):139–151,
-
[31]
Menezes, Scott A
Alfred J. Menezes, Scott A. Vanstone, and Paul C. Van Oorschot. Handbook of Applied Cryptography. CRC Press, Inc., 1st edition, 1996
1996
-
[32]
Bitcoin: A peer-to-peer electronic cash system
Satoshi Nakamoto. Bitcoin: A peer-to-peer electronic cash system. URL: https://bitcoin. org/en/bitcoin-paper
-
[33]
Blockplane: A global-scale byzantizing middleware
Faisal Nawab and Mohammad Sadoghi. Blockplane: A global-scale byzantizing middleware. In 35th International Conference on Data Engineering (ICDE) , pages 124–135. IEEE, 2019. doi:10.1109/ICDE.2019.00020
2019
-
[34]
Tamer ¨Ozsu and Patrick Valduriez
M. Tamer ¨Ozsu and Patrick Valduriez. Principles of Distributed Database Systems . Springer New York, 3th edition, 2011
2011
-
[35]
Pease, R
M. Pease, R. Shostak, and L. Lamport. Reaching agreement in the presence of faults. Journal of the ACM , 27(2):228–234, 1980. doi:10.1145/322186.322188
1980
-
[36]
Blockchain and economic development: Hype vs
Michael Pisa and Matt Juden. Blockchain and economic development: Hype vs. reality. Technical report, Center for Global Development, 2017. URL: https://www.cgdev.org/ publication/blockchain-and-economic-development-hype-vs-reality
2017
-
[37]
Blockchain – an opportunity for energy producers and consumers?,
PwC. Blockchain – an opportunity for energy producers and consumers?,
-
[38]
Practical threshold signatures
Victor Shoup. Practical threshold signatures. In Advances in Cryptology — EUROCRYPT 2000, pages 207–220. Springer Berlin Heidelberg, 2000. doi:10.1007/3-540-45539-6_15
2000 doi
-
[39]
Introduction to Distributed Algorithms
Gerard Tel. Introduction to Distributed Algorithms . Cambridge University Press, 2nd edition, 2001
2001
-
[40]
Tanenbaum
Maarten van Steen and Andrew S. Tanenbaum. Distributed Systems. Maarten van Steen, 3th edition, 2017. URL: https://www.distributed-systems.net/
2017
-
[41]
Efficient byzantine fault-tolerance
Santos Veronese Veronese, Miguel Correia, Alysson Neves Bessani, Lau Cheuk Lung, and Paulo Verissimo. Efficient byzantine fault-tolerance. IEEE Transactions on Computers , 62(1):16–30, 2013. doi:10.1109/TC.2011.221
2013 doi
-
[42]
Ethereum: a secure decentralised generalised transaction ledger
Gavin Wood. Ethereum: a secure decentralised generalised transaction ledger. EIP-150 revision. URL: https://gavwood.com/paper.pdf. 15
-
[2010]
doi:10.1109/TDSC.2008.53
2008 doi
-
[2012]
doi:10.1109/MC.2012.37
2012 doi
-
[2016]
URL: https://www.pwc.com/gx/en/industries/energy-utilities-resources/ 14 publications/opportunity-for-energy-producers.html
-
[2019]
doi:10.1109/TC.2018.2860009
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.