REVIEW 4 major objections 5 minor 25 references
Implementing Decentralized Per-Partition Automatic Failover in Azure Cosmos DB
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Azure Cosmos DB's new per-partition automatic failover restores write availability in under two minutes for every partition-set in a 4,300-partition regional power-outage test.
desk verdict An unusually candid systems paper on per-partition geo-failover in Cosmos DB; the design is plausible and the two-minute RTO is supported by real outage tests, but the shared acceptor-store dependency and missing artifacts keep it from being fully convincing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Failover Manager is a deterministic, formally specified (TLA+/TLC-verified) state machine, one per partition-set, that stores the partition-set's write region, read-lease set, and progress markers and admits compare-and-swap transitions proposed by any replica. State is persisted in a leaderless CAS Paxos replicated state machine without a write-ahead log, with acceptor state held in geographically distributed non-cross-partition-replicated Cosmos DB accounts. A heartbeat every 30 seconds triggers transitions; ungraceful failover waits for a quorum of regions to report, then picks the highest-priority region with the highest reported progress, while graceful failover quiesces writes, drains replication, and switches to the preferred region. An adaptive backoff and time-division multiplexing scheduler tunes CAS Paxos retry delays from measured Phase 2 durations to suppress dueling proposers.
What would settle it
A concrete test: take down the write region and, at the same time, take down a majority of the CAS Paxos acceptor-store regions; if partition-sets still restore write availability within two minutes, the acceptor dependency is not the binding constraint, and if they do not, the two-minute RTO claim is conditional on acceptor-store placement.
Extended reading notes
Core claim
The paper's central claim is that geo-failover in Cosmos DB can be made fine-grained and autonomous: each partition-set runs its own Failover Manager state machine, and replicas propose state transitions through CAS Paxos rounds. The state machine encodes the current write region, the set of read leases, and progress information, and transitions are triggered by partition heartbeats rather than by an operator or a control-plane workflow. The paper asserts that this removes the control plane as a scaling bottleneck and single point of failure, that the formally specified state machine guarantees availability is eventually restored and consistency levels are respected, and that the implementation achieves the stated RTO: availability restored in less than two minutes for every partition-set in the tested regional outage, with automatic failback to the preferred region after power returns.
Load-bearing premise
The failover decision's quorum lives in ordinary non-cross-partition-replicated Cosmos DB accounts, so if those acceptor stores fail or become unreachable at the same time as the data-plane region, the quorum needed to elect a new write region may be gone.
Editorial extensions
If this is right
- Region-wide write availability recovers in under two minutes for every partition-set, so an outage of one data center no longer forces account-wide failover of healthy partitions.
- Failover decisions scale with the number of backend partitions rather than with control-plane capacity, because each partition-set runs its own state machine on the machines that host it.
- Consistency levels and recovery point objectives remain honored: ungraceful failover picks the region with the highest reported progress, and dynamic read-lease quorums let a two-region account continue with one region at a configured minimum durability.
- Failed-over partition-sets automatically fail back to the user's preferred region once it recovers, using a progress table to discard only the 'false progress' written during the outage.
- The adaptive CAS Paxos scheduler keeps proposer failure rates below 0.003% even with nine concurrent proposers, making leaderless consensus practical for the heartbeat workload.
Reading between the lines
- A testable consequence of the design is that failover RTO depends on the geographic placement of the acceptor stores; placing a majority of them in the same region as the write region could violate the two-minute target, so operators should treat acceptor-store placement as a first-class availability decision.
- The same state-machine-plus-CAS-Paxos pattern could be applied to other geo-distributed systems that currently rely on a control plane for failover, provided they can tolerate the extra heartbeat load and the acceptor-store dependency.
- The paper's distinction between graceful and ungraceful failover suggests a broader design principle: any failover system that can distinguish planned from unplanned transitions can use the planned path to bound data loss and the unplanned path to bound downtime.
- Because the client SDK now uses a DNS TXT record and per-partition-set error caches instead of DNS updates, client behavior during failover becomes testable independently of DNS TTL misconfigurations.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the design and implementation of per-partition automatic failover in Azure Cosmos DB, replacing account-level geo-failover with a decentralized state machine per partition-set. The core components are a Failover Manager executing a deterministic state machine, a CAS Paxos store for persisting state machine data, heartbeat-based failure detection, graceful and ungraceful failover modes, and SDK changes to avoid dependence on DNS updates. The paper claims that write availability is restored within two minutes based on three power-outage experiments on a 3-region account with 4,300+ write-region partitions, and that an adaptive scheduling and time-division multiplexing approach reduces CAS Paxos proposer failure rates to at most 0.0028% in simulation.
Significance. If the claims are correct, this work addresses a genuine scalability and availability limitation of control-plane-coordinated geo-failover in a large cloud database, and it provides a detailed architectural blueprint for decentralized per-partition failover that could inform other geo-replicated systems. The paper's strengths include a concrete description of integration with existing Cosmos DB replication and control-plane workflows, a plausible solution to dueling proposers in CAS Paxos, and real outage tests at a non-trivial scale. However, the experimental evidence is narrow: the RTO claim rests on three runs on a single account, and the CAS Paxos evaluation is a simulation without confidence intervals. The paper also explicitly acknowledges a degenerate behavior that it says will be fixed only in the future, which limits the strength of the availability guarantee as presented.
major comments (4)
- [Section 6.1.3] The central RTO claim, stated as 'availability is restored within less than 2 minutes for every partition-set', is supported only by three power-outage runs on a single 3-region account with 4,300+ write-region partitions. The paper reports no raw recovery-time distributions, no confidence intervals, and no breakdown by partition-set. It also does not report acceptor-store latency, throttling, or CAS-conflict rates during the failover burst. Because the abstract claims the solution works 'at any scale', the experiment as presented does not establish that the RTO holds when many partition-sets across many accounts fail over concurrently and share the same acceptor-store capacity (Section 4.3.1). The authors should provide per-partition recovery-time statistics, acceptor-store load metrics under the tested outage, and either a larger-scale test or a validated model of fleet-wide failover load.
- [Section 6.2.3] The claimed reduction to a maximum 0.0028% proposer failure rate comes from a discrete-event simulation that is not sufficiently specified. No confidence intervals or variance measures are reported, the network latency distributions are described only as 'randomly assigned' and 'heterogeneous', and the simulation models 3 to 9 proposers per partition-set with seven acceptors but does not model contention from many partition-sets sharing the same acceptor stores during a regional outage. Since the RTO guarantee depends on CAS Paxos rounds completing within the 45-second lease-enforcer timeout, the simulation should include a model of shared acceptor-store load, or the authors should explicitly scope the claim to per-partition contention only.
- [Section 4.5] The paper acknowledges a degenerate behavior in the state machine: a graceful failover can succeed, then the destination region can fail, triggering an ungraceful failover, and the cycle can repeat, potentially causing a continuous outage. The text states 'We will amend our implementation to account for this', which indicates the fix is not part of the current implementation described in the paper. This is a load-bearing gap because the paper's central claim is that availability is restored within a bounded time. The authors should either present the fix as part of the implemented design and verify it, or explicitly state that the published system does not yet guarantee eventual availability in this scenario and that this is a known limitation.
- [Section 4.3.1] The design stores CAS Paxos acceptor state in a set of geographically distributed non-replicated Cosmos DB accounts that back 'all partitions globally'. The paper argues that the dependency on Cosmos DB is not circular, but the dependency remains a real availability risk: if these acceptor stores fail or throttle under load, quorum-based failover cannot proceed. The outage experiment exercises only one account's partitions and therefore does not test the shared acceptor-store dependency under the burst that would occur in a regional outage affecting many accounts. The paper should include an analysis or experiment that addresses the failure independence and load capacity of the acceptor stores, or it should qualify the RTO claim to scenarios where the acceptor stores are available and unthrottled.
minor comments (5)
- [Section 6.1] The subsection numbering is inconsistent: there are two subsections labeled 6.1.1 and two labeled 6.2.3. The sections should be renumbered for clarity.
- [Figures 7 and 8] Figure 7 and Figure 8 lack explicit axis labels and units. 'Time' should be labeled as seconds or minutes, and the y-axis should indicate the fraction or count of partition-sets.
- [Section 4.4] The TLA+ verification is described only through a short invariant snippet. No model-checking results, TLA+ specification files, or list of checked properties are included, making the verification claim difficult to assess. If the authors want the verification to be a credible contribution, they should provide more detail or artifacts.
- [Section 4.5] The phrase 'We will amend our implementation to account for this' should be changed to 'We amended' or explicitly moved to the Future Work section; as written, it reads as an admission that the current implementation does not handle the described scenario.
- [Section 6.2.2] The text says the simulator can 'compress years of system operation', but each experiment simulates one hour of operational time, with 10,000 simulations totaling about 1.14 years. This sentence should be reworded to avoid implying that a single simulation spans years.
Circularity Check
No circular derivation: the failover design is evaluated by external outage tests, an independent TLA+ specification, and simulations against a baseline; the acceptor-store dependency is explicitly discussed and is a design risk, not a circular argument.
full rationale
The paper's central claims are empirical and protocol-based, not derived from a fitted model. The RTO claim (Section 6.1.3) is supported by three power-outage tests on a 3-region account with 4,300+ write-region partitions, which are external observations rather than consequences of the paper's assumptions. The Failover Manager state machine is specified in TLA+ and model-checked with TLC (Section 4.4), an independent mechanical check of the stated safety/liveness properties; no property is assumed by defining it in terms of the result. The CAS Paxos implementation is transliterated from an external TLA+ spec (Section 4.3.1) with no self-citation chain. The adaptive scheduling improvement (Section 6.2) is evaluated by discrete-event simulation that compares the new backoff/TDM policy against an exponential-backoff baseline and reports failure rates; this is a comparative measurement, not a prediction forced by construction. The only apparent self-dependency is that CAS Paxos acceptor state is stored in non-replicated Cosmos DB accounts (Section 4.3.1). The paper explicitly addresses this: 'This might appear to create a circular dependency from Cosmos DB onto Cosmos DB. This is illusory...' That passage concerns an availability/dependency risk, not a logical derivation; the failover protocol's correctness does not assume the claim it is used to demonstrate. No load-bearing step reduces to its own inputs by definition, so the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Lease enforcement timeout =
45 seconds
- Proposer update interval =
30 seconds
- Base delay δ in initial exponential backoff =
not specified
- Number of acceptors =
7
- Number of proposers =
3, 5, 7, 9
assumptions (4)
- standard math CAS Paxos provides a replicated state machine without logs as described in [1].
- domain assumption The TLA+ model accurately models the real Failover Manager state machine and Cosmos DB replication behavior.
- domain assumption The acceptor state stores on non-replicated Cosmos DB accounts are independent of the data-plane and remain available during outages.
- domain assumption The discrete-event simulator's latency and failure models represent production network behavior.
invented entities (3)
-
Failover Manager
-
Progress table
-
Topology upsert intent
Cite this review
Pith. "Pith review of Implementing Decentralized Per-Partition Automatic Failover in Azure Cosmos DB." pith.science (2026). https://pith.science/paper/KVOQRN46
@misc{pith2026250514900,
author = {Pith},
title = {Pith review of: Implementing Decentralized Per-Partition Automatic Failover in Azure Cosmos DB},
year = {2026},
howpublished = {\url{https://pith.science/paper/KVOQRN46}},
note = {Machine review of arXiv:2505.14900}
}
read the original abstract
Azure Cosmos DB is a cloud-native distributed database, operating at a massive scale, powering Microsoft Cloud. Think 10s of millions of database partitions (replica-sets), 100+ PBs of data under management, 20M+ vCores. Failovers are an integral part of distributed databases to provide data availability during outages (partial or full regional outages). While failovers within a replica-set within a single region are well understood and commonly exercised, geo failovers in databases across regions are not as common and usually left as a disaster recovery scenario. An upcoming release of Azure Cosmos DB introduces a fine grained (partition-level) automatic failover solution for geo failovers that minimizes the Recovery Time Objective (RTO) and honors customer-chosen consistency level and Recovery Point Objective (RPO) at any scale. This is achieved thanks to a decentralized architecture which offers seamless horizontal scaling to allow us to handle outages ranging from node-level faults to full-scale regional outages. Our solution is designed to handle a broad spectrum of hardware and software faults, including node failures, crashes, power events and most network partitions, that span beyond the scope of a single fault domain or an availability zone.
Reference graph
Works this paper leans on
-
[1]
CASPaxos: Replicated State Machines without logs
D. Rystsov, "CASPaxos: Replicated State Machines without logs," 2018. [Online]. Available: https://arxiv.org/abs/1802.07000
work page Pith review arXiv 2018
-
[2]
Microsoft, "Azure Cosmos DB," [Online]. Available: https://learn.microsoft.com/en-us/azure/cosmos-db/. [Accessed 2025]
work page 2025
-
[3]
Consistency levels in Azure Cosmos DB,
Microsoft, "Consistency levels in Azure Cosmos DB," 2024. [Online]. Available: https://learn.microsoft.com/en-us/azure/cosmos-db/consistency-levels. [Accessed 2025]
work page 2024
-
[4]
Partitioning and horizontal scaling in Azure Cosmos DB,
Microsoft, "Partitioning and horizontal scaling in Azure Cosmos DB," 11 2024. [Online]. Available: https://learn.microsoft.com/en -us/azure/cosmos- db/partitioning-overview. [Accessed 2025]
work page 2024
-
[5]
Commonly asked Service Fabric questions,
Microsoft, "Commonly asked Service Fabric questions," 2024. [Online]. Available: https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric- common-questions#can-i-create-a-cluster-that-spans-multiple-azure-regions-or- my-own-datacenters. [Accessed 2025]
work page 2024
-
[6]
Windows Workflow Foundation Programming - Dynamic update,
Microsoft, "Windows Workflow Foundation Programming - Dynamic update," [Online]. Available: https://learn.microsoft.com/en - us/dotnet/framework/windows-workflow-foundation/dynamic-update. [Accessed 2025]
work page 2025
-
[7]
L. Lamport, "The Paxos Algorithm," October 2024. [Online]. Available: https://lamport.azurewebsites.net/tla/paxos-algorithm.html. [Accessed 2025]
work page 2024
-
[8]
T. Grieger, "CASPaxos -tla," [Online]. Available: https://github.com/tbg/caspaxos-tla. [Accessed 2025]
work page 2025
Show all 25 references
-
[9]
Transactions and optimistic concurrency control,
Microsoft, "Transactions and optimistic concurrency control," [Online]. Available: https://learn.microsoft.com/en -us/azure/cosmos-db/nosql/database- transactions-optimistic-concurrency
-
[10]
Deploy cloud witness for a failover cluster,
Microsoft, "Deploy cloud witness for a failover cluster," February 2025. [Online]. Available: https://learn.microsoft.com/en -us/windows-server/failover- clustering/deploy-cloud-witness. [Accessed 2025]
2025
-
[11]
The TLA+ Home Page,
L. Lamport, "The TLA+ Home Page," [Online]. Available: https://lamport.azurewebsites.net/tla/tla.html
-
[12]
Understanding cluster and pool quorum,
Microsoft, "Understanding cluster and pool quorum," February 2025. [Online]. Available: https://learn.microsoft.com/en -us/windows-server/storage/storage- spaces/quorum. [Accessed 2025]
2025
-
[13]
Serving Stale Data to Improve DNS Resiliency,
D. Lawrence, W. Kumari and P. Sood, "Serving Stale Data to Improve DNS Resiliency," March 2020. [Online]. Available: https://www.rfc - editor.org/info/rfc8767. [Accessed 2025]
2020
-
[14]
Paxos made simple.,
L. Lamport, "Paxos made simple.," ACM SIGACT News (Distributed Computing Column), vol. 121, pp. 51-58, December 2001
2001
-
[15]
Azure network round -trip latency statistics,
Microsoft, "Azure network round -trip latency statistics," [Online]. Available: https://learn.microsoft.com/en-us/azure/networking/azure-network- latency?tabs=Americas%2CWestUS. [Accessed 2025]
2025
-
[16]
Global distribution with Azure Cosmos DB - under the hood | Microsoft Learn
Microsoft, “ Global distribution with Azure Cosmos DB - under the hood | Microsoft Learn” [Online]. Available Global distribution with Azure Cosmos DB- under the hood | Microsoft Learn . [Accessed 2025]. Conference Short Name:WOODSTOCK’18
2025
-
[17]
Lamport, D
L. Lamport, D. Malkhi, and L. Zhou. Vertical Paxos and primary -backup replication. Technical report, Microsoft Research, 2009
2009
-
[18]
Lamport, D
L. Lamport, D. Malkhi, and L. Zhou. Reconfiguring a state machine. SIGACT News, 41(1), Mar. 2010
2010
-
[19]
Liskov and J
B. Liskov and J. Cowling. Viewstamped replication revis ited. Technical Report MIT-CSAIL-TR-2012-021, MIT Computer
2012
-
[20]
Andersen, and Michael Kaminsky
Iulian Moraru, David G. Andersen, and Michael Kaminsky. 2013. There is more consensus in Egalitarian parliaments. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (SOSP '13). Association for Computing Machinery, New York, N Y, USA, 358 –372. ht...
2013
-
[21]
Consistency level choices - Azure Cosmos DB | Microsoft Learn
Microsoft, “Consistency level choices - Azure Cosmos DB | Microsoft Learn ” [Online]. Available: https://learn.microsoft.com/en-us/azure/cosmos- db/consistency-levels
-
[22]
Specifying Systems : the TLA+ Language and Tools for Hardware and Software Engineers
Lamport, Leslie. Specifying Systems : the TLA+ Language and Tools for Hardware and Software Engineers. Boston :Addison-Wesley, 2003
2003
-
[23]
Raft Consensus Algorithm: an Effective Substitute for Paxos in High Throughput P2P -based Systems
Fazlali, Mohammad Reza et al. “Raft Consensus Algorithm: an Effective Substitute for Paxos in High Throughput P2P -based Systems.” ArXiv abs/1911.01231 (2019)
2019 arXiv
-
[24]
Azure Cosmos DB SQL SDK connectivity modes ,
Microsoft, “Azure Cosmos DB SQL SDK connectivity modes ,” [Online]. Available: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/sdk- connection-modes
-
[25]
Leslie Lamport. 1998. The part-time parliament. ACM Trans. Comput. Syst. 16, 2 (May 1998), 133–169. https://doi.org/10.1145/279227.279229
1998
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.