{"id":"998053c9-10cb-4a1d-8387-7e9a8f32943c","arxiv_id":"2505.14900","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Cosmos DB introduces per-partition automatic geo-failover driven by a CAS Paxos-backed Failover Manager state machine, restoring write availability in under two minutes in tests.","lead":"Azure Cosmos DB now fails over each partition independently across regions instead of moving whole accounts, using a decentralized state machine. The design could reduce recovery time after regional outages and is a real-world test of running a consensus protocol on the database itself.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RTO claim is tested against a single 4,300-partition account, but all partitions share the same CAS Paxos acceptor stores; a fleet-wide outage could saturate those stores and break the <2-minute target before per-partition correctness matters.","rationale":"The reader's conditional verdict is appropriate, and my concern does not move it. The paper presents a plausible decentralized design, and the power-outage experiment on 4,300+ partitions is genuine evidence that per-partition failover can work in a moderate-scale account. However, the central RTO claim is explicitly framed as holding 'at any scale' and for 'full-scale regional outages,' while the experimental and simulation evidence does not stress the one component that is shared across all partition-sets: the CAS Paxos acceptor store. The reader focused on the acceptor store's correlated-failure risk; I focus on its capacity and contention risk under simultaneous failover load. Both are variants of the same dependency concern, so I mark partial agreement. The proposed test is concrete: if the authors can show that the shared acceptor stores sustain the fleet-scale CAS write rate with P99 Phase 2 completion well below the 45-second lease timeout, the concern is settled. If not, the paper should either provide that evidence or qualify the RTO claim to the tested scale. This does not require rejecting the paper, but it does justify keeping the verdict conditional on additional evidence.","tokens_in":15245,"tokens_out":7612,"duration_ms":72321,"concrete_test":"Compute the required CAS Paxos write rate at fleet scale: for a regional outage affecting M partition-sets, each partition-set produces roughly one heartbeat update per 30 seconds, so the shared acceptor stores must absorb at least M/30 CAS operations per second plus failover bursts and retries. Compare this with the provisioned throughput of the non-replicated Cosmos DB accounts described in Section 4.3.1. Then run a fault-injection or large-scale simulation in which 100,000 partition-sets fail over concurrently against the same acceptor-store configuration, and measure the P95/P99 and maximum Phase 2 completion latency. If the P99 round time approaches or exceeds the 45-second lease-enforcer timeout, or if throttling events occur, the <2-minute RTO claim at scale is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central RTO claim (Section 6.1.3: availability restored in under 2 minutes for every partition-set) depends on every failover CAS Paxos round completing within the lease-enforcement window. Section 4.3.1 states that a set of geographically distributed, non-replicated Cosmos DB accounts stores the CAS Paxos acceptor state and backs all partitions globally. All partition-sets therefore share the same acceptor-store capacity. During a regional outage, every affected partition-set attempts failover roughly simultaneously, generating a burst of CAS Paxos Phase 1 and Phase 2 operations against these shared accounts. The experiment in Section 6.1 covers a single 3-region account with 4,300+ write-region partitions, but it does not report acceptor-store latency, throttling, or CAS-conflict rates under that concurrent load. The CAS Paxos evaluation in Section 6.2 is a discrete-event simulation with 3 to 9 proposers per partition-set and seven acceptors; it models per-partition proposer contention, not the shared-acceptor-store load from many partition-sets failing over at once. If the shared acceptor stores throttle or queue under fleet-scale load, CAS rounds can exceed the 45-second lease-enforcer timeout, and the <2-minute RTO will not hold at the scale claimed in the abstract. This is closely related to the reader's concern about acceptor-store dependency, but the more immediate threat is that the experiment does not exercise the shared dependency under the load that defines the scalability claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the design and implementation of per-partition automatic failover in Azure Cosmos DB, replacing account-level geo-failover with a decentralized state machine per partition-set. The core components are a Failover Manager executing a deterministic state machine, a CAS Paxos store for persisting state machine data, heartbeat-based failure detection, graceful and ungraceful failover modes, and SDK changes to avoid dependence on DNS updates. The paper claims that write availability is restored within two minutes based on three power-outage experiments on a 3-region account with 4,300+ write-region partitions, and that an adaptive scheduling and time-division multiplexing approach reduces CAS Paxos proposer failure rates to at most 0.0028% in simulation.","tokens_in":15600,"tokens_out":4180,"duration_ms":37929,"significance":"If the claims are correct, this work addresses a genuine scalability and availability limitation of control-plane-coordinated geo-failover in a large cloud database, and it provides a detailed architectural blueprint for decentralized per-partition failover that could inform other geo-replicated systems. The paper's strengths include a concrete description of integration with existing Cosmos DB replication and control-plane workflows, a plausible solution to dueling proposers in CAS Paxos, and real outage tests at a non-trivial scale. However, the experimental evidence is narrow: the RTO claim rests on three runs on a single account, and the CAS Paxos evaluation is a simulation without confidence intervals. The paper also explicitly acknowledges a degenerate behavior that it says will be fixed only in the future, which limits the strength of the availability guarantee as presented.","major_comments":[{"comment":"The central RTO claim, stated as 'availability is restored within less than 2 minutes for every partition-set', is supported only by three power-outage runs on a single 3-region account with 4,300+ write-region partitions. The paper reports no raw recovery-time distributions, no confidence intervals, and no breakdown by partition-set. It also does not report acceptor-store latency, throttling, or CAS-conflict rates during the failover burst. Because the abstract claims the solution works 'at any scale', the experiment as presented does not establish that the RTO holds when many partition-sets across many accounts fail over concurrently and share the same acceptor-store capacity (Section 4.3.1). The authors should provide per-partition recovery-time statistics, acceptor-store load metrics under the tested outage, and either a larger-scale test or a validated model of fleet-wide failover load.","section":"Section 6.1.3"},{"comment":"The claimed reduction to a maximum 0.0028% proposer failure rate comes from a discrete-event simulation that is not sufficiently specified. No confidence intervals or variance measures are reported, the network latency distributions are described only as 'randomly assigned' and 'heterogeneous', and the simulation models 3 to 9 proposers per partition-set with seven acceptors but does not model contention from many partition-sets sharing the same acceptor stores during a regional outage. Since the RTO guarantee depends on CAS Paxos rounds completing within the 45-second lease-enforcer timeout, the simulation should include a model of shared acceptor-store load, or the authors should explicitly scope the claim to per-partition contention only.","section":"Section 6.2.3"},{"comment":"The paper acknowledges a degenerate behavior in the state machine: a graceful failover can succeed, then the destination region can fail, triggering an ungraceful failover, and the cycle can repeat, potentially causing a continuous outage. The text states 'We will amend our implementation to account for this', which indicates the fix is not part of the current implementation described in the paper. This is a load-bearing gap because the paper's central claim is that availability is restored within a bounded time. The authors should either present the fix as part of the implemented design and verify it, or explicitly state that the published system does not yet guarantee eventual availability in this scenario and that this is a known limitation.","section":"Section 4.5"},{"comment":"The design stores CAS Paxos acceptor state in a set of geographically distributed non-replicated Cosmos DB accounts that back 'all partitions globally'. The paper argues that the dependency on Cosmos DB is not circular, but the dependency remains a real availability risk: if these acceptor stores fail or throttle under load, quorum-based failover cannot proceed. The outage experiment exercises only one account's partitions and therefore does not test the shared acceptor-store dependency under the burst that would occur in a regional outage affecting many accounts. The paper should include an analysis or experiment that addresses the failure independence and load capacity of the acceptor stores, or it should qualify the RTO claim to scenarios where the acceptor stores are available and unthrottled.","section":"Section 4.3.1"}],"minor_comments":[{"comment":"The subsection numbering is inconsistent: there are two subsections labeled 6.1.1 and two labeled 6.2.3. The sections should be renumbered for clarity.","section":"Section 6.1"},{"comment":"Figure 7 and Figure 8 lack explicit axis labels and units. 'Time' should be labeled as seconds or minutes, and the y-axis should indicate the fraction or count of partition-sets.","section":"Figures 7 and 8"},{"comment":"The TLA+ verification is described only through a short invariant snippet. No model-checking results, TLA+ specification files, or list of checked properties are included, making the verification claim difficult to assess. If the authors want the verification to be a credible contribution, they should provide more detail or artifacts.","section":"Section 4.4"},{"comment":"The phrase 'We will amend our implementation to account for this' should be changed to 'We amended' or explicitly moved to the Future Work section; as written, it reads as an admission that the current implementation does not handle the described scenario.","section":"Section 4.5"},{"comment":"The text says the simulator can 'compress years of system operation', but each experiment simulates one hour of operational time, with 10,000 simulations totaling about 1.14 years. This sentence should be reworded to avoid implying that a single simulation spans years.","section":"Section 6.2.2"}],"recommendation":"major_revision","confidential_remarks":"This is an industry experience report, and the peer review should judge it against that standard rather than requiring the full statistical rigor of an academic systems paper. However, the abstract's 'at any scale' claim is broader than the evidence, and the acknowledged degenerate behavior in Section 4.5 is a real correctness gap that the authors should address before publication. The paper would also benefit from a more careful treatment of the shared acceptor-store dependency, since the current wording dismisses a legitimate availability concern. I recommend major revision with the expectation that the authors can add the missing evidence and either fix or explicitly scope the degenerate-behavior issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read this if you care about what happens when a hyperscale database actually has to fail over per partition. The paper is an engineering report from Azure Cosmos DB, and the main thing to know: it is not a protocol paper, it is a deployment story. The core idea is straightforward—each partition-set runs a deterministic state machine backed by a CAS Paxos store, so partitions pick a new write region autonomously instead of failing over the whole account. What is actually new is in the details: the adaptive backoff based on an EMA of phase-2 durations plus time-division multiplexing of proposers to avoid dueling proposers, the progress table to undo false progress on failback, and the DNS TXT design that avoids depending on DNS updates during failover. The authors also do something rare: they spell out the degenerate graceful-failover loops and the exponential backoff fix, and they admit the self-dependency of using Cosmos DB as acceptor store is something they may revisit.\n\nThe real evidence is three forced power outages in East Asia on a 3-region account with 4,300+ write partitions; every partition-set recovered in under two minutes, most under one. That is a genuine, externally meaningful data point. The CAS Paxos simulation showing failure rate dropping from ~6.5% to 0.0028% at nine proposers is suggestive but weaker: no raw data, no confidence intervals, no simulator artifact.\n\nSoft spots, in proportion: first, the two-minute RTO claim has a fleet-scale gap. All partition-sets share the same geographically distributed, non-replicated Cosmos DB accounts as acceptor state. During a regional outage, every affected partition tries to run CAS Paxos against those shared stores. The 4,300-partition experiment does not report acceptor-store latency, throttling, or CAS-conflict rates under that concurrent burst, so the <2-minute target is not shown to hold when the shared dependency is actually stressed. This is the stress-test concern and it lands. Second, TLA+ verification is claimed but no spec or TLC config is shipped, so the safety properties are not independently checkable. Third, the simulation is described with enough detail to reproduce roughly, but not enough to fully trust the 0.0028% number.\n\nNone of this kills the paper. The central architecture is plausible, the outage experiment is real, and the authors are honest about limits. Who benefits: systems folks at top conferences, anyone building geo-replicated storage or automatic failover. It deserves peer review; I would send it with a request for raw data and a better treatment of shared acceptor-store saturation.","headline":"An unusually candid systems paper on per-partition geo-failover in Cosmos DB; the design is plausible and the two-minute RTO is supported by real outage tests, but the shared acceptor-store dependency and missing artifacts keep it from being fully convincing.","tokens_in":16131,"tokens_out":2451,"would_cite":true,"duration_ms":21841,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Azure Cosmos DB's new per-partition automatic failover restores write availability in under two minutes for every partition-set in a 4,300-partition regional power-outage test.","keywords":["per-partition failover","geo-failover","CAS Paxos","state machine replication","Azure Cosmos DB","recovery time objective","dueling proposers","adaptive backoff"],"falsifier":"A concrete test: take down the write region and, at the same time, take down a majority of the CAS Paxos acceptor-store regions; if partition-sets still restore write availability within two minutes, the acceptor dependency is not the binding constraint, and if they do not, the two-minute RTO claim is conditional on acceptor-store placement.","tokens_in":15097,"feed_emoji":"🔄","tokens_out":5227,"duration_ms":40766,"temperature":0.7,"pith_summary":"The paper describes a new mechanism in Azure Cosmos DB that lets each partition-set decide its own write region when the current one fails, instead of failing over an entire account through a central control plane. The authors claim this decentralized design, driven by a deterministic state machine called the Failover Manager and backed by a CAS Paxos store, restores write availability within two minutes of an outage. They report three thirty-minute power-outage tests on a 3-region account with 4,300+ write-region partitions in which every partition-set recovered in under two minutes. They also report that an adaptive scheduling scheme for CAS Paxos proposers cuts proposer failure rates to at most 0.0028%, even under high contention. If these results hold, the system meets its RTO target at scale while preserving customer-chosen consistency levels and recovery point objectives.","feed_headline":"Per-partition failover hits under 2-minute RTO at 4,300 partitions","feed_subtitle":"A decentralized state machine lets each partition pick a new write region without waiting on the control plane.","key_machinery":"The Failover Manager is a deterministic, formally specified (TLA+/TLC-verified) state machine, one per partition-set, that stores the partition-set's write region, read-lease set, and progress markers and admits compare-and-swap transitions proposed by any replica. State is persisted in a leaderless CAS Paxos replicated state machine without a write-ahead log, with acceptor state held in geographically distributed non-cross-partition-replicated Cosmos DB accounts. A heartbeat every 30 seconds triggers transitions; ungraceful failover waits for a quorum of regions to report, then picks the highest-priority region with the highest reported progress, while graceful failover quiesces writes, drains replication, and switches to the preferred region. An adaptive backoff and time-division multiplexing scheduler tunes CAS Paxos retry delays from measured Phase 2 durations to suppress dueling proposers.","core_discovery":"The paper's central claim is that geo-failover in Cosmos DB can be made fine-grained and autonomous: each partition-set runs its own Failover Manager state machine, and replicas propose state transitions through CAS Paxos rounds. The state machine encodes the current write region, the set of read leases, and progress information, and transitions are triggered by partition heartbeats rather than by an operator or a control-plane workflow. The paper asserts that this removes the control plane as a scaling bottleneck and single point of failure, that the formally specified state machine guarantees availability is eventually restored and consistency levels are respected, and that the implementation achieves the stated RTO: availability restored in less than two minutes for every partition-set in the tested regional outage, with automatic failback to the preferred region after power returns.","pith_inferences":["A testable consequence of the design is that failover RTO depends on the geographic placement of the acceptor stores; placing a majority of them in the same region as the write region could violate the two-minute target, so operators should treat acceptor-store placement as a first-class availability decision.","The same state-machine-plus-CAS-Paxos pattern could be applied to other geo-distributed systems that currently rely on a control plane for failover, provided they can tolerate the extra heartbeat load and the acceptor-store dependency.","The paper's distinction between graceful and ungraceful failover suggests a broader design principle: any failover system that can distinguish planned from unplanned transitions can use the planned path to bound data loss and the unplanned path to bound downtime.","Because the client SDK now uses a DNS TXT record and per-partition-set error caches instead of DNS updates, client behavior during failover becomes testable independently of DNS TTL misconfigurations."],"forward_implications":["Region-wide write availability recovers in under two minutes for every partition-set, so an outage of one data center no longer forces account-wide failover of healthy partitions.","Failover decisions scale with the number of backend partitions rather than with control-plane capacity, because each partition-set runs its own state machine on the machines that host it.","Consistency levels and recovery point objectives remain honored: ungraceful failover picks the region with the highest reported progress, and dynamic read-lease quorums let a two-region account continue with one region at a configured minimum durability.","Failed-over partition-sets automatically fail back to the user's preferred region once it recovers, using a progress table to discard only the 'false progress' written during the outage.","The adaptive CAS Paxos scheduler keeps proposer failure rates below 0.003% even with nine concurrent proposers, making leaderless consensus practical for the heartbeat workload."],"supporting_citations":[{"why":"Supplies the leaderless CAS Paxos protocol that backs the Failover Manager's replicated state store.","marker":"[1]"},{"why":"Defines the consistency levels whose guarantees the failover design must preserve.","marker":"[3]"},{"why":"Provides the Paxos algorithm specification used as the basis for the low-level state machine translation.","marker":"[7]"},{"why":"Provides the CASPaxos TLA+ specification used to verify the CAS Paxos implementation.","marker":"[8]"},{"why":"TLA+ and the TLC model checker are used to verify the Failover Manager state machine properties.","marker":"[11]"},{"why":"Describes dueling proposers in consensus, the problem the adaptive scheduling approach addresses.","marker":"[14]"},{"why":"Supplies prior art for dynamic quorum and minimum durability configurations used by the read-lease scheme.","marker":"[12]"},{"why":"Provides cross-region round-trip latency statistics that motivate the adaptive backoff design.","marker":"[15]"}],"fun_headline_variants":["Per-partition auto-failover hits sub-2-minute RTO in Cosmos DB","Decentralized failover lets each Cosmos DB partition recover on its own","Partition-level geo-failover: autonomous recovery without control plane","Azure Cosmos DB: per-partition failover with under 2-minute RTO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The failover decision's quorum lives in ordinary non-cross-partition-replicated Cosmos DB accounts, so if those acceptor stores fail or become unreachable at the same time as the data-plane region, the quorum needed to elect a new write region may be gone.","fun_headline_variants_meta":{"raw":{"variants":["Per-partition auto-failover hits sub-2-minute RTO in Cosmos DB","Decentralized failover lets each Cosmos DB partition recover on its own","Partition-level geo-failover: autonomous recovery without control plane","Azure Cosmos DB: per-partition failover with under 2-minute RTO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2127,"prompt_tokens":921,"completion_tokens":1206,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":1120}},"tokens_in":537,"tokens_out":1206,"duration_ms":9979,"temperature":1.0,"reasoning_tokens":1120,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:27:15.910917+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: take down the write region and, at the same time, take down a majority of the CAS Paxos acceptor-store regions; if partition-sets still restore write availability within two minutes, the acceptor dependency is not the binding constraint, and if they do not, the two-minute RTO claim is conditional on acceptor-store placement.","supporting_citations":[{"cited_title":"CASPaxos: Replicated State Machines without logs","cited_arxiv_id":"1802.07000","evidence_quote":"Supplies the leaderless CAS Paxos protocol that backs the Failover Manager's replicated state store."},{"cited_title":"Consistency levels in Azure Cosmos DB,","cited_arxiv_id":null,"evidence_quote":"Defines the consistency levels whose guarantees the failover design must preserve."},{"cited_title":"The Paxos Algorithm,","cited_arxiv_id":null,"evidence_quote":"Provides the Paxos algorithm specification used as the basis for the low-level state machine translation."},{"cited_title":"CASPaxos -tla,","cited_arxiv_id":null,"evidence_quote":"Provides the CASPaxos TLA+ specification used to verify the CAS Paxos implementation."},{"cited_title":"The TLA+ Home Page,","cited_arxiv_id":null,"evidence_quote":"TLA+ and the TLC model checker are used to verify the Failover Manager state machine properties."},{"cited_title":"Paxos made simple.,","cited_arxiv_id":null,"evidence_quote":"Describes dueling proposers in consensus, the problem the adaptive scheduling approach addresses."},{"cited_title":"Understanding cluster and pool quorum,","cited_arxiv_id":null,"evidence_quote":"Supplies prior art for dynamic quorum and minimum durability configurations used by the read-lease scheme."},{"cited_title":"Azure network round -trip latency statistics,","cited_arxiv_id":null,"evidence_quote":"Provides cross-region round-trip latency statistics that motivate the adaptive backoff design."}],"review_version":1}