{"id":"d612a5e0-0ca3-4d85-a7e9-11247b4dc625","arxiv_id":"2505.05885","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Integrating DiskANN's graph index into Cosmos DB's Bw-Tree gives low-latency, low-cost vector search inside an operational database, with claimed scale-out to billions of vectors.","lead":"Microsoft built vector search directly into Azure Cosmos DB by embedding the DiskANN graph index into the database's own storage tree, avoiding the need for a separate vector database. The paper reports sub-20 millisecond queries over 10 million vectors and claims roughly 12x to 43x lower query cost than two specialized serverless vector databases, though the cost comparison is not recall-controlled.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 43x/12x cost advantage is not established: Table 1 compares unmeasured competitor pricing without recall matching, and the latency and cost figures come from different operating points.","rationale":"The paper's engineering contribution is substantial: a clean decoupling of DiskANN from index layout, a plausible design for storing graph and quantization terms in Bw-Tree, and measurements covering latency, cost, ingestion, filtered search, and update stability. The 10M-vector results are internally consistent as a systems demonstration, and the runbook experiments give real evidence for stable recall with in-place deletion. However, the central claim that users get a 43x/12x cost advantage over Pinecone and Zilliz is the headline quantitative assertion, and it rests entirely on Table 1. That table does not measure the competitors, does not state their recall, and does not show that the listed Read Unit/vCU consumption corresponds to the same index size, dimensionality, and search-list settings used for the Cosmos DB column. This is not a disagreement with community consensus; it is a missing control in the comparison. The paper could fix this by running the same benchmark on the competitors at matched recall, or by clearly labeling the cost figures as rough public-price estimates. Until then, the cost claim should be treated as provisional, which is exactly the reader's conditional verdict. I therefore keep the verdict unchanged, while adding that the paper should also clarify the operating point behind the '<20ms' latency claim, since the cost and latency numbers currently come from different L values.","tokens_in":27449,"tokens_out":14129,"duration_ms":137303,"concrete_test":"Run a recall-matched head-to-head benchmark on the same 10M Wiki-Cohere vectors and the same 5000 queries: tune Pinecone serverless enterprise and Zilliz serverless each to recall@10=94.64 (the Cosmos DB operating point in Table 1), measure average billed Read Units/vCUs per query, and recompute Table 1. If either competitor's per-1M-query cost is materially closer to Cosmos DB than the claimed 43x/12x, the headline cost claim fails. At minimum, report Cosmos DB's p50 and average RU at L=100 and L=50 alongside Table 1 so the cost, latency, and recall operating point is unambiguous.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1's Table 1 is the sole evidence for the abstract's central cost claim of 'approximately 43x and 12x lower query cost' versus Pinecone and Zilliz serverless. The Cosmos DB column uses 70 RU per query, which in Figure 6 is the p99 RU at L=100 with recall@10=94.64. The competitor columns (32 Read Units, 55 vCUs) are taken from published pricing documentation, not from measurements, and no recall, index configuration, or search parameters are reported for those systems. If Pinecone or Zilliz achieve 94.64% recall at a different unit consumption per query, the 43x/12x ratios change materially. The comparison is further muddled by the paper's own latency claim: the abstract's '<20ms query latency' is the p50 at L=50 in Figure 6 (recall@10=91.43), while the cost claim uses L=100; no single operating point is shown to deliver both simultaneously. Because the 'cost-effective' thesis depends directly on the magnitude of these ratios, this is the load-bearing claim. It is not supported by the evidence as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the design and evaluation of a vector search capability integrated into Azure Cosmos DB NoSQL. The authors rewrite the DiskANN library to decouple its algorithms from physical index layout, store quantized vectors and graph adjacency lists as Bw-Tree index terms, and support querying, incremental updates, deletions, filtered search, and sharded multi-tenant indices through this integrated design. The evaluation uses Wiki-Cohere, MS Turing, and YFCC datasets at scales from 100K to 1 billion vectors, reporting latency, request-unit cost, recall, ingestion throughput, and recall stability under update runbooks. The central claims, stated in the abstract and introduction, are that the system supports under 20 ms query latency over 10 million vectors, maintains stable recall over updates, scales out to billions of vectors, and offers approximately 43x and 12x lower query cost than Pinecone and Zilliz serverless enterprise products.","tokens_in":27710,"tokens_out":7103,"duration_ms":69884,"significance":"If the cost and latency claims held at the stated operating points, this paper would be an important data point in the debate over specialized vector databases: it would demonstrate that a general-purpose operational database can provide high-quality approximate vector search with the availability, durability, and multi-tenancy of an existing cloud database, without replicating data to a separate system. The engineering contributions are real and clearly described: the provider-trait redesign of DiskANN, the use of Bw-Tree forward and inverted terms for index persistence, the asynchronous Rust/C++ interoperability layer, and the paginated and filter-aware search adaptations. The publication of an open-source scenario suite and reliance on independently published DiskANN algorithms are also strengths, and the latency and recall measurements at 10-million-vector scale are plausible and useful.","major_comments":[{"comment":"Table 1 is the sole evidence for the abstract's claim of approximately 43x and 12x lower query cost versus Pinecone and Zilliz. The Cosmos DB column uses the measured p99 value of 70 RU per query at searchListSize=100 (recall@10=94.64). The Pinecone and Zilliz columns (32 Read Units and 55 vCUs) are taken from published pricing documentation, not from measurements on those systems; no recall, index configuration, search-list size, or query workload is reported for them. The comparison therefore assumes, without evidence, that those list-price consumption units correspond to a system configuration achieving recall and result quality comparable to Cosmos DB's measured 94.64% recall@10. If a competitor achieves the same recall at a lower unit consumption, the 43x/12x ratios change materially. Because the cost-effectiveness thesis rests on these ratios, this is a load-bearing gap in the evaluation.","section":"4.1, Table 1"},{"comment":"The abstract states that the system 'supports <20ms query latency over an index spanning 10 million vectors,' but Figure 6 shows that at L=50 the p50 latency is 13.4ms and p95 is 17.6ms, while p99 is 20.6ms; the <20ms statement is therefore true only for p50/p95, not p99, and only at the L=50 operating point. The cost comparison in Table 1 uses a different operating point: p99 RU at L=100 (70 RU, recall@10=94.64). No single measured configuration delivers both the <20ms latency claim and the 12x/43x cost claim simultaneously. The abstract and introduction should either tie each claim to its operating point and percentile, or the claims should be softened accordingly.","section":"Abstract; Figure 6"},{"comment":"The latency and RU figures report p50/p95/p99 from what appears to be a single batch of 5000 queries, with no number of repeated runs, confidence intervals, or run-to-run variance. The cost arithmetic in Table 1 treats 70 RU per query as an exact input, but the underlying measurement is a point estimate. Given that the paper's headline claims are quantitative to two significant figures (43x, 12x, <20ms), the absence of error bars or a statement that these are single-run observations leaves the precision of the claims unverifiable. At minimum, state the experimental protocol (number of runs, warmup, cache state) and report variance or confidence intervals for the key operating points.","section":"Section 4 (Figures 6-10)"},{"comment":"The introduction highlights that ingestion 'offers cost and performance comparable to other vector databases,' but Table 2 shows Cosmos DB insertion cost is 5.4x higher than Zilliz (and only 33%/53% lower than Pinecone/DataStax) using the same unmeasured list-price methodology as Table 1. The text's suggestion that autoscale discounts reduce this gap is not part of Table 2 and no measured autoscale price is provided. The 'cost-effective' claim for write-heavy workloads is therefore not established, and the comparison should be either measured on equivalent configurations or explicitly labeled as a list-price comparison under stated assumptions.","section":"Section 4.4, Table 2; Section 1"}],"minor_comments":[{"comment":"The last legend entry reads 'L Search=20, Recall=97.15'; from the monotonic recall trend this should be L=200.","section":"Figure 6"},{"comment":"The phrase 'If the selectivity is low, i.ei, at least 5000 documents' contains a typo: 'i.ei' should be 'i.e.'.","section":"Section 3.5"},{"comment":"The sentence 'the query planner sends queries invokes the DiskANN index' is missing a conjunction and should read 'sends queries and invokes the DiskANN index'.","section":"Section 4.2"},{"comment":"The phrase 'Recall k@k' should be 'Recall@k' or 'k-recall@k' for consistency with the rest of the paper.","section":"Section 2.1"},{"comment":"The configuration paragraph says 'We use the the following parameters' with a duplicated 'the'; please fix.","section":"Section 4, Configuration"},{"comment":"The table header says 'P99 vector search query and monthly storage costs'; the caption should also state that Cosmos DB's 70 RU is the p99 value at L=100 and that the competitor columns are list-price documentation values, not measured results.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This is an industrial systems paper whose main novel contributions are the integration design and the latency/recall measurements. The underlying system appears sound and the engineering details are generally convincing. The primary obstacle is the headline cost-effectiveness claim: Table 1 compares a measured Cosmos DB operating point with unmeasured competitor list prices, and the abstract's latency and cost claims come from different operating points. This is fixable by either measuring equivalent competitor configurations or prominently reframing the comparison as a list-price comparison under stated assumptions, so I recommend major revision rather than rejection. I also note that the paper draws heavily on the authors' own prior DiskANN work; this is not circular, but the novelty relative to FreshDiskANN and Filtered-DiskANN should be stated more crisply."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the system design: a stateless rewrite of DiskANN with Provider traits so the index lives in Cosmos DB's Bw-Tree, plus the new Forward Term type, sharded DiskANN, and in-place re-quantization. That is genuinely new as an integration, and the paper does a good job explaining the engineering constraints (memory limits, latch-free updates, async Rust/C++ interop). The measured latency and recall on 10M vectors are believable: p50 under 20ms at L=50, p99 20.6ms at the same setting, with recall in the low-to-mid 90s. The recall-stability runbooks are also convincing, especially the 20-point improvement from in-place deletes on the clustered distribution-shift runbook. The sharded DiskANN results are clean and show a real benefit for tenant-filtered queries.\n\nThe soft spots are where you'd expect. Table 1 is the weakest part of the paper. The cost comparison uses public pricing for Pinecone, Zilliz, and DataStax without measuring those systems, without establishing that the configurations are comparable in recall or index settings, and the Cosmos DB figure comes from a different operating point (L=100, 94.64% recall) than the latency claim (L=50, 91.43% recall). The 43x and 12x ratios are the paper's headline, so this is a load-bearing flaw, not a nitpick. The stress-test note is correct on this. The paper also makes a couple of unsupported assertions: the new library is said to be at least as fast as the monolithic DiskANN with no benchmark shown, and the ingestion cost comparison in Table 2 has the same unmeasured-competitor problem. Minor issues: no error bars or repeated trials, and the abstract's '<20ms' is really p50, not p99.\n\nThat said, the central engineering claim—that you can build a high-quality vector index inside an operational database with competitive latency and memory profile—does hold up on the evidence. The design is coherent and the measurements are internally consistent. The cost-advantage claim needs a matched-recall competitor benchmark or at least a heavily qualified comparison. This is addressable in revision.\n\nWho is this for? Systems researchers and practitioners working on vector search in databases, and anyone evaluating whether to replicate data to a specialized vector DB. It deserves a serious referee; I'd send it out despite the weak cost comparison. I'd also suggest the authors be asked to rerun or qualify the price-based claims before publication.","headline":"A serious systems paper with a plausible core design and real measured results, but the headline 43x/12x cost claim rests on an unmeasured competitor comparison and should be treated as unproven.","tokens_in":28438,"tokens_out":1188,"would_cite":true,"duration_ms":14256,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A general-purpose operational database can host state-of-the-art vector search, the paper argues, by embedding DiskANN's graph index directly inside Cosmos DB's Bw-Tree, eliminating the need to replicate data to a specialized vector store.","keywords":["vector search","approximate nearest neighbor","DiskANN","Azure Cosmos DB","operational database","Bw-Tree","serverless vector database","query cost"],"falsifier":"Run the same 10-million-vector, 768-dimensional Wiki-Cohere queries against Pinecone and Zilliz serverless at their listed prices, measure recall@10 at comparable search settings, and compare per-1M-query cost; if either matches 94.64% recall at lower cost, the 43x and 12x claims fail.","tokens_in":27257,"feed_emoji":"⚡","tokens_out":4567,"duration_ms":42608,"temperature":0.7,"pith_summary":"The paper argues that a general-purpose, cloud-native operational database can deliver vector search on par with specialized vector databases, removing the need to replicate data out of the operational store. To prove this, the authors integrate DiskANN, a graph-based approximate nearest-neighbor library, directly into Azure Cosmos DB NoSQL by storing the index's quantized vectors and adjacency lists as terms inside Cosmos DB's own Bw-Tree index. They report under 20 ms p50 latency for searches over 10 million 768-dimensional vectors at 94.64% recall@10, stable recall across long insert/delete sequences, and roughly 43x and 12x lower per-query cost than Pinecone and Zilliz enterprise serverless offerings, with scale-out to billions of vectors. A sympathetic reader would care because, if true, it means applications can get semantic search, transactions, high availability, and multi-tenancy from one database instead of a separate vector engine.","feed_headline":"10M-vector search in under 20 ms inside Cosmos DB","feed_subtitle":"Deep DiskANN integration gives sub-20 ms vector search without copying data out of the database, and at far lower query cost.","key_machinery":"The load-bearing mechanism is a rewritten, layout-agnostic DiskANN library exposed through Provider traits, with Cosmos DB's Bw-Tree supplying persistence: quantized vectors are stored as inverted terms keyed by document id, and graph adjacency lists as new forward terms supporting blind incremental updates. Greedy graph search runs over cached quantized vectors, with a small full-precision rerank at the end; a beta-biased distance scaling (Algorithm 7) makes search filter-aware, paginated search handles hybrid predicate queries, mini-batch inserts avoid Bw-Tree duplicate-key constraints, and in-place delete (Algorithm 6) keeps recall stable under updates.","core_discovery":"The central claim is that a state-of-the-art vector index can be built inside an operational database by decoupling the index algorithm from the index layout: the DiskANN graph is rewritten so that its quantized vector terms and adjacency lists live in the database's existing Bw-Tree index and are updated durably and incrementally with each document change. This yields one DiskANN index per replica that is always in sync with the underlying documents, no separate index rebuilds or segment merges, and memory use low enough to cache only quantized terms while full-precision vectors are read from storage only for a small rerank set. The measured consequence is sub-20 ms p50 latency over 10 million vectors, query cost that grows less than 2x for a 100x increase in index size, stable recall under streaming updates (up to 20 points better than drop-only deletes on distribution-shift runbooks), roughly 43x and 12x lower query cost than Pinecone and Zilliz serverless enterprise tiers, and the ability to shard the index per tenant.","pith_inferences":["The cost comparison treats published price lists as proxies for total cost; a fair head-to-head would need identical recall targets and index configurations, so the reported 43x and 12x ratios likely bracket a range rather than a single point estimate.","The design suggests a general template: any database with a sufficiently fast, latch-free persistent index and resource governance could host a stateless DiskANN, so the same integration could be replicated in other operational stores.","Because the graph is stored as ordinary index terms, vector search inherits the database's backup, replication, and failover story for free, a property that specialized vector engines have to build separately.","One testable extension is whether the economics shift for smaller indexes or lower recall targets, where the per-partition fixed costs of the graph index may dominate the quantized-flat fallback."],"forward_implications":["Operational databases can replace the replicate-to-a-vector-DB pattern: applications keep one primary store with transactions, availability, and security SLAs, and still get approximate vector search at the same store.","Query cost scales logarithmically with partition size and linearly with partition count, so users control cost by packing vectors into as few partitions as possible.","Filter-aware search gives comparable recall to post-filtering at a fraction of the tail latency when predicates are not highly selective.","Sharded DiskANN indices make per-tenant vector search in multi-tenant collections faster and more accurate for tenant-scoped queries.","If the cost comparisons hold, the reported price gaps imply specialized serverless vector databases are priced for convenience rather than for raw query efficiency."],"supporting_citations":[{"why":"Defines the DiskANN library and its graph-based algorithms that the paper integrates into Cosmos DB.","marker":"[22]"},{"why":"Original DiskANN design showing billion-point search with quantized vectors and SSD-based storage, the algorithmic foundation being rewritten.","marker":"[38]"},{"why":"Describes the Cosmos DB engine, Bw-Tree indexing, and resource governance that host the vector index terms.","marker":"[32]"},{"why":"Open-source DiskANN implementation used as the baseline for the rewritten layout-agnostic library.","marker":"[35]"},{"why":"FreshDiskANN provides the streaming-update methodology that supports the claim of stable recall under insertions and deletions.","marker":"[36]"},{"why":"Supplies the in-place delete algorithm that the paper adapts (Algorithm 6) to keep recall stable during updates.","marker":"[44]"},{"why":"ParlayANN supplies the mini-batch update pattern used to respect Bw-Tree's constraints during parallel index updates.","marker":"[25]"},{"why":"Pinecone serverless pricing is the comparison target that yields the claim of roughly 43x lower query cost.","marker":"[30]"},{"why":"Zilliz serverless pricing is the comparison target that yields the claim of roughly 12x lower query cost.","marker":"[48]"},{"why":"DataStax vector search pricing is the third comparison target in the cost tables.","marker":"[10]"}],"fun_headline_variants":["10M vectors in under 20ms: Cosmos DB vector search","43x cheaper vector search in Cosmos DB, sub-20ms","DiskANN inside Cosmos DB: 10M vectors, <20ms, 43x cheaper","Cosmos DB vector search: sub-20ms on 10M, 43x lower cost","Sub-20ms vector search on 10M vectors at 43x lower cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline cost advantage rests on the assumption that the published prices for Pinecone, Zilliz, and DataStax correspond to setups whose recall and index settings match the 94.64% recall@10 that Cosmos DB was measured at, since the paper does not measure the competitors' recall.","fun_headline_variants_meta":{"raw":{"variants":["10M vectors in under 20ms: Cosmos DB vector search","43x cheaper vector search in Cosmos DB, sub-20ms","DiskANN inside Cosmos DB: 10M vectors, <20ms, 43x cheaper","Cosmos DB vector search: sub-20ms on 10M, 43x lower cost","Sub-20ms vector search on 10M vectors at 43x lower cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1792,"prompt_tokens":989,"completion_tokens":803,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":692}},"tokens_in":605,"tokens_out":803,"duration_ms":6795,"temperature":1.0,"reasoning_tokens":692,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:54:13.671660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 10-million-vector, 768-dimensional Wiki-Cohere queries against Pinecone and Zilliz serverless at their listed prices, measure recall@10 at comparable search settings, and compare per-1M-query cost; if either matches 94.64% recall at lower cost, the 43x and 12x claims fail.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DiskANN library and its graph-based algorithms that the paper integrates into Cosmos DB."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Original DiskANN design showing billion-point search with quantized vectors and SSD-based storage, the algorithmic foundation being rewritten."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pinecone serverless pricing is the comparison target that yields the claim of roughly 43x lower query cost."},{"cited_title":"waker\" that moves a task from an “idle","cited_arxiv_id":null,"evidence_quote":"Zilliz serverless pricing is the comparison target that yields the claim of roughly 12x lower query cost."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DataStax vector search pricing is the third comparison target in the cost tables."}],"review_version":1}