{"id":"1c58cedf-090b-4b99-921a-5e9a807a7991","arxiv_id":"2505.11783","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A disaggregated RDMA-based HNSW design using a small cached routing index and batched cluster fetches, with evaluation only against self-defined baselines.","lead":"d-HNSW is a vector search engine that stores the HNSW graph on remote memory accessed over RDMA and caches a small routing index on the compute side. It reports up to 117x speedup over a self-defined naive baseline, but without comparisons to existing ANN systems the headline performance is unverified.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 117x claim rests on a Naive d-HNSW baseline whose reported 90,271 µs network time for 3.547 RDMA round-trips per query is ~1000x slower than physically possible on the stated 100Gb testbed, so the comparison is not a valid measurement.","rationale":"The reader's weakest assumption (meta-HNSW routing) is real, but it bounds recall rather than the headline latency ratio. The most load-bearing point is that the 117x speedup is defined entirely by comparison to a baseline whose measured network latency is physically implausible. Even a robust routing scheme would not salvage a claim built on an uncalibrated baseline. I partially agree with the reader: they noted implausible latency in their rationale but selected routing as the weakest_assumption. My concrete test settles the matter: re-run with standard RDMA benchmarks; if the naive baseline cannot be recomposed from raw RDMA costs, the central performance claim is invalid. The paper's internal algorithm appears coherent; no issue with internal consistency is raised. Also, the abstract's recall of 0.87 on SIFT1M@1 conflicts with Fig. 6(b), which reports an upper recall of 0.85, further weakening the 'maintains recall' phrasing. Since REJECT was already the reader's verdict, no verdict change is needed.","tokens_in":9950,"tokens_out":6710,"duration_ms":68622,"concrete_test":"Reproduce Fig. 6(a) on the same CloudLab hardware with the paper's settings (batch 2000, efSearch 48, top-1) and instrument Naive d-HNSW per query. Independently measure raw RDMA_READ latency/bandwidth with ibv_read_lat/ibv_read_bw for the exact serialized cluster sizes implied by the number of sub-HNSW clusters. If the per-RDMA-read time is below ~100 µs, the reported 90,271 µs for 3.547 reads is erroneous and the 117x ratio collapses. A complementary check: implement Naive as one asynchronous ibv_post_read per cluster with standard pipelining; if its network latency is under a few ms, the paper's baseline is unrepresentative.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 defines Naive d-HNSW as reading each involved sub-HNSW cluster with an RDMA read, and Table 1 reports for SIFT1M@1, efSearch 48, network latency of 90,271.2 µs while Naive d-HNSW makes only 3.547 round-trips per vector query. This is ~25,000 µs per RDMA read. On the stated testbed (three compute servers, Mellanox ConnectX-6 100Gb, one memory server), a 1 MB RDMA read has wire transfer time roughly 80 µs plus RTT; even a doorbell-batched multi-region read is microseconds-to-hundreds-of-microseconds, not milliseconds. The GIST1M table is worse (Naive network 422.9 ms). Such a number is only explainable by a unit or instrumentation error, by measuring an entire batch as a single query, or by a baseline that serializes thousands of NIC operations. Since the paper's headline speedup is the ratio d-HNSW/Naive, an implausible denominator by itself inflates the 117x figure. There is no calibration of raw RDMA latency and no comparison to a non-disaggregated HNSW or any prior RDMA vector-search system, so the central performance claim is unsupported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces d-HNSW, a vector similarity search engine for RDMA-based disaggregated memory. The design partitions the dataset into sub-HNSW clusters, caches a small \"meta-HNSW\" in compute nodes for cluster routing, lays out clusters contiguously in remote memory with shared overflow regions, and uses doorbell batching plus query-aware data loading to reduce RDMA round trips. The evaluation reports latency-recall curves for SIFT1M and GIST1M against two self-defined baselines (Naive d-HNSW and d-HNSW without doorbell), claiming up to 117x lower latency while maintaining recall around 0.87 on SIFT1M@1.","tokens_in":10338,"tokens_out":5514,"duration_ms":56784,"significance":"Disaggregated memory for vector search is a timely and practically relevant problem, and the three proposed techniques (representative index caching, RDMA-friendly layout, and batched query-aware loading) are plausible building blocks. The paper also ships a substantial prototype implementation (12K LoC) and evaluates on real hardware, which is a strength. However, as submitted, the evaluation does not substantiate the central performance claims: the baselines are self-defined variants, the absolute recall figures are not benchmarked against standard HNSW, the network-latency numbers in the breakdown tables are physically implausible on the stated testbed, and the key routing assumption behind meta-HNSW is given no sensitivity analysis. If these issues are addressed, the work could be a useful contribution to the disaggregated-systems and vector-search communities.","major_comments":[{"comment":"The Naive d-HNSW network latency of 90,271.2 µs for 3.547 RDMA round trips per query implies roughly 25 ms per RDMA read. On the stated Mellanox ConnectX-6 100Gb testbed, even a multi-megabyte RDMA read should complete in well under 1 ms; the value is most likely explained by a unit error, by measuring an entire batch of 2000 queries as one query, or by an uninstrumented serialization of many NIC operations. Because the headline 117x speedup is the ratio d-HNSW/Naive, an implausible denominator directly inflates the headline claim. The authors should provide a raw RDMA latency calibration on the testbed and re-measure the baseline before any performance claim can be accepted.","section":"§4, Table 1"},{"comment":"The only comparisons are to two self-defined variants, \"Naive d-HNSW\" and \"d-HNSW without doorbell.\" There is no comparison to standard in-memory HNSW, to a non-disaggregated ANN system, or to prior RDMA- or CXL-based vector-search prototypes despite such systems being cited in the paper. As a result, the absolute recall values (0.85–0.87 on SIFT1M@1 at efSearch=48) cannot be interpreted, and the claim in the abstract and introduction that d-HNSW is \"the first\" disaggregated vector search engine is unsupported. A fair evaluation needs at least a standard HNSW recall-latency curve on the same datasets.","section":"§4, Figure 6"},{"comment":"The correctness of the meta-HNSW routing layer is assumed rather than demonstrated. The paper states that the meta-HNSW is built from 500 uniformly sampled vectors and that each query is routed to the b closest sub-HNSW clusters, but it provides no analysis or sensitivity study showing that the selected clusters contain the true top-k neighbors. If the meta-HNSW misses the correct cluster, recall is capped regardless of the local search effort. The paper should report a sensitivity analysis over sample size and b, and should separately measure recall loss caused by routing versus recall loss caused by sub-HNSW search within the selected clusters.","section":"§3.1, Figure 3"},{"comment":"The \"round trips per vector query\" metric is internally inconsistent with the reported network latencies. For d-HNSW the paper reports 0.00475 round trips per query but a network latency of 527.6 µs, which would imply roughly 111 ms per round trip; for the Naive baseline, 3.547 round trips and 90,271.2 µs imply roughly 25 ms per round trip. The relation between the doorbell batch size, the batch of 2000 queries, and the per-query latency aggregation needs a precise definition, otherwise the round-trip counts cannot be used as evidence for the speedup.","section":"§3.2, Table 1"},{"comment":"Several experimental conditions that are load-bearing for the evaluation are missing: the number and size of sub-HNSW clusters, the distribution of vectors across clusters, the actual cache hit rates, the overhead of index construction and insertion, and run-to-run variance (no error bars are shown in Figure 6). The abstract claims \"fast data indexing\" and \"low network communication overhead,\" but no indexing or insertion experiments are reported, so those claims are not evaluated.","section":"§4, experimental setup"}],"minor_comments":[{"comment":"The baseline is introduced as \"Native-HNSW\" but is called \"Naive d-HNSW\" throughout the rest of the paper; the naming should be unified.","section":"§4, first paragraph"},{"comment":"The abstract reports \"up to 117x\" speedup while Section 4 reports up to 121x on GIST1M; the inconsistency should be reconciled.","section":"Abstract and §4"},{"comment":"The latency-recall curves would be much more informative with error bars and with a labeled reference curve for a standard in-memory HNSW implementation.","section":"Figure 6"},{"comment":"There are several typographical issues, including \"evalute\" in Section 4 and the formatting of \"4.75×10^-3\" for the round-trip count; these should be fixed in a revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper reports an interesting system idea and a substantial prototype, but the evaluation needs a major rework before it can support the claimed contributions. In particular, the authors should be asked to provide raw RDMA latency measurements, a corrected Naive baseline, a comparison with standard HNSW and prior disaggregated ANN systems, and a sensitivity analysis of the meta-HNSW routing assumption. The \"first\" claim should also be verified against the existing literature cited in the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the d-HNSW design is coherent and worth discussing, but the 117x speedup is not supported. The naive baseline's network time (90,271 µs for 3.547 RDMA round-trips) works out to about 25 ms per round trip on a 100Gb ConnectX-6 testbed. A 1 MB RDMA read over that link is tens of microseconds. The denominator is off by orders of magnitude, so the headline ratio is meaningless. The stress-test note is correct.\n\nWhat's genuinely new: the combination of a sampled meta-HNSW for cluster routing (borrowed from Pyramid), serialized sub-HNSW clusters with shared overflow space in registered memory, and doorbell batching for non-contiguous reads. That is a reasonable systems design for RDMA disaggregated memory, and the paper is clearly written. The authors correctly identify the round-trip problem in graph search over RDMA.\n\nThe soft spots are mostly in the evaluation. The only baselines are self-defined: Naive d-HNSW and d-HNSW without doorbell. There is no comparison to standard local HNSW, FAISS, or any prior RDMA vector search system. Recall on SIFT1M@1 tops out around 0.85–0.87, which is low for that dataset (standard HNSW typically gets ~0.99 at efSearch 48), and there is no sensitivity analysis for the 500-sample meta-index or the choice of b. The routing assumption is plausible but unvalidated.\n\nI would also want code and data. The paper reports no release, so the latency breakdowns cannot be checked. That matters when one of the numbers is physically implausible.\n\nThe citation pattern looks fine; they cite the relevant RDMA and ANN work, including Pyramid. No obvious gaps.\n\nWho is this for? Researchers working on disaggregated memory and vector search. The system design is worth replicating or improving. But as written, the empirical claims should not be taken at face value.\n\nRecommendation: send it to peer review, but expect heavy revision. A good reviewer will ask for a correct baseline, a local HNSW comparison, and calibration of raw RDMA latency. With those, the paper could become useful.","headline":"The design is coherent but the 117x speedup is a measurement artifact; the naive baseline's RDMA latency is physically impossible, so the headline claim collapses.","tokens_in":10829,"tokens_out":2577,"would_cite":false,"duration_ms":24354,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"d-HNSW shows HNSW vector search can run over RDMA-disaggregated memory by routing each query through a small cached meta-graph and fetching only the relevant remote sub-graphs, cutting latency up to 117x at recall 0.87 on SIFT1M.","keywords":["disaggregated memory","RDMA","vector similarity search","HNSW","approximate nearest neighbor","graph-based index","doorbell batching","memory pool"],"falsifier":"Run d-HNSW on SIFT1M with the routing step in place but give the local sub-HNSW search a very large efSearch; if recall plateaus well below 1.0, the plateau measures the fraction of true neighbors that fall outside the clusters selected by the cached meta-graph. Alternatively, directly check, for a random sample of queries, whether the ground-truth top-1 neighbor lies inside the selected sub-HNSW clusters.","tokens_in":9750,"feed_emoji":"⚡","tokens_out":9217,"duration_ms":79598,"temperature":0.7,"pith_summary":"d-HNSW aims to make graph-based vector similarity search practical when the index and data live in a remote memory pool accessed over RDMA. The idea is to split HNSW into a tiny cached routing index (meta-HNSW) built from 500 sampled vectors and a set of sub-HNSW clusters stored remotely; each query routes through the cached meta-graph and then fetches only a few relevant clusters. A dedicated memory layout and batched, doorbell-coalesced RDMA reads cut the number of network round trips per query from several to near zero. The paper reports up to 117x lower latency than a naive remote HNSW while keeping recall around 0.87 on SIFT1M@1, which matters because vector datasets increasingly outgrow single-machine memory.","feed_headline":"Disaggregated-memory vector search runs 117x faster with d-HNSW","feed_subtitle":"A tiny cached routing graph picks the right remote clusters, cutting round trips from 3.5 to 0.005 per query.","key_machinery":"The meta-HNSW: a three-layer HNSW built from 500 uniformly sampled vectors and cached in the compute pool. It acts as a coarse classifier: greedy routing through its layers ends at a bottom-layer vector that selects a sub-HNSW partition, and the query then searches only the nearest few such partitions held in remote memory. The mechanism that carries the argument is the observation that HNSW's greedy search visits a short, predictable path, so a cheap replica of the upper graph can decide which remote clusters to load, while an RDMA-friendly layout and doorbell-batched reads turn the rest of the search into a small number of bulk transfers.","core_discovery":"d-HNSW claims that the greedy search path of HNSW can be exploited for memory disaggregation: instead of following pointer-chasing steps across the network, a query first traverses a small cached meta-HNSW to select the few sub-HNSW partitions likely to contain the true nearest neighbors, then performs one or a few bulk RDMA reads to load those partitions. The paper argues that this two-level lookup preserves accuracy because the meta-HNSW, built from a uniform sample, reliably classifies queries into the correct clusters. On top of this, the remote index is laid out as serialized cluster pairs with shared overflow space, and doorbell batching merges non-contiguous reads into single round trips. The reported result is up to 117x lower query latency than a naive remote HNSW at a recall of 0.87 on SIFT1M@1, with similar gains on GIST1M.","pith_inferences":["If the 500-vector routing sample generalizes, the same meta-graph idea could support multi-level partitioning for billion-scale datasets, with memory pools holding nested sub-clusters.","The recall ceiling set by routing suggests an adaptive extension: monitor how often true nearest neighbors fall outside the selected clusters and rebuild or enlarge the meta-HNSW when needed.","The reported 117x gain is against a naive remote baseline; comparing against a monolithic in-memory HNSW on the same hardware would quantify the intrinsic cost of disaggregation, a comparison the paper does not include.","Shared overflow space implies a write-heavy workload could exhaust the reserved area; a testable extension is measuring insertion throughput and overflow behavior over sustained updates."],"forward_implications":["Vector similarity search can be served from a memory pool with weak computational power, because compute nodes perform routing through the cached meta-graph and the memory node only answers bulk RDMA reads.","Query cost is set by the number of relevant sub-HNSW clusters fetched, not by total dataset size, so the approach scales to datasets that exceed any single node's memory.","Batched queries share RDMA transfers: a cluster requested by many queries is loaded once per batch, and recently used clusters are reused across batches.","Dynamic insertions are handled by overflow space shared between pairs of sub-HNSW clusters, so new vectors stay contiguous with their cluster and can be read in one transfer."],"supporting_citations":[{"why":"Supplies the HNSW graph index that d-HNSW disaggregates into a cached meta-graph and remote sub-HNSW clusters.","marker":"[20]"},{"why":"Provides the representative-index idea: a lightweight index built from sampled vectors partitions the dataset for routing.","marker":"[4]"},{"why":"Supplies RDMA design guidance that motivates one-sided reads, memory registration, and doorbell batching.","marker":"[11]"},{"why":"Establishes disaggregated database systems as the target architecture that d-HNSW builds on.","marker":"[35]"}],"fun_headline_variants":["d-HNSW cuts remote vector search latency by up to 117x","Cached meta-graph slashes RDMA round trips for vector search","Disaggregated HNSW: 117x faster vector search with meta-cache","d-HNSW: Smart sampling and RDMA batching speed up vector search","Two-level HNSW for remote memory: 117x latency win at 0.87 recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a meta-HNSW built from just 500 sampled vectors routes each query to the sub-clusters that contain its true nearest neighbors; if that routing ever fails, no additional search effort inside the chosen clusters can recover the missed neighbors.","fun_headline_variants_meta":{"raw":{"variants":["d-HNSW cuts remote vector search latency by up to 117x","Cached meta-graph slashes RDMA round trips for vector search","Disaggregated HNSW: 117x faster vector search with meta-cache","d-HNSW: Smart sampling and RDMA batching speed up vector search","Two-level HNSW for remote memory: 117x latency win at 0.87 recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3484,"prompt_tokens":995,"completion_tokens":2489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2381}},"tokens_in":611,"tokens_out":2489,"duration_ms":16894,"temperature":1.0,"reasoning_tokens":2381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:47:35.499145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run d-HNSW on SIFT1M with the routing step in place but give the local sub-HNSW search a very large efSearch; if recall plateaus well below 1.0, the plateau measures the fraction of true neighbors that fall outside the clusters selected by the cached meta-graph. Alternatively, directly check, for a random sample of queries, whether the ground-truth top-1 neighbor lies inside the selected sub-HNSW clusters.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HNSW graph index that d-HNSW disaggregates into a cached meta-graph and remote sub-HNSW clusters."},{"cited_title":"Pyramid: A general framework for distributed similarity search on large-scale datasets","cited_arxiv_id":null,"evidence_quote":"Provides the representative-index idea: a lightweight index built from sampled vectors partitions the dataset for routing."},{"cited_title":"Design guide- lines for high performance rdma systems","cited_arxiv_id":null,"evidence_quote":"Supplies RDMA design guidance that motivates one-sided reads, memory registration, and doorbell batching."},{"cited_title":"Disaggregated database systems","cited_arxiv_id":null,"evidence_quote":"Establishes disaggregated database systems as the target architecture that d-HNSW builds on."}],"review_version":1}