{"id":"c710c686-4a95-446e-8cdb-6476bbd74959","arxiv_id":"2505.07621","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A benchmark of six cloud CPUs shows that the best choice for vector search depends on the index type and quantization, with Graviton3 winning on queries per dollar and Zen4 on raw IVF throughput.","lead":"This paper benchmarks vector search on six AWS cloud CPU types, testing IVF, HNSW, and full scans under several quantization schemes. It finds the fastest CPU depends on the search method, and that a mid-range ARM chip gives the best queries per dollar.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QP$ ranking in Table 1 depends on an arbitrary categorical score; the 'Graviton3 best for most' claim may not survive alternative aggregation.","rationale":"The paper's main value is the measured data, which support the qualitative point that CPU choice interacts with index/quantization. The specific headline about Graviton3 is a ranking summary, and the only summary presented is the categorical score. Because the score's thresholds and equal weighting are arbitrary, the headline could change under reasonable alternatives. This is more load-bearing than the library-kernel issue, since the latter is disclosed and only makes Graviton3 look worse than its hardware potential. The reader's CONDITIONAL verdict remains appropriate: the paper should add a sensitivity analysis or soften the claim. I therefore keep the verdict unchanged.","tokens_in":15976,"tokens_out":12997,"duration_ms":126145,"concrete_test":"Recompute the QP$ rankings from Tables 4-6 using alternative aggregations: (a) geometric mean of each machine's QP$ relative to the per-cell best, across all settings; (b) arithmetic mean of per-index-family means (IVF, HNSW, full scan), so each index type is weighted equally; and (c) a threshold sweep (vary the +/·/- cutoff from 1.1x to 4x) to see how often Graviton3 is top. If Graviton3 remains first under (a)-(c) on all three datasets, the headline is robust; if Zen4, Graviton4, or SPR takes the lead in any of these, then the 'best for most' claim is an artifact of the Section 4 scoring system and the paper should be revised to report sensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Graviton3 gives the best QP$ 'for most indexes and quantization settings' rests on the scoring system in Section 4 / Table 1, not directly on the measured QP$ values. That scoring converts each measured QP$ into one of five arbitrary tiers (++ / + / · / - / --, with thresholds 20%, 2x, 3x) and then sums +1 for every '+' and -1 for every '-' across all settings. This aggregation has two properties that can drive the headline: it is insensitive to magnitude once a threshold is crossed, and it weights every configuration equally. For example, in the OpenAI/1536 full-scan table (Table 6), SPR beats Graviton3 by roughly 4x in FAISS SQ6/SQ4 QP$, which is scored as just one category difference, while a long run of '·' cells contributes zero to the score. A CPU that is consistently second or third can therefore outscore a CPU that wins several cells by large margins. The paper gives no sensitivity analysis, so the abstract's claim is a property of this particular scoring rule rather than a demonstrated robust property of the data. The reader's concern about missing FAISS ARM SIMD kernels is real but is acknowledged in Section 4 and would, if anything, improve Graviton3's relative standing; it is not the main threat to the ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an experimental benchmark of vector similarity search on six AWS cloud CPU microarchitectures (Graviton3, Graviton4, Zen3, Zen4, Sapphire Rapids, and Sapphire Rapids Z), using FAISS and USearch across HNSW, IVF, and full-scan workloads with multiple quantization levels and three datasets of different dimensionality. The main empirical findings are that relative CPU performance depends strongly on the index type and quantization setting, with Zen4 leading IVF/full scans in QPS, Intel Sapphire Rapids leading HNSW in QPS, and Graviton3 often being the best in queries per dollar (QP$) once instance price is considered. The paper aggregates the per-setting QPS and QP$ results into a categorical scoring table and uses this table to support the headline claim that Graviton3 gives the best 'bang for the buck' for most indexes and quantization settings, even over Graviton4.","tokens_in":16228,"tokens_out":8496,"duration_ms":80498,"significance":"The work addresses a timely and practically important question, and the raw measurements are valuable: the authors vary index type, quantization, dataset dimensionality, and CPU, and they supplement the query benchmarks with memory bandwidth and latency measurements that explain the observed differences. If the results hold, they provide clear, actionable guidance for cloud users, potentially yielding several-fold improvements in QPS or QP$ by matching CPU choice to the workload. The paper also deserves credit for explicitly acknowledging the main software-level limitation, namely the absence of optimized ARM SIMD kernels in FAISS for scalar quantization, and for using open, well-known datasets and libraries. However, the headline claim rests on a categorical scoring system whose robustness is not examined, and the experimental protocol omits variance information and exact recall operating points; these issues need to be addressed before the quantitative ranking can be considered established.","major_comments":[{"comment":"The central claim that Graviton3 gives the best QP$ 'for most indexes and quantization settings' is not read directly from the measured QP$ values but from the categorical scoring system in Table 1. That system applies arbitrary thresholds (20%, 2x, 3x) and assigns +1/-1 per symbol, making it insensitive to the magnitude of differences and weighting every configuration equally. For example, in the OpenAI/1536 FAISS full-scan SQ6 column of Table 6, SPR's QP$ is about 4.4x Graviton3's (3.1 vs. 0.7), yet this counts as only a single category difference, while a long run of middle-ground '·' cells contributes zero. No sensitivity analysis is provided, so the abstract's 'best for most' claim is a property of this particular aggregation rule rather than a demonstrated robust property of the data. I ask the authors to show how the ranking changes under alternative aggregations, such as geometric-mean QP$ across settings, counting only settings where a CPU is within a fixed fraction of the best, or using the raw QP$ ratios directly.","section":"Section 4 and Table 1"},{"comment":"All cross-CPU QPS/QP$ comparisons in Tables 4-6 are reported 'at the highest possible recall', but the paper never states the recall values, whether they are equal across microarchitectures, or how they were determined. If one CPU reaches a higher recall than another at its respective 'highest' setting, the QPS comparison mixes accuracy and speed and is not an apples-to-apples comparison. Since the central comparisons are cross-machine, the authors should fix a target recall or, alternatively, report the achievable recall per machine and show recall-vs-QPS curves for at least the settings that determine the headline ranking.","section":"Section 3.1 and Tables 4-6"},{"comment":"The paper does not report the number of repetitions per measurement, standard deviations, or run-to-run variability, and it states only that queries were run individually with k=10. For benchmark claims that drive a specific recommendation, the lack of error bars is a concern, especially because some Table 1 score totals are separated by only one point. Please add a methodology paragraph covering warmup, the number of queries per run, how many times each experiment was repeated, and how the 'highest possible recall' operating point was found, and report variance for the results that determine the ranking.","section":"Section 3 (experimental methodology)"}],"minor_comments":[{"comment":"The QP$ metric is used throughout but never explicitly defined. Please state the formula, e.g., QP$ = QPS × 3600 / (price per hour), and clarify the normalization used in each table, since Table 4 and Table 5 values are in units of 10^5 while Table 6 values appear to be in units of 10^4, with some entries such as the BQ full-scan values exceeding that order.","section":"Section 3 and Table captions"},{"comment":"The scoring description says 'each (+) is 1 and each (-) is -1' but does not explain how the '++' and '--' symbols are counted. Please state explicitly whether '++' contributes two points and '--' contributes minus two, or whether the symbols are intended only as labels for the best and worst tiers.","section":"Section 4 and Table 1"},{"comment":"The claim that FAISS uses scalar code to decode quantized vectors on ARM CPUs is important for explaining the Graviton results, but no evidence or source reference is provided. Please cite the relevant FAISS implementation or verify the claim with a small microbenchmark, so readers can assess how specific this limitation is to the tested version.","section":"Section 3"},{"comment":"When comparing Graviton3 and Graviton4, the paper states that most benchmarks show Graviton4 delivering more performance but does not cite examples. Adding one or two references here would help readers weigh the claimed novelty of this observation.","section":"Section 3.4"},{"comment":"The sentence 'Graviton3 is only pushed back in our scoring system due to the lack of symmetric kernels in FAISS' is easy to misread; it means that Graviton3's score would be even higher if FAISS had ARM-optimized kernels. Please rephrase to make the direction of the effect explicit.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a workshop-style empirical benchmark paper, and its main contribution is practical guidance rather than a new algorithmic or systems technique. The core measurements are useful, but the headline ranking is sensitive to the categorical scoring rule in Table 1, and the experimental methodology lacks error bars and explicit recall operating points. I believe these issues are addressable within the scope of the paper, so I recommend major revision rather than rejection. It would also strengthen the paper to release the benchmark scripts and raw results for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take a look at this if you care about vector search in the cloud. It's the first public benchmark I know of that compares QP$ across six AWS CPU microarchitectures (Graviton3/4, Zen3/4, Sapphire Rapids, SPR-Z) for IVF, HNSW, and full scans with several quantizations. The data are concrete: QPS and QP$ for FAISS and USearch, with memory latency/bandwidth measurements that explain the main effects. The paper does well by using real AWS on-demand prices and by showing that the \"best\" CPU flips with index type and quantization. It also correctly flags its own main caveat: FAISS decodes quantized vectors to float32 with scalar code on ARM, which hurts Gravitons on SQ.\n\nNow the soft spots. No error bars, no repetition counts, and the operating point is \"highest possible recall\" per machine, so you're comparing QP$ at different accuracy levels. More important, the headline \"Graviton3 is best for most\" is an artifact of the scoring system in Table 1. That system converts each QP$ into one of five tiers and sums +1/-1. A 4x loss and a 2x loss both get -1; a consistent second place gets zero. Graviton3 does win or tie a plurality of the raw QP$ cells, so the score isn't pure fiction, but \"most\" overstates it. In IVF SQ and full-scan SQ, Intel/AMD clearly beat Graviton3 by large margins. The paper would be more honest if it reported win counts or a geometric mean of QP$ ratios, and did a sensitivity analysis on the scoring thresholds.\n\nNone of this sinks the paper. The qualitative finding -- CPU choice matters, and it depends on the algorithm and quantization -- is solid. The Graviton3-vs-Graviton4 SVE observation is interesting and well explained. It deserves a serious referee, but the referee should push for standard benchmark hygiene: error bars, a fixed recall target (or a curve), and a less arbitrary aggregation. If you're choosing AWS instances for vector search, this is the most useful public data I've seen.\n\nRecommendation: send it to peer review, require revisions.","headline":"Useful cloud CPU benchmark, but the headline \"Graviton3 best for most\" is partly an artifact of the paper's scoring rule.","tokens_in":16717,"tokens_out":4499,"would_cite":true,"duration_ms":37100,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cloud CPU choice can triple queries per dollar for vector search, with Graviton3 usually best value.","keywords":["vector search","queries per dollar","cloud CPU benchmarking","Graviton3","SIMD kernels","quantization","HNSW","IVF"],"falsifier":"Replace FAISS's ARM scalar path for scalar-quantized vectors with a SIMD-optimized kernel, or benchmark a third library with symmetric 8-bit kernels on ARM, and rerun the IVF and full-scan measurements; if Graviton3 no longer leads in QP$ across most settings, the central ranking is an artifact of a missing kernel rather than of the hardware.","tokens_in":15773,"feed_emoji":"💸","tokens_out":8824,"duration_ms":76770,"temperature":0.7,"pith_summary":"This paper shows that the best cloud CPU for vector search depends on the index type, quantization level, and instance price, not just on raw SIMD power. The authors benchmarked IVF, HNSW, and full scans at several quantization levels on six AWS instances spanning Intel, AMD, and ARM designs, measuring both queries per second (QPS) and queries per dollar (QP$). Their central result is that no CPU wins everywhere: AMD Zen4 gives almost 3x the QPS of Intel Sapphire Rapids on IVF with float32 vectors, Intel Sapphire Rapids leads HNSW throughput, and the cheaper ARM instances, especially Graviton3, often deliver the best QP$. If the ranking holds, users can get roughly 3x more queries per dollar simply by matching the CPU to the search workload, which matters for any pay-per-query cloud deployment.","feed_headline":"On AWS, Graviton3 wins vector search on queries per dollar","feed_subtitle":"Matching CPU choice to index type and quantization can triple queries per dollar on cloud vector search.","key_machinery":"The central measurement object is QP$, queries per second divided by the instance's hourly on-demand price, which the paper uses to rank value rather than raw speed. The explanatory machinery is a two-factor account of why rankings flip: (1) the search algorithm's data-access pattern, sequential bulk reads for IVF and full scans versus cache-resident walks over HNSW graphs, combined with each microarchitecture's load bandwidth, sequential latency at L3/DRAM, and SIMD register width; and (2) whether the distance kernel computes symmetrically in the quantized domain, as in USearch, or first decodes to float32, as in FAISS, which on ARM makes scalar-quantized vectors several times slower. Together these factors explain the headline results: Zen4 dominates bandwidth-hungry scans, Sapphire Rapids wins HNSW throughput, and Graviton3 wins on cost.","core_discovery":"The paper's central claim is that the queries-per-dollar ranking of cloud CPUs is scenario-dependent, and that on current AWS pricing Graviton3 (r7g) is the best value option for most vector search settings, even beating its successor Graviton4, while AMD Zen4 (r7a) is the strongest raw performer for IVF indexes and full scans on float32 vectors. The differences are not random: they follow from the interaction between two mechanisms. Sequential, bandwidth-hungry access patterns in IVF and full scans favor CPUs with high L3/DRAM read bandwidth and low sequential latency, where Gravitons and Zens excel and Sapphire Rapids falls behind, while cache-friendlier graph traversal in HNSW favors Sapphire Rapids' high L1 load bandwidth and wider SIMD. Kernel implementation also matters: the FAISS library decodes scalar-quantized vectors back to float32 with scalar code on ARM, which depresses Graviton QPS in scalar-quantized scenarios by up to 3-6x, whereas the USearch library's symmetric quantized kernels let the same instances lead on 8-bit full scans.","pith_inferences":["If a vector library adds NEON/SVE kernels that decode 8-bit quantized vectors symmetrically on ARM, the Gravitons' scalar-quantization disadvantage would likely shrink, extending their QP$ advantage to more scenarios than the paper reports.","The single-threaded, no-update protocol may understate how batch queries or index construction affect rankings, since memory bandwidth saturation and load-port behavior differ under multi-query execution.","The durable insight is mechanistic rather than tied to today's instances: procurement for serverless vector services should be guided by data-access pattern and kernel efficiency, not by CPU generation or list price alone.","Because QP$ divides by hourly price, the absolute winning instance could shift with AWS pricing; the paper's specific instance ranking is time-sensitive, while the underlying bandwidth-versus-SIMD explanation is not."],"forward_implications":["Deploying HNSW in production should not default to the newest CPU: Intel Sapphire Rapids Z gives the highest QPS, but Zen3 and Graviton3 return more queries per dollar on high-dimensional data.","For IVF indexes and full scans on float32 or bfloat vectors, choosing AMD Zen4 over Intel Sapphire Rapids can yield roughly 3x more QPS and QP$.","Quantization level changes the ranking: at 8-bit scalar quantization Intel and AMD widen their lead, while at 1-bit and float16 the ARM instances become the cost leaders.","Users can realize about 3x more queries per dollar by matching microarchitecture to workload, an effect comparable to switching algorithms or libraries.","Newer cloud CPU generations are not automatically better for vector search: Graviton3 beats Graviton4 in many settings because its 256-bit SVE registers outperform Graviton4's 128-bit SVE registers.","A third library with symmetric 8-bit kernels on ARM would test whether Graviton3 remains the cost leader in scalar-quantized settings."],"supporting_citations":[{"why":"Supplies the FAISS implementation (v1.9.0) used for IVF and full-scan measurements, including the asymmetric float32-decoding kernels that penalize ARM on scalar-quantized vectors.","marker":"[11,33]"},{"why":"Provides the USearch library, whose symmetric quantized kernels supply the counterpoint measurements on HNSW and full scans.","marker":"[39]"},{"why":"Defines the IVF partition-based index and product quantization, the index category whose sequential access pattern drives one side of the comparison.","marker":"[18]"},{"why":"Defines HNSW, the graph index whose cache-resident traversal drives the other side of the comparison.","marker":"[25]"},{"why":"Documents Arm Neoverse V1 (Graviton3) SIMD capabilities, load throughput, and latencies used to explain Graviton3's cost advantage.","marker":"[5]"},{"why":"Documents Arm Neoverse V2 (Graviton4) capabilities, showing why the 128-bit SVE registers make Graviton4 slower than Graviton3 in some settings.","marker":"[6]"},{"why":"Provides the microarchitecture throughput and latency data for Intel and AMD CPUs used to explain Zen4 and Sapphire Rapids behavior.","marker":"[12]"},{"why":"Supports the premise that vector search is largely data-access bound, which motivates using cache bandwidth and memory latency as the main explanatory variables.","marker":"[21,31,34,41]"}],"fun_headline_variants":["Cloud CPU choice can triple vector search value","Graviton3 beats Graviton4 in vector search QP$","Vector search: the best cloud CPU depends on index","AMD Zen4 tops IVF, Graviton3 wins value"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings depend on the two benchmarked libraries representing each CPU's achievable performance; in particular, FAISS's scalar ARM code for scalar-quantized vectors may understate what Graviton instances could deliver, so a better-optimized library could change which CPU looks cheapest.","fun_headline_variants_meta":{"raw":{"variants":["Cloud CPU choice can triple vector search value","Graviton3 beats Graviton4 in vector search QP$","Vector search: the best cloud CPU depends on index","AMD Zen4 tops IVF, Graviton3 wins value"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00054,"raw_usage":{"total_tokens":2583,"prompt_tokens":932,"completion_tokens":1651,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1585}},"tokens_in":548,"tokens_out":1651,"duration_ms":10339,"temperature":1.0,"reasoning_tokens":1585,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:11:28.858317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace FAISS's ARM scalar path for scalar-quantized vectors with a SIMD-optimized kernel, or benchmark a third library with symmetric 8-bit kernels on ARM, and rerun the IVF and full-scan measurements; if Graviton3 no longer leads in QP$ across most settings, the central ranking is an artifact of a missing kernel rather than of the hardware.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents Arm Neoverse V1 (Graviton3) SIMD capabilities, load throughput, and latencies used to explain Graviton3's cost advantage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents Arm Neoverse V2 (Graviton4) capabilities, showing why the 128-bit SVE registers make Graviton4 slower than Graviton3 in some settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the microarchitecture throughput and latency data for Intel and AMD CPUs used to explain Zen4 and Sapphire Rapids behavior."}],"review_version":1}