{"id":"ff1dad3a-5820-4103-9592-6025297295a6","arxiv_id":"2411.14006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A benchmark showing that ANN search algorithm performance rankings shift between server and edge devices, and that low-cost edge hardware can match expensive counterparts on CPU workloads.","lead":"This paper benchmarks five approximate nearest neighbor search setups on edge devices such as NVIDIA Jetsons and Raspberry Pis, measuring queries per second, insertion and deletion speed, and power consumption. It reports that algorithm rankings differ between server and edge hardware, and that a low-cost device like the Raspberry Pi 4 can be competitive with far more expensive Jetson boards.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed server-optimized hyperparameters and PQ compression settings across all edge devices may confound the central device-comparison claim; the paper reports no sensitivity analysis to determine whether rankings are artifacts.","rationale":"The central claim is an empirical generalization about algorithm performance across hardware. For that generalization to be valid, the comparisons must isolate hardware effects from configuration effects. The paper's own methodology (Section 4.2) fixes both construction-time hyperparameters and PQ compression at values chosen on a 28-core/2TB server, then transfers the same index to all edge devices. This design is reasonable for a 'build once, deploy anywhere' scenario, but it creates a specific, testable threat: the compression level is capped by the least capable device, so the more powerful devices are running an unnecessarily lossy representation. The claim that 'more powerful devices do not always yield more efficient execution' could be an artifact of this capping. The reader's weakest_assumption identified the same class of issue (server-optimized parameters may not be near-optimal on edge devices); I agree partially and sharpen it to the fixed PQ setting, which is explicitly stated and directly couples the memory constraint to compression quality. The absence of reported hyperparameter values and lack of any sensitivity test means the concern cannot be dismissed internally. This does not warrant rejection: the observed effects could be real, and the paper's conclusions are plausible. It does warrant maintaining the conditional verdict, with the condition that the authors either supply a sensitivity analysis (or per-device tuning results) or at minimum disclose all parameter settings and code so the benchmark can be reproduced. I therefore recommend 'UNCHANGED' relative to the reader's CONDITIONAL verdict.","tokens_in":7707,"tokens_out":7721,"duration_ms":75777,"concrete_test":"Re-run the CPU QPS-accuracy sweeps (Figures 2 and 3, left) on at least the Raspberry Pi 4, Jetson Nano, Xavier AGX, and Orin using two configurations: (i) the server-optimized settings as in the paper, and (ii) settings re-optimized per device, including PQ bytes per vector chosen for each device's memory budget and HNSW M/efConstruction re-tuned on a device-specific validation split. Compare the resulting Pareto frontiers. If the per-device frontiers change the observed rankings—for example, if Orin overtakes Xavier or Pi 4 no longer matches Jetson at matched accuracy—the paper's central claim is an artifact of the fixed server-side configuration. Also require reporting the exact tuned parameter values to make the test reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 1) rests on device-level comparisons of QPS/accuracy tradeoffs in Figures 2–3. The paper states (Section 4.2) that 'construction and parameter optimization were conducted on a server' and that 'PQ was applied across all algorithms to ensure compatibility with memory constraints of edge devices.' This implies a single, server-tuned configuration (index construction parameters such as HNSW M/efConstruction, IVF nlist, PQ subquantizer settings) was transferred to all seven devices without per-device tuning or a sensitivity analysis. Because the devices span 512 MB (Pi Zero) to 64 GB (Jetson Orin) with qualitatively different cache hierarchies and core counts, a single compression level may disproportionately handicap the more capable devices: for instance, a PQ setting chosen to fit the weakest device needlessly degrades accuracy on the Orin, and a graph built with server-optimal M may exploit the server's large cache in a way that does not transfer. Under these conditions, the observation that the Raspberry Pi 4 matches Jetson devices, or that Xavier beats Orin, could reflect the fixed configuration rather than intrinsic algorithm/device performance. The paper does not report the tuned hyperparameter values, the PQ bit allocation, or any robustness check (such as per-device parameter optimization or a few alternative settings), so the reader cannot tell whether the central conclusions are configuration-specific artifacts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an experimental comparison of approximate nearest neighbor search (ANNS) algorithms on edge devices for real-time smart-city object tracking. Using street-camera data from Helsinki (Section 3.4), the authors extract ResNet50 embeddings from detected bounding boxes and evaluate FAISS implementations of IVF, LSH, HNSW-PQ, and PQ, plus DiskANN/Vamana, on seven edge devices (Jetson Nano, TX2, Xavier AGX, Orin, Raspberry Pi 4, Pi 3, Pi Zero) and a server. The evaluation metrics are QPS, classification accuracy, insertion/deletion throughput (only for DiskANN), and power consumption. The central claim (Section 1) is that algorithm performance patterns differ between edge devices and servers, that stronger edge devices do not always yield faster execution, and that inexpensive devices can match more expensive counterparts. The paper concludes with suggestions for future optimization.","tokens_in":8019,"tokens_out":6370,"duration_ms":56693,"significance":"The paper addresses a practical and understudied question: whether ANN benchmarks obtained on server hardware transfer to edge devices. Its strengths are the breadth of real-world hardware (seven devices), the use of real street-camera data, the inclusion of CPU and GPU execution paths, and the explicit attention to cost-effectiveness. If the central claims are supported after the missing analysis is supplied, the paper would provide useful engineering guidance and a caution against extrapolating server benchmarks to edge deployments. The contribution is incremental over existing ANN survey and benchmark literature, but the edge-device focus is a useful addition. The main limitation is that the descriptive conclusions currently rest on unreported configuration details, a dataset-size inconsistency, and single-point measurements without variance information.","major_comments":[{"comment":"The central claim of device-dependent rankings rests on measurements taken with one fixed configuration: \"Construction and parameter optimization were conducted on a server\" and \"PQ was applied across all algorithms to ensure compatibility with memory constraints of edge devices.\" The paper does not report the tuned hyperparameter values (e.g., HNSW M/efConstruction/efSearch, IVF nlist/nprobe) nor the PQ codebook settings, and it provides no sensitivity analysis or per-device tuning. Because the devices differ by orders of magnitude in memory (512 MB to 64 GB) and cache architecture, the observed gaps (e.g., Pi 4 three times faster than Jetson Nano, Xavier 1.5 times faster than Orin) could be artifacts of a single server-tuned and compression setting rather than intrinsic algorithm or device behavior. Please report the settings and add either per-device parameter optimization or a robustness check across a few parameter choices.","section":"§4.2, Figs. 2–3"},{"comment":"Section 3.4 states that the preprocessing pipeline produced 161,805 valid bounding boxes, split 85% for training and 15% for evaluation, and gives no indication of augmentation to a larger size; Table 3, however, reports construction times and index sizes for \"~1M vectors.\" This is roughly a factor-of-six discrepancy that affects the interpretation of all scalability claims and the index-size numbers. Please clarify the actual number of vectors used in each index and explain any augmentation or duplication step that produces ~1M vectors.","section":"§3.4 vs. Table 3"},{"comment":"None of the QPS/accuracy figures or the text reports the number of repeated runs, error bars, confidence intervals, or per-device variance. Quantitative statements such as \"it performed over three times better than the Jetson Nano\" and \"the Jetson Xavier also outperformed the Jetson Orin by 1.5 times\" are presented without supporting tables of exact values or measures of spread. Since the goal is to compare algorithms and devices, the absence of variance information leaves the reader unable to judge whether the reported rankings are statistically meaningful.","section":"§4.2, Figs. 2–3"},{"comment":"The abstract and Section 3.2 promise additional metrics of insertion/deletion latency and power consumption, but the experiment does not deliver a comparative evaluation of these metrics. Insertion/deletion results are shown only for DiskANN in Figure 4, and there is no comparison of other algorithms' insertion/deletion throughput. Power consumption is only listed as hardware specifications in Tables 1 and 2 and asserted in one sentence (\"GPU-based inference consumes double the power of CPU-based inference\") without a measurement methodology or results. Please either add the missing measurements and figures or explicitly qualify the claims to the metrics actually measured.","section":"§3.2, §4.2"}],"minor_comments":[{"comment":"The title and Section 2 describe the study as comparing graph-based ANNS algorithms, but the evaluated set includes LSH and PQ, which are not graph-based; please either adjust the terminology or explain why non-graph methods are included.","section":"§1, §2.2"},{"comment":"The reason FAISS was not supported on Raspberry Pi 3 and Pi Zero is not stated; please specify whether this is due to ARM architecture, memory limits, or library dependencies.","section":"§4.2"},{"comment":"The embedding pipeline mentions data augmentation (random cropping, horizontal flipping, brightness adjustments), but the paper does not state whether augmented samples were added to the ANN index or used only for training; this affects the interpretation of the dataset size.","section":"§3.4"},{"comment":"Software versions are not reported; please list FAISS and DiskANN versions, the operating system, and the CPU/GPU driver versions used on each device to enable reproducibility.","section":"§4.1"},{"comment":"The figures are referenced as displaying comparisons, but the text does not specify the exact operating point (e.g., efSearch or nprobe) used for each curve; please describe the swept parameters and the axes in the captions.","section":"§4.2, Fig. 2"},{"comment":"The accuracy metric is a classification accuracy based on neighbor labels, not recall@K of the ANN index; this choice is reasonable for the application, but it should be stated explicitly that the metric conflates embedding quality with ANN accuracy.","section":"§3.1"},{"comment":"No statement is provided on code or data availability; please include one to allow the benchmark to be reproduced.","section":"n/a"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a benchmark rather than a theoretical contribution, so the main risk is not novelty but reproducibility and confounded device comparison. The central claims are plausible and the edge-device focus is genuinely useful, but the current version lacks the measurement detail needed for a definitive reference benchmark. The revision must resolve the dataset-size inconsistency, report the tuned hyperparameters and library versions, add variance information or repeated runs, and either measure or qualify the power and insertion/deletion claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper gives the community something genuinely new—measured ANN search throughput, insertion/deletion latency, and power draw for HNSW, IVF, LSH, and PQ across a meaningful range of edge hardware, from Pi Zero to Jetson Orin, on real camera data. The observation that the Raspberry Pi 4 competes with far pricier Jetsons on CPU search, and that Xavier can beat Orin for HNSW-PQ, is the kind of empirical nugget practitioners will care about. The paper is also honest enough to report that construction and parameter tuning were done on a big server, not on the devices.\n\nThe problems are real but not fatal. The biggest one is exactly what the stress-test flagged: with one server-tuned configuration and one PQ compression level applied on every device, the across-device rankings in Figures 2–3 may be artifacts of that fixed setup. A PQ setting that fits a 512 MB Pi Zero could needlessly cripple accuracy or speed on a 64 GB Orin, and a graph built with server-optimized M may not transfer its cache behavior. The authors do not report the tuned hyperparameters, the PQ bit allocation, or any sensitivity check, so we cannot tell whether \"cost-effective devices match expensive ones\" is a property of the algorithms or of the configuration. That is a real limitation and it sits at the center of the paper's main claim.\n\nA smaller but concrete inconsistency: the text says the dataset had 161,805 bounding boxes, yet Table 3 describes indexing ~1M vectors. Either they used a different set for the index or there is an error; the paper should explain this. Also, the title says \"graph-based\" but the comparison includes IVF, LSH, and plain PQ, which are not graph methods. Minor, but sloppy.\n\nThe missing code, data, error bars, and library versions make replication impossible as submitted. That said, the work is not a toy; the effort to run on seven devices with real traffic data is considerable, and the findings are plausible enough to warrant a proper review, not a desk reject.\n\nWho gets value: people choosing ANN libraries for edge deployments, and anyone building edge benchmarks themselves—they will find the pitfalls instructive. I would not cite it yet in my own work, but if the authors release artifacts and add a sensitivity analysis, I would. My recommendation: send to peer review, but make the methods and the fixed-configuration caveat explicit before publication.","headline":"Fresh benchmark data on ANN search across seven edge devices, but fixed server-tuned parameters and missing methodology limits the strength of the cross-device conclusions.","tokens_in":8538,"tokens_out":2286,"would_cite":false,"duration_ms":22632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that approximate nearest neighbor search performance on edge devices does not follow server rankings: cheaper, less powerful devices can match or beat costly ones, so practitioners should benchmark on the actual…","keywords":["approximate nearest neighbor search","graph-based index","edge devices","HNSW","product quantization","benchmark","NVIDIA Jetson","Raspberry Pi"],"falsifier":"Re-run the same benchmark with hyperparameters optimized separately on each device; if the Raspberry Pi 4's threefold advantage over the Jetson Nano disappears or reverses when each device uses its own tuned settings, the paper's central device-ranking claim would be undercut.","tokens_in":7529,"feed_emoji":"🔍","tokens_out":4046,"duration_ms":35128,"temperature":0.7,"pith_summary":"The paper reports an experimental comparison of five graph-based approximate nearest neighbor (ANN) search implementations on seven edge devices, using real street-camera embeddings. Its central claim is that performance rankings measured on a server do not carry over to edge hardware: cheaper, less powerful devices can outperform more expensive ones for the same algorithm, and a mid-priced Raspberry Pi 4 can match or beat several NVIDIA Jetson boards in CPU search throughput. The authors argue that practitioners should therefore evaluate ANN algorithms on the actual edge hardware they plan to deploy, with attention to insertion and deletion costs and power consumption, not just query speed. The study also shows that DiskANN supports true vector deletion but its uncompressed index was too large to fit on the edge devices tested.","feed_headline":"Raspberry Pi 4 outranks costly Jetsons in nearest-neighbor search","feed_subtitle":"Five ANN algorithms on seven devices show server rankings flip on edge hardware; benchmark before you buy.","key_machinery":"The evaluation is carried by a fixed benchmark protocol: indices for FAISS-based IVF, LSH, HNSW-PQ, and PQ, plus DiskANN's Vamana graph, are constructed once on a 28-core server with 2 TB RAM and then copied to the edge devices, where query throughput, insertion and deletion latency, and power draw are measured. The central objects compared are the graph-based index structures themselves, HNSW's hierarchical small-world graph and Vamana's navigable graph, alongside inverted-file and hashing baselines. Because hyperparameters are tuned only on the server, the protocol isolates device behavior under identical index configurations rather than per-device optimal tuning.","core_discovery":"On its street-camera dataset of about one million 2048-dimensional embeddings, the paper finds that HNSW-PQ generally outperforms IVF-PQ on edge devices even though the reverse pattern appears on a server CPU, that the Raspberry Pi 4 achieves over three times the CPU queries-per-second of the Jetson Nano at about a quarter of the price, that the Jetson Xavier AGX outperforms the newer Jetson Orin for HNSW-PQ despite fewer cores, and that GPU acceleration triples throughput while roughly doubling power consumption. These results support the claim that device-specific benchmarking, including dynamic insert and delete pipelines, is needed before choosing an ANN algorithm for edge deployment.","pith_inferences":["A likely explanation the paper does not test is that memory-hierarchy details, such as cache size, matter more than core count for these graph traversals; a controlled study varying only cache size could confirm this.","For battery-powered deployments, queries-per-second per watt may be a better objective than raw QPS, which would strengthen the case for low-power boards even further.","The fixed-server hyperparameter protocol may systematically disadvantage algorithms whose optimal parameters depend on hardware; tuning per device could change both the algorithm ranking and the device ranking.","The inability to run DiskANN on edge devices points to an open problem: compressed on-disk graph indexes with true deletion that fit within edge memory budgets."],"forward_implications":["Practitioners should treat server-side ANN benchmarks as only a rough guide and measure query throughput, update latency, and power on the target edge device before deployment.","Cost-effective devices such as the Raspberry Pi 4 are viable platforms for CPU-based real-time ANN search, potentially lowering the hardware cost of smart-city and tracking systems.","Rankings among algorithms can invert between server and edge: HNSW-PQ leads on edge devices while IVF-PQ can lead on the server at low accuracy, so algorithm choice must be hardware-specific.","Dynamic workloads need explicit testing of insertion and deletion paths; DiskANN offers true deletion but its uncompressed index did not fit on the tested edge devices, limiting its use there."],"supporting_citations":[{"why":"Defines the HNSW graph structure used as the primary graph-based method in the benchmark.","marker":"[13]"},{"why":"Supplies the Vamana graph and DiskANN system, including the true deletion mechanism evaluated in the pipeline tests.","marker":"[14]"},{"why":"Defines the inverted-file search method that serves as one of the FAISS baseline algorithms.","marker":"[15]"},{"why":"Provides the FAISS library implementations of IVF, LSH, HNSW, and PQ used for all edge-device experiments.","marker":"[19]"},{"why":"Establishes the prior comprehensive comparison of graph-based ANN algorithms that this paper contrasts with its edge-device focus.","marker":"[17]"},{"why":"Introduces product quantization, the compression technique applied across the FAISS algorithms to fit indices on edge devices.","marker":"[4]"},{"why":"Supplies the ResNet50 feature extractor that produced the 2048-dimensional embeddings used in the dataset.","marker":"[23]"}],"fun_headline_variants":["Benchmarking ANN on edge: Pi 4 outdoes Jetson at quarter price","HNSW-PQ beats IVF-PQ on edge devices, server flips","Older Jetson Xavier beats Orin for HNSW-PQ on edge","GPU triples ANN throughput but doubles power draw on edge","Server ANN rankings flip on edge: benchmark before you buy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that hyperparameters optimized once on a large server remain near-optimal for every edge device, so the observed device rankings are not artifacts of a single fixed configuration.","fun_headline_variants_meta":{"raw":{"variants":["Benchmarking ANN on edge: Pi 4 outdoes Jetson at quarter price","HNSW-PQ beats IVF-PQ on edge devices, server flips","Older Jetson Xavier beats Orin for HNSW-PQ on edge","GPU triples ANN throughput but doubles power draw on edge","Server ANN rankings flip on edge: benchmark before you buy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":3084,"prompt_tokens":797,"completion_tokens":2287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":2190}},"tokens_in":413,"tokens_out":2287,"duration_ms":16789,"temperature":1.0,"reasoning_tokens":2190,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:37:48.212171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same benchmark with hyperparameters optimized separately on each device; if the Raspberry Pi 4's threefold advantage over the Jetson Nano disappears or reverses when each device uses its own tuned settings, the paper's central device-ranking claim would be undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the HNSW graph structure used as the primary graph-based method in the benchmark."},{"cited_title":"Subramanya, Suhas, et al., Diskann: Fast accurate billion-point nearest neighbor search on a single node, in: Advances in Neural Information Processing Systems 32, 2019","cited_arxiv_id":null,"evidence_quote":"Supplies the Vamana graph and DiskANN system, including the true deletion mechanism evaluated in the pipeline tests."},{"cited_title":"Cöster, M","cited_arxiv_id":null,"evidence_quote":"Defines the inverted-file search method that serves as one of the FAISS baseline algorithms."},{"cited_title":"Jegou, M","cited_arxiv_id":null,"evidence_quote":"Introduces product quantization, the compression technique applied across the FAISS algorithms to fit indices on edge devices."}],"review_version":1}