REVIEW 4 major objections 7 minor 26 references
Experimental comparison of graph-based approximate nearest neighbor search algorithms on edge devices
T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that approximate nearest neighbor search performance on edge devices does not follow server rankings: cheaper, less powerful devices can match or beat costly ones, so practitioners should benchmark on the actual…
desk verdict Fresh benchmark data on ANN search across seven edge devices, but fixed server-tuned parameters and missing methodology limits the strength of the cross-device conclusions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The evaluation is carried by a fixed benchmark protocol: indices for FAISS-based IVF, LSH, HNSW-PQ, and PQ, plus DiskANN's Vamana graph, are constructed once on a 28-core server with 2 TB RAM and then copied to the edge devices, where query throughput, insertion and deletion latency, and power draw are measured. The central objects compared are the graph-based index structures themselves, HNSW's hierarchical small-world graph and Vamana's navigable graph, alongside inverted-file and hashing baselines. Because hyperparameters are tuned only on the server, the protocol isolates device behavior under identical index configurations rather than per-device optimal tuning.
What would settle it
Re-run the same benchmark with hyperparameters optimized separately on each device; if the Raspberry Pi 4's threefold advantage over the Jetson Nano disappears or reverses when each device uses its own tuned settings, the paper's central device-ranking claim would be undercut.
Extended reading notes
Core claim
On its street-camera dataset of about one million 2048-dimensional embeddings, the paper finds that HNSW-PQ generally outperforms IVF-PQ on edge devices even though the reverse pattern appears on a server CPU, that the Raspberry Pi 4 achieves over three times the CPU queries-per-second of the Jetson Nano at about a quarter of the price, that the Jetson Xavier AGX outperforms the newer Jetson Orin for HNSW-PQ despite fewer cores, and that GPU acceleration triples throughput while roughly doubling power consumption. These results support the claim that device-specific benchmarking, including dynamic insert and delete pipelines, is needed before choosing an ANN algorithm for edge deployment.
Load-bearing premise
The results assume that hyperparameters optimized once on a large server remain near-optimal for every edge device, so the observed device rankings are not artifacts of a single fixed configuration.
Editorial extensions
If this is right
- Practitioners should treat server-side ANN benchmarks as only a rough guide and measure query throughput, update latency, and power on the target edge device before deployment.
- Cost-effective devices such as the Raspberry Pi 4 are viable platforms for CPU-based real-time ANN search, potentially lowering the hardware cost of smart-city and tracking systems.
- Rankings among algorithms can invert between server and edge: HNSW-PQ leads on edge devices while IVF-PQ can lead on the server at low accuracy, so algorithm choice must be hardware-specific.
- Dynamic workloads need explicit testing of insertion and deletion paths; DiskANN offers true deletion but its uncompressed index did not fit on the tested edge devices, limiting its use there.
Reading between the lines
- A likely explanation the paper does not test is that memory-hierarchy details, such as cache size, matter more than core count for these graph traversals; a controlled study varying only cache size could confirm this.
- For battery-powered deployments, queries-per-second per watt may be a better objective than raw QPS, which would strengthen the case for low-power boards even further.
- The fixed-server hyperparameter protocol may systematically disadvantage algorithms whose optimal parameters depend on hardware; tuning per device could change both the algorithm ranking and the device ranking.
- The inability to run DiskANN on edge devices points to an open problem: compressed on-disk graph indexes with true deletion that fit within edge memory budgets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an experimental comparison of approximate nearest neighbor search (ANNS) algorithms on edge devices for real-time smart-city object tracking. Using street-camera data from Helsinki (Section 3.4), the authors extract ResNet50 embeddings from detected bounding boxes and evaluate FAISS implementations of IVF, LSH, HNSW-PQ, and PQ, plus DiskANN/Vamana, on seven edge devices (Jetson Nano, TX2, Xavier AGX, Orin, Raspberry Pi 4, Pi 3, Pi Zero) and a server. The evaluation metrics are QPS, classification accuracy, insertion/deletion throughput (only for DiskANN), and power consumption. The central claim (Section 1) is that algorithm performance patterns differ between edge devices and servers, that stronger edge devices do not always yield faster execution, and that inexpensive devices can match more expensive counterparts. The paper concludes with suggestions for future optimization.
Significance. The paper addresses a practical and understudied question: whether ANN benchmarks obtained on server hardware transfer to edge devices. Its strengths are the breadth of real-world hardware (seven devices), the use of real street-camera data, the inclusion of CPU and GPU execution paths, and the explicit attention to cost-effectiveness. If the central claims are supported after the missing analysis is supplied, the paper would provide useful engineering guidance and a caution against extrapolating server benchmarks to edge deployments. The contribution is incremental over existing ANN survey and benchmark literature, but the edge-device focus is a useful addition. The main limitation is that the descriptive conclusions currently rest on unreported configuration details, a dataset-size inconsistency, and single-point measurements without variance information.
major comments (4)
- [§4.2, Figs. 2–3] The central claim of device-dependent rankings rests on measurements taken with one fixed configuration: "Construction and parameter optimization were conducted on a server" and "PQ was applied across all algorithms to ensure compatibility with memory constraints of edge devices." The paper does not report the tuned hyperparameter values (e.g., HNSW M/efConstruction/efSearch, IVF nlist/nprobe) nor the PQ codebook settings, and it provides no sensitivity analysis or per-device tuning. Because the devices differ by orders of magnitude in memory (512 MB to 64 GB) and cache architecture, the observed gaps (e.g., Pi 4 three times faster than Jetson Nano, Xavier 1.5 times faster than Orin) could be artifacts of a single server-tuned and compression setting rather than intrinsic algorithm or device behavior. Please report the settings and add either per-device parameter optimization or a robustness check across a few parameter choices.
- [§3.4 vs. Table 3] Section 3.4 states that the preprocessing pipeline produced 161,805 valid bounding boxes, split 85% for training and 15% for evaluation, and gives no indication of augmentation to a larger size; Table 3, however, reports construction times and index sizes for "~1M vectors." This is roughly a factor-of-six discrepancy that affects the interpretation of all scalability claims and the index-size numbers. Please clarify the actual number of vectors used in each index and explain any augmentation or duplication step that produces ~1M vectors.
- [§4.2, Figs. 2–3] None of the QPS/accuracy figures or the text reports the number of repeated runs, error bars, confidence intervals, or per-device variance. Quantitative statements such as "it performed over three times better than the Jetson Nano" and "the Jetson Xavier also outperformed the Jetson Orin by 1.5 times" are presented without supporting tables of exact values or measures of spread. Since the goal is to compare algorithms and devices, the absence of variance information leaves the reader unable to judge whether the reported rankings are statistically meaningful.
- [§3.2, §4.2] The abstract and Section 3.2 promise additional metrics of insertion/deletion latency and power consumption, but the experiment does not deliver a comparative evaluation of these metrics. Insertion/deletion results are shown only for DiskANN in Figure 4, and there is no comparison of other algorithms' insertion/deletion throughput. Power consumption is only listed as hardware specifications in Tables 1 and 2 and asserted in one sentence ("GPU-based inference consumes double the power of CPU-based inference") without a measurement methodology or results. Please either add the missing measurements and figures or explicitly qualify the claims to the metrics actually measured.
minor comments (7)
- [§1, §2.2] The title and Section 2 describe the study as comparing graph-based ANNS algorithms, but the evaluated set includes LSH and PQ, which are not graph-based; please either adjust the terminology or explain why non-graph methods are included.
- [§4.2] The reason FAISS was not supported on Raspberry Pi 3 and Pi Zero is not stated; please specify whether this is due to ARM architecture, memory limits, or library dependencies.
- [§3.4] The embedding pipeline mentions data augmentation (random cropping, horizontal flipping, brightness adjustments), but the paper does not state whether augmented samples were added to the ANN index or used only for training; this affects the interpretation of the dataset size.
- [§4.1] Software versions are not reported; please list FAISS and DiskANN versions, the operating system, and the CPU/GPU driver versions used on each device to enable reproducibility.
- [§4.2, Fig. 2] The figures are referenced as displaying comparisons, but the text does not specify the exact operating point (e.g., efSearch or nprobe) used for each curve; please describe the swept parameters and the axes in the captions.
- [§3.1] The accuracy metric is a classification accuracy based on neighbor labels, not recall@K of the ANN index; this choice is reasonable for the application, but it should be stated explicitly that the metric conflates embedding quality with ANN accuracy.
- [n/a] No statement is provided on code or data availability; please include one to allow the benchmark to be reproduced.
Circularity Check
No significant circularity: this is a measured benchmark comparison, not a derivation whose outputs are forced by its inputs.
full rationale
The paper is an experimental evaluation. It constructs indices using FAISS and DiskANN, optimizes parameters on a server, transfers the indices to edge devices, and reports measured QPS, accuracy, insertion/deletion latency, and power consumption. The central claims—that algorithm performance patterns differ between edge devices and servers, and that cheaper devices can match more expensive ones—are direct observations of these measurements, not quantities derived from fitted parameters. The statement in Section 4.2 that 'Construction and parameter optimization were conducted on a server' and that 'PQ was applied across all algorithms' identifies a potential confound for the device-comparison claim: a single server-tuned configuration may not be equally appropriate across devices with very different memory and cache characteristics. However, that is a validity or robustness concern, not circular reasoning: the paper does not predict edge-device performance from the fitted parameters, and no equation or construction reduces a reported result to its own input. The accuracy measure is a standard k-NN classification accuracy and is not defined in terms of the algorithms' outputs in a way that would force the comparison. Citations to HNSW, DiskANN, FAISS, and related work are standard external references, not self-citations carrying the argument. In summary, no pattern from self-definitional through renaming-known-result applies, and the benchmark is self-contained as a measurement study.
Assumptions & free parameters
free parameters (5)
- HNSW hyperparameters (M, efConstruction, efSearch)
- IVF hyperparameters (nlist, nprobe)
- LSH hyperparameters (number of hash tables, hash length)
- PQ codebook sizes and subspace counts
- PSL pseudo-label score threshold =
0.5
assumptions (4)
- domain assumption Pseudo-labels generated by Co-DETR are treated as ground-truth class labels for accuracy evaluation.
- domain assumption K-NN label voting accuracy is a valid proxy for approximate nearest neighbor search quality.
- domain assumption ResNet50 embeddings trained on ImageNet are suitable for representing the traffic camera images for nearest neighbor search.
- domain assumption Server-optimized hyperparameters transfer optimally to all edge devices without adjustment.
Cite this review
Pith. "Pith review of Experimental comparison of graph-based approximate nearest neighbor search algorithms on edge devices." pith.science (2026). https://pith.science/paper/4IS7HLSN
@misc{pith2026241114006,
author = {Pith},
title = {Pith review of: Experimental comparison of graph-based approximate nearest neighbor search algorithms on edge devices},
year = {2026},
howpublished = {\url{https://pith.science/paper/4IS7HLSN}},
note = {Machine review of arXiv:2411.14006}
}
read the original abstract
In this paper, we present an experimental comparison of various graph-based approximate nearest neighbor (ANN) search algorithms deployed on edge devices for real-time nearest neighbor search applications, such as smart city infrastructure and autonomous vehicles. To the best of our knowledge, this specific comparative analysis has not been previously conducted. While existing research has explored graph-based ANN algorithms, it has often been limited to single-threaded implementations on standard commodity hardware. Our study leverages the full computational and storage capabilities of edge devices, incorporating additional metrics such as insertion and deletion latency of new vectors and power consumption. This comprehensive evaluation aims to provide valuable insights into the performance and suitability of these algorithms for edge-based real-time tracking systems enhanced by nearest-neighbor search algorithms.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
A. N. Papadopoulos, Y. Manolopoulos, Nearest Neighbor Search: A Database Perspective, Springer Science & Business Media, 2006
work page 2006
-
[3]
Z. Lulu, G. Guohua, L. Kang, H. Ajing, Images matching algorithm based on surf and fast approximate nearest neighbor search, Application Research of Computers 30 (2013) 921–923
work page 2013
- [4]
-
[5]
F. F. Xu, U. Alon, G. Neubig, Why do nearest neighbor language models work?, arXiv preprint arXiv:2301.02828 (2023). ArXiv:2301.02828 [cs]
work page Pith review arXiv 2023
-
[6]
N. Banerjee, R. C. Connolly, D. Lisin, J. Briggs, M. E. Munich, View management for lifelong visual maps, in: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2019, pp. 7871–7878
work page 2019
-
[7]
T. Liu, C. Rosenberg, H. A. Rowley, Clustering billions of images with large scale nearest neighbor search, in: 2007 IEEE Workshop on Applications of Computer Vision (WACV ’07), IEEE, 2007, pp. 28–28
work page 2007
-
[8]
P. Li, W. Zhao, C. Wang, Q. Xia, A. Wu, L. Peng, Practice with graph-based ann algorithms on sparse data: Chi-square two-tower model, hnsw, sign cauchy projections, arXiv preprint arXiv:2306.07607 (2023). ArXiv:2306.07607 [cs, stat]
work page Pith review arXiv 2023
Show all 26 references
-
[9]
Terasawa, Y
K. Terasawa, Y. Tanaka, Spherical lsh for approximate nearest neighbor search on unit hypersphere, in: Workshop on Algorithms and Data Structures, Springer, 2007, pp. 27–38
2007
-
[10]
Arora, S
A. Arora, S. Sinha, P. Kumar, A. Bhattacharya, Hd-index: Pushing the scalability- accuracy boundary for approximate knn search in high-dimensional spaces, arXiv preprint arXiv:1804.06829 (2018)
2018 arXiv
-
[11]
Zhang, Y
M. Zhang, Y. He, Grip: Multi-store capacity-optimized high-performance nearest neighbor search for vector search engine, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019, pp. 1673–1682
2019
-
[12]
C. Fu, C. Xiang, C. Wang, D. Cai, Fast approximate nearest neighbor search with the navigating spreading-out graph, Proceedings of the VLDB Endowment 12 (2019) 461–474
2019
-
[13]
Y. A. Malkov, D. A. Yashunin, Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs, IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (2018) 824–836
2018
-
[14]
Subramanya, Suhas, et al., Diskann: Fast accurate billion-point nearest neighbor search on a single node, in: Advances in Neural Information Processing Systems 32, 2019
J. Subramanya, Suhas, et al., Diskann: Fast accurate billion-point nearest neighbor search on a single node, in: Advances in Neural Information Processing Systems 32, 2019
2019
-
[15]
Cöster, M
R. Cöster, M. Svensson, Inverted file search algorithms for collaborative filtering, in: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, 2002, pp. 246–252
2002
-
[16]
Dasgupta, R
A. Dasgupta, R. Kumar, T. Sarlós, Fast locality-sensitive hashing, in: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, 2011, pp. 1073–1081
2011
-
[17]
M. Wang, X. Xu, Q. Yue, Y. Wang, A comprehensive survey and experimental comparison of graph-based approximate nearest neighbor search, arXiv preprint arXiv:2101.12631 (2021)
2021 arXiv
-
[18]
Jiang, P
X. Jiang, P. Hu, Y. Li, C. Yuan, I. Masood, H. Jelodar, M. Rabbani, Y. Wang, A survey of real-time approximate nearest neighbor query over streaming data for fog computing, Journal of Parallel and Distributed Computing 116 (2018) 50–62
2018
-
[19]
Johnson, M
J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with gpus, IEEE Transactions on Big Data 7 (2019) 535–547
2019
-
[20]
Z. Zong, G. Song, Y. Liu, Detrs with collaborative hybrid assignments training, in: Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 6748–6758
2023
-
[21]
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer, 2014, pp...
2014
-
[22]
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255
2009
-
[23]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Pro- ceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
2016
-
[24]
Nguyen, D
H.-H. Nguyen, D. N.-N. Tran, L. H. Pham, J. W. Jeon, Optimizing monocular driving assistance for real-time processing on jetson agx xavier, IEEE Access (2024)
2024
-
[25]
P. Pace, G. Aloi, R. Gravina, G. Caliciuri, G. Fortino, A. Liotta, An edge-based architecture to support efficient applications for healthcare industry 4.0, IEEE Transactions on Industrial Informatics 15 (2018) 481–489
2018
-
[26]
Kavalionak, C
H. Kavalionak, C. Gennaro, G. Amato, C. Vairo, C. Perciante, C. Meghini, F. Falchi, Dis- tributed video surveillance using smart cameras, Journal of Grid Computing 17 (2019) 59–77
2019
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.