Pith. sign in

REVIEW 4 major objections 7 minor 26 references

Experimental comparison of graph-based approximate nearest neighbor search algorithms on edge devices

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read This paper claims that approximate nearest neighbor search performance on edge devices does not follow server rankings: cheaper, less powerful devices can match or beat costly ones, so practitioners should benchmark on the actual…

desk verdict Fresh benchmark data on ANN search across seven edge devices, but fixed server-tuned parameters and missing methodology limits the strength of the cross-device conclusions. read the letter →

arxiv 2411.14006 v1 pith:4IS7HLSN submitted 2024-11-21 cs.DS cs.ARcs.CVcs.DCcs.PF

classification cs.DScs.ARcs.CVcs.DCcs.PF
keywords approximatenearestneighborsearchgraph-basedindexedgedevicesHNSWproductquantizationbenchmarkNVIDIAJetsonRaspberryPi
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports an experimental comparison of five graph-based approximate nearest neighbor (ANN) search implementations on seven edge devices, using real street-camera embeddings. Its central claim is that performance rankings measured on a server do not carry over to edge hardware: cheaper, less powerful devices can outperform more expensive ones for the same algorithm, and a mid-priced Raspberry Pi 4 can match or beat several NVIDIA Jetson boards in CPU search throughput. The authors argue that practitioners should therefore evaluate ANN algorithms on the actual edge hardware they plan to deploy, with attention to insertion and deletion costs and power consumption, not just query speed. The study also shows that DiskANN supports true vector deletion but its uncompressed index was too large to fit on the edge devices tested.

What carries the argument

The evaluation is carried by a fixed benchmark protocol: indices for FAISS-based IVF, LSH, HNSW-PQ, and PQ, plus DiskANN's Vamana graph, are constructed once on a 28-core server with 2 TB RAM and then copied to the edge devices, where query throughput, insertion and deletion latency, and power draw are measured. The central objects compared are the graph-based index structures themselves, HNSW's hierarchical small-world graph and Vamana's navigable graph, alongside inverted-file and hashing baselines. Because hyperparameters are tuned only on the server, the protocol isolates device behavior under identical index configurations rather than per-device optimal tuning.

What would settle it

Re-run the same benchmark with hyperparameters optimized separately on each device; if the Raspberry Pi 4's threefold advantage over the Jetson Nano disappears or reverses when each device uses its own tuned settings, the paper's central device-ranking claim would be undercut.

Watch

Extended reading notes

Core claim

On its street-camera dataset of about one million 2048-dimensional embeddings, the paper finds that HNSW-PQ generally outperforms IVF-PQ on edge devices even though the reverse pattern appears on a server CPU, that the Raspberry Pi 4 achieves over three times the CPU queries-per-second of the Jetson Nano at about a quarter of the price, that the Jetson Xavier AGX outperforms the newer Jetson Orin for HNSW-PQ despite fewer cores, and that GPU acceleration triples throughput while roughly doubling power consumption. These results support the claim that device-specific benchmarking, including dynamic insert and delete pipelines, is needed before choosing an ANN algorithm for edge deployment.

Load-bearing premise

The results assume that hyperparameters optimized once on a large server remain near-optimal for every edge device, so the observed device rankings are not artifacts of a single fixed configuration.

Editorial extensions

If this is right

  • Practitioners should treat server-side ANN benchmarks as only a rough guide and measure query throughput, update latency, and power on the target edge device before deployment.
  • Cost-effective devices such as the Raspberry Pi 4 are viable platforms for CPU-based real-time ANN search, potentially lowering the hardware cost of smart-city and tracking systems.
  • Rankings among algorithms can invert between server and edge: HNSW-PQ leads on edge devices while IVF-PQ can lead on the server at low accuracy, so algorithm choice must be hardware-specific.
  • Dynamic workloads need explicit testing of insertion and deletion paths; DiskANN offers true deletion but its uncompressed index did not fit on the tested edge devices, limiting its use there.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely explanation the paper does not test is that memory-hierarchy details, such as cache size, matter more than core count for these graph traversals; a controlled study varying only cache size could confirm this.
  • For battery-powered deployments, queries-per-second per watt may be a better objective than raw QPS, which would strengthen the case for low-power boards even further.
  • The fixed-server hyperparameter protocol may systematically disadvantage algorithms whose optimal parameters depend on hardware; tuning per device could change both the algorithm ranking and the device ranking.
  • The inability to run DiskANN on edge devices points to an open problem: compressed on-disk graph indexes with true deletion that fit within edge memory budgets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper reports an experimental comparison of approximate nearest neighbor search (ANNS) algorithms on edge devices for real-time smart-city object tracking. Using street-camera data from Helsinki (Section 3.4), the authors extract ResNet50 embeddings from detected bounding boxes and evaluate FAISS implementations of IVF, LSH, HNSW-PQ, and PQ, plus DiskANN/Vamana, on seven edge devices (Jetson Nano, TX2, Xavier AGX, Orin, Raspberry Pi 4, Pi 3, Pi Zero) and a server. The evaluation metrics are QPS, classification accuracy, insertion/deletion throughput (only for DiskANN), and power consumption. The central claim (Section 1) is that algorithm performance patterns differ between edge devices and servers, that stronger edge devices do not always yield faster execution, and that inexpensive devices can match more expensive counterparts. The paper concludes with suggestions for future optimization.

Significance. The paper addresses a practical and understudied question: whether ANN benchmarks obtained on server hardware transfer to edge devices. Its strengths are the breadth of real-world hardware (seven devices), the use of real street-camera data, the inclusion of CPU and GPU execution paths, and the explicit attention to cost-effectiveness. If the central claims are supported after the missing analysis is supplied, the paper would provide useful engineering guidance and a caution against extrapolating server benchmarks to edge deployments. The contribution is incremental over existing ANN survey and benchmark literature, but the edge-device focus is a useful addition. The main limitation is that the descriptive conclusions currently rest on unreported configuration details, a dataset-size inconsistency, and single-point measurements without variance information.

major comments (4)
  1. [§4.2, Figs. 2–3] The central claim of device-dependent rankings rests on measurements taken with one fixed configuration: "Construction and parameter optimization were conducted on a server" and "PQ was applied across all algorithms to ensure compatibility with memory constraints of edge devices." The paper does not report the tuned hyperparameter values (e.g., HNSW M/efConstruction/efSearch, IVF nlist/nprobe) nor the PQ codebook settings, and it provides no sensitivity analysis or per-device tuning. Because the devices differ by orders of magnitude in memory (512 MB to 64 GB) and cache architecture, the observed gaps (e.g., Pi 4 three times faster than Jetson Nano, Xavier 1.5 times faster than Orin) could be artifacts of a single server-tuned and compression setting rather than intrinsic algorithm or device behavior. Please report the settings and add either per-device parameter optimization or a robustness check across a few parameter choices.
  2. [§3.4 vs. Table 3] Section 3.4 states that the preprocessing pipeline produced 161,805 valid bounding boxes, split 85% for training and 15% for evaluation, and gives no indication of augmentation to a larger size; Table 3, however, reports construction times and index sizes for "~1M vectors." This is roughly a factor-of-six discrepancy that affects the interpretation of all scalability claims and the index-size numbers. Please clarify the actual number of vectors used in each index and explain any augmentation or duplication step that produces ~1M vectors.
  3. [§4.2, Figs. 2–3] None of the QPS/accuracy figures or the text reports the number of repeated runs, error bars, confidence intervals, or per-device variance. Quantitative statements such as "it performed over three times better than the Jetson Nano" and "the Jetson Xavier also outperformed the Jetson Orin by 1.5 times" are presented without supporting tables of exact values or measures of spread. Since the goal is to compare algorithms and devices, the absence of variance information leaves the reader unable to judge whether the reported rankings are statistically meaningful.
  4. [§3.2, §4.2] The abstract and Section 3.2 promise additional metrics of insertion/deletion latency and power consumption, but the experiment does not deliver a comparative evaluation of these metrics. Insertion/deletion results are shown only for DiskANN in Figure 4, and there is no comparison of other algorithms' insertion/deletion throughput. Power consumption is only listed as hardware specifications in Tables 1 and 2 and asserted in one sentence ("GPU-based inference consumes double the power of CPU-based inference") without a measurement methodology or results. Please either add the missing measurements and figures or explicitly qualify the claims to the metrics actually measured.
minor comments (7)
  1. [§1, §2.2] The title and Section 2 describe the study as comparing graph-based ANNS algorithms, but the evaluated set includes LSH and PQ, which are not graph-based; please either adjust the terminology or explain why non-graph methods are included.
  2. [§4.2] The reason FAISS was not supported on Raspberry Pi 3 and Pi Zero is not stated; please specify whether this is due to ARM architecture, memory limits, or library dependencies.
  3. [§3.4] The embedding pipeline mentions data augmentation (random cropping, horizontal flipping, brightness adjustments), but the paper does not state whether augmented samples were added to the ANN index or used only for training; this affects the interpretation of the dataset size.
  4. [§4.1] Software versions are not reported; please list FAISS and DiskANN versions, the operating system, and the CPU/GPU driver versions used on each device to enable reproducibility.
  5. [§4.2, Fig. 2] The figures are referenced as displaying comparisons, but the text does not specify the exact operating point (e.g., efSearch or nprobe) used for each curve; please describe the swept parameters and the axes in the captions.
  6. [§3.1] The accuracy metric is a classification accuracy based on neighbor labels, not recall@K of the ANN index; this choice is reasonable for the application, but it should be stated explicitly that the metric conflates embedding quality with ANN accuracy.
  7. [n/a] No statement is provided on code or data availability; please include one to allow the benchmark to be reproduced.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is a measured benchmark comparison, not a derivation whose outputs are forced by its inputs.

full rationale

The paper is an experimental evaluation. It constructs indices using FAISS and DiskANN, optimizes parameters on a server, transfers the indices to edge devices, and reports measured QPS, accuracy, insertion/deletion latency, and power consumption. The central claims—that algorithm performance patterns differ between edge devices and servers, and that cheaper devices can match more expensive ones—are direct observations of these measurements, not quantities derived from fitted parameters. The statement in Section 4.2 that 'Construction and parameter optimization were conducted on a server' and that 'PQ was applied across all algorithms' identifies a potential confound for the device-comparison claim: a single server-tuned configuration may not be equally appropriate across devices with very different memory and cache characteristics. However, that is a validity or robustness concern, not circular reasoning: the paper does not predict edge-device performance from the fitted parameters, and no equation or construction reduces a reported result to its own input. The accuracy measure is a standard k-NN classification accuracy and is not defined in terms of the algorithms' outputs in a way that would force the comparison. Citations to HNSW, DiskANN, FAISS, and related work are standard external references, not self-citations carrying the argument. In summary, no pattern from self-definitional through renaming-known-result applies, and the benchmark is self-contained as a measurement study.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several unverified assumptions: pseudo-labels are treated as ground truth, classification accuracy is used as a proxy for ANN quality, and server-tuned hyperparameters are assumed to be appropriate on every edge device. None of these assumptions are validated with sensitivity analysis, and the tuned parameter values are not disclosed.

free parameters (5)
  • HNSW hyperparameters (M, efConstruction, efSearch)
    Tuned on the server to maximize accuracy, but exact values are not reported, and the tuned settings are transferred unchanged to all edge devices.
  • IVF hyperparameters (nlist, nprobe)
    Tuned on the server, not reported, and reused on edge devices without device-specific adjustment.
  • LSH hyperparameters (number of hash tables, hash length)
    Tuned on the server, not reported, and reused on edge devices.
  • PQ codebook sizes and subspace counts
    Part of the FAISS index configuration, tuned on the server, not reported.
  • PSL pseudo-label score threshold = 0.5
    The threshold for filtering Co-DETR bounding boxes is chosen by hand in Section 3.4 and affects the composition of the dataset.
assumptions (4)
  • domain assumption Pseudo-labels generated by Co-DETR are treated as ground-truth class labels for accuracy evaluation.
    Section 3.4 uses these pseudo-labels as ground truth; errors in the detection model propagate into the accuracy measurements.
  • domain assumption K-NN label voting accuracy is a valid proxy for approximate nearest neighbor search quality.
    Section 3.1 defines accuracy as classification accuracy based on retrieved neighbor labels, not recall@k or distance-based metrics, so the reported accuracy may not reflect true ANN retrieval quality.
  • domain assumption ResNet50 embeddings trained on ImageNet are suitable for representing the traffic camera images for nearest neighbor search.
    Section 3.4 uses a pre-trained ResNet50 without fine-tuning on the target domain; domain shift could affect the usefulness of the embeddings.
  • domain assumption Server-optimized hyperparameters transfer optimally to all edge devices without adjustment.
    Section 4.2 states that parameter optimization was done on the server; the paper assumes this setup is fair for all devices, which is not validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Experimental comparison of graph-based approximate nearest neighbor search algorithms on edge devices." pith.science (2026). https://pith.science/paper/4IS7HLSN

@misc{pith2026241114006,
  author       = {Pith},
  title        = {Pith review of: Experimental comparison of graph-based approximate nearest neighbor search algorithms on edge devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4IS7HLSN}},
  note         = {Machine review of arXiv:2411.14006}
}
read the original abstract

In this paper, we present an experimental comparison of various graph-based approximate nearest neighbor (ANN) search algorithms deployed on edge devices for real-time nearest neighbor search applications, such as smart city infrastructure and autonomous vehicles. To the best of our knowledge, this specific comparative analysis has not been previously conducted. While existing research has explored graph-based ANN algorithms, it has often been limited to single-threaded implementations on standard commodity hardware. Our study leverages the full computational and storage capabilities of edge devices, incorporating additional metrics such as insertion and deletion latency of new vectors and power consumption. This comprehensive evaluation aims to provide valuable insights into the performance and suitability of these algorithms for edge-based real-time tracking systems enhanced by nearest-neighbor search algorithms.

Figures

Figures reproduced from arXiv: 2411.14006 by the authors.

Figure 1
Figure 1. Server comparison using CPU between QPS and accuracy for each algorithm. Notably, DiskANN, which does not use any data compression and operates on an index built from the original data vectors, shows competitive performance relative to other approximate nearest neighbor search (ANNS) methods that utilize Product Quantization (PQ) for data compression. Furthermore, it is evident that FAISS algorithms, even with compr… view at source ↗
Figure 2
Figure 2. Edge device comparison using CPU [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. CPU(left) and GPU(right) performance comparison across devices devices, so we evaluated it on our server. DiskANN allows true vector deletion, unlike FAISS, which uses "soft" deletion—vectors are marked as deleted but remain in the index structure. We developed a test pipeline for insertion and deletion with DiskANN, simulating real-world data scenarios [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: DiskANN pipeline performance capabilities of DiskANN, particularly its support for true deletion of vectors. This feature proves invaluable for maintaining model relevance for potential shifts in data distribution over time and supports continuous on-device learning. L…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 20 canonical work pages

  1. [1]

    Sarwar, G

    B. Sarwar, G. Karypis, J. Konstan, J. Riedl, Itembased collaborative filtering recommendation algorithms, in: Proceedings of the 10th International Conference on World Wide Web, 2001, pp. 285–295

  2. [2]

    A. N. Papadopoulos, Y. Manolopoulos, Nearest Neighbor Search: A Database Perspective, Springer Science & Business Media, 2006

  3. [3]

    Z. Lulu, G. Guohua, L. Kang, H. Ajing, Images matching algorithm based on surf and fast approximate nearest neighbor search, Application Research of Computers 30 (2013) 921–923

  4. [4]

    Jegou, M

    H. Jegou, M. Douze, C. Schmid, Product quantization for nearest neighbor search, IEEE transactions on pattern analysis and machine intelligence 33 (2010) 117–128

  5. [5]

    F. F. Xu, U. Alon, G. Neubig, Why do nearest neighbor language models work?, arXiv preprint arXiv:2301.02828 (2023). ArXiv:2301.02828 [cs]

  6. [6]

    Banerjee, R

    N. Banerjee, R. C. Connolly, D. Lisin, J. Briggs, M. E. Munich, View management for lifelong visual maps, in: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2019, pp. 7871–7878

  7. [7]

    T. Liu, C. Rosenberg, H. A. Rowley, Clustering billions of images with large scale nearest neighbor search, in: 2007 IEEE Workshop on Applications of Computer Vision (WACV ’07), IEEE, 2007, pp. 28–28

  8. [8]

    P. Li, W. Zhao, C. Wang, Q. Xia, A. Wu, L. Peng, Practice with graph-based ann algorithms on sparse data: Chi-square two-tower model, hnsw, sign cauchy projections, arXiv preprint arXiv:2306.07607 (2023). ArXiv:2306.07607 [cs, stat]

Show all 26 references
  1. [9]

    Terasawa, Y

    K. Terasawa, Y. Tanaka, Spherical lsh for approximate nearest neighbor search on unit hypersphere, in: Workshop on Algorithms and Data Structures, Springer, 2007, pp. 27–38

  2. [10]

    Arora, S

    A. Arora, S. Sinha, P. Kumar, A. Bhattacharya, Hd-index: Pushing the scalability- accuracy boundary for approximate knn search in high-dimensional spaces, arXiv preprint arXiv:1804.06829 (2018)

  3. [11]

    Zhang, Y

    M. Zhang, Y. He, Grip: Multi-store capacity-optimized high-performance nearest neighbor search for vector search engine, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019, pp. 1673–1682

  4. [12]

    C. Fu, C. Xiang, C. Wang, D. Cai, Fast approximate nearest neighbor search with the navigating spreading-out graph, Proceedings of the VLDB Endowment 12 (2019) 461–474

  5. [13]

    Y. A. Malkov, D. A. Yashunin, Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs, IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (2018) 824–836

  6. [14]

    Subramanya, Suhas, et al., Diskann: Fast accurate billion-point nearest neighbor search on a single node, in: Advances in Neural Information Processing Systems 32, 2019

    J. Subramanya, Suhas, et al., Diskann: Fast accurate billion-point nearest neighbor search on a single node, in: Advances in Neural Information Processing Systems 32, 2019

  7. [15]

    Cöster, M

    R. Cöster, M. Svensson, Inverted file search algorithms for collaborative filtering, in: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, 2002, pp. 246–252

  8. [16]

    Dasgupta, R

    A. Dasgupta, R. Kumar, T. Sarlós, Fast locality-sensitive hashing, in: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, 2011, pp. 1073–1081

  9. [17]

    M. Wang, X. Xu, Q. Yue, Y. Wang, A comprehensive survey and experimental comparison of graph-based approximate nearest neighbor search, arXiv preprint arXiv:2101.12631 (2021)

  10. [18]

    Jiang, P

    X. Jiang, P. Hu, Y. Li, C. Yuan, I. Masood, H. Jelodar, M. Rabbani, Y. Wang, A survey of real-time approximate nearest neighbor query over streaming data for fog computing, Journal of Parallel and Distributed Computing 116 (2018) 50–62

  11. [19]

    Johnson, M

    J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with gpus, IEEE Transactions on Big Data 7 (2019) 535–547

  12. [20]

    Z. Zong, G. Song, Y. Liu, Detrs with collaborative hybrid assignments training, in: Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 6748–6758

  13. [21]

    T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer, 2014, pp...

  14. [22]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255

  15. [23]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Pro- ceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

  16. [24]

    Nguyen, D

    H.-H. Nguyen, D. N.-N. Tran, L. H. Pham, J. W. Jeon, Optimizing monocular driving assistance for real-time processing on jetson agx xavier, IEEE Access (2024)

  17. [25]

    P. Pace, G. Aloi, R. Gravina, G. Caliciuri, G. Fortino, A. Liotta, An edge-based architecture to support efficient applications for healthcare industry 4.0, IEEE Transactions on Industrial Informatics 15 (2018) 481–489

  18. [26]

    Kavalionak, C

    H. Kavalionak, C. Gennaro, G. Amato, C. Vairo, C. Perciante, C. Meghini, F. Falchi, Dis- tributed video surveillance using smart cameras, Journal of Grid Computing 17 (2019) 59–77

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.