REVIEW 4 major objections 5 minor 23 references
Fast and Scalable Gene Embedding Search: A Comparative Study of FAISS and ScaNN
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read For short microbial gene fragments, searching learned DNA embeddings with approximate-nearest-neighbor libraries is faster and more accurate than alignment-based search, and FAISS outperforms ScaNN on both counts.
desk verdict Useful engineering benchmark with a real soft spot: the top-1 accuracy metric is underdefined and the FAISS-vs-ScaNN gap is not demonstrably apples-to-apples. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the retrieval pipeline: 400 bp gene fragments are encoded into 1,024-dimensional embeddings by the MetaBERTA-BigBird transformer model, and those embeddings are indexed by the two approximate-nearest-neighbor libraries so that each query's nearest database vector can be retrieved quickly. The quality measure is top-1 accuracy, defined by labeling each query with the taxonomic label of the nearest database vector. The library comparison is carried by sweeping index configurations: FAISS's Flat, IVF, PCA, OPQ, and PQ variants against ScaNN's brute-force, asymmetric-hashing, partitioning, and reordering variants. In these sweeps, PCA-preserved Flat indexes keep embedding fidelity, while quantization-heavy indexes trade accuracy for speed and memory.
What would settle it
Recompute top-1 accuracy separately for the 7,000 in-domain and 7,000 out-of-domain queries, and verify whether any database sequence shares a phylum label with each out-of-domain query; if out-of-domain accuracy is near zero, the reported 31-36% accuracy mostly measures in-domain retrieval, not novelty detection.
Extended reading notes
Core claim
The paper's central claim is that embedding-based retrieval frameworks like FAISS and ScaNN consistently outperform alignment-based approaches for short-fragment similarity search in terms of both speed and accuracy, and that FAISS is the stronger of the two libraries. The authors report that FAISS with the PCA64,Flat configuration achieves the highest accuracy (0.362), while ScaNN's best configuration reaches 0.310 and MMseqs2 only 0.018; a FAISS variant with PCA plus IVF (PCAWR64,IVF4096,Flat) keeps accuracy at 0.327 while cutting search time to roughly 0.3 seconds, about 25 times faster than the most accurate configuration. They further find that FAISS's distance distributions give slightly better contrast between in-domain and out-of-domain queries, which they read as marginally better utility for novelty detection. The conclusion states that the results support the use of embedding-based retrieval frameworks like FAISS for scalable and biologically meaningful similarity search in large genomic datasets.
Load-bearing premise
The results rest on the assumption that top-1 nearest-neighbor taxonomic labels are a valid accuracy measure for both in-domain and out-of-domain queries, even though the paper never specifies the taxonomic level used, whether the database contains any representatives of the out-of-domain phyla, or how ties are resolved.
Editorial extensions
If this is right
- A practitioner searching short metagenomic fragments can expect embedding retrieval to return a taxonomically matching neighbor far more often than MMseqs2-style alignment (36% vs 1.8% top-1 accuracy on this benchmark).
- FAISS Flat or PCA-Flat indexes are the accuracy-first default; switching to a PCA-plus-IVF configuration trades a few accuracy points for a roughly 25x speedup, so library choice becomes a speed-quality dial.
- Aggressive product quantization costs real accuracy on gene embeddings, so memory savings should be weighed against reduced retrieval fidelity.
- The mild separation FAISS shows between in-domain and out-of-domain query distances suggests that distance thresholds could be used for novelty screening, though the separation is not clean.
- ScaNN's main tunable parameters (leaves, quantization thresholds) affect runtime far more than retrieval accuracy, meaning tuning effort is best spent on latency budgets.
Reading between the lines
- A separate accuracy report for in-domain and out-of-domain queries would clarify whether the headline 31-36% figures mostly come from in-domain retrieval; if out-of-domain queries lack same-phylum neighbors in the database, their accuracy contribution is undefined, and the novelty-detection claim rests on the distance-distribution plots rather than the top-1 metric.
- The paper's own numbers suggest the embedding model, not the ANN library, is the dominant factor in retrieval quality — both libraries land near 30-36% top-1 accuracy while alignment is at 2% — so FAISS's edge over ScaNN may be largely an effect of index choice.
- Because the paper notes all runs were on CPU, a GPU-based replication could change the speed ranking; ScaNN's anisotropic quantization is designed for massive-scale throughput, so the comparison may look different at billion scale.
- A testable extension is to evaluate recall at k against curated homology labels instead of taxonomy, since taxonomy is a coarse proxy for functional similarity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks two approximate nearest-neighbor (ANN) libraries, FAISS and ScaNN, for retrieval over 1,024-dimensional gene-fragment embeddings produced by the MetaBERTA-BigBird model. The search database is the 165,615-sequence Scorpio-Gene-Taxa test set, and the query sets consist of 7,000 in-domain fragments and 7,000 fragments from phyla absent from training. The authors sweep FAISS index/preprocessing combinations and ScaNN scoring/partition/reordering settings, report top-1 accuracy, indexing/search time and memory in Table 1, and compare both libraries against MMseqs2. They conclude that embedding-based retrieval, especially FAISS, outperforms alignment-based search on short fragments and that FAISS generally beats ScaNN in accuracy, speed, and novelty-detection utility.
Significance. If the comparison were valid, the paper would provide useful practical guidance for choosing ANN infrastructure for genomic embedding search, and the head-to-head FAISS/ScaNN tuning sweeps would be a useful reference. The authors deserve credit for using a biology-relevant embedding model and dataset, for reporting memory and runtime alongside accuracy, for a systematic parameter sweep including SHAP analysis, and for explicitly stating the CPU-only setting. However, the evaluation's central accuracy measure is never defined, the out-of-domain query design can force zero accuracy by construction, and the best configurations are selected on the same query sets used for the final numbers. These issues make the reported accuracy gaps, including the headline 0.362 vs 0.018 comparison, not currently interpretable.
major comments (4)
- [§3 (Materials and Methods), §4 (Results), Table 1] The top-1 accuracy used throughout Figures 2–3 and Table 1 is never defined. Please state the exact scoring rule: what label is transferred from the retrieved database sequence to the query (taxonomic rank, genus, phylum), how ties among equal distances are broken, and whether accuracy is averaged over the combined 14,000 queries or reported separately per query set. This matters because the out-domain queries come from phyla absent from the training set; if the 165,615-sequence test database contains no representatives of those phyla, then every out-domain query has zero (or undefined) label-transfer accuracy by construction. In that case the reported 0.31–0.36 values mostly reflect the in-domain half, and the embedding-vs-MMseqs2 gap is not a valid measure of retrieval quality. Please report per-set accuracies and verify label availability.
- [§4 (Similarity Search Performance Comparison), Table 1] The figure of 0.362 for FAISS PCA64,Flat exceeds the 0.33 reported for ScaNN brute-force full-precision search, which the text describes as an accuracy ceiling. If the FAISS run uses 64-dimensional PCA-reduced vectors, then the ScaNN brute-force run (presumably on full 1,024-dimensional vectors) is not the correct ceiling, and the two numbers are not comparable. Please specify the exact vector representation used by each configuration and either match representations or justify why the comparison is meaningful despite the dimensionality difference. This is essential for the FAISS-vs-ScaNN accuracy comparison.
- [§4 (Results), Table 1] The best-performing FAISS and ScaNN configurations were selected from parameter sweeps evaluated on the same 14,000 queries used for the final reported accuracy. This selection-on-the-test-set procedure inflates the apparent advantage of the chosen configurations and makes the 0.362 vs 0.310 gap between FAISS and ScaNN difficult to interpret. Please use a separate validation set for configuration selection, or report the full sweep with multiple query replicates and variability estimates. The ScaNN figure shows error bars but FAISS results do not, and Table 1 reports single numbers without variance.
- [§3 (Materials and Methods), Table 1] The MMseqs2 baseline is not described in enough detail to support the claim that alignment-based methods are inferior. Please report the MMseqs2 database construction, search mode, sensitivity parameter, e-value threshold, number of threads, and how the top-1 hit was converted into an accuracy score. Without these details, the 1.8% accuracy could reflect an unfavorable or misconfigured baseline rather than a fundamental limitation of alignment. At minimum, state whether the MMseqs2 search was nucleotide-nucleotide and whether the same label-transfer rule was applied.
minor comments (5)
- [§4 (Similarity Search Performance Comparison)] The text introduces FAISS configurations PCA64,IMI2x10,Flat and IMI2x10,Flat with runtimes of 0.12 s and 0.76 s, but these names do not appear in the Methods or in Figure 2. Please either add them to the configuration list or remove them from the text.
- [§4 (FAISS Parameter Sensitivity), Figure 2] The text first states that PCAW64,Flat and PCAWR64,Flat reach 0.362, then later attributes the highest accuracy to PCA64,Flat; Table 1 repeats 'PCA64,Flat'. Please reconcile the naming and report which configuration the headline number belongs to.
- [Figure 3 and Table 1] ScaNN's Table 1 accuracy (0.310) does not match the brute-force value (0.33) highlighted in Figure 3. Please state explicitly which ScaNN configuration Table 1 reports and why it was chosen.
- [§4, last paragraph] The claim that FAISS provides 'more reliable distance-based scoring and novelty inference' is based only on visual inspection of overlapping KDE plots. Please add a quantitative separation measure (e.g., AUROC or Wasserstein distance) if this claim is retained.
- [References and figure captions] There are minor typographical inconsistencies: 'The F AISS Library' in the references and 'Faiss' in the Figure 2 caption should be 'FAISS', and parameter names such as 'Leaves Search' and 'Leaves_Search' are used interchangeably.
Circularity Check
No significant circularity: the FAISS/ScaNN/MMseqs2 comparison is an external benchmark; self-built embeddings and an underspecified top-1 metric are validity concerns, not circular reductions.
full rationale
This is an empirical benchmark, not a derivation, so most circularity patterns do not apply. The central comparison is among external libraries (FAISS, ScaNN, MMseqs2) on fixed query and database embeddings; none of the reported accuracies or runtimes is an algebraic consequence of the definitions. The dataset (Scorpio-Gene-Taxa) and embedding model (MetaBERTA-BigBird) are cited from the authors' own prior work, but those citations supply benchmark inputs rather than proof of the library ranking, and both ANN libraries are evaluated on the same embeddings, so the self-citation does not by itself force the FAISS-over-ScaNN result. A legitimate validity concern is that the best configurations were selected from a parameter sweep using the same query sets on which final accuracy is reported, and the top-1 'classification accuracy' is never defined (no taxonomic rank, tie rule, or statement about whether out-of-domain queries have same-label database neighbors); these issues affect whether 0.362 vs 0.310 vs 0.018 is a meaningful accuracy comparison, but they are not circular reductions. The unexplained result that FAISS PCA64,Flat (0.362) exceeds ScaNN brute-force (0.33) similarly points to a comparability problem, not to a definitional equivalence. No circular step can be pinned to a specific equation or fitted parameter; score 2 reflects the presence of self-citations in the benchmark inputs while the library comparison itself remains externally grounded.
Assumptions & free parameters
free parameters (4)
- FAISS PCA dimensions (PCA64/PCAW64/PCAWR64) =
64
- FAISS IVF cells (IVF4096) =
4096
- FAISS nprobe range =
100 to 1000, exact per-config values not reported
- ScaNN Leaves / Leaves_Search / Q_Thresh / Dim/Block / Train_Iter =
not reported
assumptions (4)
- domain assumption The Scorpio-Gene-Taxa (short fragment) test set has correct taxonomic labels at a consistent, relevant level.
- domain assumption MetaBERTA-BigBird embeddings preserve the biological similarity needed for retrieval and novelty detection.
- domain assumption The taxonomic label of the nearest neighbor is an adequate proxy for retrieval correctness.
- domain assumption FAISS and ScaNN are used in a representative way, and MMseqs2's nucleotide-only configuration is a fair baseline.
Cite this review
Pith. "Pith review of Fast and Scalable Gene Embedding Search: A Comparative Study of FAISS and ScaNN." pith.science (2026). https://pith.science/paper/VSMXC6Q3
@misc{pith2026250716978,
author = {Pith},
title = {Pith review of: Fast and Scalable Gene Embedding Search: A Comparative Study of FAISS and ScaNN},
year = {2026},
howpublished = {\url{https://pith.science/paper/VSMXC6Q3}},
note = {Machine review of arXiv:2507.16978}
}
read the original abstract
The exponential growth of DNA sequencing data has outpaced traditional heuristic-based methods, which struggle to scale effectively. Efficient computational approaches are urgently needed to support large-scale similarity search, a foundational task in bioinformatics for detecting homology, functional similarity, and novelty among genomic and proteomic sequences. Although tools like BLAST have been widely used and remain effective in many scenarios, they suffer from limitations such as high computational cost and poor performance on divergent sequences. In this work, we explore embedding-based similarity search methods that learn latent representations capturing deeper structural and functional patterns beyond raw sequence alignment. We systematically evaluate two state-of-the-art vector search libraries, FAISS and ScaNN, on biologically meaningful gene embeddings. Unlike prior studies, our analysis focuses on bioinformatics-specific embeddings and benchmarks their utility for detecting novel sequences, including those from uncharacterized taxa or genes lacking known homologs. Our results highlight both computational advantages (in memory and runtime efficiency) and improved retrieval quality, offering a promising alternative to traditional alignment-heavy tools.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
F., Gish, W., Miller, W., Myers, E
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., and Lipman, D. J. Basic local alignment search tool. Journal of molecular biology, 215 0 (3): 0 403--410, 1990
work page 1990
-
[3]
B., Lespiau, J.-B., Damoc, B., Clark, A., et al
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp.\ 2206--2240. PMLR, 2022
work page 2022
-
[4]
H., Oteri, F., Dallago, C., Trop, E., de Almeida, B
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, pp.\ 1--11, 2024
work page 2024
-
[5]
E., Lomeli, M., Hosseini, L., and Jegou, H
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazare, P. E., Lomeli, M., Hosseini, L., and Jegou, H. The FAISS Library, 2024
work page 2024
-
[6]
FitzGerald, N., Botha, J. A., Gillick, D., Bikel, D. M., Kwiatkowski, T., and McCallum, A. Moleman: Mention-only linking of entities with a mention annotation network, 2022
work page 2022
-
[7]
Accelerating large-scale inference with anisotropic vector quantization
Guo, R., Sun, P., Lindgren, E., Geng, Q., Simcha, D., Chern, F., and Kumar, S. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning, pp.\ 3887--3896. PMLR, 2020
work page 2020
-
[8]
Ji, Y., Zhou, Z., Liu, H., and Davuluri, R. V. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37 0 (15): 0 2112--2120, 2021
work page 2021
Show all 23 references
-
[9]
Billion-scale similarity search with gpus
Johnson, J., Douze, M., and J \'e gou, H. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7 0 (3): 0 535--547, 2019
2019
-
[10]
Product quantization for nearest neighbor search
Jégou, H., Douze, M., and Schmid, C. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33 0 (1): 0 117--128, 2011. doi:10.1109/TPAMI.2010.57
2011 doi
-
[11]
Correlation between the gut microbiome and neurodegenerative diseases: a review of metagenomics evidence
Liu, X., Liu, Y., Liu, J., Zhang, H., Shan, C., Guo, Y., Gong, X., Cui, M., Li, X., and Tang, M. Correlation between the gut microbiome and neurodegenerative diseases: a review of metagenomics evidence. Neural Regeneration Research, 19 0 (4): 0 833--845, 2024
2024
-
[12]
Lundberg, S. M. and Lee, S.-I. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017
2017
-
[13]
P., Mulaudzi, T
Mathebela, P., Damane, B. P., Mulaudzi, T. V., Mkhize-Khwitshana, Z. L., Gaudji, G. R., and Dlamini, Z. Influence of the microbiome metagenomics and epigenomics on gastric cancer. International Journal of Molecular Sciences, 23 0 (22): 0 13750, 2022
2022
-
[14]
Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution
Nguyen, E., Poli, M., Faizi, M., Thomas, A., Wornow, M., Birch-Sykes, C., Massaroli, S., Patel, A., Rabideau, C., Bengio, Y., et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024
2024
-
[15]
Leveraging large language models for metagenomic analysis
Refahi, M., Sokhansanj, B., and Rosen, G. Leveraging large language models for metagenomic analysis. In 2023 IEEE signal processing in medicine and biology symposium (SPMB), pp.\ 1--6. IEEE, 2023
2023
-
[16]
A., Mell, J
Refahi, M., Sokhansanj, B. A., Mell, J. C., Brown, J. R., Yoo, H., Hearne, G., and Rosen, G. L. Enhancing nucleotide sequence representations in genomic analysis with contrastive optimization. Communications biology, 8 0 (1): 0 517, 2025
2025
-
[17]
and Thompson, G
Safaei, Z. and Thompson, G. L. Caspase-dependent cell death and hdac4 translocation following microsecond pulsed electric field ( spef) exposure in mcf-7 breast cancer cells. Bioelectromagnetics, 46 0 (4): 0 e70009, 2025
2025
-
[18]
Caduceus: Bi-directional equivariant long-range dna sequence modeling
Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov, V. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024
2024 arXiv
-
[19]
Nearest neighbor search on embeddings rapidly identifies distant protein relations
Sch \"u tze, K., Heinzinger, M., Steinegger, M., and Rost, B. Nearest neighbor search on embeddings rapidly identifies distant protein relations. Frontiers in Bioinformatics, 2: 0 1033775, 2022
2022
-
[20]
and S \"o ding, J
Steinegger, M. and S \"o ding, J. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35 0 (11): 0 1026--1028, 2017
2017
-
[21]
ClusTCR: a python interface for rapid clustering of large sets of CDR3 sequences with unknown antigen specificity
Valkiers, S., Van Houcke, M., Laukens, K., and Meysman, P. ClusTCR: a python interface for rapid clustering of large sets of CDR3 sequences with unknown antigen specificity . Bioinformatics, 37 0 (24): 0 4865--4867, 06 2021. ISSN 1367-4803. doi:10.1093/bioinformatics/btab446. ...
2021 doi
-
[22]
Approximate nearest neighbor negative contrastive learning for dense text retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., and Overwijk, A. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808, 2020
2007 arXiv
-
[23]
Dnabert-2: Efficient foundation model and benchmark for multi-species genome
Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R., and Liu, H. Dnabert-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006, 2023
2023 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.