TEngineDB-V: An OLAP-Native Vector Search System for Large-$k$ Workloads at Tencent

Anqi Liang; Fan Wu; Guangxu Cheng; Guoliang Li; Jijun Du; Kai Wang; Peng Chen; Pengcheng Zhang; Xiaobo Zhang; Xuanhe Zhou

REVIEW 4 major objections 5 minor 60 references

TEngineDB-V: An OLAP-Native Vector Search System for Large-$k$ Workloads at Tencent

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vector search becomes a relational operator: store the index as tables, and large-k queries run up to 145x faster.

desk verdict The architecture is more interesting than the headline speedups; send it to referees, but make the authors defend the baseline tuning and the update story. read the letter →

arxiv 2608.00650 v1 pith:P7Q2D2G7 submitted 2026-08-01 cs.DB

Xufei Wu , Pengcheng Zhang , Yitong Song , Xiaobo Zhang , Anqi Liang , Kai Wang , Jijun Du , Yidi Xiong

show 6 more authors

Guangxu Cheng Zhe Chen Peng Chen Guoliang Li Xuanhe Zhou Fan Wu

This is my paper

classification cs.DB

keywords vectorsearchlarge-kretrievalOLAPIVFPQrelationalexecutionproductquantizationhybridqueryoptimizationsegment-decoupledindex

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TEngineDB-V claims that large-scale analytical vector search — retrieving 10^3 to 10^5 nearest neighbors for downstream aggregation, filtering, and joins — is poorly served by both specialized vector databases, which cap k to protect tail latency, and OLAP systems, which embed per-segment vector indexes as black boxes that amplify I/O and compute by a factor of N×k. The proposed fix is architectural: build one global IVF index over the whole table, store its centroids, codebooks, and PQ codes as ordinary relational tables, and run the search as a pipeline of standard SQL operators. The paper reports up to a 145x speedup over a leading OLAP system at k=10^4 and up to a 52x improvement over its own previous segment-coupled engine on a 10-billion-vector production workload, while introducing DPPQ, a quantization scheme that keeps recall high without the expensive raw-vector refinement stage. If these numbers hold, the right place for vector search in analytical workloads is inside the relational engine, not as a sidecar component.

What carries the argument

Segment-decoupled global IVF index materialized as three relational tables — IVF centroids, PQ codebooks, and quantized vectors — with search decomposed into IVF prune, LUT compute, and distance estimate, fused into one SIMD operator. DPPQ augments this with direction-aware quantization (splitting each sub-vector into a norm and a unit direction, then PQ-encoding the direction) and hierarchical residual refinement over multiple epochs, reconstructing distances via the law of cosines.

What would settle it

Measure end-to-end latency on a workload where 10% of vectors are inserted or deleted each day, counting both the daily full rebuild and the staleness window; if rebuild cost or the fraction of stale-result queries exceeds the large-k query speedup, the central claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that the N×k amplification of segment-coupled indexing is not inevitable: a global index that spans all segments and is stored as sharded relational tables lets a query probe only relevant clusters, join them to quantized vectors, and estimate distances with a broadcast lookup table. This turns IVFPQ search into ordinary OLAP work — columnar scans, hash joins, runtime filters, late materialization — so the engine's existing optimizations apply directly. DPPQ then addresses the accuracy cost of PQ by optimizing directional alignment rather than raw reconstruction error and layering residual refinement, which the paper shows yields higher recall under the same bit budg

Load-bearing premise

The load-bearing assumption is that the global index can be rebuilt asynchronously once per day with a full background scan, so the workload must be read-mostly with updates arriving on the scale of days; otherwise queries return stale results and the index becomes a bottleneck.

Editorial extensions

If this is right

Large-k queries (k up to 10^5) can run inside an OLAP engine at sub-second to few-second latency, so analytics pipelines can treat similarity retrieval as a first-class SQL step rather than a pre-computation outside the database.
Read/compute amplification stops growing with the number of segments: search cost scales with probed clusters, so tables with thousands of segments no longer pay the N×k penalty.
Quantization accuracy improves without a separate raw-vector refinement phase, meaning the same bit budget yields higher recall at large k.
Hybrid filtered queries can be automatically planned: an index-aware rewrite enumerates pre-filter vs post-filter, build-side choice, and broadcast vs shuffle, and a cost model picks among them.
Because the index is in tables, it inherits the engine's columnar compression, fine-grained caching, and late materialization — lowering both storage footprint and memory pressure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not isolate how much of the speedup comes from the global index layout versus the relational execution engine; a reader could ablate by running the same global index in a segment-coupled engine to see the residual gap.
Because the index refresh is daily at most, the architecture implicitly assumes read-mostly analytics; a natural extension is incremental index maintenance (update only affected clusters) to handle higher update rates without staleness.
The table-centric design means index data becomes subject to the same backup, replication, and cache-policy machinery as base data, which could simplify operations but also makes index size part of the cache budget.
DPPQ's direction+residual scheme is expressed entirely in relational operators, so the same quantization idea could be transferred to other approximate indexes that currently need custom execution paths.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The system architecture is the contribution, not the 145x or 52x numbers. The idea of materializing a segment-decoupled global IVFPQ index as relational tables—IVF centroids, PQ codebooks, quantized vectors—and then running IVFPQ as scans, joins, projections and top-k is genuinely new relative to the per-segment black-box integrations in StarRocks, Milvus, and pgvector. The paper does this well: the execution model is coherent, the late-materialization join is clean, the columnar FastScan adaptation addresses a real impedance mismatch, and the lessons-learned section gives an honest account of why segment-coupled indexing failed at Tencent's scale. DPPQ is plausible and the recall-vs-bit-budget measurements over GloVe, SIFT1M and GIST are useful empirical evidence, even if the comparison to RaBitQ is not deep.

The soft spots are mostly in the experimental protocol, and the stress-test note is correct. All systems are given the same IVFPQ parameters, but for segment-coupled systems those parameters are per local index, so the effective index quality is not comparable; the paper never reports nprobe per system, and the 145x speedup is quoted at k=10^4 with no recall stated for that figure. The production 52x is at recall 0.8, which the abstract omits. These numbers could shrink under re-tuning, and a referee should ask for the full latency-recall curves at every k and explicit probe counts. The eventual-consistency model—full background rebuild at most once per day, deleted IDs may still appear—is a real limitation that should be in the abstract, not buried in Section 4.2. The cost model weights are hand-set and there is no sensitivity analysis. There is an artifact link in the paper, but it points to 'relify' and is not obviously this system; no artifact appendix or build instructions are included.

None of this undermines the central architecture. The gains come from a physical design plus DPPQ, not from circular reasoning, and the production results, if verified, are valuable. This is a paper for systems researchers and DBMS builders who care about vector search as a first-class analytical operation. I would bring it to a reading group; it will provoke discussion. I would send it to peer review, with the expectation of major revisions on the experimental validation and a proper artifact.

Referee Report

4 major / 5 minor

Summary. The paper presents TEngineDB-V, a vector search extension of Tencent's OLAP engine TEngineDB, targeting large-k workloads (k=10^3--10^5). The central architectural claim is that vector search should become a first-class analytical primitive: instead of per-segment local vector indexes with scatter--gather execution, TEngineDB-V materializes a global, segment-decoupled IVFPQ/IVFDPPQ index as relational tables and executes search as composable relational operators. The paper also introduces DPPQ, a direction-aware progressive product quantization scheme, and an index-aware distributed cost model that enumerates FANNS execution plans. The evaluation reports up to 145x speedup over StarRocks on SIFT1B at k=10^4, and up to 52x improvement over a legacy TEngineDB in a 10-billion-vector production deployment.

Significance. If the experimental claims are validated, this is a significant systems contribution. The idea of replacing black-box segment-coupled vector indexes with table-centric global indexes, and expressing IVF/PQ search through relational operators, directly attacks the N×k amplification problem and enables native OLAP optimizations. The paper is also valuable for its candid discussion of the limitations of segment-coupled indexing and the lessons learned from the legacy system. The production-scale case study at 10B vectors is a rare and potentially strong piece of evidence. However, the current manuscript does not yet establish the key speedups convincingly because the baseline configuration, operating points, and memory accounting are under-specified. The DPPQ bit-budget comparison, in particular, appears to omit a substantial per-vector storage cost.

major comments (4)

[§7.1, Figure 8b] The headline 145.5× speedup is reported at k=10^4 on SIFT1B, but no recall value is given for this point. Since the paper compares systems at recall 0.8--0.9 elsewhere, and latency can vary by orders of magnitude across operating points, the speedup is not interpretable without recall@k. Please report recall@k and per-system nprobe for every latency point in the k-scalability experiments, and specifically for the k=10^4 point that is quoted in the abstract.
[§7.1, parameter settings] The statement that all systems except DiskANN use the same IVFPQ parameters (3000 clusters, 32 subspaces, 8 bits on SIFT1B) is problematic for segment-coupled baselines. For StarRocks, PGVector, and Milvus, each segment builds a local index on only a shard of the data (~18M vectors with 56 shards on SIFT1B). A 3000-cluster local index trained on such a subset will have much noisier centroids than a globally trained 3000-cluster index, potentially forcing far larger nprobe values. Please report the actual per-segment cluster count, the training data size per local index, the per-system nprobe, and the tuning procedure. Without this information, the reported speedups over StarRocks and PGVector may be artifacts of an underconfigured baseline.
[§5.2, §7.4.1] The DPPQ bit-budget comparison appears to omit a large component of the actual storage cost. Equation (1) stores a scalar magnitude ρ^(j) per subvector, and Section 7.4.1 admits that DPPQ requires 'one additional FP32 scaling factor per subspace at each refinement epoch.' For the Wikipedia-style setting with m=192 subvectors and E=5 epochs, this adds 192×32×5 = 30,720 bits per vector, far exceeding the nominal 400-bit budget used in Figure 11. The claim that DPPQ and PQ/RaBitQ are compared 'under the same bit budget' is therefore not credible unless the scaling factors are counted. Please report total stored bits per vector, including all scaling factors and residual codebooks, or justify why these factors should be excluded.
[§4.2, §7.5] The asynchronous full-rebuild refresh model (at most once per day, full Spark rebuild on any change) and the resulting eventual consistency are acknowledged, but no measurements are provided for rebuild cost, staleness, or index footprint. The production claim at 10B vectors is made for a read-mostly workload; for workloads with updates, the rebuild cost could dominate the total cost of ownership. Please add index construction/refresh time and memory/SSD footprint, and explicitly state the update rate at which the design remains practical.

minor comments (5)

[Abstract, §7.5] The 52× production improvement is the recall=0.8 point from Figure 13; at recall=0.88 the improvement over Legacy TEngineDB is 14×. Please state the operating point whenever quoting this number.
[§5.2] The phrase 'higher recall upper bound' is imprecise. Figure 11 reports empirical recall, not an upper bound. Rephrase to avoid implying a theoretical guarantee.
[§7.1] Please clarify whether the SIFT1B and Wikipedia experiments are single-node or distributed. The text says 'the machine' for the main experiments and 'a distributed cluster of 10 machines' for the case study, but the '56-shard configuration' makes it ambiguous for the baselines.
[Table 3] The cost-model table is difficult to parse as rendered; align the cost expressions under the correct plan columns and ensure all symbols are defined before use.
[PVLDB Artifact Availability] The artifact link is given, but the paper does not describe versions, commands, or parameter files. Please add a short reproducibility statement, including how the baseline indexes were configured and tuned.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the system architecture and DPPQ claims are supported by independent design reasoning and public benchmark evaluations, not by re-importing their conclusions.

full rationale

The central claims of TEngineDB-V are (1) that a segment-decoupled global IVFPQ index materialized as relational tables avoids the N×k scatter-gather amplification of per-segment indexes, (2) that relational execution with OLAP optimizations is more efficient for large-k workloads, and (3) that DPPQ improves quantization accuracy under an equal bit budget. None of these claims reduces to a definition or to a fitted parameter. The scatter-gather elimination is an architectural argument: a global index lets queries probe relevant clusters instead of every segment, and the paper supports this with end-to-end latency experiments on public datasets (Wikipedia, SIFT1B) and production data. The DPPQ accuracy claims are empirical comparisons against PQ and RaBitQ on public benchmarks (GloVe, SIFT1M, GIST), which are not derived from the experimental outcome itself. The production comparison against 'Legacy TEngineDB' is self-referential in the sense of comparing the new system to the authors' own earlier system, but this is a standard baseline comparison and not circular: the legacy system is an independent implementation with a different indexing architecture, not a restatement of TEngineDB-V's conclusions. The paper contains several self-citations (e.g., [11], [28], [43], [45], [46]), but they are used only as related-work context or auxiliary support (e.g., disk-resident indexing techniques, graph-based methods), not as load-bearing justification for the paper's central design or performance claims. The acknowledged eventual-consistency limitation and the reliance on infrequent-update workloads are scope restrictions that weaken generality, but they are explicitly stated and do not constitute circularity. Concerns about baseline tuning, unreported per-system probe counts, and the lack of recall reporting at the k=10^4 point are experimental-validation risks, not circular-reasoning steps. Under the rules requiring quotation and specific reduction before flagging circularity, no such reduction can be exhibited here.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central claims rest on the design assumptions listed above. Axioms used: the eventual-consistency refresh model, the decomposability and efficiency of IVFPQ-as-relational-operators, the fairness of the experimental comparison, and the adequacy of the cost model. The invented index tables and DPPQ are implementation entities with plausible independent evidence (they could be measured externally), which is why the paper is not fully circular. All free parameters are configuration choices.

free parameters (4)

Cost model weights (w_cpu=0.5, w_mem=2.0, w_net=1.5) = w_cpu=0.5, w_mem=2.0, w_net=1.5
Section 6.2: 'The weights are calibrated to reflect the architectural characteristics of TEngineDB-V. By default, we set wcpu = 0.5, wmem = 2.0, and wnet = 1.5'. These are hand-chosen, not derived or learned from data. The plan-selection results (Figure 10) depend on them.
Post-filter K_ann selection rule (2k at 50% selectivity) = 2k for estimated 50% selectivity; 20% yield threshold
Section 6.2: 'adaptively sets K_ann based on the predicate selectivity estimated by the optimizer (e.g., retrieving 2k candidates for an estimated 50% selectivity)' and configurable fallback threshold defaulting to 20% of k. These are heuristic constants without derivation.
IVF/PQ index parameters (nlist, nbits, subspaces) for each dataset = e.g., SIFT1B: 3000 clusters, 32 subspaces, 8-bit; Wikipedia: 6000 clusters, 192 subspaces, 8-bit
Experiment settings (Section 7.1). These are chosen by hand during benchmark setup and affect all recall/latency numbers. They are standard tuning parameters, but the specific values are not derived from any principle.
DPPQ epochs E and subspaces M = E=2-5 in convergence figures; various M
Algorithm 1 takes epoch count E and subspace count M as inputs; Figure 12 scans them. These are free parameters tuned per dataset and affect the reported recall improvements.

assumptions (5)

domain assumption A global index refreshed at most once per day with eventual consistency is adequate for the target workloads.
Section 4.2: 'adopts an eventual consistency model... This design is well-suited for analytical workloads with infrequent updates (days/weeks).' If update rates are higher, the claimed production applicability breaks. This is acknowledged as an assumption.
domain assumption IVFPQ-based search with PQ-style quantization can be decomposed into relational operators without materialization that destroys the gains.
Section 5.1. The relational execution plan (CROSS_JOIN of LUT with quantized vectors) could in principle create massive intermediate results; the paper's contribution is claiming the fused operator avoids this. This is the core technical bet, unproven outside this implementation.
domain assumption The reported latency-recall points are comparable across systems.
Section 7.1: 'All evaluated systems are open-source and compared at medium recall levels (0.8-0.9)'. But Figures 7-9 compare at slightly different recall points, and the 145x speedup specifically compares at a point where competitors failed (Milvus OOM on SIFT1B). If the comparison protocol is unfair, the speedup numbers are misleading.
domain assumption The cost model's plan-cost formulas are accurate predictors of real distributed execution cost.
Section 6.2 Table 3. The cost formulas treat CPU cost as simple arithmetic (e.g., Mp*C_dist) and ignore memory-bandwidth contention, stragglers, and disk I/O for index blocks. No validation of the cost model's prediction accuracy against actual measured latencies.
standard math Standard relational algebra and query optimization principles (Cascades, Volcano, hash joins, runtime filters) apply to the new index-table scans.
The paper relies on conventional database theory: vectorized execution (MonetDB/X100), Cascades/Volcano optimization frameworks, and hash-join correctness. These are standard and not ad hoc.

invented entities (2)

Global vector index tables (IVF centroid table, PQ codebook table, quantized vector table) independent evidence
purpose: New storage substrate: materializing a global index as relational tables to enable native OLAP optimizations and eliminate scatter-gather.
The index tables are a concrete storage design that could be used and measured by anyone implementing the described architecture. Their benefit is backed by the experimental claims, though code is not shipped to verify.
DPPQ (Directional Progressive Product Quantization) independent evidence
purpose: A new quantization scheme combining direction-aware quantization and hierarchical residual refinement to improve recall under equal bit budget.
DPPQ can be reproduced from Algorithm 1 and tested on public datasets (GloVe, SIFT, GIST). No external code is provided in this submission, so independent evidence is only theoretical.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TEngineDB-V: An OLAP-Native Vector Search System for Large-$k$ Workloads at Tencent." pith.science (2026). https://pith.science/paper/P7Q2D2G7

@misc{pith2026260800650,
  author       = {Pith},
  title        = {Pith review of: TEngineDB-V: An OLAP-Native Vector Search System for Large-$k$ Workloads at Tencent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P7Q2D2G7}},
  note         = {Machine review of arXiv:2608.00650}
}

abstract

Vector search systems are essential infrastructure for modern data-driven applications. Large-$k$ analytical vector search, which retrieves $k=10^3$--$10^5$ results for analytics (e.g., aggregation, filtering, joins), is increasingly important for emerging workloads, including LLM data management and advertising analysis at Tencent. Existing systems remain inadequate: specialized vector databases often cap $k$ (e.g., $k \leq 10^4$) to satisfy tail-latency constraints and offer limited analytical support, while OLAP systems typically embed per-segment vector indexes as black boxes, causing severe read/compute amplification and preventing native query optimization. This paper presents TEngineDB-V, an OLAP-native vector search system for large-$k$ workloads. TEngineDB-V makes vector search a first-class analytical primitive in Tencent's OLAP engine through a global segment-decoupled index materialized as relational tables, eliminating scatter-gather execution, reducing amplification, and enabling native storage optimizations. It decomposes IVFPQ-based search into relational operators, integrates OLAP optimizations, and introduces DPPQ, which combines direction-aware quantization with hierarchical residual refinement to improve recall while preserving relational efficiency. TEngineDB-V further incorporates index-aware query rewriting and a distributed-aware cost model for efficient distributed execution. Experiments show that TEngineDB-V achieves up to a $145\times$ speedup over competitive systems such as StarRocks, and up to a $52\times$ improvement in 10-billion-scale production deployments.

Figures

Figures reproduced from arXiv: 2608.00650 by the authors.

**Figure 3.** The Architecture of TEngineDB-V. IVFPQ uses 𝑘-means clustering [25] to partition vectors into 𝑘 clusters and searches the top-𝑁𝑝𝑟𝑜𝑏𝑒 clusters closest to the query. To further accelerate distance computation within each cluster, Product Quantization (PQ) is applied to encode the vectors. The 𝑑-dimensional vector is partitioned into 𝑚 disjoint subspaces, each of dimension 𝑑/𝑚. For each subspace, a separate codebook wi… view at source ↗

**Figure 4.** Storage architectural comparison. and enable SIMD-friendly computation. Sharding by CID further ensures that only relevant shards are accessed during IVF probing. Native Optimizations. Our table-centric design enables the global index to fully leverage native engine optimizations along the storage and execution pipeline. (1) Columnar Compression. Since index tables are stored in columnar format, they automatically … view at source ↗

Figures from the paper (8 more)

**Figure 5.** Figure 5: Relational execution in TEngineDB-V. quantization approach with strong empirical performance, but its processing relies on complex operations, including per-cluster normalization, matrix multiplication, and bitwise operations, which are not naturally adaptable to stan…

**Figure 6.** Figure 6: Execution Plans for FANNS. 𝑤net = 1.5, assigning higher penalties to memory pressure and network communication in distributed environments. Recall that we classify the eight candidate FANNS plans along three orthogonal dimensions. For each dimension, we then analyze th…

**Figure 7.** Figure 7: Latency–Recall Trade-off Across Systems. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Latency Scalability with Increasing 𝐾. DiskANN utilize the IVFPQ index. For the SIFT1B dataset, the data space is partitioned into 3000 clusters, and vectors are compressed using 32 subspaces with 8-bit quantization per subspace. For the Wikipedia dataset, the data is …

**Figure 11.** Figure 11: Comparison of different quantization algorithms. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

**Figure 10.** Figure 10: Performance of the eight FANNS execution plans [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

**Figure 12.** Figure 12: Convergence analysis of DPPQ. 0.80 0.82 0.84 0.86 0.88 recall@20, 000 10 3 10 4 10 5 L a t e n cy(m s) Tencent-Image TEngineDB-V Legacy TEngineDB StarRocks Milvus [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

**Figure 13.** Figure 13: Production performance at 10-billion scale. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 3 linked inside Pith

[1]

Meta AI. 2017. FAISS. https://ai.facebook.com/tools/faiss

2017
[2]

Fabien André, Anne-Marie Kermarrec, and Nicolas Le Scouarnec. 2016. Cache locality is not enough: High-performance nearest neighbor search with product quantization fast scan. InPVLDB, Vol. 9. 12

2016
[3]

Fabien André, Anne-Marie Kermarrec, and Nicolas Le Scouarnec. 2019. Quicker adc: Unlocking the hidden potential of product quantization with simd.IEEE TPAMI43, 5 (2019), 1666–1677

2019
[4]

Apache Doris Community. 2025. Vector Search in Apache Doris. https://doris. apache.org/zh-CN/docs/4.x/ai/vector-search/overview/. Accessed: 2025-12

2025
[5]

Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. 2020. ANN- Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems87 (2020), 101374

2020
[6]

Peter A Boncz, Marcin Zukowski, and Niels Nes. 2005. MonetDB/X100: Hyper- Pipelining Query Execution.. InCidr, Vol. 5. 225–237

2005
[7]

Andrei Broder and Michael Mitzenmacher. 2004. Network applications of bloom filters: A survey.Internet mathematics1, 4 (2004), 485–509

2004
[8]

Cheng Chen, Chenzhe Jin, Yunan Zhang, Sasha Podolsky, Chun Wu, Szu- Po Wang, Eric Hanson, Zhou Sun, Robert Walzer, and Jianguo Wang. 2024. Singlestore-v: An integrated vector database system in singlestore.PVLDB17, 12 (2024)

2024

Show all 60 references

[9]

2018.SPTAG: A library for fast approximate nearest neighbor search

Qi Chen, Haidong Wang, Mingqin Li, Gang Ren, Scarlett Li, Jeffery Zhu, Jason Li, Chuanjie Liu, Lintao Zhang, and Jingdong Wang. 2018.SPTAG: A library for fast approximate nearest neighbor search. Microsoft Research, GitHub repository: https://github.com/Microsoft/SPTAG

2018
[10]

Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. 2021. SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search.arXiv preprint arXiv:2111.08566(2021)

2021 arXiv
[11]

Xiaoyu Chen, Jinxiu Qu, Yitong Song, Shuhang Lu, Huiling Li, Minghui Jiang, Wei Zhou, Jianliang Xu, Xuanhe Zhou, and Fan Wu. 2026. Disk-Resident Graph ANN Search: An Experimental Evaluation.arXiv preprint arXiv:2603.01779(2026)

2026
[12]

Yanqi Chen, Xiao Yan, Alexandra Meliou, and Eric Lo. 2025. DiskJoin: Large-scale Vector Similarity Join with SSD.Proceedings of the ACM on Management of Data 3, 6 (2025), 1–27

2025
[13]

ClickHouse, Inc. 2024. Approximate Nearest Neighbor Search with Vector Simi- larity Indexes. https://clickhouse.com/docs/en/engines/table-engines/mergetree- family/annindexes. Accessed: 2025-12

2024
[14]

Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, et al. 2016. The snowflake elastic data warehouse. InProceed- ings of the 2016 International Conference on Manage...

2016
[15]

Elastic. 2023. kNN Search and Vector Fields in Elasticsearch. https://www. elastic.co/guide/en/elasticsearch/reference/current/knn-search.html. Accessed: 2025-12

2023
[16]

Cong Fu, Chao Xiang, Changxu Wang, and Deng Cai. 2019. Fast approximate nearest neighbor search with the navigating spreading-out graph. InPVLDB, Vol. 12. VLDB Endowment, 416–474

2019
[17]

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 2023. Datacomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing ...

2023
[18]

Jianyang Gao, Yutong Gou, Yuexuan Xu, Yongyi Yang, Cheng Long, and Raymond Chi-Wing Wong. 2025. Practical and asymptotically optimal quantization of high-dimensional vectors in euclidean space for approximate nearest neighbor search.ACM SIGMOD3, 3 (2025), 1–26

2025
[19]

Jianyang Gao and Cheng Long. 2024. RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search.ACM SIGMOD2, 3 (2024), 1–27

2024
[20]

Goetz Graefe et al. 1995. The cascades framework for query optimization.IEEE Data Eng. Bull.18, 3 (1995), 19–29

1995
[21]

Goetz Graefe and William J McKenna. 1993. The volcano optimizer genera- tor: Extensibility and efficient search. InProceedings of IEEE 9th international conference on data engineering. IEEE, 209–218

1993
[22]

Yuxing Han, Yu Lin, Yifeng Dong, Xuanhe Zhou, Xindong Peng, Xinhui Tian, Zhiyuan You, Yingzhong Guo, Xi Chen, Weiping Qu, et al. 2026. ByteHouse: A Cloud-Native OLAP Engine with Incremental Computation and Multi-Modal Retrieval.ACM SIGMOD(2026)

2026
[23]

Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. 2019. Diskann: Fast accurate billion-point nearest neighbor search on a single node.Advances in Neural Information Processing Systems32 (2019)

2019
[24]

Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search.IEEE TPAMI33, 1 (2010), 117–128

2010
[25]

K Krishna and M Narasimha Murty. 1999. Genetic K-means algorithm.IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)29, 3 (1999), 433–439

1999
[26]

Hai Lan, Shixun Huang, Zhifeng Bao, and Renata Borovica-Gajic. 2024. Cardi- nality estimation for similarity search on high-dimensional data objects: The impact of reference objects.Proceedings of the VLDB Endowment18, 3 (2024), 544–556

2024
[27]

Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. 2019. Approximate nearest neighbor search on high dimensional data—experiments, analyses, and improvement.TKDE32, 8 (2019), 1475–1488

2019
[28]

Anqi Liang, Pengcheng Zhang, Bin Yao, Zhongpu Chen, Yitong Song, and Guangxu Cheng. 2024. UNIFY: Unified Index for Range Filtered Approximate Nearest Neighbors Search.Proceedings of the VLDB Endowment18, 4 (2024), 1118–1130

2024
[29]

Guy M Lohman. 1988. R* optimizer validation and performance evaluation for distributed queries.Readings in Database Systems(1988), 219

1988
[30]

Youzhong Ma, Shijie Jia, and Yongxin Zhang. 2017. A novel approach for high- dimensional vector similarity join query.Concurrency and Computation: Practice and Experience29, 5 (2017), e3952

2017
[31]

Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.IEEE TPAMI42, 4 (2018), 824–836

2018
[32]

Zhaojie Niu, Xinhui Tian, Xindong Peng, and Xing Chen. 2025. BlendHouse: A cloud-native vector database system in ByteHouse. InIEEE ICDE. IEEE, 4332– 4345

2025
[33]

Yun Peng, Byron Choi, Tsz Nam Chan, Jianye Yang, and Jianliang Xu. 2023. Efficient Approximate Nearest Neighbor Search in Multi-dimensional Databases. ACM SIGMOD1, 1 (2023), 1–27

2023
[34]

pgvector authors. 2023. pgvector: Open-source vector similarity search for Postgres. https://github.com/pgvector/pgvector. Accessed: 2025-12

2023
[35]

2021.Pinecone.io

Pinecone. 2021.Pinecone.io. https://www.pinecone.io/

2021
[36]

2021.qdrant.tech

Qdrant. 2021.qdrant.tech. https://qdrant.tech/

2021
[37]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al
[38]

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen
[39]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695

2022
[40]

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 22500–22510

2023
[41]

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural inform...

2022
[42]

Mohamed A Soliman, Lyublena Antova, Venkatesh Raghavan, Amr El-Helw, Zhongxian Gu, Entong Shen, George C Caragea, Carlos Garcia-Alvarado, Foyzur Rahman, Michalis Petropoulos, et al . 2014. Orca: a modular query optimizer architecture for big data. InProceedings of the 2014 ACM...

2014
[43]

Yitong Song, Kai Wang, Bin Yao, Zhida Chen, Jiong Xie, and Feifei Li. 2024. Effi- cient Reverse𝑘 Approximate Nearest Neighbor Search Over High-Dimensional Vectors. In2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 4262–4274

2024
[44]

Yitong Song, Bin Yao, Zhida Chen, Xin Yang, Jiong Xie, Feifei Li, and Mengshi Chen. 2025. Efficient top-k spatial-range-constrained approximate nearest neigh- bor search on geo-tagged high-dimensional vectors.The VLDB Journal34, 1 (2025), 14

2025
[45]

Yitong Song, Pengcheng Zhang, Chao Gao, Bin Yao, Kai Wang, Zongyuan Wu, and Lin Qu. 2025. TRIM: Accelerating High-Dimensional Vector Similarity Search with Enhanced Triangle-Inequality-Based Pruning.ACM SIGMOD3, 6 (2025), 1–26

2025
[46]

Yitong Song, Xuanhe Zhou, Christian S Jensen, and Jianliang Xu. 2026. Vector Search for the Future: From Memory-Resident, Static Heterogeneous Storage, to Cloud-Native Architectures. (2026), 580–587

2026
[47]

StarRocks Community. 2024. Vector Index in StarRocks. https://docs.starrocks. io/docs/using_starrocks/vector-index/. Accessed: 2025-12

2024
[48]

Ji Sun, Guoliang Li, James Pan, Jiang Wang, Yongqing Xie, Ruicheng Liu, and Wen Nie. 2025. GaussDB-Vector: A Large-Scale Persistent Real-Time Vector Database for LLM Applications.PVLDB18, 12 (2025), 4951–4963

2025
[49]

The Apache Software Foundation. 2025. Apache Arrow: A cross-language de- velopment platform for in-memory data. https://arrow.apache.org/. Accessed: 2025-10-24

2025
[50]

Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xi- angyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al. 2021. Milvus: A purpose-built vector data management system. InACM SIGMOD. 2614–2627

2021
[51]

Mengzhao Wang, Lingwei Lv, Xiaoliang Xu, Yuxiang Wang, Qiang Yue, and Jiongkang Ni. 2022. Navigable Proximity Graph-Driven Native Hybrid Queries with Structured and Unstructured Constraints.arXiv preprint arXiv:2203.13601 (2022)

2022 arXiv
[52]

Mengzhao Wang, Weizhi Xu, Xiaomeng Yi, Songlin Wu, Zhangyang Peng, Xi- angyu Ke, Yunjun Gao, Xiaoliang Xu, Rentong Guo, and Charles Xie. 2024. Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High- Dimensional Vector Similarity Search on Data Segment.ACM SIGM...

2024
[53]

2019.Weaviate.io

Weaviate. 2019.Weaviate.io. https://weaviate.io/developers/weaviate/concepts/ vector-index

2019
[54]

Chuangxian Wei, Bin Wu, Sheng Wang, Renjie Lou, Chaoqun Zhan, Feifei Li, and Yuanzhe Cai. 2020. Analyticdb-v: A hybrid analytical engine towards query fusion for structured and unstructured data.PVLDB13, 12 (2020), 3152–3165

2020
[55]

Jiadong Xie, Jeffrey Xu Yu, and Yingfan Liu. 2025. Fast approximate similarity join in vector databases.Proceedings of the ACM on Management of Data3, 3 (2025), 1–26

2025
[56]

Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster computing with working sets. In2nd USENIX workshop on hot topics in cloud computing (HotCloud 10)

2010
[57]

Yunan Zhang, Shige Liu, and Jianguo Wang. 2024. Are there fundamental limitations in supporting vector data management in relational databases? A case study of PostgreSQL. InIEEE ICDE. IEEE

2024
[58]

Chaoji Zuo, Miao Qiao, Wenchao Zhou, Feifei Li, and Dong Deng. 2024. SeRF: Segment Graph for Range-Filtering Approximate Nearest Neighbor Search.ACM SIGMOD2, 1 (2024), 1–26

2024
[2021]

In International conference on machine learning

Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763
[2022]

Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.061251, 2 (2022), 3

2022 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Meta AI. 2017. FAISS. https://ai.facebook.com/tools/faiss

2017

[2] [2]

Fabien André, Anne-Marie Kermarrec, and Nicolas Le Scouarnec. 2016. Cache locality is not enough: High-performance nearest neighbor search with product quantization fast scan. InPVLDB, Vol. 9. 12

2016

[3] [3]

Fabien André, Anne-Marie Kermarrec, and Nicolas Le Scouarnec. 2019. Quicker adc: Unlocking the hidden potential of product quantization with simd.IEEE TPAMI43, 5 (2019), 1666–1677

2019

[4] [4]

Apache Doris Community. 2025. Vector Search in Apache Doris. https://doris. apache.org/zh-CN/docs/4.x/ai/vector-search/overview/. Accessed: 2025-12

2025

[5] [5]

Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. 2020. ANN- Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems87 (2020), 101374

2020

[6] [6]

Peter A Boncz, Marcin Zukowski, and Niels Nes. 2005. MonetDB/X100: Hyper- Pipelining Query Execution.. InCidr, Vol. 5. 225–237

2005

[7] [7]

Andrei Broder and Michael Mitzenmacher. 2004. Network applications of bloom filters: A survey.Internet mathematics1, 4 (2004), 485–509

2004

[8] [8]

Cheng Chen, Chenzhe Jin, Yunan Zhang, Sasha Podolsky, Chun Wu, Szu- Po Wang, Eric Hanson, Zhou Sun, Robert Walzer, and Jianguo Wang. 2024. Singlestore-v: An integrated vector database system in singlestore.PVLDB17, 12 (2024)

2024