Pith. sign in

REVIEW 4 major objections 5 minor 60 references

Vector search becomes a relational operator: store the index as tables, and large-k queries run up to 145x faster.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An OLAP-native vector search system materializes a segment-decoupled global IVFPQ index as relational tables and executes large-k search via relational operators, reporting up to 145x speedup over StarRocks and 52x over legacy TEngineDB.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The architecture is more interesting than the headline speedups; send it to referees, but make the authors defend the baseline tuning and the update story. the 4 major comments →

arxiv 2608.00650 v1 pith:P7Q2D2G7 submitted 2026-08-01 cs.DB

TEngineDB-V: An OLAP-Native Vector Search System for Large-$k$ Workloads at Tencent

Xufei Wu , Pengcheng Zhang , Yitong Song , Xiaobo Zhang , Anqi Liang , Kai Wang , Jijun Du , Yidi Xiong
show 6 more authors
Guangxu Cheng Zhe Chen Peng Chen Guoliang Li Xuanhe Zhou Fan Wu
This is my paper
classification cs.DB
keywords vector searchlarge-k retrievalOLAPIVFPQrelational executionproduct quantizationhybrid query optimizationsegment-decoupled index
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TEngineDB-V claims that large-scale analytical vector search — retrieving 10^3 to 10^5 nearest neighbors for downstream aggregation, filtering, and joins — is poorly served by both specialized vector databases, which cap k to protect tail latency, and OLAP systems, which embed per-segment vector indexes as black boxes that amplify I/O and compute by a factor of N×k. The proposed fix is architectural: build one global IVF index over the whole table, store its centroids, codebooks, and PQ codes as ordinary relational tables, and run the search as a pipeline of standard SQL operators. The paper reports up to a 145x speedup over a leading OLAP system at k=10^4 and up to a 52x improvement over its own previous segment-coupled engine on a 10-billion-vector production workload, while introducing DPPQ, a quantization scheme that keeps recall high without the expensive raw-vector refinement stage. If these numbers hold, the right place for vector search in analytical workloads is inside the relational engine, not as a sidecar component.

Core claim

The central discovery is that the N×k amplification of segment-coupled indexing is not inevitable: a global index that spans all segments and is stored as sharded relational tables lets a query probe only relevant clusters, join them to quantized vectors, and estimate distances with a broadcast lookup table. This turns IVFPQ search into ordinary OLAP work — columnar scans, hash joins, runtime filters, late materialization — so the engine's existing optimizations apply directly. DPPQ then addresses the accuracy cost of PQ by optimizing directional alignment rather than raw reconstruction error and layering residual refinement, which the paper shows yields higher recall under the same bit budg

What carries the argument

Segment-decoupled global IVF index materialized as three relational tables — IVF centroids, PQ codebooks, and quantized vectors — with search decomposed into IVF prune, LUT compute, and distance estimate, fused into one SIMD operator. DPPQ augments this with direction-aware quantization (splitting each sub-vector into a norm and a unit direction, then PQ-encoding the direction) and hierarchical residual refinement over multiple epochs, reconstructing distances via the law of cosines.

Load-bearing premise

The load-bearing assumption is that the global index can be rebuilt asynchronously once per day with a full background scan, so the workload must be read-mostly with updates arriving on the scale of days; otherwise queries return stale results and the index becomes a bottleneck.

What would settle it

Measure end-to-end latency on a workload where 10% of vectors are inserted or deleted each day, counting both the daily full rebuild and the staleness window; if rebuild cost or the fraction of stale-result queries exceeds the large-k query speedup, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Large-k queries (k up to 10^5) can run inside an OLAP engine at sub-second to few-second latency, so analytics pipelines can treat similarity retrieval as a first-class SQL step rather than a pre-computation outside the database.
  • Read/compute amplification stops growing with the number of segments: search cost scales with probed clusters, so tables with thousands of segments no longer pay the N×k penalty.
  • Quantization accuracy improves without a separate raw-vector refinement phase, meaning the same bit budget yields higher recall at large k.
  • Hybrid filtered queries can be automatically planned: an index-aware rewrite enumerates pre-filter vs post-filter, build-side choice, and broadcast vs shuffle, and a cost model picks among them.
  • Because the index is in tables, it inherits the engine's columnar compression, fine-grained caching, and late materialization — lowering both storage footprint and memory pressure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate how much of the speedup comes from the global index layout versus the relational execution engine; a reader could ablate by running the same global index in a segment-coupled engine to see the residual gap.
  • Because the index refresh is daily at most, the architecture implicitly assumes read-mostly analytics; a natural extension is incremental index maintenance (update only affected clusters) to handle higher update rates without staleness.
  • The table-centric design means index data becomes subject to the same backup, replication, and cache-policy machinery as base data, which could simplify operations but also makes index size part of the cache budget.
  • DPPQ's direction+residual scheme is expressed entirely in relational operators, so the same quantization idea could be transferred to other approximate indexes that currently need custom execution paths.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents TEngineDB-V, a vector search extension of Tencent's OLAP engine TEngineDB, targeting large-k workloads (k=10^3--10^5). The central architectural claim is that vector search should become a first-class analytical primitive: instead of per-segment local vector indexes with scatter--gather execution, TEngineDB-V materializes a global, segment-decoupled IVFPQ/IVFDPPQ index as relational tables and executes search as composable relational operators. The paper also introduces DPPQ, a direction-aware progressive product quantization scheme, and an index-aware distributed cost model that enumerates FANNS execution plans. The evaluation reports up to 145x speedup over StarRocks on SIFT1B at k=10^4, and up to 52x improvement over a legacy TEngineDB in a 10-billion-vector production deployment.

Significance. If the experimental claims are validated, this is a significant systems contribution. The idea of replacing black-box segment-coupled vector indexes with table-centric global indexes, and expressing IVF/PQ search through relational operators, directly attacks the N×k amplification problem and enables native OLAP optimizations. The paper is also valuable for its candid discussion of the limitations of segment-coupled indexing and the lessons learned from the legacy system. The production-scale case study at 10B vectors is a rare and potentially strong piece of evidence. However, the current manuscript does not yet establish the key speedups convincingly because the baseline configuration, operating points, and memory accounting are under-specified. The DPPQ bit-budget comparison, in particular, appears to omit a substantial per-vector storage cost.

major comments (4)
  1. [§7.1, Figure 8b] The headline 145.5× speedup is reported at k=10^4 on SIFT1B, but no recall value is given for this point. Since the paper compares systems at recall 0.8--0.9 elsewhere, and latency can vary by orders of magnitude across operating points, the speedup is not interpretable without recall@k. Please report recall@k and per-system nprobe for every latency point in the k-scalability experiments, and specifically for the k=10^4 point that is quoted in the abstract.
  2. [§7.1, parameter settings] The statement that all systems except DiskANN use the same IVFPQ parameters (3000 clusters, 32 subspaces, 8 bits on SIFT1B) is problematic for segment-coupled baselines. For StarRocks, PGVector, and Milvus, each segment builds a local index on only a shard of the data (~18M vectors with 56 shards on SIFT1B). A 3000-cluster local index trained on such a subset will have much noisier centroids than a globally trained 3000-cluster index, potentially forcing far larger nprobe values. Please report the actual per-segment cluster count, the training data size per local index, the per-system nprobe, and the tuning procedure. Without this information, the reported speedups over StarRocks and PGVector may be artifacts of an underconfigured baseline.
  3. [§5.2, §7.4.1] The DPPQ bit-budget comparison appears to omit a large component of the actual storage cost. Equation (1) stores a scalar magnitude ρ^(j) per subvector, and Section 7.4.1 admits that DPPQ requires 'one additional FP32 scaling factor per subspace at each refinement epoch.' For the Wikipedia-style setting with m=192 subvectors and E=5 epochs, this adds 192×32×5 = 30,720 bits per vector, far exceeding the nominal 400-bit budget used in Figure 11. The claim that DPPQ and PQ/RaBitQ are compared 'under the same bit budget' is therefore not credible unless the scaling factors are counted. Please report total stored bits per vector, including all scaling factors and residual codebooks, or justify why these factors should be excluded.
  4. [§4.2, §7.5] The asynchronous full-rebuild refresh model (at most once per day, full Spark rebuild on any change) and the resulting eventual consistency are acknowledged, but no measurements are provided for rebuild cost, staleness, or index footprint. The production claim at 10B vectors is made for a read-mostly workload; for workloads with updates, the rebuild cost could dominate the total cost of ownership. Please add index construction/refresh time and memory/SSD footprint, and explicitly state the update rate at which the design remains practical.
minor comments (5)
  1. [Abstract, §7.5] The 52× production improvement is the recall=0.8 point from Figure 13; at recall=0.88 the improvement over Legacy TEngineDB is 14×. Please state the operating point whenever quoting this number.
  2. [§5.2] The phrase 'higher recall upper bound' is imprecise. Figure 11 reports empirical recall, not an upper bound. Rephrase to avoid implying a theoretical guarantee.
  3. [§7.1] Please clarify whether the SIFT1B and Wikipedia experiments are single-node or distributed. The text says 'the machine' for the main experiments and 'a distributed cluster of 10 machines' for the case study, but the '56-shard configuration' makes it ambiguous for the baselines.
  4. [Table 3] The cost-model table is difficult to parse as rendered; align the cost expressions under the correct plan columns and ensure all symbols are defined before use.
  5. [PVLDB Artifact Availability] The artifact link is given, but the paper does not describe versions, commands, or parameter files. Please add a short reproducibility statement, including how the baseline indexes were configured and tuned.

Circularity Check

0 steps flagged

No significant circularity: the system architecture and DPPQ claims are supported by independent design reasoning and public benchmark evaluations, not by re-importing their conclusions.

full rationale

The central claims of TEngineDB-V are (1) that a segment-decoupled global IVFPQ index materialized as relational tables avoids the N×k scatter-gather amplification of per-segment indexes, (2) that relational execution with OLAP optimizations is more efficient for large-k workloads, and (3) that DPPQ improves quantization accuracy under an equal bit budget. None of these claims reduces to a definition or to a fitted parameter. The scatter-gather elimination is an architectural argument: a global index lets queries probe relevant clusters instead of every segment, and the paper supports this with end-to-end latency experiments on public datasets (Wikipedia, SIFT1B) and production data. The DPPQ accuracy claims are empirical comparisons against PQ and RaBitQ on public benchmarks (GloVe, SIFT1M, GIST), which are not derived from the experimental outcome itself. The production comparison against 'Legacy TEngineDB' is self-referential in the sense of comparing the new system to the authors' own earlier system, but this is a standard baseline comparison and not circular: the legacy system is an independent implementation with a different indexing architecture, not a restatement of TEngineDB-V's conclusions. The paper contains several self-citations (e.g., [11], [28], [43], [45], [46]), but they are used only as related-work context or auxiliary support (e.g., disk-resident indexing techniques, graph-based methods), not as load-bearing justification for the paper's central design or performance claims. The acknowledged eventual-consistency limitation and the reliance on infrequent-update workloads are scope restrictions that weaken generality, but they are explicitly stated and do not constitute circularity. Concerns about baseline tuning, unreported per-system probe counts, and the lack of recall reporting at the k=10^4 point are experimental-validation risks, not circular-reasoning steps. Under the rules requiring quotation and specific reduction before flagging circularity, no such reduction can be exhibited here.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claims rest on the design assumptions listed above. Axioms used: the eventual-consistency refresh model, the decomposability and efficiency of IVFPQ-as-relational-operators, the fairness of the experimental comparison, and the adequacy of the cost model. The invented index tables and DPPQ are implementation entities with plausible independent evidence (they could be measured externally), which is why the paper is not fully circular. All free parameters are configuration choices.

free parameters (4)
  • Cost model weights (w_cpu=0.5, w_mem=2.0, w_net=1.5) = w_cpu=0.5, w_mem=2.0, w_net=1.5
    Section 6.2: 'The weights are calibrated to reflect the architectural characteristics of TEngineDB-V. By default, we set wcpu = 0.5, wmem = 2.0, and wnet = 1.5'. These are hand-chosen, not derived or learned from data. The plan-selection results (Figure 10) depend on them.
  • Post-filter K_ann selection rule (2k at 50% selectivity) = 2k for estimated 50% selectivity; 20% yield threshold
    Section 6.2: 'adaptively sets K_ann based on the predicate selectivity estimated by the optimizer (e.g., retrieving 2k candidates for an estimated 50% selectivity)' and configurable fallback threshold defaulting to 20% of k. These are heuristic constants without derivation.
  • IVF/PQ index parameters (nlist, nbits, subspaces) for each dataset = e.g., SIFT1B: 3000 clusters, 32 subspaces, 8-bit; Wikipedia: 6000 clusters, 192 subspaces, 8-bit
    Experiment settings (Section 7.1). These are chosen by hand during benchmark setup and affect all recall/latency numbers. They are standard tuning parameters, but the specific values are not derived from any principle.
  • DPPQ epochs E and subspaces M = E=2-5 in convergence figures; various M
    Algorithm 1 takes epoch count E and subspace count M as inputs; Figure 12 scans them. These are free parameters tuned per dataset and affect the reported recall improvements.
axioms (5)
  • domain assumption A global index refreshed at most once per day with eventual consistency is adequate for the target workloads.
    Section 4.2: 'adopts an eventual consistency model... This design is well-suited for analytical workloads with infrequent updates (days/weeks).' If update rates are higher, the claimed production applicability breaks. This is acknowledged as an assumption.
  • domain assumption IVFPQ-based search with PQ-style quantization can be decomposed into relational operators without materialization that destroys the gains.
    Section 5.1. The relational execution plan (CROSS_JOIN of LUT with quantized vectors) could in principle create massive intermediate results; the paper's contribution is claiming the fused operator avoids this. This is the core technical bet, unproven outside this implementation.
  • domain assumption The reported latency-recall points are comparable across systems.
    Section 7.1: 'All evaluated systems are open-source and compared at medium recall levels (0.8-0.9)'. But Figures 7-9 compare at slightly different recall points, and the 145x speedup specifically compares at a point where competitors failed (Milvus OOM on SIFT1B). If the comparison protocol is unfair, the speedup numbers are misleading.
  • domain assumption The cost model's plan-cost formulas are accurate predictors of real distributed execution cost.
    Section 6.2 Table 3. The cost formulas treat CPU cost as simple arithmetic (e.g., Mp*C_dist) and ignore memory-bandwidth contention, stragglers, and disk I/O for index blocks. No validation of the cost model's prediction accuracy against actual measured latencies.
  • standard math Standard relational algebra and query optimization principles (Cascades, Volcano, hash joins, runtime filters) apply to the new index-table scans.
    The paper relies on conventional database theory: vectorized execution (MonetDB/X100), Cascades/Volcano optimization frameworks, and hash-join correctness. These are standard and not ad hoc.
invented entities (2)
  • Global vector index tables (IVF centroid table, PQ codebook table, quantized vector table) independent evidence
    purpose: New storage substrate: materializing a global index as relational tables to enable native OLAP optimizations and eliminate scatter-gather.
    The index tables are a concrete storage design that could be used and measured by anyone implementing the described architecture. Their benefit is backed by the experimental claims, though code is not shipped to verify.
  • DPPQ (Directional Progressive Product Quantization) independent evidence
    purpose: A new quantization scheme combining direction-aware quantization and hierarchical residual refinement to improve recall under equal bit budget.
    DPPQ can be reproduced from Algorithm 1 and tested on public datasets (GloVe, SIFT, GIST). No external code is provided in this submission, so independent evidence is only theoretical.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of TEngineDB-V: An OLAP-Native Vector Search System for Large-$k$ Workloads at Tencent." pith.science (2026). https://pith.science/paper/P7Q2D2G7

@misc{pith2026260800650,
  author       = {Pith},
  title        = {Pith review of: TEngineDB-V: An OLAP-Native Vector Search System for Large-$k$ Workloads at Tencent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P7Q2D2G7}},
  note         = {Machine review of arXiv:2608.00650}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Vector search systems are essential infrastructure for modern data-driven applications. Large-$k$ analytical vector search, which retrieves $k=10^3$--$10^5$ results for analytics (e.g., aggregation, filtering, joins), is increasingly important for emerging workloads, including LLM data management and advertising analysis at Tencent. Existing systems remain inadequate: specialized vector databases often cap $k$ (e.g., $k \leq 10^4$) to satisfy tail-latency constraints and offer limited analytical support, while OLAP systems typically embed per-segment vector indexes as black boxes, causing severe read/compute amplification and preventing native query optimization. This paper presents TEngineDB-V, an OLAP-native vector search system for large-$k$ workloads. TEngineDB-V makes vector search a first-class analytical primitive in Tencent's OLAP engine through a global segment-decoupled index materialized as relational tables, eliminating scatter-gather execution, reducing amplification, and enabling native storage optimizations. It decomposes IVFPQ-based search into relational operators, integrates OLAP optimizations, and introduces DPPQ, which combines direction-aware quantization with hierarchical residual refinement to improve recall while preserving relational efficiency. TEngineDB-V further incorporates index-aware query rewriting and a distributed-aware cost model for efficient distributed execution. Experiments show that TEngineDB-V achieves up to a $145\times$ speedup over competitive systems such as StarRocks, and up to a $52\times$ improvement in 10-billion-scale production deployments.

Figures

Figures reproduced from arXiv: 2608.00650 by Anqi Liang, Fan Wu, Guangxu Cheng, Guoliang Li, Jijun Du, Kai Wang, Peng Chen, Pengcheng Zhang, Xiaobo Zhang, Xuanhe Zhou, Xufei Wu, Yidi Xiong, Yitong Song, Zhe Chen.

Figure 1
Figure 1. Figure 1: QPS comparison of vector indexes under varying [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: The Architecture of TEngineDB-V. IVFPQ uses 𝑘-means clustering [25] to partition vectors into 𝑘 clusters and searches the top-𝑁𝑝𝑟𝑜𝑏𝑒 clusters closest to the query. To further accelerate distance computation within each cluster, Product Quantization (PQ) is applied to encode the vectors. The 𝑑-dimensional vector is partitioned into 𝑚 disjoint subspaces, each of dimension 𝑑/𝑚. For each subspace, a separate c… view at source ↗
Figure 4
Figure 4. Figure 4: Storage architectural comparison. and enable SIMD-friendly computation. Sharding by CID further ensures that only relevant shards are accessed during IVF probing. Native Optimizations. Our table-centric design enables the global index to fully leverage native engine optimizations along the stor￾age and execution pipeline. (1) Columnar Compression. Since index tables are stored in columnar format, they auto… view at source ↗
Figure 5
Figure 5. Figure 5: Relational execution in TEngineDB-V. quantization approach with strong empirical performance, but its processing relies on complex operations, including per-cluster nor￾malization, matrix multiplication, and bitwise operations, which are not naturally adaptable to standard relational operators and may be less efficient when adapted to relational execution. Decomposing IVFPQ Search to Relational Operators. … view at source ↗
Figure 6
Figure 6. Figure 6: Execution Plans for FANNS. 𝑤net = 1.5, assigning higher penalties to memory pressure and network communication in distributed environments. Recall that we classify the eight candidate FANNS plans along three orthogonal dimensions. For each dimension, we then analyze the corresponding CPU, memory, and network bottlenecks, with the overall cost breakdown summarized in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Latency–Recall Trade-off Across Systems. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Latency Scalability with Increasing 𝐾. DiskANN utilize the IVFPQ index. For the SIFT1B dataset, the data space is partitioned into 3000 clusters, and vectors are compressed using 32 subspaces with 8-bit quantization per subspace. For the Wikipedia dataset, the data is divided into 6000 clusters, utilizing 192 subspaces with 8-bit quantization. For DiskANN, the graph degree is set to 48, the build list size… view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of different quantization algorithms. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance of the eight FANNS execution plans [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Convergence analysis of DPPQ. 0.80 0.82 0.84 0.86 0.88 recall@20, 000 10 3 10 4 10 5 L a t e n cy(m s) Tencent-Image TEngineDB-V Legacy TEngineDB StarRocks Milvus [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Production performance at 10-billion scale. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 3 linked inside Pith

  1. [1]

    Meta AI. 2017. FAISS. https://ai.facebook.com/tools/faiss

    2017

  2. [2]

    Fabien André, Anne-Marie Kermarrec, and Nicolas Le Scouarnec. 2016. Cache locality is not enough: High-performance nearest neighbor search with product quantization fast scan. InPVLDB, Vol. 9. 12

    2016

  3. [3]

    Fabien André, Anne-Marie Kermarrec, and Nicolas Le Scouarnec. 2019. Quicker adc: Unlocking the hidden potential of product quantization with simd.IEEE TPAMI43, 5 (2019), 1666–1677

    2019

  4. [4]

    Apache Doris Community. 2025. Vector Search in Apache Doris. https://doris. apache.org/zh-CN/docs/4.x/ai/vector-search/overview/. Accessed: 2025-12

    2025

  5. [5]

    Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. 2020. ANN- Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems87 (2020), 101374

    2020

  6. [6]

    Peter A Boncz, Marcin Zukowski, and Niels Nes. 2005. MonetDB/X100: Hyper- Pipelining Query Execution.. InCidr, Vol. 5. 225–237

    2005

  7. [7]

    Andrei Broder and Michael Mitzenmacher. 2004. Network applications of bloom filters: A survey.Internet mathematics1, 4 (2004), 485–509

    2004

  8. [8]

    Cheng Chen, Chenzhe Jin, Yunan Zhang, Sasha Podolsky, Chun Wu, Szu- Po Wang, Eric Hanson, Zhou Sun, Robert Walzer, and Jianguo Wang. 2024. Singlestore-v: An integrated vector database system in singlestore.PVLDB17, 12 (2024)

    2024

  9. [9]

    2018.SPTAG: A library for fast approximate nearest neighbor search

    Qi Chen, Haidong Wang, Mingqin Li, Gang Ren, Scarlett Li, Jeffery Zhu, Jason Li, Chuanjie Liu, Lintao Zhang, and Jingdong Wang. 2018.SPTAG: A library for fast approximate nearest neighbor search. Microsoft Research, GitHub repository: https://github.com/Microsoft/SPTAG

    2018

  10. [10]

    Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. 2021. SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search.arXiv preprint arXiv:2111.08566(2021)

    Pith/arXiv arXiv 2021

  11. [11]

    Xiaoyu Chen, Jinxiu Qu, Yitong Song, Shuhang Lu, Huiling Li, Minghui Jiang, Wei Zhou, Jianliang Xu, Xuanhe Zhou, and Fan Wu. 2026. Disk-Resident Graph ANN Search: An Experimental Evaluation.arXiv preprint arXiv:2603.01779(2026)

    arXiv 2026

  12. [12]

    Yanqi Chen, Xiao Yan, Alexandra Meliou, and Eric Lo. 2025. DiskJoin: Large-scale Vector Similarity Join with SSD.Proceedings of the ACM on Management of Data 3, 6 (2025), 1–27

    2025

  13. [13]

    ClickHouse, Inc. 2024. Approximate Nearest Neighbor Search with Vector Simi- larity Indexes. https://clickhouse.com/docs/en/engines/table-engines/mergetree- family/annindexes. Accessed: 2025-12

    2024

  14. [14]

    Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, et al. 2016. The snowflake elastic data warehouse. InProceed- ings of the 2016 International Conference on Management of Data. 215–226

    2016

  15. [15]

    Elastic. 2023. kNN Search and Vector Fields in Elasticsearch. https://www. elastic.co/guide/en/elasticsearch/reference/current/knn-search.html. Accessed: 2025-12

    2023

  16. [16]

    Cong Fu, Chao Xiang, Changxu Wang, and Deng Cai. 2019. Fast approximate nearest neighbor search with the navigating spreading-out graph. InPVLDB, Vol. 12. VLDB Endowment, 416–474

    2019

  17. [17]

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 2023. Datacomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing Systems36 (2023), 27092– 27112

    2023

  18. [18]

    Jianyang Gao, Yutong Gou, Yuexuan Xu, Yongyi Yang, Cheng Long, and Raymond Chi-Wing Wong. 2025. Practical and asymptotically optimal quantization of high-dimensional vectors in euclidean space for approximate nearest neighbor search.ACM SIGMOD3, 3 (2025), 1–26

    2025

  19. [19]

    Jianyang Gao and Cheng Long. 2024. RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search.ACM SIGMOD2, 3 (2024), 1–27

    2024

  20. [20]

    Goetz Graefe et al. 1995. The cascades framework for query optimization.IEEE Data Eng. Bull.18, 3 (1995), 19–29

    1995

  21. [21]

    Goetz Graefe and William J McKenna. 1993. The volcano optimizer genera- tor: Extensibility and efficient search. InProceedings of IEEE 9th international conference on data engineering. IEEE, 209–218

    1993

  22. [22]

    Yuxing Han, Yu Lin, Yifeng Dong, Xuanhe Zhou, Xindong Peng, Xinhui Tian, Zhiyuan You, Yingzhong Guo, Xi Chen, Weiping Qu, et al. 2026. ByteHouse: A Cloud-Native OLAP Engine with Incremental Computation and Multi-Modal Retrieval.ACM SIGMOD(2026)

    2026

  23. [23]

    Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. 2019. Diskann: Fast accurate billion-point nearest neighbor search on a single node.Advances in Neural Information Processing Systems32 (2019)

    2019

  24. [24]

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search.IEEE TPAMI33, 1 (2010), 117–128

    2010

  25. [25]

    K Krishna and M Narasimha Murty. 1999. Genetic K-means algorithm.IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)29, 3 (1999), 433–439

    1999

  26. [26]

    Hai Lan, Shixun Huang, Zhifeng Bao, and Renata Borovica-Gajic. 2024. Cardi- nality estimation for similarity search on high-dimensional data objects: The impact of reference objects.Proceedings of the VLDB Endowment18, 3 (2024), 544–556

    2024

  27. [27]

    Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. 2019. Approximate nearest neighbor search on high dimensional data—experiments, analyses, and improvement.TKDE32, 8 (2019), 1475–1488

    2019

  28. [28]

    Anqi Liang, Pengcheng Zhang, Bin Yao, Zhongpu Chen, Yitong Song, and Guangxu Cheng. 2024. UNIFY: Unified Index for Range Filtered Approximate Nearest Neighbors Search.Proceedings of the VLDB Endowment18, 4 (2024), 1118–1130

    2024

  29. [29]

    Guy M Lohman. 1988. R* optimizer validation and performance evaluation for distributed queries.Readings in Database Systems(1988), 219

    1988

  30. [30]

    Youzhong Ma, Shijie Jia, and Yongxin Zhang. 2017. A novel approach for high- dimensional vector similarity join query.Concurrency and Computation: Practice and Experience29, 5 (2017), e3952

    2017

  31. [31]

    Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.IEEE TPAMI42, 4 (2018), 824–836

    2018

  32. [32]

    Zhaojie Niu, Xinhui Tian, Xindong Peng, and Xing Chen. 2025. BlendHouse: A cloud-native vector database system in ByteHouse. InIEEE ICDE. IEEE, 4332– 4345

    2025

  33. [33]

    Yun Peng, Byron Choi, Tsz Nam Chan, Jianye Yang, and Jianliang Xu. 2023. Efficient Approximate Nearest Neighbor Search in Multi-dimensional Databases. ACM SIGMOD1, 1 (2023), 1–27

    2023

  34. [34]

    pgvector authors. 2023. pgvector: Open-source vector similarity search for Postgres. https://github.com/pgvector/pgvector. Accessed: 2025-12

    2023

  35. [35]

    2021.Pinecone.io

    Pinecone. 2021.Pinecone.io. https://www.pinecone.io/

    2021

  36. [36]

    2021.qdrant.tech

    Qdrant. 2021.qdrant.tech. https://qdrant.tech/

    2021

  37. [37]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al

  38. [38]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen

  39. [39]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695

    2022

  40. [40]

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 22500–22510

    2023

  41. [41]

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems35 (2022), 36479–36494

    2022

  42. [42]

    Mohamed A Soliman, Lyublena Antova, Venkatesh Raghavan, Amr El-Helw, Zhongxian Gu, Entong Shen, George C Caragea, Carlos Garcia-Alvarado, Foyzur Rahman, Michalis Petropoulos, et al . 2014. Orca: a modular query optimizer architecture for big data. InProceedings of the 2014 ACM SIGMOD international conference on Management of data. 337–348

    2014

  43. [43]

    Yitong Song, Kai Wang, Bin Yao, Zhida Chen, Jiong Xie, and Feifei Li. 2024. Effi- cient Reverse𝑘 Approximate Nearest Neighbor Search Over High-Dimensional Vectors. In2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 4262–4274

    2024

  44. [44]

    Yitong Song, Bin Yao, Zhida Chen, Xin Yang, Jiong Xie, Feifei Li, and Mengshi Chen. 2025. Efficient top-k spatial-range-constrained approximate nearest neigh- bor search on geo-tagged high-dimensional vectors.The VLDB Journal34, 1 (2025), 14

    2025

  45. [45]

    Yitong Song, Pengcheng Zhang, Chao Gao, Bin Yao, Kai Wang, Zongyuan Wu, and Lin Qu. 2025. TRIM: Accelerating High-Dimensional Vector Similarity Search with Enhanced Triangle-Inequality-Based Pruning.ACM SIGMOD3, 6 (2025), 1–26

    2025

  46. [46]

    Yitong Song, Xuanhe Zhou, Christian S Jensen, and Jianliang Xu. 2026. Vector Search for the Future: From Memory-Resident, Static Heterogeneous Storage, to Cloud-Native Architectures. (2026), 580–587

    2026

  47. [47]

    StarRocks Community. 2024. Vector Index in StarRocks. https://docs.starrocks. io/docs/using_starrocks/vector-index/. Accessed: 2025-12

    2024

  48. [48]

    Ji Sun, Guoliang Li, James Pan, Jiang Wang, Yongqing Xie, Ruicheng Liu, and Wen Nie. 2025. GaussDB-Vector: A Large-Scale Persistent Real-Time Vector Database for LLM Applications.PVLDB18, 12 (2025), 4951–4963

    2025

  49. [49]

    The Apache Software Foundation. 2025. Apache Arrow: A cross-language de- velopment platform for in-memory data. https://arrow.apache.org/. Accessed: 2025-10-24

    2025

  50. [50]

    Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xi- angyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al. 2021. Milvus: A purpose-built vector data management system. InACM SIGMOD. 2614–2627

    2021

  51. [51]

    Mengzhao Wang, Lingwei Lv, Xiaoliang Xu, Yuxiang Wang, Qiang Yue, and Jiongkang Ni. 2022. Navigable Proximity Graph-Driven Native Hybrid Queries with Structured and Unstructured Constraints.arXiv preprint arXiv:2203.13601 (2022)

    Pith/arXiv arXiv 2022

  52. [52]

    Mengzhao Wang, Weizhi Xu, Xiaomeng Yi, Songlin Wu, Zhangyang Peng, Xi- angyu Ke, Yunjun Gao, Xiaoliang Xu, Rentong Guo, and Charles Xie. 2024. Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High- Dimensional Vector Similarity Search on Data Segment.ACM SIGMOD2, 1 (2024), 1–27

    2024

  53. [53]

    2019.Weaviate.io

    Weaviate. 2019.Weaviate.io. https://weaviate.io/developers/weaviate/concepts/ vector-index

    2019

  54. [54]

    Chuangxian Wei, Bin Wu, Sheng Wang, Renjie Lou, Chaoqun Zhan, Feifei Li, and Yuanzhe Cai. 2020. Analyticdb-v: A hybrid analytical engine towards query fusion for structured and unstructured data.PVLDB13, 12 (2020), 3152–3165

    2020

  55. [55]

    Jiadong Xie, Jeffrey Xu Yu, and Yingfan Liu. 2025. Fast approximate similarity join in vector databases.Proceedings of the ACM on Management of Data3, 3 (2025), 1–26

    2025

  56. [56]

    Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster computing with working sets. In2nd USENIX workshop on hot topics in cloud computing (HotCloud 10)

    2010

  57. [57]

    Yunan Zhang, Shige Liu, and Jianguo Wang. 2024. Are there fundamental limitations in supporting vector data management in relational databases? A case study of PostgreSQL. InIEEE ICDE. IEEE

    2024

  58. [58]

    Chaoji Zuo, Miao Qiao, Wenchao Zhou, Feifei Li, and Dong Deng. 2024. SeRF: Segment Graph for Range-Filtering Approximate Nearest Neighbor Search.ACM SIGMOD2, 1 (2024), 1–26

    2024

  59. [2021]

    In International conference on machine learning

    Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763

  60. [2022]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.061251, 2 (2022), 3

    Pith/arXiv arXiv 2022

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.