{"id":"db44e1f3-f447-497e-b890-e562b0dc66ec","arxiv_id":"2501.05006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CHASE natively integrates ANN vector search into a compiled relational engine, cuts redundant similarity computation in hybrid queries, and reports up to 7,500x speedups.","lead":"This paper describes CHASE, a database query engine that combines relational filtering with vector similarity search and compiles hybrid queries into machine code. The authors report speedups over existing vector database systems ranging from 17% to 7,500x on a million-image benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 7,500x Q4 speedup is not apples-to-apples: CHASE runs an approximate ANN scan at recall 0.95 while baselines are exact, and the timing protocol excludes planning and compilation, so the magnitude of the central performance claim is not established.","rationale":"The reader's weakest assumption was that measurement choices in Section 7.1—excluded compilation time and unequal recall—undermine the headline speedups. My review reaches the same conclusion and sharpens it: the most dramatic number, the 7,500x Q4 speedup, is specifically the case where CHASE is approximate (recall 0.95) and baselines are exact, and where compilation is not charged. That is a load-bearing concern for the abstract's central claim. It is not a fundamental contradiction in the system design: the plan rewrites, operator optimizations, and LingoDB-based compilation are plausible and partially grounded in prior work, and the paper honestly discloses the timing protocol and recall values. The right remedy is conditional acceptance with the evaluation redone under matched recall and end-to-end timing, rather than rejection. I therefore keep the reader's CONDITIONAL verdict unchanged. External support from LingoDB and VBASE is acknowledged; the issue is measurement fairness, not circular derivation or fitted predictions.","tokens_in":21347,"tokens_out":3402,"duration_ms":37741,"concrete_test":"Re-run Q4 under an end-to-end protocol: charge CHASE's query compilation and planning time in every latency measurement, and report recall for both sides at matched quality. Concretely, vary CHASE's HNSW ef_search until its recall reaches 1.0 against the same ground truth used for the exact baselines, then compare latency; separately, if approximate execution is allowed, run pgvector or PASE with an HNSW configuration that reaches CHASE's reported recall 0.95 and compare. If the CHASE advantage drops below 2x, or if compilation time dominates the 26 ms figure, the 7,500x headline overstates the benefit attributable to the native design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the performance advantage of native hybrid-query design, capped by the 'extraordinary 7500x' result for Q4. That result rests on two measurement asymmetries. First, Section 7.1 states that execution time explicitly excludes planning, data loading, and compilation time. Compilation is one of CHASE's three stated core techniques (Section 3, Section 6), so for one-shot queries the omitted compilation latency is part of the real cost of CHASE's approach and is not charged to any baseline. Second, the Q4 comparison mixes approximate and exact execution: Section 7.3.3 reports CHASE recall 0.95, while baselines that cannot use an ANN index are described as exact with recall represented as '-' in Table 7. The 7,500x ratio is therefore partly the speedup from replacing an exact 100-million-pair brute-force join with an approximate index scan that resolves a rewritten per-user top-k query, not purely from native plan rewriting and code generation. The same issue appears in Q1: Table 3 shows pgvector recall 0.52 at selectivity 0.03 while CHASE has recall 1.0, despite the text claiming recall was aligned by adjusting ef_search. If compilation is charged and recall is equalized, the headline speedups, especially 7,500x, could shrink by orders of magnitude; the direction of the comparison may survive, but the central quantitative claim would need substantial revision. No code or data artifact is provided (the PVLDB artifact URL is a placeholder), so the reported timings and recall values cannot currently be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CHASE, a query engine for hybrid SQL queries over relational attributes and high-dimensional vectors. It identifies six hybrid-query templates, rewrites logical plans using new map and updateState operators, implements optimized index-scan and range-search physical operators, and compiles query plans to machine code via the LingoDB framework. On the LAION-1M dataset, it reports execution-time speedups of 17%–33% over VBASE for Q1–Q2, roughly 64% for Q3, 7,500x for Q4, 33%–46% for Q5, and 3.1x–4.04x for Q6, with recall values around 0.93–1.0. The central claim is that treating vectors as first-class citizens throughout planning, operators, and compilation yields substantial performance gains over existing hybrid-query systems.","tokens_in":21658,"tokens_out":7028,"duration_ms":69354,"significance":"If the reported results were obtained under fair, equal-recall, end-to-end comparisons, the paper would be a meaningful contribution: it gives a concrete taxonomy of hybrid queries, identifies plan-level redundancies in existing engines, and proposes reusable operators and a compilation path for vector workloads. The plan rewrites in Figures 4–6 and the operator algorithms in Section 5 are concrete and useful, and the paper explicitly credits VBASE for the range-search algorithm and LingoDB for the compilation framework. However, the central quantitative claims are not yet established: the evaluation excludes compilation time, the headline Q4 speedup mixes approximate CHASE execution with exact baselines, and the Q1 recall-alignment claim is contradicted by Table 3. The contribution is promising but needs substantial experimental revision before publication.","major_comments":[{"comment":"Section 7.1 states that execution time excludes planning, data loading, and compilation time, but compilation is one of the three core techniques claimed in Sections 3 and 6. For one-shot queries, CHASE's per-query latency includes compilation, and none of the baselines is charged for it in the reported numbers. The speedup magnitudes, especially the 7,500x Q4 figure, are therefore not end-to-end. Please report end-to-end wall-clock times or, at minimum, measure and report compilation time separately and show how the speedups change when that cost is included.","section":"7.1 Metric; 6 Code Generation"},{"comment":"The Q4 comparison is not apples-to-apples: CHASE uses an approximate ANN scan with recall 0.95, while pgvector, PASE, VBASE, and LingoDB-V are described as exact baselines whose recall is shown as '-' in Tables 6 and 7. The 7,500x speedup therefore includes the effect of replacing exact matching with approximate search, not only the effect of native plan rewriting and code generation. The claimed reduction from 100 million to 94,000 similarity computations is a property of ANN approximation and should be separated from the system-design contribution. Please report an exact CHASE variant or an equal-recall comparison (for example, raising HNSW ef_search until CHASE matches exact recall) before attributing the speedup to native design.","section":"7.3.3, Table 7"},{"comment":"The text in Section 7.3.1 says that to align recall across all systems, the authors manually adjust the ef_search parameter for pgvector and PASE, but Table 3 shows pgvector with recall 0.52 at selectivity 0.03 while CHASE, PASE, and VBASE report recall 0.98–1.0. This contradiction means recall was not equalized in the reported runs, so the Q1 speedup at low selectivity is not a like-for-like comparison. Please re-run with matched recall, disclose the ef_search values used, or report results at several explicit recall levels.","section":"7.3.1, Table 3"},{"comment":"The correctness of the updateState operator and the R1-to-R2 range reduction rests on the uniform-distribution assumption stated in Section 4.3, but no recall or result-equivalence measurement is reported for the updateState experiments at thresholds 0.7, 0.6, and 0.5: Figure 9 shows only execution time. Since the operator's purpose is to shrink the scanned range without changing query results, the paper must verify on real data that the reduced range returns the same top-K results per category, or report the recall at each threshold when the updateState operator is active.","section":"4.3, 5.3, Figure 9"},{"comment":"The artifact block contains the placeholder 'URL_TO_YOUR_ARTIFACTS,' so no code, data, or runnable system is actually provided for the empirical claims. In addition, the timing tables report only mean execution times without variance or a per-query breakdown. Given that the measurement protocol has several asymmetries, the absence of artifacts and error statistics makes independent verification impossible. Please provide a working artifact and include standard deviations or full result distributions for the reported execution times.","section":"PVLDB Artifact Availability"}],"minor_comments":[{"comment":"The Abstract reports speedups of '13% to an extraordinary 7500 times,' while the Introduction reports '17% to 7,500×'; please make these consistent.","section":"Abstract; Section 1"},{"comment":"The caption contains the typo 'query qlans'; it should read 'query plans.'","section":"Figure 1 caption"},{"comment":"The pseudocode in Algorithm 1 has a dangling else after the nested if blocks (lines 3–12); please re-indent so the intended control flow is unambiguous.","section":"5.2, Algorithm 1"},{"comment":"The notation 'filteredK c, queuec' appears to contain subscripting artifacts; please typeset the variables consistently.","section":"5.3, Algorithm 2"},{"comment":"Table 5's header reads 'Branches misses' instead of 'Branch misses,' and the branch-miss-rate row would be clearer if it were expressed as a percentage with consistent decimal places.","section":"7.3.1, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper targets a real and increasingly important problem, and the proposed optimizations are plausible, but the evaluation as written substantially overstates the central speedup claim. The 7,500x result compares approximate ANN execution against exact baselines, compilation time is excluded even though compilation is a core contribution, and the Q1 recall-alignment claim is internally inconsistent. I would consider acceptance after the authors rerun the experiments with equal recall and end-to-end timing and provide a working artifact. The current evidence does not support the headline quantitative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The CHASE system is a genuine attempt to treat vectors as first-class citizens across planning, operator implementation, and code generation. The query taxonomy (VKNN-SF, DR-SF, W-VKNN-SF) is useful and the plan rewrites are clearly explained. The map operator for score reuse and the updateState operator for early range termination are sensible extensions of VBASE's relaxed monotonicity, and building on LingoDB for MLIR/LLVM compilation is a reasonable way to bring vector operations into a compiled engine. The design is coherent and the paper does a good job of showing where existing systems waste computation.\n\nThe soft spot is the evaluation, and it touches the central claim. The headline 7,500x speedup for Q4 is not apples-to-apples: CHASE runs an approximate ANN scan at recall 0.95 while the baselines do an exact brute-force join (their recall is reported as '-', meaning 1.0). Part of that speedup is simply the cost of approximation. Compilation time is explicitly excluded from execution time even though compilation is one of the paper's three core techniques; for one-shot queries that omission flatters CHASE. Table 3 also shows pgvector recall 0.52 at selectivity 0.03 while CHASE has recall 1.0, despite the text claiming recall was aligned by adjusting ef_search—that discrepancy needs an explanation. There are no error bars and the artifact link is a placeholder, so the numbers cannot be independently verified right now.\n\nThese are addressable weaknesses, not fatal ones. The design is plausible and the optimizations are sound in principle. But the central quantitative claim is not established as stated. A revision that reports end-to-end time including compilation, equalizes recall across all comparisons, publishes the artifact, and adds variance would turn this into a solid systems paper.\n\nThis is worth a serious referee: the problem is important, the design is thoughtful, and the ideas merit careful review. I would not desk-reject it, but reviewers should push hard on the evaluation methodology and require the artifact before publication.","headline":"A coherent native hybrid-query engine with useful optimizations, but the headline speedup rests on approximate-versus-exact comparison and excluded compilation time.","tokens_in":22213,"tokens_out":2978,"would_cite":true,"duration_ms":28925,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CHASE argues that a database purpose-built for hybrid structured-and-vector queries, with plan rewriting, score-reusing operators, and machine-code compilation, outperforms plugin-style vector search by 13% to 7,500x.","keywords":["hybrid queries","approximate nearest neighbor","vector search","query compilation","logical plan rewriting","relational database","structured and unstructured data","ANN index"],"falsifier":"Run the six hybrid queries end-to-end on a cold process, including planning and compilation time, with every baseline's recall tuned to match CHASE's; if the 7,500x gap for the KNN-join query collapses or reverses, or if the range-query baselines catch up once recall is equalized, the central performance claim fails.","tokens_in":21109,"feed_emoji":"⚡","tokens_out":7775,"duration_ms":68381,"temperature":0.7,"pith_summary":"The paper argues that today's databases treat vector similarity as an add-on: they bolt an approximate-nearest-neighbor index onto the scan operator and leave the rest of the query plan untouched. CHASE instead makes the vector a first-class citizen across the whole pipeline, rewriting logical plans, adding operators that reuse similarity scores, and compiling the final plan to machine code. The claim is that this end-to-end approach removes redundant distance computations and unlocks ANN acceleration for query shapes that today fall back to brute force. On six hybrid query templates over a million-row real dataset, the reported effect ranges from 13% to 7,500x speedups over existing systems.","feed_headline":"Vectors as first-class citizens speed hybrid queries 7,500x","feed_subtitle":"A plan-rewriting database reuses similarity scores and compiles to machine code, beating plugin-style vector search.","key_machinery":"The load-bearing device is the rewritten hybrid query plan itself. For KNN-like queries, a map operator materializes the similarity scores computed by the ANN index scan into a 'sim' column that the sort and limit operators reuse instead of recomputing; for window-KNN queries, the ORDER BY is decoupled from the window and a limit is inserted to form an index-accelerated VKNN-SF sub-query per partition; for range and category queries, a range-search index scan and an updateState operator track per-category completion and stop the scan early. The same rewritten plan is lowered through intermediate dialects to machine code, so the plan itself, not just operator internals, carries the optimization.","core_discovery":"CHASE's central claim is that hybrid query performance is governed not by the ANN index alone but by the interaction between the index scan and the rest of the query plan. For top-k queries, the ANN scan already computes similarity scores that later sort operators recompute; CHASE introduces a map operator that carries those scores into a temporary column so sorting and limiting reuse them. For window-KNN joins, rewriting the plan turns a brute-force join into per-group ANN probes. For range and category-partitioned queries, a range-search index scan plus an updateState operator tracks per-category completion and terminates the traversal early without scanning the full threshold range. These plan-level and operator-level changes are then compiled into machine code, so the speedups come from removing redundant computation at every stage rather than from any single index trick.","pith_inferences":["Editorial inference: the paper reports execution time after planning, loading, and compilation are complete, so the 7,500x figure is an execution-only bound; charging compile time would shrink the one-shot speedup.","Editorial inference: the map and updateState rewriting patterns sit above a generic ANN index interface, so the same plan rewriting could be applied on top of other approximate-nearest-neighbor algorithms, not only the graph index used in the experiments.","Editorial inference: a benchmark that equalizes recall across all baselines, including those reported as exact, would be the decisive test of whether the remaining speedups reflect the plan rewriting rather than accuracy differences.","Editorial inference: if similarity scores are cached as a virtual column, then repeated or incremental hybrid queries could reuse prior scores, a direction the paper does not explore."],"forward_implications":["If CHASE is right, vector-extended databases can support range and window-KNN hybrid queries without falling back to brute-force scans, shrinking entity-centric join cost from $O(|A|\\cdot|B|)$ to roughly $O(C\\cdot|A|)$.","Similarity scores become reusable data rather than transient distance computations, so sorting, ranking, and window operations no longer recompute distances.","Compiled execution of vector plans cuts branch misses, cache misses, and instruction counts relative to iterator-model engines, so the gains should persist across different hardware.","The query classification (vector-KNN with filters, distance-range with filters, windowed vector-KNN with filters) gives database designers a template for deciding which rewriting applies to which SQL shape."],"supporting_citations":[{"why":"Supplies the relaxed-monotonicity retrieval and range-search algorithm that CHASE adapts for its index scan; also a main accuracy-matched baseline.","marker":"[43]"},{"why":"Represents the plugin-style ANN index scan with a conservatively large K' that CHASE argues wastes computation.","marker":"[41]"},{"why":"Represents a simple plugin-style vector extension used as a baseline for top-k and join queries.","marker":"[31]"},{"why":"Represents a plugin-style analytical engine in the class of systems CHASE claims are suboptimal for hybrid queries.","marker":"[38]"},{"why":"Provides the code-generation framework that CHASE extends to compile rewritten plans to machine code.","marker":"[14, 15]"},{"why":"Documents iterator-model interpretation overhead, motivating the shift to compiled query plans.","marker":"[26]"},{"why":"Provides the graph-based ANN index used for similarity search in the experiments.","marker":"[25]"}],"fun_headline_variants":["CHASE rewrites hybrid query plans to skip redundant similarity scores","Native database engine compiles hybrid queries to machine code, up to 7500x","Plan rewriting and score reuse give CHASE 7500x speedups on hybrid queries","Hybrid queries get 7500x speedup when database plans reuse ANN scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported speedups assume that measuring only execution time after planning and compilation are complete is the right comparison, and that the baselines are run at comparable recall.","fun_headline_variants_meta":{"raw":{"variants":["CHASE rewrites hybrid query plans to skip redundant similarity scores","Native database engine compiles hybrid queries to machine code, up to 7500x","Plan rewriting and score reuse give CHASE 7500x speedups on hybrid queries","Hybrid queries get 7500x speedup when database plans reuse ANN scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000438,"raw_usage":{"total_tokens":2216,"prompt_tokens":928,"completion_tokens":1288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":1203}},"tokens_in":544,"tokens_out":1288,"duration_ms":9905,"temperature":1.0,"reasoning_tokens":1203,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:20:26.939229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the six hybrid queries end-to-end on a cold process, including planning and compilation time, with every baseline's recall tuned to match CHASE's; if the 7,500x gap for the KNN-join query collapses or reverses, or if the range-query baselines catch up once recall is equalized, the central performance claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the relaxed-monotonicity retrieval and range-search algorithm that CHASE adapts for its index scan; also a main accuracy-matched baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents a simple plugin-style vector extension used as a baseline for top-k and join queries."}],"review_version":1}