{"id":"2d93f8fe-6490-4252-8c30-d188b42907b8","arxiv_id":"2608.03199","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SieveIVF accelerates embedding deduplication by stopping each IVF query after W consecutive threshold-empty partitions, preserving batching with continuous regrouping and a lookahead scheduler.","lead":"SieveIVF is a new execution strategy for IVF vector search that stops probing a query after a run of partitions with no similar neighbors above a threshold. On four Tencent and two public benchmarks it is several times faster than fixed-probe IVF, with a small recall loss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stopping-rule evidence is about last-hit rank, not the consecutive-empty gaps that actually trigger W-stops; a direct gap-distribution check is needed.","rationale":"The reader's weakest assumption is the front-loading property: threshold-qualified neighbors appear early enough that W consecutive empty results signal leaving the productive region. I agree this is the load-bearing empirical assumption, but the paper's Section 3.3 measurement (last productive rank) does not directly quantify the consecutive-empty-run lengths that the stopping rule in Section 4.1 actually uses. A query with a small last rank can still have a large internal gap, causing the rule to stop before a later hit; conversely, a query with a large last rank but dense hits might not be harmed. Therefore the workload characterization is incomplete as evidence for the mechanism. The final recall numbers (Section 6.2) do validate the tradeoff on the evaluated workloads, so this is not a rejection: it is a missing diagnostic that would make the central claim more robust. The reader's CONDITIONAL verdict already flags reproducibility gaps (no code, no error bars), and my concern adds a specific analytical gap but does not change the verdict. I set verdict_should_be UNCHANGED because the paper is otherwise transparent, includes a matched-recall comparison, and the disclosed recall losses are part of the claim. If the requested gap-distribution test were run and showed large tail gaps, I would move toward CONDITIONAL with a stricter requirement; if it showed W safely above the gap tail, the concern is settled. The agreement is partial because the reader identified the broad assumption but not the mismatch between the measured metric (last rank) and the rule's trigger (consecutive empty runs).","tokens_in":17154,"tokens_out":15813,"duration_ms":197404,"concrete_test":"For each query in each workload, using exact ground-truth neighbors and stored exact-centroid partitions, compute the ordered list of probe ranks containing at least one threshold-qualified neighbor and the gaps between consecutive such ranks (including the initial gap from rank 1). Report the distribution (median, 90th/99th percentile, max) of these gaps, and check whether W=8 is at or above the 90th percentile on each workload. Then verify that queries with any gap > W account for the pooled recall loss reported at W=8. If the 90th-percentile gap is below W on all workloads, the stopping rule is well-matched; if not, the current Figure 1(b) evidence is insufficient and the recall loss could be concentrated in tail queries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central speed/recall tradeoff depends on the stopping rule in Section 4.1: a query stops after W consecutive threshold-empty partitions. For this to preserve recall, the gaps between consecutive threshold-qualified partitions in probe order must generally be smaller than W. The paper's motivating evidence, Section 3.3 / Figure 1(b), reports only the rank of the last IVF partition containing a threshold-filtered exact top-10 neighbor. That 'front-loading' metric is not the quantity the rule actually consumes: a query whose neighbors lie at ranks 1 and 16 has last rank 16 (front-loaded relative to 90), but with W=8 the rule stops at rank 9 and misses the rank-16 neighbor. Thus small last-rank percentiles are neither necessary nor sufficient for W-consecutive-empty stops to be safe. The reported recall losses (e.g., 1.13 pp on TABLE at W=8) empirically bound the damage on these six workloads, and the exact-assignment precondition in Section 4.4 is disclosed, but the paper has not directly measured the gap distribution that determines when the rule misfires. Without that distribution, the characterization in Section 3 does not establish the mechanism; the recall numbers do that only for the specific W and workloads tested, leaving the rule's robustness across workload geometries underspecified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SieveIVF is a threshold-aware IVF executor for embedding-based training-data deduplication. Instead of probing a fixed number of partitions for every query, it stops a query after W consecutive partition searches return no candidate above the similarity threshold, exploiting the empirical observation that qualifying neighbors tend to appear early in the query–centroid probe order. The core systems contribution is preserving partition-major batching under data-dependent per-query stopping: continuous batching dynamically regroups ready queries by partition, and a lookahead scheduler enqueues only the prefix already committed by the stopping rule, with a proof that dispatch timing does not change stopping decisions or returned results. The paper evaluates on four 10M-vector proprietary Hunyuan workloads and two public 100M datasets, reporting 4.1–7.6x and 6.1–8.4x speedups over fixed-probe IVF at W=8 with small pooled filtered top-10 recall losses, plus an ablation separating pruning, batching, lookahead, and an automatic W-calibration mode.","tokens_in":17446,"tokens_out":5571,"duration_ms":68572,"significance":"If the reported results hold, the paper makes a practical contribution: it shows that an application predicate known before search can guide physical IVF work allocation without changing the index format, the within-partition search, or the bounded top-k interface. The evaluation is carefully controlled—same index, same nprobe, k, threshold, and concurrency for both executors—and the use of two public 100M datasets mitigates reliance on the proprietary workloads. The ablation in Section 6.3 cleanly attributes the speedup to pruning, batching, and lookahead, and the lookahead equivalence argument is convincing. The transparent single-knob design (W) and the disclosed exact-centroid-assignment precondition in Section 4.4 are strengths. The main weakness is that the workload characterization used to motivate the stopping rule measures last-hit rank, not the consecutive-empty gap distribution that the rule actually consumes; this is a fixable but load-bearing gap in the mechanism evidence.","major_comments":[{"comment":"The stopping rule consumes runs of W consecutive threshold-empty partitions, but the characterization evidence in Figure 1(b) reports only the rank of the last IVF partition containing a threshold-filtered exact top-10 neighbor. These are not the same quantity: a query whose qualifying neighbors lie at ranks 1 and 16 has last rank 16 yet with W=8 will be stopped at rank 9 and miss the rank-16 neighbor; conversely, small last-rank percentiles do not guarantee that gaps between consecutive hits are smaller than W. Section 3.5 therefore overstates what the presented data establish when it says that a run of empty results is 'runtime evidence' of leaving the productive region. The end-to-end recall losses in Section 6.2 empirically bound the damage on the six tested workloads, but the paper should directly measure the gap distribution—e.g., the distribution of distances between consecutive t","section":"Section 6.1 / 6.2"},{"comment":"The primary quality metric in Section 6.1 defines pooled filtered top-k recall as an absolute recall against exact ground truth, but the main results in the abstract and Section 6.2 report 'recall loss' without stating the baseline. Figure 15 later says 'recall loss against pooled filtered exact top-10 ground truth,' while Section 6.5 compares 'relative to each index's own fixed-probe recall.' These are different quantities, and the interpretation of the headline 0.03–1.13pp and 1.43–2.29pp numbers depends on which baseline is used. Given that fixed probing at nprobe=90 may itself have substantial absolute recall loss, the paper should explicitly state the baseline for every reported loss and, ideally, report both absolute recall and loss relative to fixed probing.","section":"Section 6.2"},{"comment":"The characterization in Section 3.3 is based on exact top-10 neighbors, while the stopping rule operates on the approximate RaBitQ search's threshold-empty runs. The paper notes that exact-GT partition placement differs from approximate search recall, but does not show how the approximate search's empty-run distribution relates to the exact-GT front-loading evidence. A direct measurement of the consecutive-empty-run lengths experienced by the actual executor on the evaluation workloads would close this gap and would also strengthen the external-validity argument for the two public 100M datasets.","section":"Section 3.5"}],"minor_comments":[{"comment":"The caption of Figure 4 is dense and the notation r1..r8, t1..t6 is not defined in the caption. Please clarify that r_i are partition ranks and t_i are dispatch rounds, or label the figure more explicitly.","section":"Section 4.3"},{"comment":"Figure 10(b) reports 'the first measured point above 1x is τ=0.35 for WEB, 0.90 for TABLE, and 0.40 for STEM and SCENE.' The relationship between these thresholds and the application thresholds used elsewhere could be stated more clearly, since loose thresholds are explicitly outside the intended operating range.","section":"Section 6.4"},{"comment":"The sentence 'No single window is optimal for every workload. We use W=8 in later experiments as a common operating point' could be softened: Figure 5(a) and 5(b) show that W=8 is a reasonable compromise, but the 'optimal' wording may be ambiguous without a defined cost function over speed and recall.","section":"Section 6.2"},{"comment":"The calibrated mode uses a 2% sample and a 0.5% sample-relative loss target, and the paper honestly states that this is not a confidence bound. It would be helpful to report the variance of the selected W across repeated samples, since the shard-level variation in Figure 14(b) suggests sensitivity to the sample.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the stopping-rule evidence lands: the authors should add a direct gap-distribution measurement. The rest of the paper is solid systems work, and the ambiguity about the recall-loss baseline is easily fixed in the text. The proprietary nature of the Hunyuan workloads is mitigated by the two public 100M datasets. I see no reason to doubt the empirical claims, but the mechanism story needs the additional evidence before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading it. First, the core idea is genuinely useful and carefully executed: SieveIVF stops each query after W consecutive threshold-empty IVF partitions, and the systems contribution is preserving partition-major batching through continuous batching plus a lookahead scheduler. The lookahead equivalence proof is clean, and the ablation in Section 6.3 separates pruning, batching, and lookahead cleanly. Second, the empirical claims are the load-bearing part, and they are direct measurements on six workloads with the same index and search parameters. At W=8 they report 4.1-7.6x speedups on four 10M proprietary workloads and 6.1-8.4x on two public 100M workloads, with recall losses mostly below 1.5 percentage points. These numbers are plausible and well-presented, and the full W sweep shows an honest tradeoff.\n\nThe evaluation is fair by systems standards: same-index comparison, workload-tuned baselines, a defined pooled filtered recall metric, and a deliberate test of the exact-assignment precondition. The authors disclose the speed/recall tradeoff and the need for exact centroid assignment, and Appendix A's online calibration is a nice addition, clearly scoped as sample-relative rather than a guarantee.\n\nNow the soft spots, in proportion. The biggest one is the one the stress-test flags: the workload characterization reports the rank of the last threshold-qualified partition, but the stopping rule consumes gaps between hits, not last-rank. A query with neighbors at ranks 1 and 16 has last rank 16 and looks front-loaded, yet W=8 would stop at rank 9 and miss the second neighbor. So Figure 1(b) doesn't actually establish the mechanism. That said, the paper's recall numbers do empirically bound the damage on the six test workloads, so this is a characterization gap rather than a fatal flaw. It leaves the rule's robustness across new workload geometries under-specified, and the authors do not provide the gap distribution that would close it. Minor issues: no error bars or variance reporting, proprietary primary workloads, and no code release, which make exact reproduction impossible. None of these undercut the measured claims.\n\nThis paper deserves a serious referee. It is a pragmatic, honest systems contribution that would likely need revision (add variance reporting, ideally release code, and either add a direct gap-distribution check or soften the characterization claims), but the central argument holds. I'd bring it to a reading group and would cite it in my own work on adaptive vector search.","headline":"Solid, well-scoped systems paper: a threshold-aware IVF executor with measured 4-8x speedups at small recall cost; the stopping-rule characterization has a real gap, but the empirical claims hold on their own.","tokens_in":17924,"tokens_out":1194,"would_cite":true,"duration_ms":16549,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SieveIVF makes each IVF query stop after a run of empty threshold partitions, cutting deduplication search time by 4-8x while keeping top-10 recall loss small.","keywords":["training data deduplication","vector search","IVF","threshold retrieval","early stopping","continuous batching","lookahead scheduling","approximate nearest neighbor search"],"falsifier":"Run SieveIVF with $W=8$ against fixed-probe IVF on a workload whose exact threshold-filtered top-10 neighbors are not front-loaded, for example a workload where the 90th percentile last productive partition rank is far above the values in Figure 1(b) or where the first eight partitions store well under 90% of qualifying neighbors. If pooled filtered top-10 recall loss rises materially beyond the reported 0.03-2.29 percentage points while speedup also falls below the reported 4-8x range, the front-loading premise is falsified.","tokens_in":17068,"feed_emoji":"⚡","tokens_out":8257,"duration_ms":85437,"temperature":0.7,"pith_summary":"Fixed-probe inverted-file (IVF) search spends the same partition budget on every query, even when only neighbors above a similarity threshold can be duplicate candidates. SieveIVF is a threshold-aware executor that watches each query's local search results: after $W$ consecutive partitions return no qualifying candidate, it stops that query early. Because qualifying neighbors concentrate near the front of the partition order in the workloads studied, most queries need far fewer than the fixed 90 probes. To keep batched IVF efficient under per-query stopping, the paper contributes continuous batching, which dynamically groups ready queries by partition, and a lookahead scheduler that enqueues only ranks already committed by the stopping rule. The measured result, with $W=8$ and identical indexes and search parameters, is a 4.1-7.6x speedup on four 10M workloads and 6.1-8.4x speedup on two public 100M workloads, at pooled filtered top-10 recall losses of 0.03-1.13 and 1.43-2.29 percentage points.","feed_headline":"Stop after 8 empty partitions: IVF dedup runs 4-8x faster","feed_subtitle":"Per-query probe budgets cut partition visits by up to 91% while keeping pooled top-10 recall loss below 2.3 percentage points.","key_machinery":"The central mechanism is the consecutive-empty window $W$: after IVF partitions are searched in query-centroid distance order, a query stops when $W$ consecutive partitions return no candidate above the similarity threshold $\\tau$, and any qualifying hit resets the counter. This converts the measured front-loading of threshold-qualified neighbors into a predictor-free, per-query probe budget. Continuous batching preserves partition-major execution by dynamically regrouping ready queries by partition, while lookahead scheduling admits only ranks already committed by the stopping rule, keeping low-concurrency batches from serializing.","core_discovery":"Fixed-probe inverted-file (IVF) search gives every query the same number of partition probes ($n_{\\text{probe}}$), even though in training-data deduplication only neighbors above an application similarity threshold $\\tau$ can become duplicate edges. SieveIVF replaces that fixed budget with a per-query stopping rule: walk the partitions in the same centroid-distance order, and stop after $W$ consecutive partition searches return no candidate satisfying $\\tau$; any qualifying hit resets the counter. The systems contribution is making this rule fast at batch scale without changing the index or the bounded top-$k$ interface: continuous batching keeps queries grouped by the partition they are rea","pith_inferences":["A natural extension not claimed by the paper: the threshold-empty-run signal should transfer to any search structure with a deterministic partition or segment order, such as coarse quantizers in product-quantization indexes or shard-ordered searches, as a general predicate-aware early-termination pattern.","The front-loading property is empirical, and the paper's own dense TABLE workload shows the smallest speedup and largest recall loss; on workloads with diffuse threshold-neighbor placement, $W=8$ would either stop too late, killing the speedup, or stop too early, losing recall.","The appendix's in-call calibration selects $W$ per batch and per shard from a sample without exact ground truth; if that holds as data distributions drift, the method becomes self-tuning, but its sample-relative loss target is not a guarantee on full-call exact recall.","A deployment could use the reported microbatch-occupancy effect as a diagnostic: when mean queries-per-microbatch falls, lookahead has little room to help, so the main tuning levers become $W$ and threshold selectivity."],"forward_implications":["Deduplication pipelines can adapt per-query probe counts at runtime without choosing one fixed $n_{\\text{probe}}$ that must fit both sparse and dense query distributions.","The speed/recall tradeoff is controlled by one integer $W$: small $W$ prunes aggressively (up to 15x speedup at $W=2$, with larger recall loss), while large $W$ approaches fixed probing with near-zero loss.","Savings do not come only from queries with no qualifying neighbors: even queries with ten threshold-qualified neighbors skip 70-90% of partition visits, because front-loading matters more than neighbor count.","The same execution mechanism scales to 100M vectors and works across different embedding dimensions, and it still beats a recall-matched fixed-probe baseline by 1.09-1.22x.","Capping the maximum probe budget at $n_{\\text{probe}}=24$ still leaves SieveIVF 1.53-2.39x faster with 0.02-0.72 percentage points recall loss."],"supporting_citations":[{"why":"Defines semantic redundancy in embedding space and motivates threshold-based candidate retrieval for training-data deduplication.","marker":"[1]"},{"why":"Provides the DiskJoin threshold-join path compared as an alternative deduplication strategy.","marker":"[8]"},{"why":"Supplies the RaBitQ compressed-vector representation used by the IVF index in all experiments.","marker":"[10]"},{"why":"Describes the conventional fixed-probe, partition-major batching baseline that SieveIVF adapts and compares against.","marker":"[14]"},{"why":"The open lakehouse vector-search system in which SieveIVF is implemented; supplies the IVF layout, within-partition search, and worker pool.","marker":"[17]"},{"why":"Provides the HNSW-based approximate build-time assignment variant whose recall degradation is measured to justify exact centroid assignment.","marker":"[19]"},{"why":"Source of the 512-dimensional public 100M workload used in the scale-out evaluation.","marker":"[20]"},{"why":"Source of the 96-dimensional public 100M workload used in the scale-out evaluation.","marker":"[24]"}],"fun_headline_variants":["IVF dedup speeds up 4-8x by stopping after 8 empty partitions","Stop counting probes: threshold-aware IVF runs 8x faster","Per-query probe stop: 4-8x faster dedup, <2.3% recall loss","Adaptive IVF: stop when no hits, 4-8x faster dedup"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that threshold-qualified neighbors are front-loaded in the IVF partition order, so a run of $W$ empty partitions is reliable evidence that a query has left the productive region; the paper also relies on exact centroid assignment at build time so the query-time partition order matches storage.","fun_headline_variants_meta":{"raw":{"variants":["IVF dedup speeds up 4-8x by stopping after 8 empty partitions","Stop counting probes: threshold-aware IVF runs 8x faster","Per-query probe stop: 4-8x faster dedup, <2.3% recall loss","Adaptive IVF: stop when no hits, 4-8x faster dedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1709,"prompt_tokens":802,"completion_tokens":907,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":815}},"tokens_in":546,"tokens_out":907,"duration_ms":8642,"temperature":1.0,"reasoning_tokens":815,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:42:13.440616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SieveIVF with $W=8$ against fixed-probe IVF on a workload whose exact threshold-filtered top-10 neighbors are not front-loaded, for example a workload where the 90th percentile last productive partition rank is far above the values in Figure 1(b) or where the first eight partitions store well under 90% of qualifying neighbors. If pooled filtered top-10 recall loss rises materially beyond the reported 0.03-2.29 percentage points while speedup also falls below the reported 4-8x range, the front-loading premise is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DiskJoin threshold-join path compared as an alternative deduplication strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The open lakehouse vector-search system in which SieveIVF is implemented; supplies the IVF layout, within-partition search, and worker pool."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the 512-dimensional public 100M workload used in the scale-out evaluation."}],"review_version":1}