{"id":"82910ecb-f7d1-451a-ae12-7a8a35127afe","arxiv_id":"2505.01164","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CaGR-RAG reorders RAG queries into groups with overlapping cluster accesses and prefetches next-group clusters, improving cache hits and cutting p99 latency by up to 51.55% in disk-based IVF search.","lead":"This paper presents a way to speed up disk-based vector search in RAG systems by grouping incoming queries that need similar chunks of the index and prefetching the next group's data. In tests on three benchmarks, it cut 99th percentile search latency by up to 51.55% compared with a caching baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prefetch pollution is the load-bearing risk: Algorithm 1 Step 4 unconditionally inserts next-group clusters, so the claimed 51.55% tail-latency gain is contingent on cache capacity large enough to avoid evicting clusters needed by later groups.","rationale":"The reader's verdict is CONDITIONAL, and this pass does not change that. The paper proposes a concrete, plausible optimization—reordering queries by Jaccard overlap of their cluster sets and prefetching the next group's first-query clusters. The evaluation shows consistent cache-hit and latency improvements over EdgeRAG on three datasets, and Figure 7 includes an ablation of both modules. The weakest point is the unconditional prefetch: Algorithm 1 Step 4 inserts C(qF(Gi+1)) into the cache without checking whether the cache has spare capacity or whether the evicted entries will be needed again. The paper's own Section 4.4 observation that prefetching at high thresholds can reduce hit rates confirms this failure mode exists. Since the experiments fix the cache at 40 entries and nprobe at 10, the central claim is not shown to generalize across cache sizes; a small-cache run or an eviction-reuse counter would settle whether the reported 51.55% tail-latency reduction is robust or is an artifact of a cache large enough to absorb prefetch pollution. This is a correctness and robustness concern, not an inconsistency with prior consensus. The contribution may well hold, but the current evidence supports CONDITIONAL acceptance.","tokens_in":9759,"tokens_out":7109,"duration_ms":71606,"concrete_test":"Run the hotpotqa workload of Section 4 with total cache entries varied over {10, 20, 40, 80}, keeping nprobe=10 and θ=0.5 fixed. Measure p99 latency and cache hit ratio for EdgeRAG and CaGR-RAG. If at cache=10 or 20 CaGR-RAG's hit ratio drops below EdgeRAG, or its p99 advantage disappears or reverses, then the unconditional prefetch in Algorithm 1 Step 4 is the cause. As a second check, instrument the cache to count, at each group transition, how many clusters evicted by the prefetch are subsequently re-referenced before the next prefetch; if that eviction-reuse count exceeds the number of prefetched clusters actually used, prefetch pollution is demonstrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Load-bearing concern: The reported gains rest on the assumption that prefetching C(qF(Gi+1)) at group transitions never reduces the cache hit ratio. Algorithm 1 Step 4 (line 35) executes S ← S ∪ C(qF(Gi+1)) unconditionally. With total cache entries fixed at 40 (Section 4.1) and nprobe=10, a group whose queries share only the threshold similarity θ=0.5 can have a working set close to cache capacity; inserting the next group's up to 10 clusters then forces evictions of entries that may be re-referenced by later groups. This is not hypothetical: Section 4.4 reports that at a 90% Jaccard threshold 'frequent cache evictions may reduce the cache hit rate (e.g., Query 260 in Figure 5),' i.e., the prefetch module can be counterproductive. The evaluation does not vary cache capacity, does not report per-group working-set sizes, and does not state the eviction policy used in CaGR-RAG (only the baseline EdgeRAG's cost-aware scheme is described). The claim that CaGR-RAG 'consistently maintains a higher cache hit ratio' is therefore conditional on cache capacity being large relative to group working sets; if prefetch pollution dominates for smaller caches or larger nprobe, the headline 51.55% tail-latency improvement may invert.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CaGR-RAG, a mechanism for disk-based IVF vector search in RAG systems that reorders incoming queries into groups with similar cluster access patterns, measured by the Jaccard overlap of the clusters each query probes, and that prefetches the clusters of the first query of the next group at group transitions. The authors evaluate CaGR-RAG against a single baseline, EdgeRAG, on three BEIR datasets (nq, hotpotqa, fever) with the all-miniLM-L6-v2 embedding model. They report that CaGR-RAG consistently achieves higher cache hit ratios and reduces the 99th percentile tail latency by up to 51.55% (on hotpotqa).","tokens_in":10034,"tokens_out":3126,"duration_ms":31435,"significance":"If the findings hold, the paper makes a useful, simple contribution: it shows that query reordering based on cluster-access similarity, combined with one-step-ahead prefetching, can reduce disk I/O and tail latency in disk-based vector search for RAG. The mechanism is orthogonal to cache replacement policy, uses standard IVF indexes, and is evaluated on public benchmarks. The paper also includes pseudocode (Algorithm 1) and concrete experimental configurations, which helps reproducibility. The main caveat is that the evaluation's support for the headline claim is weaker than the abstract suggests, because the prefetch step can be counterproductive in the regime the paper itself identifies, and because the experimental comparison is limited to one baseline with no variance reporting.","major_comments":[{"comment":"Equation (3) defines a group as Gk = {qi in Q | J(qi,qj) >= theta for all qj in Gk}, but Algorithm 1 line 8 assigns qi to a group if max(J(qi,qj)) >= theta, i.e., if the query is sufficiently similar to at least one existing member. These two definitions produce different groupings, and the algorithm's weaker condition can place a query into a group with which it shares clusters only marginally, undermining the stated rationale of grouping queries with shared cluster access patterns. The paper should state which definition is actually used and, ideally, measure how the discrepancy affects cache hit ratio.","section":"§3.2, Eq. (3) and Algorithm 1"},{"comment":"The prefetch operation in Algorithm 1 (line 35) is unconditional: S <- S ∪ C(qF(Gi+1)) for every group transition. With the experimental cache capacity of 40 entries and nprobe=10, the clusters required by the next group's first query can evict entries still needed by the remaining queries of the current group or by later groups. The paper's own Section 4.4 reports that at a 90% Jaccard threshold, frequent cache evictions may reduce the cache hit rate (citing Query 260 in Figure 5), which confirms that the prefetch module can be counterproductive. The evaluation does not vary cache capacity or nprobe, does not report per-group working-set sizes, and does not state the eviction policy used for CaGR-RAG (only EdgeRAG's cost-aware policy is described). The claim of consistently higher cache hit ratio is therefore contingent on cache capacity being large relative to group working sets; for smaller caches the reported gains may invert. Please add a cache-size sweep and specify the eviction policy.","section":"Algorithm 1 Step 4 and §4.4"},{"comment":"The experimental comparison uses only one baseline, EdgeRAG's cost-aware cache scheme, and reports single point estimates without error bars or multiple runs. Because the batch sizes are randomly generated between 20 and 100, the reported 51.55% tail-latency improvement and the cache hit ratio comparisons in Figures 4-6 could be sensitive to the particular random draw. At minimum, the authors should report the mean and variance over several seeds, and they should consider an additional baseline such as LRU or FIFO to isolate the effect of grouping from the choice of eviction policy.","section":"§4.1 and §4.3"},{"comment":"The paper states the claim 'CaGR-RAG reduces the 99th percentile tail latency by up to 51.55% on hotpotqa' but only plots tail CDFs and reports the point difference. The zoomed-in inset covers the 95th-100th percentile range, but the exact 99th percentile values reported in the text (0.936 vs 0.4621 sec for nq, 1.5365 vs 0.7445 sec for hotpotqa, 1.287 vs 0.7584 sec for fever) should be presented in a table together with the number of queries, batch composition, and the range across runs. This would make the headline claim auditable.","section":"§4.3, Figure 6"}],"minor_comments":[{"comment":"There is a typo: 'using using Equation 3' should be 'using Equation 3'.","section":"§3.2, line after Eq. (2)"},{"comment":"The x-axis of Figure 7 is labeled 'Distance Threshold (%)' and the text says 'Jaccard distance closer to 100 indicates greater similarity,' which is confusing because Section 4.1 defines the threshold as a Jaccard similarity of 0.5. Clarify whether the threshold is a similarity or a distance and keep the notation consistent.","section":"Figure 7 and §4.4"},{"comment":"In Equation (5), the variable k is used both for the number of clusters in Eq. (4) and for the number of groups in Eq. (5); use different symbols to avoid ambiguity.","section":"Eq. (5)"},{"comment":"Figures 4 and 5 show only a subset of queries (IDs 100-200 or 250-300). State explicitly how these windows were selected and whether the full trace exhibits the same behavior, to rule out cherry-picking.","section":"§4.2, Figures 4 and 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is short and workshop-style, and the central technical idea is plausible but the evaluation currently does not support the strong 'consistently maintains a higher cache hit ratio' claim without a cache-size sweep and a specified eviction policy. The authors should also consider releasing code and data to strengthen reproducibility. The paper's fit with a systems venue is reasonable, but the presentation needs to move from a proof-of-concept to a more complete empirical study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuine: reorder RAG queries by Jaccard overlap in the IVF clusters they touch, and prefetch the next group's first query's clusters. That is new in disk-based ANN search, and the experiments show real gains. The motivation is well-argued with actual embedding-model observations, and the mechanism is simple and plausible. The evaluation on three BEIR datasets shows consistent cache hit and latency improvements, and the module ablation shows prefetch helps.\n\nBut the evidence is a bit thinner than the headline. Only one baseline (EdgeRAG), no error bars, and no sensitivity analysis on cache capacity or nprobe. The cache replacement policy for CaGR-RAG is never specified, so the comparison to EdgeRAG's cost-aware cache is not fully apples-to-apples. More worrying, Equation 3 defines a group as all members pairwise similar, but Algorithm 1 assigns a query to a group if it is similar to any member—that is a real discrepancy that should be fixed. The threshold sensitivity is only shown for hotpotqa, and the 90% case actually shows the prefetch can hurt, which the paper admits. That supports the concern that prefetch pollution is possible, though the headline claim is for the 0.5 threshold and this cache size.\n\nNone of these are fatal. The central mechanism holds up, and the 51.55% improvement is plausible for this type of optimization. But the current evidence does not justify the 'consistently maintains a higher cache hit ratio' claim as a general statement; it holds for the tested configuration.\n\nThis is for systems folks working on disk-based ANN or RAG infrastructure. A serious referee would want the missing details, but the paper deserves that review. I'd send it to peer review with a request for major revision: add a second baseline, report variance, specify the eviction policy, and run a cache-capacity sweep.","headline":"A genuinely new query-ordering idea for disk-based IVF in RAG that shows real gains, but the evaluation needs more care before the claims generalize.","tokens_in":10579,"tokens_out":2101,"would_cite":true,"duration_ms":21141,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CaGR-RAG groups RAG queries by cluster similarity and reports up to 51.55% lower 99th-percentile tail latency in disk-based vector search.","keywords":["Retrieval Augmented Generation","Disk-based Vector Search","query grouping","cache","prefetching","Jaccard similarity","IVF index","tail latency"],"falsifier":"Run the same three workloads with cache capacity reduced below one group's working set (for example, fewer cache entries than $nprobe$); if the 99th-percentile latency improvement shrinks or reverses, the prefetch step is causing the evictions the design must avoid. A second test: use a synthetic query stream whose cluster sets are independent (Jaccard similarity near zero between all pairs); CaGR-RAG should then perform no better than the baseline, and any gain would indicate the comparison is measuring something other than query locality.","tokens_in":9563,"feed_emoji":"⚡","tokens_out":6945,"duration_ms":66699,"temperature":0.7,"pith_summary":"Disk-based vector search must read clusters from storage for each query, and cache misses create long tails in RAG response times. The paper tries to establish that queries are not independent: embedding models place similarly structured questions near each other, so the cluster sets different queries read overlap heavily. CaGR-RAG exploits that by grouping queries with high Jaccard overlap and prefetching the next group's first-query clusters, cutting redundant disk reads. If correct, it brings 99th-percentile tail latency down by up to 51.55% and keeps cache hit ratios consistently above a cost-aware baseline, with no change to accuracy.","feed_headline":"Query grouping cuts RAG disk-search tail latency by 51%","feed_subtitle":"Batching similar queries and prefetching the next group's clusters lifts cache hits without changing search accuracy.","key_machinery":"The load-bearing object is the per-query cluster set $C(q_i)$ and the Jaccard similarity $J(q_i,q_j)=|C(q_i)\\cap C(q_j)|/|C(q_i)\\cup C(q_j)|$, which defines grouping via agglomerative clustering with threshold $\\theta$. The second mechanism is the group descriptor $D$: for each group it stores the member queries, the clusters they need, and the clusters of the first query of the next group; the vector database uses this descriptor to prefetch $C(q_{F(G_{i+1})})$ into the cache just as the group switches. The prefetch is unconditional in Algorithm 1 (Step 4 inserts $C(q_{F(G_{i+1})})$ into the cache state), which is what makes cache capacity relative to group working sets the key constraint.","core_discovery":"The paper claims that queries in a RAG workload are not independent: embedding models map similarly structured questions to nearby vector regions, so the sets of disk clusters read for different queries overlap substantially. CaGR-RAG uses the Jaccard similarity of those cluster sets to reorder incoming queries into groups, and then, when one group finishes, prefetches the clusters of the first query of the next group. In evaluation the scheme is reported to reduce 99th-percentile tail latency by up to 51.55% and to keep cache hit ratios consistently above the cost-aware baseline across three datasets.","pith_inferences":["Beyond the paper: the algorithm as described operates on a batch of queries; converting it into an online scheduler for a live stream would require predicting the next group's head query before the batch is known, perhaps by maintaining a sliding window of recent queries.","Beyond the paper: since cluster file sizes vary widely, prefetching could be made size-aware, prefetching the high-value small clusters of the next group first rather than fetching the first query's full cluster set.","Beyond the paper: a direct testable extension is to make the prefetch conditional on free cache capacity; the paper's own 90%-threshold result indicates that when groups are small and evictions frequent, prefetching can reduce hit rates."],"forward_implications":["RAG retrieval over disk-resident IVF indexes can be accelerated without modifying the embedding model, the search algorithm, or the answer-generation stage; only the order in which queries are dispatched changes.","Reordering by Jaccard similarity converts a stream of seemingly independent queries into bursts that reuse the same cached clusters, so the same cache holds more useful vectors and fewer disk reads happen on the critical path.","Grouping alone is not enough: the prefetch of the next group's first-query clusters is what removes the cache-miss spike at group transitions, cutting tail latency by up to 3.1x over grouping alone at low similarity thresholds.","Because the scheme is described as compatible with any cache replacement policy, it can be layered on existing cost-aware or LRU-style caches rather than requiring a new storage engine."],"supporting_citations":[{"why":"The cost-aware cache scheme used as the comparison baseline; CaGR-RAG's reported gains are measured against it.","marker":"[12]"},{"why":"The IVF index structure and two-level search procedure that defines which clusters each query reads.","marker":"[14]"},{"why":"The hierarchical agglomerative clustering algorithm used to build query groups from pairwise Jaccard similarities.","marker":"[11]"},{"why":"The benchmark suite supplying the three query workloads on which cache hit ratios and latencies are measured.","marker":"[15]"},{"why":"The embedding model used to encode queries and build the vector indexes, grounding the locality assumption.","marker":"[19]"},{"why":"The multi-hop QA dataset used for module-ablation and threshold experiments, including the 51.55% tail-latency result.","marker":"[20]"}],"fun_headline_variants":["Grouping similar queries in RAG cuts tail latency 51%","CaGR-RAG batches queries to cut disk I/O and tail latency","Prefetching next query group's clusters lifts RAG cache hits","Context-aware grouping for disk-based RAG: 51% less tail latency","RAG disk search: group queries by cluster access to cut latency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scheme assumes the cache can hold the current group's working set plus the next group's first-query clusters, so the unconditional prefetch in Algorithm 1 does not evict clusters the current group still needs.","fun_headline_variants_meta":{"raw":{"variants":["Grouping similar queries in RAG cuts tail latency 51%","CaGR-RAG batches queries to cut disk I/O and tail latency","Prefetching next query group's clusters lifts RAG cache hits","Context-aware grouping for disk-based RAG: 51% less tail latency","RAG disk search: group queries by cluster access to cut latency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001059,"raw_usage":{"total_tokens":4382,"prompt_tokens":821,"completion_tokens":3561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":3467}},"tokens_in":437,"tokens_out":3561,"duration_ms":24944,"temperature":1.0,"reasoning_tokens":3467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:25:00.210027+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three workloads with cache capacity reduced below one group's working set (for example, fewer cache entries than $nprobe$); if the 99th-percentile latency improvement shrinks or reverses, the prefetch step is causing the evictions the design must avoid. A second test: use a synthetic query stream whose cluster sets are independent (Jaccard similarity near zero between all pairs); CaGR-RAG should then perform no better than the baseline, and any gain would indicate the comparison is measuring something other than query locality.","supporting_citations":[{"cited_title":"In: IEEE International Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"The IVF index structure and two-level search procedure that defines which clusters each query reads."},{"cited_title":"In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021), https://openreview.net/forum? id=wCu6T5xFjeJ","cited_arxiv_id":null,"evidence_quote":"The benchmark suite supplying the three query workloads on which cache hit ratios and latencies are measured."},{"cited_title":"NIPS ’20, Curran Associates Inc., Red Hook, NY, USA (2020)","cited_arxiv_id":null,"evidence_quote":"The embedding model used to encode queries and build the vector indexes, grounding the locality assumption."}],"review_version":1}