Pith. sign in

REVIEW 4 major objections 4 minor 21 references

CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read CaGR-RAG groups RAG queries by cluster similarity and reports up to 51.55% lower 99th-percentile tail latency in disk-based vector search.

desk verdict A genuinely new query-ordering idea for disk-based IVF in RAG that shows real gains, but the evaluation needs more care before the claims generalize. read the letter →

arxiv 2505.01164 v1 pith:CQMCYORB submitted 2025-05-02 cs.DC

classification cs.DC
keywords RetrievalAugmentedGenerationDisk-basedVectorSearchquerygroupingcacheprefetchingJaccardsimilarityIVFindextaillatency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Disk-based vector search must read clusters from storage for each query, and cache misses create long tails in RAG response times. The paper tries to establish that queries are not independent: embedding models place similarly structured questions near each other, so the cluster sets different queries read overlap heavily. CaGR-RAG exploits that by grouping queries with high Jaccard overlap and prefetching the next group's first-query clusters, cutting redundant disk reads. If correct, it brings 99th-percentile tail latency down by up to 51.55% and keeps cache hit ratios consistently above a cost-aware baseline, with no change to accuracy.

What carries the argument

The load-bearing object is the per-query cluster set $C(q_i)$ and the Jaccard similarity $J(q_i,q_j)=|C(q_i)\cap C(q_j)|/|C(q_i)\cup C(q_j)|$, which defines grouping via agglomerative clustering with threshold $\theta$. The second mechanism is the group descriptor $D$: for each group it stores the member queries, the clusters they need, and the clusters of the first query of the next group; the vector database uses this descriptor to prefetch $C(q_{F(G_{i+1})})$ into the cache just as the group switches. The prefetch is unconditional in Algorithm 1 (Step 4 inserts $C(q_{F(G_{i+1})})$ into the cache state), which is what makes cache capacity relative to group working sets the key constraint.

What would settle it

Run the same three workloads with cache capacity reduced below one group's working set (for example, fewer cache entries than $nprobe$); if the 99th-percentile latency improvement shrinks or reverses, the prefetch step is causing the evictions the design must avoid. A second test: use a synthetic query stream whose cluster sets are independent (Jaccard similarity near zero between all pairs); CaGR-RAG should then perform no better than the baseline, and any gain would indicate the comparison is measuring something other than query locality.

Watch

Extended reading notes

Core claim

The paper claims that queries in a RAG workload are not independent: embedding models map similarly structured questions to nearby vector regions, so the sets of disk clusters read for different queries overlap substantially. CaGR-RAG uses the Jaccard similarity of those cluster sets to reorder incoming queries into groups, and then, when one group finishes, prefetches the clusters of the first query of the next group. In evaluation the scheme is reported to reduce 99th-percentile tail latency by up to 51.55% and to keep cache hit ratios consistently above the cost-aware baseline across three datasets.

Load-bearing premise

The scheme assumes the cache can hold the current group's working set plus the next group's first-query clusters, so the unconditional prefetch in Algorithm 1 does not evict clusters the current group still needs.

Editorial extensions

If this is right

  • RAG retrieval over disk-resident IVF indexes can be accelerated without modifying the embedding model, the search algorithm, or the answer-generation stage; only the order in which queries are dispatched changes.
  • Reordering by Jaccard similarity converts a stream of seemingly independent queries into bursts that reuse the same cached clusters, so the same cache holds more useful vectors and fewer disk reads happen on the critical path.
  • Grouping alone is not enough: the prefetch of the next group's first-query clusters is what removes the cache-miss spike at group transitions, cutting tail latency by up to 3.1x over grouping alone at low similarity thresholds.
  • Because the scheme is described as compatible with any cache replacement policy, it can be layered on existing cost-aware or LRU-style caches rather than requiring a new storage engine.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the algorithm as described operates on a batch of queries; converting it into an online scheduler for a live stream would require predicting the next group's head query before the batch is known, perhaps by maintaining a sliding window of recent queries.
  • Beyond the paper: since cluster file sizes vary widely, prefetching could be made size-aware, prefetching the high-value small clusters of the next group first rather than fetching the first query's full cluster set.
  • Beyond the paper: a direct testable extension is to make the prefetch conditional on free cache capacity; the paper's own 90%-threshold result indicates that when groups are small and evictions frequent, prefetching can reduce hit rates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CaGR-RAG, a mechanism for disk-based IVF vector search in RAG systems that reorders incoming queries into groups with similar cluster access patterns, measured by the Jaccard overlap of the clusters each query probes, and that prefetches the clusters of the first query of the next group at group transitions. The authors evaluate CaGR-RAG against a single baseline, EdgeRAG, on three BEIR datasets (nq, hotpotqa, fever) with the all-miniLM-L6-v2 embedding model. They report that CaGR-RAG consistently achieves higher cache hit ratios and reduces the 99th percentile tail latency by up to 51.55% (on hotpotqa).

Significance. If the findings hold, the paper makes a useful, simple contribution: it shows that query reordering based on cluster-access similarity, combined with one-step-ahead prefetching, can reduce disk I/O and tail latency in disk-based vector search for RAG. The mechanism is orthogonal to cache replacement policy, uses standard IVF indexes, and is evaluated on public benchmarks. The paper also includes pseudocode (Algorithm 1) and concrete experimental configurations, which helps reproducibility. The main caveat is that the evaluation's support for the headline claim is weaker than the abstract suggests, because the prefetch step can be counterproductive in the regime the paper itself identifies, and because the experimental comparison is limited to one baseline with no variance reporting.

major comments (4)
  1. [§3.2, Eq. (3) and Algorithm 1] Equation (3) defines a group as Gk = {qi in Q | J(qi,qj) >= theta for all qj in Gk}, but Algorithm 1 line 8 assigns qi to a group if max(J(qi,qj)) >= theta, i.e., if the query is sufficiently similar to at least one existing member. These two definitions produce different groupings, and the algorithm's weaker condition can place a query into a group with which it shares clusters only marginally, undermining the stated rationale of grouping queries with shared cluster access patterns. The paper should state which definition is actually used and, ideally, measure how the discrepancy affects cache hit ratio.
  2. [Algorithm 1 Step 4 and §4.4] The prefetch operation in Algorithm 1 (line 35) is unconditional: S <- S ∪ C(qF(Gi+1)) for every group transition. With the experimental cache capacity of 40 entries and nprobe=10, the clusters required by the next group's first query can evict entries still needed by the remaining queries of the current group or by later groups. The paper's own Section 4.4 reports that at a 90% Jaccard threshold, frequent cache evictions may reduce the cache hit rate (citing Query 260 in Figure 5), which confirms that the prefetch module can be counterproductive. The evaluation does not vary cache capacity or nprobe, does not report per-group working-set sizes, and does not state the eviction policy used for CaGR-RAG (only EdgeRAG's cost-aware policy is described). The claim of consistently higher cache hit ratio is therefore contingent on cache capacity being large relative to group working sets; for smaller caches the reported gains may invert. Please add a cache-size sweep and specify the eviction policy.
  3. [§4.1 and §4.3] The experimental comparison uses only one baseline, EdgeRAG's cost-aware cache scheme, and reports single point estimates without error bars or multiple runs. Because the batch sizes are randomly generated between 20 and 100, the reported 51.55% tail-latency improvement and the cache hit ratio comparisons in Figures 4-6 could be sensitive to the particular random draw. At minimum, the authors should report the mean and variance over several seeds, and they should consider an additional baseline such as LRU or FIFO to isolate the effect of grouping from the choice of eviction policy.
  4. [§4.3, Figure 6] The paper states the claim 'CaGR-RAG reduces the 99th percentile tail latency by up to 51.55% on hotpotqa' but only plots tail CDFs and reports the point difference. The zoomed-in inset covers the 95th-100th percentile range, but the exact 99th percentile values reported in the text (0.936 vs 0.4621 sec for nq, 1.5365 vs 0.7445 sec for hotpotqa, 1.287 vs 0.7584 sec for fever) should be presented in a table together with the number of queries, batch composition, and the range across runs. This would make the headline claim auditable.
minor comments (4)
  1. [§3.2, line after Eq. (2)] There is a typo: 'using using Equation 3' should be 'using Equation 3'.
  2. [Figure 7 and §4.4] The x-axis of Figure 7 is labeled 'Distance Threshold (%)' and the text says 'Jaccard distance closer to 100 indicates greater similarity,' which is confusing because Section 4.1 defines the threshold as a Jaccard similarity of 0.5. Clarify whether the threshold is a similarity or a distance and keep the notation consistent.
  3. [Eq. (5)] In Equation (5), the variable k is used both for the number of clusters in Eq. (4) and for the number of groups in Eq. (5); use different symbols to avoid ambiguity.
  4. [§4.2, Figures 4 and 5] Figures 4 and 5 show only a subset of queries (IDs 100-200 or 250-300). State explicitly how these windows were selected and whether the full trace exhibits the same behavior, to rule out cherry-picking.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CaGR-RAG is an algorithm plus external empirical evaluation; claims are not derived from their own definitions or self-citations.

full rationale

The paper makes no first-principles derivation that reduces to its inputs. The central claims are empirical measurements comparing CaGR-RAG against an external baseline (EdgeRAG). The query grouping and prefetch mechanisms are defined by Equations 1–5 and Algorithm 1, and the reported cache hit ratios and latencies are measured outcomes, not quantities fitted from the same data. The similarity threshold θ=0.5 and cache size 40 are fixed experimental settings, not tuned to produce the headline 51.55% tail-latency reduction. The paper contains no self-citations and imports no uniqueness theorem from the authors' prior work. Although grouping queries by Jaccard similarity naturally promotes cache reuse, Section 4.4 explicitly shows that this is not guaranteed: at a 90% threshold, 'frequent cache evictions may reduce the cache hit rate (e.g., Query 260 in Figure 5)', which demonstrates that the reported improvements are contingent empirical findings rather than definitional tautologies. The acknowledged limitations are performance caveats, not circular reasoning.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger contains the hand-set experimental parameters and the load-bearing workload assumptions. The method has no fitted equations, but the empirical claims rest on batch reordering being allowed, on cluster sets being known before search, on cache capacity being sufficient for prefetching, and on the embedding model placing similar queries near each other in vector space.

free parameters (3)
  • Jaccard similarity threshold theta = 0.5 (swept 0.1-0.9 in Figure 7)
    Controls how aggressively queries are grouped; set to 0.5 in main experiments without a stated selection procedure, and performance is sensitive to it in Figure 7.
  • nprobe = 10 (varied 10-40 in Figure 2)
    Number of clusters loaded per query; affects cluster-set overlap and cache pressure, and is fixed at 10 for the main evaluation.
  • cache capacity = 40 cluster entries (50 in the motivation experiment)
    Hand-chosen capacity; the benefit of grouping and prefetching depends on capacity relative to nprobe and group sizes.
assumptions (4)
  • domain assumption Query batches of 20 to 100 can be reordered without violating latency requirements.
    The grouping algorithm computes pairwise similarities over a whole batch; if queries must be answered in arrival order, the scheme cannot be applied (Section 4.1 Traffic).
  • domain assumption Cluster sets C(q_i) are available from the in-memory first-level quantizer search before any disk I/O.
    Algorithm 1 assumes C(q_i) is retrieved for every query at step 1; this is true for IVF indexes but not for graph-based indexes where neighbors are discovered during search.
  • domain assumption Prefetched clusters fit in cache without evicting clusters the current group still needs.
    Algorithm 1 Step 4 unconditionally adds prefetched clusters to cache state S; the paper notes in Section 4.4 that frequent evictions can degrade hit rate when groups are small.
  • domain assumption Embedding models map structurally or semantically similar queries to nearby vector regions, so cluster-set overlap reflects query context.
    The motivation in Section 2.4 demonstrates this empirically for three models, but it is a statistical property, not a guarantee; the method's effectiveness depends on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems." pith.science (2026). https://pith.science/paper/CQMCYORB

@misc{pith2026250501164,
  author       = {Pith},
  title        = {Pith review of: CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQMCYORB}},
  note         = {Machine review of arXiv:2505.01164}
}
read the original abstract

Modern embedding models capture both semantic and syntactic structures of queries, often mapping different queries to similar regions in vector space. This results in non-uniform cluster access patterns in disk-based vector search systems, particularly in Retrieval Augmented Generation (RAG) framework. While existing approaches optimize individual queries, they overlook the impact of cluster access patterns, failing to account for the locality effects of queries that access similar clusters. This oversight reduces cache efficiency and increases search latency due to excessive disk I/O. To address this, we introduce CaGR-RAG, a context-aware query grouping mechanism that organizes queries based on shared cluster access patterns. Additionally, it incorporates opportunistic cluster prefetching to minimize cache misses during transitions between query groups, further optimizing retrieval performance. Experimental results show that CaGR-RAG reduces 99th percentile tail latency by up to 51.55% while consistently maintaining a higher cache hit ratio than the baseline.

Figures

Figures reproduced from arXiv: 2505.01164 by the authors.

Figure 1
Figure 1. Cluster accessed pattern per an embedding model. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The cumulative distribution function of search latency per nprobe and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Illustration of disk-based IVF search for the baseline and CaGR-RAG, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Cache utilization of EdgeRAG and CaGR-RAG under three datasets. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: The relationship between file size reading from disk, search latency, and [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Search latency comparison between EdgeRAG and CaGR-RAG for dif [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: 99th percentile tail latency of query grouping (QG) and query grouping [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages

  1. [1]

    In: Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)

    Bang, F.: GPTCache: An open-source semantic cache for LLM applications en- abling faster answers and cost savings. In: Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023). pp. 212–

  2. [2]

    Chen, C., Jin, C., Zhang, Y., Podolsky, S., Wu, C., Wang, S.P., Han- son, E., Sun, Z., Walzer, R., Wang, J.: Singlestore-v: An integrated vec- tor database system in singlestore. Proc. VLDB Endow. 17(12), 3772–3785 (Aug 2024). https://doi.org/10.14778/3685800.3685805, https://doi. org/10.14778/3685800.3685805

  3. [3]

    In: Proceedings of the 35th International Conference on Neural Information Processing Systems

    Chen, Q., Zhao, B., Wang, H., Li, M., Liu, C., Li, Z., Yang, M., Wang, J.: Spann: highly-efficient billion-scale approximate nearest neighbor search. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. NIPS ’21, Curran Associates Inc., Red Hook, NY, USA (2021)

  4. [4]

    Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library (2024)

  5. [5]

    Packt Publishing (2013)

    Garg, N.: Apache Kafka. Packt Publishing (2013)

  6. [6]

    Guo, R., Luan, X., Xiang, L., Yan, X., Yi, X., Luo, J., Cheng, Q., Xu, W., Luo, J., Liu, F., Cao, Z., Qiao, Y., Wang, T., Tang, B., Xie, C.: Manu: a cloud na- tive vector database management system. Proc. VLDB Endow.15(12), 3548–3561 (Aug 2022). https://doi.org/10.14778/3554821.3554843, https://doi. org/10.14778/3554821.3554843

  7. [7]

    ACM Transactions on Information Systems 43(2), 1–55 (2025)

    Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al.: A survey on hallucination in large language mod- els: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43(2), 1–55 (2025)

  8. [8]

    Advances in neural information processing Systems32 (2019)

    Jayaram Subramanya, S., Devvrit, F., Simhadri, H.V., Krishnawamy, R., Kadekodi, R.: Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in neural information processing Systems32 (2019)

Show all 21 references
  1. [9]

    Transactions of the Association for Computational Linguistics 7, 452–466 (2019)

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.W., Dai, A.M., Uszkoreit, J., Le, Q., Petrov, S.: Natural questions: A benchmark for question ans...

  2. [10]

    arXiv:2308.03281 (2023)

    Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., Zhang, M.: Towards general text embeddings with multi-stage contrastive learning. arXiv:2308.03281 (2023)

  3. [11]

    arXiv preprint arXiv:1109.2378 (2011)

    Müllner, D.: Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378 (2011)

  4. [12]

    arXiv preprint arXiv:2412.21023 (2024)

    Seemakhupt, K., Liu, S., Khan, S.: Edgerag: Online-indexed rag for edge devices. arXiv preprint arXiv:2412.21023 (2024)

  5. [13]

    Shen, M., Umar, M., Maeng, K., Suh, G.E., Gupta, U.: Towards understand- ing systems trade-offs in retrieval-augmented generation model inference (2024), https://arxiv.org/abs/2412.11854

  6. [14]

    In: IEEE International Conference on Computer Vision

    Sivic,J.,Zisserman,A.:VideoGoogle:Atextretrievalapproachtoobjectmatching in videos. In: IEEE International Conference on Computer Vision. vol. 2, pp. 1470– 1477 (2003)

  7. [15]

    In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021), https://openreview.net/forum? id=wCu6T5xFjeJ

    Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., Gurevych, I.: BEIR: A het- erogeneous benchmark for zero-shot evaluation of information retrieval models. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021), h...

  8. [16]

    In: NAACL-HLT (2018)

    Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: FEVER: a large-scale dataset for fact extraction and VERification. In: NAACL-HLT (2018)

  9. [17]

    arXiv preprint arXiv:2402.05672 (2024)

    Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., Wei, F.: Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024)

  10. [18]

    Wang, M., Xu, W., Yi, X., Wu, S., Peng, Z., Ke, X., Gao, Y., Xu, X., Guo, R., Xie, C.: Starling: An i/o-efficient disk-resident graph index framework for high- dimensional vector similarity search on data segment. Proc. ACM Manag. Data 2(1)(Mar2024). https://doi.org/10.1145/36...

  11. [19]

    NIPS ’20, Curran Associates Inc., Red Hook, NY, USA (2020)

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: Minilm: deep self- attention distillation for task-agnostic compression of pre-trained transformers. NIPS ’20, Curran Associates Inc., Red Hook, NY, USA (2020)

  12. [20]

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., Manning, C.D.: Hotpotqa: A dataset for diverse, explainable multi-hop question answering (2018), https://arxiv.org/abs/1809.09600

  13. [218]

    https: //doi.org/10.18653/v1/2023.nlposs-1.24, https://aclanthology.org/ 2023.nlposs-1.24/

    Association for Computational Linguistics, Singapore (Dec 2023). https: //doi.org/10.18653/v1/2023.nlposs-1.24, https://aclanthology.org/ 2023.nlposs-1.24/

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.