Pith. sign in

REVIEW 3 major objections 5 minor 82 references

RAG-Stack: Co-Optimizing RAG Serving Performance and Quality

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAG-Stack claims that a hybrid cost-model-driven search finds RAG quality–performance Pareto frontiers that cover 52.5% to 153.2% more of the normalized objective space than existing configuration-search methods at the same evaluation budge

desk verdict A genuinely useful framework for co-optimizing RAG quality and serving performance, but the headline hypervolume gains are likely inflated by comparing predicted performance for RAG-Stack against measured performance for baselines. read the letter →

arxiv 2608.03487 v1 pith:UDZ6WVSF submitted 2026-08-04 cs.DB cs.AIcs.IR

Haiqiang Zhang , Yuanqing Lei , Wanting Li , Tao Zhang , Wenqi Jiang This is my paper
classification cs.DBcs.AIcs.IR
keywords retrieval-augmentedgenerationParetofrontiermulti-objectiveBayesianoptimizationperformancecostmodelHNSWvectorsearchsystemdeploymentRAGserving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RAG-Stack is a framework for finding the Pareto frontier between answer quality and serving performance of retrieval-augmented generation (RAG) pipelines, searching both algorithm choices (chunking, retrieval index, reranking, generator, workflow) and deployment choices (placement, batching, parallelism) in one loop. Its central claim is that this joint search can be done without deploying each candidate: a hybrid analytical-machine-learning cost model predicts the best achievable latency and throughput for each algorithm configuration, so the optimizer spends its scarce budget on quality evaluations only. With the same number of optimization iterations, the frontiers RAG-Stack finds cover 52.5% (RAGEval) and 153.2% (MS MARCO) more of the normalized quality–performance space than state-of-the-art configuration-search baselines, averaged over three seeds. If correct, this means a RAG deployment can be co-tuned for quality and speed in a few hours, and an existing frontier can be retargeted to new hardware by re-scoring the cost model instead of re-deploying everything.

What carries the argument

The load-bearing mechanism is the split between algorithm and system design spaces, plus RAG-CM, a four-layer cost model that predicts serving performance without deployment. RAG-CM's algorithm layer turns each executed workflow into hardware-agnostic operator work profiles (operation counts, bytes, access patterns); a roofline model maps those profiles to time on the given hardware; a communication layer prices data movement between stages; and an assembly layer runs a closed discrete-event simulation over the system design space to return the best predicted deployment. The HNSW retrieval model is the one machine-learned piece: two gradient-boosted regressors predict per-query distance comp

What would settle it

Run RAG-CM against a truly new corpus and a hardware generation outside the two servers tested, with no re-calibration beyond the per-corpus constant, and compare predicted versus measured HNSW latency rankings across a grid of M and efSearch values. If the Spearman correlation of predicted versus measured order falls well below the reported 0.95–0.98 range, or if the predicted optimal efSearch flips relative to measured throughput, the transfer claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the quality–performance Pareto frontier of a RAG system can be found efficiently by splitting the configuration space into an algorithm part (choices that change the answer) and a system part (choices that only change how the answer is served), then searching the algorithm part with a stage-aware multi-objective Bayesian optimizer while a cost model exhaustively searches deployments without deploying any. Evidence: on RAGEval and MS MARCO, with identical budgets, RAG-Stack's frontiers average 0.514 and 0.635 normalized hypervolume versus 0.364 and 0.259 for the strongest baseline — per-seed gains of 52.5% and 153.2% — and the calibrated cost mo

Load-bearing premise

RAG-Stack rests on the assumption that the cost model's learned predictor, trained once on synthetic data with only a corpus-geometry feature and a per-corpus scaling constant, will keep ranking index configurations correctly on new corpora and on new hardware; if that generalization fails, the whole search inherits the error.

Editorial extensions

If this is right

  • On a new RAG application, the same evaluation budget yields a Pareto frontier covering roughly half again to two-and-a-half times more of the normalized quality–performance space than existing configuration-search methods.
  • Migrating an optimized RAG system to different hardware can reuse archived quality measurements: 20 additional evaluations with a calibrated cost model recover a frontier 182.2% larger than re-optimizing from scratch.
  • Because ranking, not absolute error, is what guides search, a cost model with 11–15% MAPE and 0.95+ Spearman correlation is sufficient to drive deployment selection.
  • End-to-end search finishes in about 4.6 hours on RAGEval and 3.6 hours on MS MARCO on a four-H100 server, because no candidate is deployed to measure performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The system-space abstraction (workflow schema plus per-request traces) is general enough that the same co-optimization loop could be applied to other multi-stage LLM serving pipelines, such as multi-agent workflows, even though the paper only evaluates sequential and agentic RAG.
  • The reported quality gap between RAGEval and MS MARCO suggests public benchmarks may understate the serving cost of answer quality; a practitioner should re-tune on fresh data before trusting a public-benchmark frontier.
  • The HNSW predictor's reliance on local intrinsic dimensionality implies a strong test: corpora with extreme geometry (nearly uniform or heavily clustered embeddings) should be used as a stress test for the transfer claim.
  • Hardware planning before purchase is a natural use: RAG-CM can score candidate machines from a hardware description, which the paper motivates but does not evaluate as a standalone decision tool.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. RAG-Stack targets the quality-performance Pareto frontier of RAG deployments. It separates the design space into algorithm and system parameters. RAG-PE is a multi-objective Bayesian optimizer (LogNEHVI) augmented with stage-level diagnostics and heterogeneous candidate generation; RAG-IR abstracts executed workflows into a system-agnostic intermediate representation; RAG-CM combines analytical models for retrieval/operators, roofline/Amdahl performance modeling, communication pricing, and a closed-loop assembly that searches system settings. The evaluation claims that, under equal budgets, RAG-Stack frontiers cover 52.5% (RAGEval) and 153.2% (MS MARCO) more normalized hypervolume than GP+LogNEHVI, SMAC, and Greedy-Forward, that RAG-CM has 11.2-15.3% MAPE with Spearman 0.95-0.98, and that system transfer yields 182.2% hypervolume improvement over from-scratch optimization.

Significance. If the claims hold, the work is significant: it attacks a real problem, combines a credible cost model with a stage-aware optimizer, and includes a broad cost-model validation (137-1080 operator configs, 231 end-to-end configs, two systems) with high rank correlations. The optimizer ablation is well designed (77/80 post-warm-start budgets). Release of artifacts is a strength. The main reservation is methodological fairness of the end-to-end comparison because RAG-Stack's performance signal is predicted while baselines are measured.

major comments (3)
  1. [§3, §7.2, Fig. 8] The headline hypervolume comparison appears to mix predicted and measured performance. §3 states that for every candidate 'performance is always predicted by RAG-CM—no candidate is ever deployed to measure it,' while §7.2 says Figure 8 compares 'measured' frontiers and the caption re-measures only selected RAG-Stack configurations. Baselines are deployed and measured on every iteration (§7.1). If the hypervolume values (0.514 vs 0.364; 0.635 vs 0.259) use predicted throughput for RAG-Stack points, the comparison is not between measured frontiers. Rank correlations in Table 4 do not bound absolute bias; a systematic bias in the high-throughput region could shift RAG-Stack points and inflate the reported 52.5%/153.2% gains. Please compute hypervolume from measured throughput for every non-dominated RAG-Stack point, report per-seed error bars and signed bias, and state precisely which point
  2. [§7.1 Baseline system space; §7.2] The baseline search space is not the same as RAG-Stack's. Baselines search algorithm space plus only 8 deployment presets × 6 batch presets, 'expert-curated to be strong on SysA,' while RAG-CM exhaustively searches the full system space in Table 1 for each algorithm configuration. Thus part of the end-to-end gain may come from RAG-CM's larger system design space rather than RAG-PE's algorithm search, even though both sides search the same algorithm space. The optimizer ablation (Fig. 9) controls for this by holding RAG-CM fixed, but the §7.2 claim is presented as end-to-end. Please use the same system space for baselines or decompose the gain into the optimizer's contribution and the system-space search's contribution.
  3. [§6.1.2; §7.5] The HNSW workload predictor relies on g-hat trained on synthetic high-LID data plus a per-corpus scalar c_M obtained by calibration. The manuscript does not specify how c_M is calibrated, whether calibrating requires building/running the target corpus index, or how robust predictions are to c_M. Since the transfer headline (182.2%) uses a calibrated variant on SysB, while the uncalibrated variant yields 108.4%, the calibration protocol and its cost materially affect the transfer claim. Please provide the calibration procedure, its cost, and a sensitivity analysis.
minor comments (5)
  1. [§6.1.2] 'Amsaleg MLE' is mentioned without a citation; please add one.
  2. [Fig. 3] The right panel's 'median QPS ratio' is not fully defined; specify whether ratios are medians of per-config ratios or ratio of medians, and define the IQR error bars.
  3. [Fig. 11] The legend labels 'Pool win' and 'Forced Init' are unclear; add definitions or a caption explanation.
  4. [Table 3] The RAG-Stack MS MARCO runtime has a large standard deviation (±1.79 h); please comment on the source of this variance.
  5. [§8] The trace-driven limitation for agentic RAG is acknowledged; consider scoping the abstract's 'without deploying every candidate' claim to sequential RAG or quantifying agentic coverage in the evaluation.

Circularity Check

1 steps flagged · score 4.0 of 10

One calibrated per-corpus constant in RAG-CM is fitted to target-corpus measurements, so a portion of the HNSW 'prediction' is an input; however, the core optimizer and system-space search remain independent.

  1. fitted input called prediction [Section 6.1.2 (Graph Indices)]
    "Concretely, the HNSW operator model estimates ˆn_dis = c_M ˆg_dis(z) and ˆn_hops = ˆg_hops(z) with feature vector z=(d,M,ef_s,LID): a learned predictor ˆg emits the workload counts ˆn=(ˆn_dis, ˆn_hops), and a factor c_M obtained through one-time per-corpus calibration fixes the scale of ˆn_dis."

    c_M is a per-corpus scalar obtained by calibrating to measured data on the target corpus. It fixes the absolute scale of n_dis, so HNSW latency/QPS predictions inherit a fitted constant from the same corpus the model is asked to predict. Table 4 reports MAPE for the calibrated model, so the accuracy claim is not an out-of-sample test of the uncalibrated predictor. The fitted scale does not drive within-corpus ordering (Spearman is preserved), but it does set absolute throughput and influences cross-index frontier placement, so the 'prediction' is partially fitted to its target.

full rationale

No definitional equivalence or self-citation chain forces the main result. RAG-PE is a standard MOBO loop with stage-guided candidates; RAG-IR is a workload abstraction; and RAG-CM's analytical layers (roofline, Amdahl, communication, discrete-event assembly) are independent of the target Pareto frontier. The only fitted input that leaks target data into a 'prediction' is the HNSW per-corpus scale c_M (and analogously the IVF imbalance factor f), which are calibrated on measured per-query work. This is disclosed and affects absolute scale more than configuration ordering, so it is a partial, not total, circularity. Separately, the end-to-end comparison in Sec. 7.2 computes hypervolume from RAG-CM-predicted throughput for most RAG-Stack points while baselines are measured; that is a validity/fairness risk rather than a circular derivation, and it is not scored here as a circular step.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The system is a modelling/optimization framework, so its overhead is concentrated in learned/calibrated performance-model components (c_M, f, g-hat regressors, LID) and in domain assumptions about the algorithm/system split and the validity of a closed-loop steady-state serving model. No new physical entity is postulated. The free-parameter count is moderate for a systems paper and the paper is honest about the calibrations.

free parameters (5)
  • c_M (HNSW scan-count calibration factor) = per-corpus calibration, exact values in artifact
    Section 6.1.2: 'a factor c_M obtained through one-time per-corpus calibration fixes the scale of n_dis'. This is fitted to measured distance counts on the target corpus.
  • f (IVF cell-imbalance factor) = measured from built index per (corpus, embedding)
    Section 6.1.1: 'we correct with a cell-imbalance factor f measured from the built index... obtained from the index's per-list sizes and the nearest-list assignments of a query sample'. This is data-derived, though the paper argues it is corpus-only and cached.
  • g_dis / g_hops gradient-boosted regressor weights = trained on synthetic data, exact weights in artifact
    Section 6.1.2: the workload predictor g-hat is trained to predict n_dis and n_hops; the paper claims training on high-LID synthetic data, but the regressor has fitted parameters.
  • LID (local intrinsic dimensionality) estimate = Amsaleg MLE on a query sample
    Used as a feature in the HNSW predictor, estimated from a query sample (Section 6.1.2). Not a free parameter in the classical sense, but an estimated input.
  • RAG-CM hardware tuning / calibration constants = e.g., T_ovh, pi_eff, beta_eff per hardware
    The calibrated transfer variant in Section 7.5 'additionally calibrates RAG-CM on the target machine', indicating hardware-dependent fitted constants.
assumptions (5)
  • domain assumption System parameters do not affect answer quality (I_Q subseteq I_P split).
    Section 4.1.1, the entire algorithm/system decomposition rests on the claim that batching, parallelism, thread counts and hardware leave the logical computation and thus the answer unchanged; floating-point reduction-order perturbations are dismissed as not changing the logical computation.
  • domain assumption The GP+LogNEHVI acquisition and the mixed Matérn/Hamming kernel provide an adequate surrogate over the hierarchical conditional RAG space.
    Section 4.2.1, borrowed from BoTorch; the paper does not formalize why this kernel class captures RAG stage interactions, only that it outperforms SMAC/ParEGO empirically.
  • domain assumption RAGAS answer correctness, context recall/precision and faithfulness, scored by DeepSeek-V4-Flash, is a valid quality objective and valid stage diagnostics.
    Section 7.1, quality metric choice; no calibration of the LLM-judge against human ratings is reported.
  • domain assumption Saturated closed-loop serving with fixed concurrency is the right performance model for RAG deployment comparison.
    Section 6.5.1, the assembly simulates a saturated closed-loop system and reports throughput at saturation; open-system latency under real arrival patterns is deliberately excluded.
  • domain assumption The discrete-event simulation captures continuous batching, prefix caching, decoupled batch limits and shared-resource contention faithfully.
    Section 6.5.2 lists these mechanisms; the simulation is validated indirectly via Table 4 but the simulator itself is assumed to faithfully represent a real serving system (vLLM, FAISS, etc.).
invented entities (3)
  • RAG-IR workload representation (workflow schema + per-request execution DAG) independent evidence
    purpose: Bridges RAG-PE and RAG-CM, decoupling logical work from deployment.
    It is an abstraction, not a physical entity; its validity is evidenced by the RAG-CM accuracy measurements in Table 4.
  • HNSW workload predictor g-hat with LID feature independent evidence
    purpose: Predicts n_dis and n_hops for unbuilt HNSW indexes.
    It is a model with fitted parameters, but it is evaluated on held-out corpora (SIFT1M, GloVe, GIST1M, Deep1M, ELI5, TriviaQA) in Table 4.
  • Agentic analyzer (sub-metric based candidate generator)
    purpose: Produces stage-guided candidate configurations for RAG-PE.
    Internal optimizer heuristic; its value is shown only through the ablation contributions in Figure 11.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG-Stack: Co-Optimizing RAG Serving Performance and Quality." pith.science (2026). https://pith.science/paper/UDZ6WVSF

@misc{pith2026260803487,
  author       = {Pith},
  title        = {Pith review of: RAG-Stack: Co-Optimizing RAG Serving Performance and Quality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UDZ6WVSF}},
  note         = {Machine review of arXiv:2608.03487}
}
read the original abstract

Retrieval-augmented generation (RAG), which augments large language model (LLM) generation with information retrieved from databases, has become a widely used approach for knowledge-intensive applications. Modern RAG systems, however, expose many configuration choices, such as retrieval indexes, model selections, and how models invoke retrieval. Each configuration yields a different trade-off between answer quality and serving performance, making it challenging to choose the optimal setting for a specific application deployment. We present RAG-Stack, a framework for efficiently discovering quality-performance Pareto frontiers across diverse RAG applications and serving systems. RAG-Stack consists of RAG-PE, an iterative design-space exploration algorithm that selects the next RAG configuration to evaluate; RAG-IR, a workload abstraction for diverse RAG algorithms; and RAG-CM, a performance model that predicts the optimal deployment and serving performance on the given hardware. Together, these components allow RAG-Stack to search the joint algorithm-system configuration space without deploying every candidate and to transfer an existing Pareto frontier to a new serving system. Given the same number of optimization iterations across diverse datasets, the Pareto frontiers found by RAG-Stack cover 52.5% to 153.2% more of the normalized quality-performance space than those found by state-of-the-art configuration-search methods evaluated over the same RAG design space.

Figures

Figures reproduced from arXiv: 2608.03487 by the authors.

Figure 1
Figure 1. Modern RAG system comprises diverse model [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview ofRAG-Stack and its two operating modes. degrade quality when noisy passages reach a weaker generator with￾out reranking. By contrast, global RAG optimization captures these interactions by jointly tuning parameters across all stages using end-to-end accuracy as the optimization signal [12]. However, this end-to-end signal does not reveal which stageis responsible for a gain or loss, potentially leading to … view at source ↗
Figure 3
Figure 3. Median quality–throughput tension across algo [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Cross-stage interactions in the algorithm design [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Algorithm design space [23, 29, 36, 51, 54] searched by RAG-PE. The highlighted selected nodes are one example, showing how a single evaluation’s algorithm configuration is assembled: each trial fixes one value at every active branch of the hierarchy (retrieval-side in…
Figure 6
Figure 6. Figure 6: The optimizer of RAG-PE. 4.2.2 Our optimizer: stage–global co-awareMOBO. RAG-PE extends conventional MOBO for RAG with stage-aware diagnostics, het￾erogeneous candidate generation, and global arbitration, making exploration more directed while preserving end-to-end Par…
Figure 7
Figure 7. Figure 7: Overview of RAG-CM. A: analytical; ML: fused machine learning and analytical [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Optimizer ablation: dominated hypervolume versus [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 12
Figure 12. Figure 12: System transfer from SysA to SysB. The transferred [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 11
Figure 11. Figure 11: Per-channel contributions to RAG-PE’s results in §7.3: shares of all quality evaluations (left) and final hypervolume (right) [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 24 canonical work pages

  1. [1]

    Algorithmicsuperintelligence/Openevolve

    2026. Algorithmicsuperintelligence/Openevolve. Algorithmic SuperIntelligence Labs

    2026

  2. [2]

    FlagOpen/FlagEmbedding

    2026. FlagOpen/FlagEmbedding. FlagOpen

    2026

  3. [3]

    Marker-Inc-Korea/AutoRAG

    2026. Marker-Inc-Korea/AutoRAG. Markr.AI

    work page 2026

  4. [4]

    Sebastian Ament, Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. 2025. Unexpected Improvements to Expected Improve- ment for Bayesian Optimization. https://doi.org/10.48550/arXiv.2310.20708 arXiv:2310.20708 [cs]

    arXiv doi:10.48550/arxiv.2310.20708 2025

  5. [5]

    Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023. Task-Aware Retrieval with Instructions. InFindings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Can...

    doi:10.18653/v1/2023.findings-acl.225 2023

  6. [6]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. https://doi.org/10.48550/arXiv.2310.11511 arXiv:2310.11511 [cs]

    arXiv doi:10.48550/arxiv.2310.11511 2023

  7. [7]

    Raul Astudillo and Peter Frazier. 2019. Bayesian Optimization of Composite Functions. InProceedings of the 36th International Conference on Machine Learning. PMLR, 354–363

    work page 2019

  8. [8]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv doi:10.48550/arxiv.2309.16609 2023

Show all 82 references
  1. [9]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A Human Generated MAchine Reading COmprehension Da...

    2018 doi arXiv

  2. [10]

    Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. 2020. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. InAdvances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 21...

    2020

  3. [11]

    Abhimanyu Bambhaniya, Ritik Raj, Geonhwa Jeong, Souvik Kundu, Sudarshan Srinivasan, Suvinay Subramanian, Midhilesh Elavazhagan, Madhu Kumar, and Tushar Krishna. 2025. Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM Models. https://doi.org/10.48...

    2025 doi arXiv

  4. [12]

    Matthew Barker, Andrew Bell, Evan Thomas, James Carr, Thomas Andrews, and Umang Bhatt. 2025. Faster, Cheaper, Better: Multi-Objective Hyperparameter Op- timization for LLM and RAG Systems. https://doi.org/10.48550/arXiv.2502.18635 arXiv:2502.18635 [cs]

    2025 doi arXiv

  5. [13]

    Maciej Besta, Lorenzo Paleari, Jia Hao Andrea Jiang, Robert Gerstenberger, You Wu, Jón Gunnar Hannesson, Patrick Iff, Ales Kubicek, Piotr Nyczyk, Diana Khimey, Nils Blach, Haiqiang Zhang, Tao Zhang, Peiran Ma, Grzegorz Kwaśniewski, Marcin Copik, Hubert Niewiadomski, and Torste...

    2025 doi

  6. [14]

    Hans-Georg Beyer and Hans-Paul Schwefel. 2002. Evolution Strategies – A Comprehensive Introduction.Natural Computing1, 1 (March 2002), 3–52. https://doi.org/10.1023/A:1015059928466

    2002 doi

  7. [15]

    Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

  8. [16]

    Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024. FinTextQA: A Dataset for Long-form Financial Question Answering. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

    2024 doi

  9. [17]

    Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. 2021. Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improve- ment. https://doi.org/10.48550/arXiv.2105.08195 arXiv:2105.08195 [cs.LG]

    2021 doi arXiv

  10. [18]

    K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II.IEEE Transactions on Evolutionary Computation6, 2 (April 2002), 182–197. https://doi.org/10.1109/4235.996017

    2002

  11. [19]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2025. The Faiss Library. https://doi.org/10.48550/arXiv.2401.08281 arXiv:2401.08281 [cs]

    2025 doi arXiv

  12. [20]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated Evaluation of Retrieval Augmented Generation. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Nikola...

    2024 doi

  13. [21]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason We- ston, and Michael Auli. 2019. ELI5: Long Form Question Answering. https://doi.org/10.48550/arXiv.1907.09190 arXiv:1907.09190 [cs.CL]

    2019 doi arXiv

  14. [22]

    Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. 2024. AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation. https://doi.org/10.48550/arXiv.2406.19251 arXiv:2406.192...

    2024 doi arXiv

  15. [23]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Pre- cise Zero-Shot Dense Retrieval without Relevance Labels. https: //doi.org/10.48550/arXiv.2212.10496 arXiv:2212.10496 [cs.IR]

    2022 doi arXiv

  16. [24]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Ji- awei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. https://doi.org/10.48550/arXiv.2312.10997 arXiv:2312.10997 [cs.CL]

    2024 doi arXiv

  17. [25]

    Nikolaus Hansen. 2023. The CMA Evolution Strategy: A Tutorial. https://doi.org/10.48550/arXiv.1604.00772 arXiv:1604.00772 [cs.LG]

    2023 doi arXiv

  18. [26]

    Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. 2018. PipeDream: Fast and Efficient Pipeline Parallel DNN Training. https://doi.org/10.48550/arXiv.1806.03377 arXiv:1806.03377 [cs.DC]

    2018 doi arXiv

  19. [27]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism. https://doi.org/10.48550/arXiv.1811.069...

    2019 doi arXiv

  20. [28]

    Pooyan Jamshidi, Miguel Velez, Christian Kästner, and Norbert Siegmund. 2018. Learning to Sample: Exploiting Similarities across Environments to Learn Per- formance Models for Configurable Systems. InProceedings of the 2018 26th ACM Joint Meeting on European Software Engineeri...

    2018

  21. [29]

    Herve Jégou, Matthijs Douze, and Cordelia Schmid. 2011. Product Quantization for Nearest Neighbor Search.IEEE Transactions on Pattern Analysis and Machine Intelligence33, 1 (Jan. 2011), 117–128. https://doi.org/10.1109/TPAMI.2010.57

    2011 doi

  22. [30]

    Wenqi Jiang, Suvinay Subramanian, Cat Graves, Gustavo Alonso, Amir Yazdanbakhsh, and Vidushi Dadu. 2025. RAGO: Systematic Perfor- mance Optimization for Retrieval-Augmented Generation Serving. https://doi.org/10.48550/arXiv.2503.14649 arXiv:2503.14649 [cs]

    2025 doi arXiv

  23. [31]

    Wenqi Jiang, Marco Zeller, Roger Waleffe, Torsten Hoefler, and Gus- tavo Alonso. 2025. Chameleon: A Heterogeneous and Disaggre- gated Accelerator System for Retrieval-Augmented Language Models. https://doi.org/10.48550/arXiv.2310.09949 arXiv:2310.09949 [cs]

    2025 doi arXiv

  24. [32]

    Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska

  25. [33]

    Jiajie Jin, Yutao Zhu, Guanting Dong, Yuyao Zhang, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, Zhicheng Dou, and Ji-Rong Wen. 2025. FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research. InCompanion Proceedings of the ACM on Web Conference 202...

    2025

  26. [34]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. 13 https://doi.org/10.48550/arXiv.1705.03551 arXiv:1705.03551 [cs.CL]

    2017 doi arXiv

  27. [35]

    Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabas Poczos

  28. [36]

    Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Ef- fective Passage Search via Contextualized Late Interaction over BERT. https://doi.org/10.48550/arXiv.2004.12832 arXiv:2004.12832 [cs.IR]

    2020 doi arXiv

  29. [37]

    Junkyum Kim and Divya Mahajan. 2026. VectorLiteRAG: Latency- Aware and Fine-Grained Resource Partitioning for Efficient RAG. https://doi.org/10.48550/arXiv.2504.08930 arXiv:2504.08930 [cs]

    2026 doi

  30. [38]

    J. Knowles. 2006. ParEGO: A Hybrid Algorithm with on-Line Landscape Approximation for Expensive Multiobjective Optimization Problems. IEEE Transactions on Evolutionary Computation10, 1 (Feb. 2006), 50–66. https://doi.org/10.1109/TEVC.2005.851274

    2006

  31. [39]

    Akshay Kudva, Wei-Ting Tang, and Joel A. Paulson. 2026. Multi-Objective Bayesian Optimization for Networked Black-Box Systems: A Path to Greener Profits and Smarter Designs. https://doi.org/10.48550/arXiv.2502.14121 arXiv:2502.14121 [stat]

    2026 doi

  32. [40]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. https://doi.org/10.48550/arXiv.2309.06180 arXiv:2309.06180 [cs.LG]

    2023 doi arXiv

  33. [41]

    Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. 2025. GPTuner: An LLM- Based Database Tuning System.SIGMOD Rec.54, 1 (April 2025), 101–110. https://doi.org/10.1145/3733620.3733641

    2025

  34. [42]

    Julien-Charles Lévesque, Audrey Durand, Christian Gagné, and Robert Sabourin

  35. [43]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InProceedings o...

    2020

  36. [44]

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala

  37. [45]

    Shaobo Li, Yirui Zhou, Yuan Xu, Kevin Chen, Daniel Waddington, Swaminathan Sundararaman, Hubertus Franke, and Jian Huang. 2026. RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems. https://doi.org/10.48550/arXiv.2603.10765 arXiv:2603.10765 [cs.PF]

    2026 doi

  38. [46]

    In 2017 International Joint Conference on Neural Networks (IJCNN)

    Bayesian Optimization for Conditional Hyperparameter Spaces. In 2017 International Joint Conference on Neural Networks (IJCNN). 286–293. https://doi.org/10.1109/IJCNN.2017.7965867

    2017

  39. [47]

    Ning Liang, Fabian Wenz, Jana Giceva, and Lisa Wu Wills. 2025. Athena: A Plug-and-Play Advisor for Retrieval-Augmented Generation Using VectorDB. In 2025 IEEE International Symposium on Workload Characterization (IISWC). 28–41. https://doi.org/10.1109/IISWC66894.2025.00013

    2025

  40. [48]

    Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhopf, René Sass, and Frank Hutter

  41. [49]

    Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. 2024. Large Language Models to Enhance Bayesian Optimization. https://doi.org/10.48550/arXiv.2402.03921 arXiv:2402.03921 [cs]

    2024 doi arXiv

  42. [50]

    Zhichao Lu, Gautam Sreekumar, Erik Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, and Vishnu Naresh Boddeti. 2021. Neural Architecture Transfer.IEEE Transactions on Pattern Analysis and Machine Intelligence43, 9 (Sept. 2021), 2971–2989. https://doi.org/10.1109/TPAMI.2021.3052758

    2021

  43. [51]

    Yangning Li, Weizhi Zhang, Yuyao Yang, Wei-Chieh Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, Chunkit Chan, Yankai Chen, Zhongfen Deng, Yinghui Li, Hai-Tao Zheng, Dongyuan Li, Renhe Jiang, Ming Zhang, Yangqiu Song, and Philip S. Yu. 2025. T...

    2025 doi arXiv

  44. [52]

    Alexander Novikov, Ngân V ˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Push...

    2025 doi arXiv

  45. [53]

    Miles Olson, Elizabeth Santorella, Louis C. Tiao, Sait Cakmak, Mia Garrard, Samuel Daulton, Zhiyuan Jerry Lin, Sebastian Ament, Bernard Beckerman, Eric Onofrey, Paschal Igusti, Cristian Lara, Benjamin Letham, Cesar Cardoso, Shiyun Sunny Shen, Andy Chenyuan Lin, Matthew Grange,...

    2025

  46. [54]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. InFi...

    2024 doi

  47. [55]

    Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. 2019. A Flex- ible Framework for Multi-Objective Bayesian Optimization Using Random Scalarizations. https://doi.org/10.48550/arXiv.1805.12168 arXiv:1805.12168 [cs]

    2019 doi arXiv

  48. [56]

    Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu. 2025. Long$^2$RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall. https://doi.org/10.48550/arXiv.2410.23000 arXiv:2410.23000 [cs]

    2025 doi arXiv

  49. [57]

    Malkov and D

    Yu A. Malkov and D. A. Yashunin. 2018. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. https://doi.org/10.48550/arXiv.1603.09320 arXiv:1603.09320 [cs]

    2018 doi arXiv

  50. [58]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. https://doi.org/10.48550/arXiv.1908.10084 arXiv:1908.10084 [cs.CL]

    2019 doi arXiv

  51. [59]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Lan- guage Models Using Model Parallelism. https://doi.org/10.48550/arXiv.1909.08053 arXiv:1909.08053 [cs.CL]

    2020 doi arXiv

  52. [60]

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval Augmentation Reduces Hallucination in Conversation. InFindings of the Association for Computational Linguistics: EMNLP 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-...

    2021 doi

  53. [61]

    Vasilakos

    Aditi Singh, Abul Ehtesham, Saket Kumar, Tala Talaei Khoei, and Athanasios V. Vasilakos. 2026. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. https://doi.org/10.48550/arXiv.2501.09136 arXiv:2501.09136 [cs]

    2026 doi arXiv

  54. [62]

    Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kalu- arachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering.Transactions of the Association for Com...

    2023 doi

  55. [63]

    Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Shaoting Feng, Ganesh Ananthanarayanan, Ravi Netravali, and Junchen Jiang. 2025. METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation. https://doi.org/10.48550/arXiv.2412.10543 arXiv:2412.10543 [cs.LG]

    2025 doi

  56. [64]

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong

  57. [65]

    Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, Kun Yu, Yuxing Yuan, Yinghao Zou, Jiquan Long, Yudong Cai, Zhenxiang Li, Zhifeng Zhang, Yihua Mo, Jun Gu, Ruiyi Jiang, Yi Wei, and Charles Xie. 2021. M...

    2021

  58. [66]

    Shuhei Watanabe. 2026. Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance. https://doi.org/10.48550/arXiv.2304.11127 arXiv:2304.11127 [cs.LG]

    2026 doi arXiv

  59. [67]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2025. WizardLM: Empow- ering Large Pre-Trained Language Models to Follow Complex Instructions. https://doi.org/10.48550/arXiv.2304.12244 arXiv:2304.12244 [cs]

    2025 doi arXiv

  60. [68]

    Artem Babenko Yandex and Victor Lempitsky. 2016. Efficient Indexing of Billion- Scale Datasets of Deep Descriptors. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2055–2063. https://doi.org/10.1109/CVPR.2016.226

    2016 doi

  61. [69]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. MP- Net: Masked and Permuted Pre-training for Language Understanding. https://doi.org/10.48550/arXiv.2004.09297 arXiv:2004.09297 [cs.CL]

    2020 doi arXiv

  62. [70]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538

    2022

  63. [71]

    InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.)

    FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Asso- ciation for Computational Linguistics, Bangkok, Thailand, 13697–13720...

    2024 doi

  64. [72]

    Tianyang Zhang, Zhuoxuan Jiang, Shengguang Bai, Tianrui Zhang, Lin Lin, Yang Liu, and Jiawei Ren. 2024. RAG4ITOps: A Supervised Fine-Tunable and Comprehensive RAG Framework for IT Operations and Maintenance. In Proceedings of the 2024 Conference on Empirical Methods in Natural...

    2024 doi

  65. [73]

    Tao Zhang, Kaixian Qu, Zhibin Li, Jiajun Wu, Marco Hutter, Manling Li, and Fan Shi

  66. [74]

    Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework. https://doi.org/10.48550/arXiv.2408.01262 arX...

    2025 doi arXiv

  67. [76]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. https://doi.org/10.48550/arXiv.2210.03629 arXiv:2210.03629 [cs.CL]

    2023 doi arXiv

  68. [78]

    Qingfu Zhang and Hui Li. 2007. MOEA/D: A Multiobjective Evolutionary Algo- rithm Based on Decomposition.IEEE Transactions on Evolutionary Computation 14 11, 6 (Dec. 2007), 712–731. https://doi.org/10.1109/TEVC.2007.892759

    2007

  69. [2017]

    https://doi.org/10.48550/arXiv.1703.06240 arXiv:1703.06240 [stat]

    Multi-Fidelity Bayesian Optimisation with Continuous Approximations. https://doi.org/10.48550/arXiv.1703.06240 arXiv:1703.06240 [stat]

    doi arXiv

  70. [2020]

    https://doi.org/10.48550/arXiv.2006.15704 arXiv:2006.15704 [cs.DC]

    PyTorch Distributed: Experiences on Accelerating Data Parallel Training. https://doi.org/10.48550/arXiv.2006.15704 arXiv:2006.15704 [cs.DC]

    2006 doi arXiv

  71. [2022]

    https://doi.org/10.48550/arXiv.2109.09831 arXiv:2109.09831 [cs]

    SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization. https://doi.org/10.48550/arXiv.2109.09831 arXiv:2109.09831 [cs]

    doi arXiv

  72. [2024]

    InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.)

    M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computationa...

    2024 doi

  73. [2025]

    InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD ’25)

    PipeRAG: Fast Retrieval-Augmented Generation via Adaptive Pipeline Parallelism. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD ’25). Association for Computing Machinery, New York, NY, USA, 589–600. https://doi.org/10.1145/369062...

  74. [2026]

    https://doi.org/10.48550/arXiv.2604.18463 arXiv:2604.18463 [cs.AI]

    Using Large Language Models for Embodied Planning Introduces Systematic Safety Risks. https://doi.org/10.48550/arXiv.2604.18463 arXiv:2604.18463 [cs.AI]

    doi arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.