{"id":"9a321251-b0e2-422f-9e50-f5c448a81d39","arxiv_id":"2608.03487","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RAG-Stack jointly optimizes RAG algorithm choices and serving-system settings via sub-metric-aware multi-objective Bayesian optimization plus an analytical performance model, reporting Pareto frontiers covering 52.5% to 153.2% more normalized quality-performance space than baselines.","lead":"RAG-Stack is a framework that automatically searches for the best quality-versus-speed settings in retrieval-augmented generation (RAG) systems, combining a smart search algorithm with a cost model that predicts serving performance without deploying every candidate. A smart generalist might read this to see how Bayesian optimization and analytical performance models can jointly tune both the AI pipeline and the serving hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RAG-Stack's headline hypervolume gains may be computed from predicted, not measured, performance; only selected frontier points are re-measured (Fig. 8).","rationale":"The reader's conditional verdict already centers RAG-CM reliability, but their weakest_assumption focuses on HNSW cost-model generalization across corpora and hardware. My stress-test identifies a sharper and more directly headline-threatening risk: the end-to-end comparison may use RAG-CM's predicted performance for RAG-Stack while baselines are measured, and only selected RAG-Stack configurations are re-measured. Table 4 validates RAG-CM on separate datasets and reports MAPE plus Spearman correlation, but high rank correlation does not rule out a systematic absolute bias that would inflate hypervolume. The optimizer ablation (§7.3) shows RAG-PE alone adds only 13.8–14.8% hypervolume over baselines at budget 50, which suggests the much larger end-to-end gains (52.5%, 153.2%) may be driven primarily by the predicted-performance/system-search component—making the fairness of the comparison and the predicted-vs-measured distinction central. This does not change the conditional verdict, but the condition should explicitly require measured performance for all reported frontier points and signed prediction-error reporting, not merely additional seeds or error bars.","tokens_in":26172,"tokens_out":7005,"duration_ms":83350,"concrete_test":"Deploy every non-dominated configuration on RAG-Stack's final frontiers (and all baseline points) on SysA under the saturated closed-loop measurement protocol used in §7.2, then recompute normalized hypervolume using measured QPS for both RAG-Stack and baselines. Also report signed predicted-minus-measured QPS for the frontier points and seed-level hypervolume differences. If the 52.5%/153.2% gains shrink below seed-level noise or roughly halve, the central claim should be restated as a predicted-performance result or conditioned on full measured-frontier validation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The end-to-end claim (§7.2, Fig. 8) states that RAG-Stack frontiers cover 52.5%/153.2% more normalized quality-performance space than baselines 'evaluated over the same RAG design space.' But RAG-Stack never deploys candidates during search: serving performance is always RAG-CM's prediction (§3), and Fig. 8's caption says only 'Selected RAG-Stack configurations are re-measured on SysA.' Baselines, by contrast, must be deployed and measured at every iteration (§7.1 baseline system space). Hypervolume is computed from normalized quality and raw throughput (§7.2). If RAG-CM's 11.2–15.3% MAPE is optimistically biased in the high-throughput region—e.g., systematic under-prediction of batching or communication overhead—RAG-Stack's frontier points shift right in the quality-performance plane, inflating hypervolume relative to measured baselines. Table 4's Spearman correlations (0.95–0.98) show good ranking but do not bound absolute bias, and no signed-bias or per-frontier error bars are reported. The comparison is further complicated because baseline system space is compressed to 8 deployment × 6 batch presets, so the two sides are not literally searching the same design space. The most load-bearing risk is therefore not HNSW cross-corpus generalization per se, but whether the headline numbers survive replacing predicted performance with measured performance for all frontier points.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"RAG-Stack targets the quality-performance Pareto frontier of RAG deployments. It separates the design space into algorithm and system parameters. RAG-PE is a multi-objective Bayesian optimizer (LogNEHVI) augmented with stage-level diagnostics and heterogeneous candidate generation; RAG-IR abstracts executed workflows into a system-agnostic intermediate representation; RAG-CM combines analytical models for retrieval/operators, roofline/Amdahl performance modeling, communication pricing, and a closed-loop assembly that searches system settings. The evaluation claims that, under equal budgets, RAG-Stack frontiers cover 52.5% (RAGEval) and 153.2% (MS MARCO) more normalized hypervolume than GP+LogNEHVI, SMAC, and Greedy-Forward, that RAG-CM has 11.2-15.3% MAPE with Spearman 0.95-0.98, and that system transfer yields 182.2% hypervolume improvement over from-scratch optimization.","tokens_in":26602,"tokens_out":6707,"duration_ms":74019,"significance":"If the claims hold, the work is significant: it attacks a real problem, combines a credible cost model with a stage-aware optimizer, and includes a broad cost-model validation (137-1080 operator configs, 231 end-to-end configs, two systems) with high rank correlations. The optimizer ablation is well designed (77/80 post-warm-start budgets). Release of artifacts is a strength. The main reservation is methodological fairness of the end-to-end comparison because RAG-Stack's performance signal is predicted while baselines are measured.","major_comments":[{"comment":"The headline hypervolume comparison appears to mix predicted and measured performance. §3 states that for every candidate 'performance is always predicted by RAG-CM—no candidate is ever deployed to measure it,' while §7.2 says Figure 8 compares 'measured' frontiers and the caption re-measures only selected RAG-Stack configurations. Baselines are deployed and measured on every iteration (§7.1). If the hypervolume values (0.514 vs 0.364; 0.635 vs 0.259) use predicted throughput for RAG-Stack points, the comparison is not between measured frontiers. Rank correlations in Table 4 do not bound absolute bias; a systematic bias in the high-throughput region could shift RAG-Stack points and inflate the reported 52.5%/153.2% gains. Please compute hypervolume from measured throughput for every non-dominated RAG-Stack point, report per-seed error bars and signed bias, and state precisely which point","section":"§3, §7.2, Fig. 8"},{"comment":"The baseline search space is not the same as RAG-Stack's. Baselines search algorithm space plus only 8 deployment presets × 6 batch presets, 'expert-curated to be strong on SysA,' while RAG-CM exhaustively searches the full system space in Table 1 for each algorithm configuration. Thus part of the end-to-end gain may come from RAG-CM's larger system design space rather than RAG-PE's algorithm search, even though both sides search the same algorithm space. The optimizer ablation (Fig. 9) controls for this by holding RAG-CM fixed, but the §7.2 claim is presented as end-to-end. Please use the same system space for baselines or decompose the gain into the optimizer's contribution and the system-space search's contribution.","section":"§7.1 Baseline system space; §7.2"},{"comment":"The HNSW workload predictor relies on g-hat trained on synthetic high-LID data plus a per-corpus scalar c_M obtained by calibration. The manuscript does not specify how c_M is calibrated, whether calibrating requires building/running the target corpus index, or how robust predictions are to c_M. Since the transfer headline (182.2%) uses a calibrated variant on SysB, while the uncalibrated variant yields 108.4%, the calibration protocol and its cost materially affect the transfer claim. Please provide the calibration procedure, its cost, and a sensitivity analysis.","section":"§6.1.2; §7.5"}],"minor_comments":[{"comment":"'Amsaleg MLE' is mentioned without a citation; please add one.","section":"§6.1.2"},{"comment":"The right panel's 'median QPS ratio' is not fully defined; specify whether ratios are medians of per-config ratios or ratio of medians, and define the IQR error bars.","section":"Fig. 3"},{"comment":"The legend labels 'Pool win' and 'Forced Init' are unclear; add definitions or a caption explanation.","section":"Fig. 11"},{"comment":"The RAG-Stack MS MARCO runtime has a large standard deviation (±1.79 h); please comment on the source of this variance.","section":"Table 3"},{"comment":"The trace-driven limitation for agentic RAG is acknowledged; consider scoping the abstract's 'without deploying every candidate' claim to sequential RAG or quantifying agentic coverage in the evaluation.","section":"§8"}],"recommendation":"major_revision","confidential_remarks":"The central concern is fixable by re-measuring all frontier points; given the available artifact, this is a revision rather than a rejection. I would also ask the editor to ensure the baseline-system-space asymmetry is addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely useful: split RAG configuration into algorithm parameters (which affect quality) and system parameters (which don't), then search the algorithm space with a multi-objective Bayesian optimizer while a hybrid ML-analytical cost model exhaustively searches the system space for each candidate. That division of labor directly addresses a real pain point in RAG serving. The cost-model validation (Table 4) is the strongest part: 231 end-to-end configurations across two systems, Spearman correlations above 0.95, MAPE 11–15%. The optimizer ablation (Figure 9) controls for the rest of the system and shows RAG-PE is consistently ahead. The paper also ships code and data, and it clearly states the trace-driven limitation of agentic RAG.\n\nThe biggest issue is the one the stress-test note flags: the headline hypervolume gains (52.5% and 153.2%) appear to be computed from RAG-CM predicted throughput for RAG-Stack, while baselines are measured every iteration. The caption says only “selected RAG-Stack configurations are re-measured on SysA,” and the method section says performance is always predicted. If the cost model’s 11–15% MAPE is biased in the high-throughput region, RAG-Stack’s frontier points shift right in the quality-throughput plane, inflating hypervolume relative to a measured baseline. Spearman correlations don’t bound absolute bias, and the paper doesn’t report signed bias or per-frontier error bars. This isn’t a fabrication, but it means the headline numbers are not yet trustworthy.\n\nRelatedly, the baseline system space is compressed to 8 deployment presets × 6 batch presets, while RAG-CM searches a much richer space. The paper justifies this as necessary for the baselines, but it means the two sides are not literally searching the same design space, and the abstract’s claim “evaluated over the same RAG design space” is misleading.\n\nSecond, the HNSW cost model depends on a per-corpus calibration factor c_M and a learned regressor trained largely on synthetic high-LID data. The paper validates on 6–7 corpora and two server types, which is acceptable for a systems paper, but the generalization story is thinner than the rest. The evidence base is also narrow: two datasets, one quality judge, three seeds.\n\nThis is for anyone working on RAG serving systems or DB/ML co-design. A serious editor should send it to review: the system is well-designed, the evaluation is extensive but not conclusive, and the concerns are addressable in a revision. I’d bring it to a reading group, and I’d cite the cost model and the algorithm/system split even while being careful about the headline numbers.","headline":"A genuinely useful framework for co-optimizing RAG quality and serving performance, but the headline hypervolume gains are likely inflated by comparing predicted performance for RAG-Stack against measured performance for baselines.","tokens_in":27111,"tokens_out":5349,"would_cite":true,"duration_ms":53502,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAG-Stack claims that a hybrid cost-model-driven search finds RAG quality–performance Pareto frontiers that cover 52.5% to 153.2% more of the normalized objective space than existing configuration-search methods at the same evaluation budge","keywords":["retrieval-augmented generation","Pareto frontier","multi-objective Bayesian optimization","performance cost model","HNSW","vector search","system deployment search","RAG serving"],"falsifier":"Run RAG-CM against a truly new corpus and a hardware generation outside the two servers tested, with no re-calibration beyond the per-corpus constant, and compare predicted versus measured HNSW latency rankings across a grid of M and efSearch values. If the Spearman correlation of predicted versus measured order falls well below the reported 0.95–0.98 range, or if the predicted optimal efSearch flips relative to measured throughput, the transfer claim fails.","tokens_in":26051,"feed_emoji":"⚖️","tokens_out":8617,"duration_ms":88867,"temperature":0.7,"pith_summary":"RAG-Stack is a framework for finding the Pareto frontier between answer quality and serving performance of retrieval-augmented generation (RAG) pipelines, searching both algorithm choices (chunking, retrieval index, reranking, generator, workflow) and deployment choices (placement, batching, parallelism) in one loop. Its central claim is that this joint search can be done without deploying each candidate: a hybrid analytical-machine-learning cost model predicts the best achievable latency and throughput for each algorithm configuration, so the optimizer spends its scarce budget on quality evaluations only. With the same number of optimization iterations, the frontiers RAG-Stack finds cover 52.5% (RAGEval) and 153.2% (MS MARCO) more of the normalized quality–performance space than state-of-the-art configuration-search baselines, averaged over three seeds. If correct, this means a RAG deployment can be co-tuned for quality and speed in a few hours, and an existing frontier can be retargeted to new hardware by re-scoring the cost model instead of re-deploying everything.","feed_headline":"RAG-Stack finds 52–153% more of the quality–speed frontier","feed_subtitle":"A hybrid cost model predicts server throughput without deploying every candidate, so Pareto search covers more ground per trial.","key_machinery":"The load-bearing mechanism is the split between algorithm and system design spaces, plus RAG-CM, a four-layer cost model that predicts serving performance without deployment. RAG-CM's algorithm layer turns each executed workflow into hardware-agnostic operator work profiles (operation counts, bytes, access patterns); a roofline model maps those profiles to time on the given hardware; a communication layer prices data movement between stages; and an assembly layer runs a closed discrete-event simulation over the system design space to return the best predicted deployment. The HNSW retrieval model is the one machine-learned piece: two gradient-boosted regressors predict per-query distance comp","core_discovery":"On the paper's own terms, the discovery is that the quality–performance Pareto frontier of a RAG system can be found efficiently by splitting the configuration space into an algorithm part (choices that change the answer) and a system part (choices that only change how the answer is served), then searching the algorithm part with a stage-aware multi-objective Bayesian optimizer while a cost model exhaustively searches deployments without deploying any. Evidence: on RAGEval and MS MARCO, with identical budgets, RAG-Stack's frontiers average 0.514 and 0.635 normalized hypervolume versus 0.364 and 0.259 for the strongest baseline — per-seed gains of 52.5% and 153.2% — and the calibrated cost mo","pith_inferences":["The system-space abstraction (workflow schema plus per-request traces) is general enough that the same co-optimization loop could be applied to other multi-stage LLM serving pipelines, such as multi-agent workflows, even though the paper only evaluates sequential and agentic RAG.","The reported quality gap between RAGEval and MS MARCO suggests public benchmarks may understate the serving cost of answer quality; a practitioner should re-tune on fresh data before trusting a public-benchmark frontier.","The HNSW predictor's reliance on local intrinsic dimensionality implies a strong test: corpora with extreme geometry (nearly uniform or heavily clustered embeddings) should be used as a stress test for the transfer claim.","Hardware planning before purchase is a natural use: RAG-CM can score candidate machines from a hardware description, which the paper motivates but does not evaluate as a standalone decision tool."],"forward_implications":["On a new RAG application, the same evaluation budget yields a Pareto frontier covering roughly half again to two-and-a-half times more of the normalized quality–performance space than existing configuration-search methods.","Migrating an optimized RAG system to different hardware can reuse archived quality measurements: 20 additional evaluations with a calibrated cost model recover a frontier 182.2% larger than re-optimizing from scratch.","Because ranking, not absolute error, is what guides search, a cost model with 11–15% MAPE and 0.95+ Spearman correlation is sufficient to drive deployment selection.","End-to-end search finishes in about 4.6 hours on RAGEval and 3.6 hours on MS MARCO on a four-H100 server, because no candidate is deployed to measure performance."],"supporting_citations":[{"why":"Supplies the stage-wise greedy RAG optimizer that RAG-PE is compared against and that motivates the stage-interaction problem.","marker":"[3]"},{"why":"Provides the MS MARCO dataset used to measure the end-to-end Pareto frontier and quality objective.","marker":"[9]"},{"why":"Provides the multi-objective RAG hyperparameter optimizer with a fixed deployment that RAG-Stack improves upon and compares against.","marker":"[12]"},{"why":"Supplies the noisy expected-hypervolume-improvement acquisition and multi-objective Bayesian optimization foundation that RAG-PE extends and that serves as the strongest baseline.","marker":"[17]"},{"why":"Supplies the vector-search index implementations whose configuration knobs define the retrieval portion of the algorithm design space.","marker":"[19]"},{"why":"Supplies the RAGAS answer-correctness metric and the stage-level diagnostics used as quality objective and sub-metrics.","marker":"[20]"},{"why":"Supplies the RAG serving-system optimization baseline and the batch/parallelism mechanisms modeled by RAG-CM's assembly layer.","marker":"[30]"},{"why":"Supplies a baseline Bayesian optimizer with a random-forest surrogate and scalarized objectives used in the comparison.","marker":"[48]"},{"why":"Supplies the HNSW graph index algorithm whose workload counts (distance computations and hops) RAG-CM must predict.","marker":"[51]"},{"why":"Supplies the RAGEval dataset and fresh-data control used in the end-to-end evaluation.","marker":"[74]"}],"fun_headline_variants":["RAG-Stack: up to 2.5x more of the quality-speed frontier","Smarter RAG search: 52–153% wider Pareto frontier","RAG-Stack boosts Pareto coverage by up to 153%","Efficient RAG tuning finds 1.5–2.5x more trade-offs","RAG-Stack: co-optimize quality and speed without full search"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"RAG-Stack rests on the assumption that the cost model's learned predictor, trained once on synthetic data with only a corpus-geometry feature and a per-corpus scaling constant, will keep ranking index configurations correctly on new corpora and on new hardware; if that generalization fails, the whole search inherits the error.","fun_headline_variants_meta":{"raw":{"variants":["RAG-Stack: up to 2.5x more of the quality-speed frontier","Smarter RAG search: 52–153% wider Pareto frontier","RAG-Stack boosts Pareto coverage by up to 153%","Efficient RAG tuning finds 1.5–2.5x more trade-offs","RAG-Stack: co-optimize quality and speed without full search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":2843,"prompt_tokens":792,"completion_tokens":2051,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1947}},"tokens_in":536,"tokens_out":2051,"duration_ms":14589,"temperature":1.0,"reasoning_tokens":1947,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:05:04.571821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RAG-CM against a truly new corpus and a hardware generation outside the two servers tested, with no re-calibration beyond the per-corpus constant, and compare predicted versus measured HNSW latency rankings across a grid of M and efSearch values. If the Spearman correlation of predicted versus measured order falls well below the reported 0.95–0.98 range, or if the predicted optimal efSearch flips relative to measured throughput, the transfer claim fails.","supporting_citations":[{"cited_title":"Marker-Inc-Korea/AutoRAG","cited_arxiv_id":null,"evidence_quote":"Supplies the stage-wise greedy RAG optimizer that RAG-PE is compared against and that motivates the stage-interaction problem."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a baseline Bayesian optimizer with a random-forest surrogate and scalarized objectives used in the comparison."}],"review_version":1}