{"id":"382b88a0-9ae2-48a4-8cba-2f2b83ecba46","arxiv_id":"2606.02581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Per-query utility routing over discrete retrieval-depth bundles yields 26% fewer billed tokens than always-heavy RAG and 34% lower latency than always-direct inference at matched lexical quality on a 28-query micro-benchmark.","lead":"CA-RAG routes each RAG query to one of four retrieval-depth bundles by maximizing a simple quality-minus-cost-and-latency utility. On a 28-query toy benchmark it cuts tokens 26% versus always-heavy retrieval and latency 34% versus always-direct while holding a lexical quality proxy flat.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Quality-equivalence claim rests on hand-set priors and a lexical proxy that cannot detect grounding failures on this self-referential corpus.","rationale":"The reader correctly isolates the weakest assumption: hand-specified priors + lexical proxy + micro-benchmark. My concern is the same one, sharpened to the precise place it becomes load-bearing for the strongest claim. The token and latency numbers are internally consistent with the logged CSVs and the routing diversity (Fig. 1) is real; the paper is a clean, reproducible systems micro-study. The only soft spot that can overturn the joint claim is the quality side. Because the authors already flag this in Limitations and the evaluation is transparent, the appropriate verdict remains CONDITIONAL rather than REJECT: the empirical savings hold inside the stated micro-benchmark, but external validation with a stronger quality metric is required before production claims can be taken at face value. No change to the reader’s verdict is needed.","tokens_in":12841,"tokens_out":582,"duration_ms":6795,"concrete_test":"Replace the lexical quality_proxy with an independent LLM-as-judge (or human) rubric that scores factual grounding against the 15-sentence corpus for every query under both the default router and fixed-heavy; recompute Table III. If mean quality for the router falls >0.05 below fixed-heavy, or if any of the four direct_llm queries that drive the bulk of the token savings receive a grounding score <0.5, the equivalence claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (abstract + Table III) is that the default router saves 26% tokens vs fixed-heavy and 34% latency vs fixed-direct while maintaining equivalent answer quality (0.80 vs 0.81). That equivalence is load-bearing: without it the paper only shows a cheaper, faster policy that may be worse. Quality is operationalized solely by (i) hand-specified priors in Table I (0.52/0.66/0.74/0.82) that directly drive argmax selection via Eq. (1), and (ii) a post-hoc lexical token-overlap proxy. The 15-sentence corpus is a self-description of CA-RAG itself and the 28 questions are author-written about that same text (Appendices D–E). On such a corpus a fluent parametric answer can achieve high lexical overlap without any retrieval, so the proxy cannot falsify the claim that deeper retrieval is unnecessary for the definitional queries that produce most of the savings. The authors acknowledge the proxy and scale limits (§IX), yet still assert quality parity as part of the headline result. If the proxy is uncorrelated with factual grounding, the 26%/34% savings are real but the joint claim collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes CA-RAG, a per-query router that selects among four discrete strategy bundles (direct_llm, light_rag, medium_rag, heavy_rag) by maximizing a linear utility (Eq. 1) that combines a hand-specified quality prior with normalized latency and billed-token penalties. Implemented with FAISS dense retrieval and OpenAI APIs, it is evaluated on a 28-query / 15-passage micro-benchmark. The headline result (abstract, Table III) is that the default router uses 26.4% fewer billed tokens than fixed-heavy retrieval and 34.3% lower mean latency than fixed-direct inference while keeping a lexical quality proxy essentially unchanged (0.80 vs 0.81). Supporting analyses cover strategy mix, token decomposition, weight sensitivity, and per-query deltas, all regenerated from logged CSVs.","tokens_in":13162,"tokens_out":1626,"duration_ms":26226,"significance":"If the joint cost–latency–quality claim holds under stronger evaluation, the paper would supply a practical, auditable template for production RAG routing: discrete bundles, explicit utility weights as operator levers, and full token accounting including query embeddings. Strengths that already stand are the transparent utility formulation, complete billing model (Eq. 2), weight-sensitivity demonstration that the same catalog supports multiple operating points, and end-to-end reproducibility from CSV artifacts and CLI commands. Even with the current micro-benchmark, the work is a useful systems-style characterization of retrieval-depth tradeoffs rather than a new retriever architecture. The main significance risk is that the quality-parity half of the claim is not yet measured at a standard that would support the abstract’s joint assertion.","major_comments":[{"comment":"Abstract and Table III assert “equivalent answer quality” (router 0.80 vs fixed-heavy 0.81) as part of the central joint claim. Quality is operationalized only by (i) hand-specified priors in Table I (0.52/0.66/0.74/0.82) that directly drive arg max selection via Eq. (1), and (ii) a post-hoc lexical token-overlap proxy. Lexical overlap cannot detect grounding failures or hallucinations, and on this corpus a fluent parametric answer can score well without retrieval. The Limitations section acknowledges the proxy, yet the headline still treats quality parity as established. Either replace/augment the proxy with a calibrated LLM judge or human ratings on a held-out set, or substantially soften the quality claim so that the paper reports cost/latency savings under a weak proxy rather than “equivalent quality.”","section":"Abstract, Table III, §V.C Eq. (1), Table I, §IX"},{"comment":"The evaluation corpus is a 15-sentence self-description of CA-RAG itself, and the 28 questions are author-written about that same text (Appendices D–E, §VI.A). Definitional queries that drive most token savings (Figs. 15–17, Appendix G) are exactly those for which parametric knowledge of the paper’s own terminology yields high lexical overlap without retrieval. This design cannot falsify the claim that shallow or zero retrieval is quality-preserving for the queries that produce the 26% savings. At minimum, re-run or validate on a standard open-domain set (e.g., Natural Questions / HotpotQA subsets) or an external enterprise corpus, and report quality under a non-lexical metric; otherwise scope all quality statements strictly to this micro-benchmark.","section":"§VI.A, Appendices D–E, Figs. 15–17"},{"comment":"With n=28 and high per-query variance (Table V: latency std 1675 ms on mean 2512 ms; cost std 94.5 on mean 258.5), aggregate percentages in the abstract and Table III lack uncertainty quantification. No bootstrap CIs, paired tests, or leave-one-out stability are reported, despite the Limitations section noting that “bootstrap confidence intervals and paired significance tests are warranted.” For a load-bearing 26%/34% claim, add at least paired bootstrap intervals on cost and latency deltas vs fixed-heavy and fixed-direct, and report whether quality-proxy differences are distinguishable from noise.","section":"Table III, Table V, §IX"},{"comment":"Heuristic complexity c(q) (§V.A) correlates only weakly with billed tokens (r≈0.21–0.22, Table VII / Fig. 9), yet it modulates quality priors and is presented as the main query signal. The paper correctly flags this as a limitation, but the routing story still depends on these free parameters (α, β, Lmax, Kmax) plus the four quality/latency priors. A short ablation—routing with complexity disabled, or with priors set equal—would show how much of the strategy mix and savings is driven by the hand-tuned prior table versus the complexity signal. Without that, it is hard to separate the contribution of the utility framework from prior engineering on this specific 28-query set.","section":"§V.A, Table I, Table VII, Fig. 9"}],"minor_comments":[{"comment":"Table VII reports cost–latency r=0.66 and cost–utility r=−0.50, while the prose in §VII.D cites r=0.41 and r=−0.38. Align the text with the table or explain the discrepancy.","section":"§VII.D, Table VII"},{"comment":"Fig. 3’s claim that direct_llm has the highest latency variance is interesting; a one-sentence note on completion-length statistics (or a small table of mean completion tokens by strategy) would make the explanation fully checkable against Fig. 5.","section":"§VII.B, Fig. 3, Fig. 5"},{"comment":"Related work on adaptive retrieval / query routing (beyond bandits) is thin; a short comparison to recent adaptive-RAG or cascade/router papers would help position the discrete-bundle contribution.","section":"§II"},{"comment":"Notation: ˆLnorm and ˆCnorm are said to be normalized to [0,1] across B, but the exact min–max or z-score procedure is not stated; one sentence would remove ambiguity for reimplementation.","section":"§V.C Eq. (1)"},{"comment":"Appendix G strategy assignments are useful; cross-referencing a few of them in the per-query delta discussion would help readers map savings to query types without flipping to the appendix.","section":"§VII.G, Appendix G"}],"recommendation":"major_revision","confidential_remarks":"Fit is reasonable for an IR/systems venue interested in production RAG, but novelty is incremental relative to the broader adaptive-retrieval literature; the paper’s real value is the transparent cost accounting and reproducible micro-study. I would not reject on novelty alone if quality measurement and scale are fixed. The self-referential corpus is the single most important fix: without an external evaluation, the quality half of the abstract is not publishable as stated. If the authors only soften claims and keep the micro-benchmark, the paper becomes a solid workshop-style systems note rather than a full journal empirical study—editor may wish to set expectations on venue level."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this is a careful systems micro-study, not a new retrieval method. On a 28-query, 15-passage self-describing corpus the discrete utility router does exercise all four depth bundles and posts the advertised 26% token cut vs always-heavy and 34% latency cut vs always-direct. Those numbers match the logged CSVs. That is the real result.\n\nWhat is actually new is the packaging: a small catalog of strategy bundles (direct through top-k=10), a linear utility with full billed-token accounting including query embeddings, and weight knobs that move the operating point without changing code. Adaptive retrieval and cost-aware serving already exist in the literature they cite; the contribution is the concrete, auditable operational template and the per-query delta analysis showing savings concentrate on simple queries. Reproducibility is done right—CLI, CSVs, figures generated from logs, appendices with the full question set and strategy assignments. That is real engineering hygiene and worth credit.\n\nThe soft spot is load-bearing and the stress-test is right about it. “Equivalent quality” is a lexical token-overlap proxy plus hand-set priors (0.52/0.66/0.74/0.82). On a corpus that is literally a description of CA-RAG itself, a fluent parametric answer can score well without retrieval, so the proxy cannot falsify the claim that deep retrieval was unnecessary for the definitional queries that drive most of the savings. The authors flag the proxy and the scale limits in §IX; they still put quality parity in the abstract and Table III. n=28 also has no statistical power. None of that invents a circularity in the utility math—Eq. 1 is ordinary prior-plus-penalty routing—but it does mean the joint cost–latency–quality claim only holds inside this micro-benchmark.\n\nWho gets value: production RAG engineers who want a transparent control surface and a checklist for token accounting, weight calibration, and failure modes. Not someone looking for a new retriever or a general scientific result on adaptive retrieval. Math and citations are fine for what the paper is; free parameters are explicit.\n\nI would send it to peer review as a short systems/IR note. Expect referees to demand a stronger quality metric and an external corpus. Engage if you care about cost-aware serving; skip if you need foundational IR.","headline":"Clean, reproducible micro-study of discrete cost-aware RAG routing; the token/latency savings are real on the toy set, but the quality-parity claim rests on a weak proxy.","tokens_in":13771,"tokens_out":599,"would_cite":false,"duration_ms":11949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Per-query routing over discrete retrieval depths cuts billed tokens 26% versus always-heavy RAG and latency 34% versus always-direct inference, with quality held level.","keywords":["retrieval-augmented generation","cost-aware inference","query routing","strategy bundles","token efficiency","utility maximization","latency–quality tradeoff","production RAG"],"falsifier":"Re-run the identical router and fixed baselines on a public multi-hop QA set (e.g., HotpotQA) with an LLM-as-judge or human quality metric; if the router still shows statistically significant token and latency savings at matched quality, the central claim holds.","tokens_in":13662,"feed_emoji":"⚖️","tokens_out":852,"duration_ms":9025,"temperature":0.7,"pith_summary":"Static RAG setups force every query through the same retrieval depth, so simple definitional questions burn tokens on unused context while complex analytical ones get too little grounding. This paper argues that the fix is a transparent per-query router that chooses, for each question, one of four fixed \"strategy bundles\" ranging from pure LLM inference to top-10 dense retrieval. The router maximises a scalar utility that trades an estimated quality prior against normalised latency and billed-token penalties; operators steer the tradeoff simply by changing the three weights. On a 28-query micro-benchmark the default router uses every bundle, spends 26% fewer tokens than always-heavy retrieval and finishes 34% faster than always-direct inference, while a lexical quality proxy stays essentially unchanged. Savings concentrate on the simpler queries, and the same bundle catalog supports latency-first or cost-first operating points without any code change. The result is an auditable, weight-tunable foundation for production systems that treat token cost and latency as first-class constraints.","feed_headline":"Router cuts RAG tokens 26% and latency 34% at equal quality","feed_subtitle":"One weight vector steers four retrieval depths; savings concentrate on simple queries.","key_machinery":"Strategy bundles paired with the scalar utility Ub = wQ Q̂b(q) − wL L̂normb − wC Ĉnormb. Each bundle fixes a retrieval depth (k = 0, 3, 5 or 10) and a shared generation profile; the router simply picks the arg-max bundle for every query.","core_discovery":"A per-query router that maximises a linear utility over a discrete catalog of retrieval-depth bundles can simultaneously reduce billed tokens by roughly one-quarter relative to always-heavy retrieval and cut mean latency by roughly one-third relative to always-direct inference, while preserving answer quality on the reported benchmark.","pith_inferences":["Because the heuristic complexity signal correlates only weakly with actual cost (r≈0.21), replacing it with an embedding-novelty or intent classifier is likely the highest-leverage next engineering step.","The bimodal retrieval-confidence distribution already visible in the logs suggests a cheap fallback rule: when max cosine similarity falls below a threshold, force the direct-LLM bundle and avoid poorly grounded generation.","If the same utility surface is re-calibrated online with a contextual bandit, the hand-specified priors could be retired and the framework would become self-tuning under live traffic."],"forward_implications":["Operators can move between latency-first, cost-first and quality-first regimes by changing only three scalar weights, without redesigning retrieval or generation code.","Token and latency savings are largest on definitional queries, so production systems should pair the router with complexity-aware guardrails or intent classifiers.","The same discrete-bundle interface can later host richer signals (entity density, retrieval-confidence fallbacks, bandit-updated priors) without changing the routing API.","Full per-query CSV telemetry makes cost–quality audits and post-hoc recalibration routine rather than ad-hoc."],"fun_headline_variants":["CA-RAG router cuts tokens 26% and latency 34% at matched quality","Per-query depth router saves 26% RAG tokens, 34% mean latency","Utility-based routing trims RAG tokens 26% latency 34% equally","Discrete bundle router yields 26% fewer tokens 34% less latency","Cost-aware RAG routing cuts billed tokens 26% latency 34%"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The claim of quality parity rests on hand-written quality priors and a simple lexical-overlap proxy evaluated on a 15-sentence, author-written corpus of only 28 queries.","fun_headline_variants_meta":{"raw":{"variants":["CA-RAG router cuts tokens 26% and latency 34% at matched quality","Per-query depth router saves 26% RAG tokens, 34% mean latency","Utility-based routing trims RAG tokens 26% latency 34% equally","Discrete bundle router yields 26% fewer tokens 34% less latency","Cost-aware RAG routing cuts billed tokens 26% latency 34%"]},"model":"grok-4.5","effort":"low","cost_usd":0.003878,"raw_usage":{"total_tokens":1255,"prompt_tokens":817,"num_sources_used":0,"completion_tokens":88,"cost_in_usd_ticks":38780000,"prompt_tokens_details":{"text_tokens":817,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":350,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":817,"tokens_out":88,"duration_ms":5914,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T18:31:02.513623+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the identical router and fixed baselines on a public multi-hop QA set (e.g., HotpotQA) with an LLM-as-judge or human quality metric; if the router still shows statistically significant token and latency savings at matched quality, the central claim holds.","supporting_citations":[],"review_version":1}