{"id":"78ed21a1-6f7d-4bf4-8771-736ce6b04cbc","arxiv_id":"2506.19783","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A strategy-guided reinforcement learning framework, SAGE, improves dense retrieval query rewriting on HotpotQA and NFCorpus, achieving NDCG@10 gains over baseline rewrites while generating substantially shorter queries.","lead":"Query rewriting is the step that turns a vague user question into the precise phrase a retrieval system can match. This paper shows that steering a large language model with five hand-written rewriting strategies, and training it with reinforcement learning, lifts retrieval scores on two benchmarks while using far fewer tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA empirical claim rests entirely on NDCG@10 from BGE-base-en-v1.5, a retriever the paper itself says may have trained on these benchmarks; unremediated, that makes the reported gains retriever-specific artifacts rather than evidence of general query-rewriting quality.","rationale":"The reader's weakest assumption—that the NDCG@10 reward signal is uncontaminated—is also the most load-bearing assumption in the paper. Every headline number in Table 1 and every RL training signal in Section 2.3 flows through BGE-base-en-v1.5. If that retriever has seen the benchmark test queries, the policy is being trained and evaluated against a contaminated instrument. The authors themselves flag this in Section 4.3, which makes the omission of a cross-retriever evaluation of the trained SAGE policies a genuine gap rather than a hypothetical. The SOTA overclaim on HotpotQA (0.6955 versus GPT-4.1's 0.7118) is an independent, objectively checkable error, but it is less fundamental than the contamination issue because the framework's relative gains over DeepRetrieval and its token-efficiency findings would still stand if only the SOTA wording were corrected. The contamination issue, in contrast, threatens the validity of the central empirical demonstration. A cross-retriever evaluation would settle it: if the trained rewrites transfer, the concern is answered; if not, the reported state-of-the-art result is an artifact of the evaluation instrument. I therefore recommend keeping the reader's CONDITIONAL verdict: the paper is promising but requires this additional evidence, artifact release, and corrected SOTA wording before its central claim can be accepted.","tokens_in":13438,"tokens_out":6362,"duration_ms":63643,"concrete_test":"Run the rewrites generated by the final SAGE-SCS and SAGE-CRS policies on the NFCorpus and HotpotQA test queries through a retriever that was not used during RL training (e.g., Contriever, or BGE-large if it is verified not to be in the training loop), and compare NDCG@10 against the original queries and against the DeepRetrieval baseline. If SAGE's margins shrink to noise or reverse under this retriever, the reported SOTA is an artifact of reward-signal overfitting to BGE-base-en-v1.5; if the margins persist, the contamination concern is mitigated. As a complementary audit, check whether BGE-base-en-v1.5's training corpus overlaps with the test queries of these benchmarks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SAGE establishes new state-of-the-art NDCG@10 and that SCS/CRS improve query rewriting—depends on the reward signal being a faithful measure of query quality. In this paper, BGE-base-en-v1.5 is used both as the RL reward (Section 2.3) and as the sole evaluation retriever for all SAGE results (Table 1). Section 4.3 concedes that 'many powerful retrievers have been trained on these benchmarks,' which means the retriever may have memorized query–document associations from NFCorpus and HotpotQA test sets. If so, the optimal rewrite under the reward is the one that most closely matches the retriever's training-time queries, and the reported gains over the original-query and DeepRetrieval baselines could be artifacts of retriever familiarity rather than transferable query-quality improvements. No cross-retriever evaluation of the trained SAGE policies is reported: the Contriever results in Figure 6 are only for the prompting upper bound (SGP), not for the RL-trained models. The framework's practical claim of 'robust information retrieval' is therefore unsupported. Separately, the abstract's 'state-of-the-art' phrasing is contradicted by Table 1: GPT-4.1 achieves 0.7118 NDCG@10 on HotpotQA, above SAGE-SCS's 0.6955, so the SOTA claim is inaccurate as written regardless of contamination.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAGE, a strategy-guided reinforcement learning framework for query rewriting in dense retrieval. The method defines five hand-crafted rewriting strategies (semantic expansion, entity disambiguation, sub-question decomposition, concise rewriting, neutralized claim reformulation), embeds them in a GRPO training loop, and introduces two reward-shaping mechanisms: Strategic Credit Shaping (SCS), which scales rewards by the rank of the chosen strategy's average performance, and Contrastive Reward Shaping (CRS), which subtracts a batch baseline. To counter reward hacking, the authors modify the prompt to encourage exploration and add a penalty for outputs identical to the original query. Experiments on HotpotQA and NFCorpus with the BGE-base-en-v1.5 retriever report NDCG@10 improvements over the original query and over the DeepRetrieval baseline, together with a large reduction in average output token count, which the authors describe as an emergent efficiency behavior. The paper also reports a prompting-only upper-bound study (SGP) across four datasets with two retrievers.","tokens_in":13923,"tokens_out":3026,"duration_ms":30205,"significance":"If the empirical claims hold, the paper makes a useful contribution: it shows that constraining the action space to a small set of interpretable strategies can make RL-based query rewriting more sample-efficient and more transparent, and the reward-shaping schemes (SCS, CRS) are simple and potentially reusable ideas. The controlled comparison against DeepRetrieval re-evaluated in the same setting, the ablation of the exploration penalty, and the detailed training-dynamics curves are strengths. The claimed efficiency gain (a factor of three to five fewer tokens than the original query) is practically relevant. However, the headline claim of 'state-of-the-art' is contradicted by the paper's own numbers, and the use of a single retriever for both reward and evaluation raises a contamination risk that is acknowledged only partially. The contribution is therefore not yet established at the level claimed.","major_comments":[{"comment":"The claim that SAGE 'achieves state-of-the-art NDCG@10 results' is contradicted by the numbers in Table 1. On HotpotQA, GPT-4.1 scores 0.7118, which is above SAGE-SCS's 0.6955; on NFCorpus, SAGE-CRS's 0.4035 is the highest, but SAGE-Direct (0.3776) is below GPT-o4-mini (0.3809). The abstract and contribution list should be reworded to state precisely what is claimed: e.g., 'best among RL-trained 4B models' or 'competitive with much larger proprietary models.' As written, the SOTA claim is factually inaccurate on one of the two main datasets.","section":"Section 3.2, Table 1"},{"comment":"The reward signal during RL training and the final evaluation both use the same retriever, BGE-base-en-v1.5. Section 4.3 itself notes that 'many powerful retrievers have been trained on these benchmarks,' which means the NDCG@10 reward may be biased toward rewrites that match the retriever's training-time query patterns. The paper reports no cross-retriever evaluation of the trained SAGE policies; Figure 6 (Contriever) covers only the prompting upper bound (SGP), not the RL-trained models. Reporting SAGE-SCS and SAGE-CRS NDCG@10 with at least one held-out retriever (e.g., Contriever or a BGE-large variant) is essential to support the claim that SAGE improves general query quality rather than merely exploiting retriever-specific artifacts.","section":"Section 4.3 and Table 1"},{"comment":"All comparisons are point estimates with no error bars, no significance tests, and no description of how the test set is split or whether the reported numbers are from a single run or averaged over seeds. Several margins are very small, e.g., SAGE-CRS 0.6918 vs. GPT-o4-mini 0.6915 on HotpotQA, and SAGE-Direct 0.3776 vs. GPT-4.1 0.3711 on NFCorpus, so without variance information the ranking of methods is not statistically supported. Please add confidence intervals, significance tests, or at least multi-seed standard deviations.","section":"Section 3.2, Tables 1 and 2"},{"comment":"The exploration penalty hyperparameter is reported inconsistently. The text in Section 4.3 says 'subtracting a fixed value p=0.05,' but Table 3 reports the condition as 'Proactive Prompt + Penalty (p = 0.02).' In addition, the paper does not describe how p was chosen or whether the observed NDCG@10 drop under the penalty (0.694 to 0.692 in Table 3) is within noise, given that no error bars are provided. Please reconcile the value and discuss sensitivity to this hyperparameter.","section":"Section 2.5, Section 4.3, Table 3"},{"comment":"The paper assumes, without evidence, that the five hand-crafted strategies define a sufficient and complete action space for query rewriting in dense retrieval. The limitation section notes that manual analysis is used to craft strategies, but it does not discuss the risk that a useful strategy (e.g., temporal specification, negation handling, or clarification for ambiguous user intent) may be missing, which would cap the attainable performance of SAGE. A brief analysis of which strategies are selected on each dataset and whether any generated rewrites fall outside the five categories would make the sufficiency assumption more credible.","section":"Section 2.2 and Table 4"}],"minor_comments":[{"comment":"The text says the model is fine-tuned using 'Verl (Sheng et al., 2024) and GROP (Shao et al., 2024)'; 'GROP' should be 'GRPO'.","section":"Appendix A.3"},{"comment":"The caption contains a typo: 'Constractive Reward Shaping' should be 'Contrastive Reward Shaping.'","section":"Figure 2 caption"},{"comment":"The NFCorpus NDCG@10 for SAGE (Direct) is reported as 0.3776 in Table 1 and 0.3775 in Table 2. Please check which value is correct and ensure consistency across tables.","section":"Tables 1 and 2"},{"comment":"The phrase 'detailed is demonstrated in Appendix A.1' is ungrammatical; it should read 'as detailed in Appendix A.1' or 'as demonstrated in Appendix A.1.'","section":"Section 4.3"},{"comment":"Several entries in the reference list use the truncated form 'and 1 others' (e.g., Achiam et al., 2023; Guo et al., 2025; Shao et al., 2024). Please expand the author lists to comply with journal style, and ensure all citations in the text have a corresponding full entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the ablation structure is sound, but the paper currently overstates its results: the SOTA claim is refuted by Table 1, and the single-retriever evaluation creates a contamination risk that the authors themselves acknowledge. These issues are fixable with a qualified claims rewrite, additional cross-retriever evaluation, and proper uncertainty reporting, so I recommend major revision rather than rejection. The inconsistency around the exploration penalty value (p=0.02 vs 0.05) should also be resolved before the paper can be considered publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper is a genuine attempt to make RL-based query rewriting practical, and the core idea is sound. What's actually new: encoding five explicit rewriting strategies (semantic expansion, entity disambiguation, sub-question decomposition, concise rewriting, neutralized claim reformulation) into the prompt and policy, then adding two reward-shaping schemes (SCS and CRS) on top of GRPO, plus an exploration penalty to stop the model from just copying the query. The authors also show scaling laws for strategy-guided prompting across four datasets, and report a striking efficiency result: SAGE-SCS uses 66 tokens per rewrite on HotpotQA versus 232 for DeepRetrieval and 1598 for the base Qwen3-4B, with NDCG@10 slightly better than the original query. If that efficiency holds, it is a real production consideration.\n\nCredit where due: they openly discuss reward hacking, run ablations showing the penalty increases modification rate but slightly hurts NDCG@10, provide full prompt templates in the appendix, and check the prompting upper bound with Contriever. That is honest work.\n\nThe soft spots are real. The \"state-of-the-art\" claim is factually wrong as written: Table 1 shows GPT-4.1 at 0.7118 on HotpotQA, above SAGE's 0.6955. The gains over the original-query baseline are small—roughly 0.03 NDCG@10—and no error bars or significance tests are reported. More importantly, the same retriever (BGE-base-en-v1.5) is used for both RL reward and final evaluation, and the paper itself notes many powerful retrievers have been trained on these benchmark collections. The Contriever check covers only the prompting SGP, not the RL-trained policies, so the contamination risk is real and unmitigated. The exploration penalty also has a small internal inconsistency: p=0.05 in Section 4.3 versus p=0.02 in Table 3. No code or data is released.\n\nIn proportion: the central mechanism—strategy-guided RL with shaped rewards—is plausible and the empirical pattern is consistent, but the headline number should be read as retriever-specific until a cross-retriever evaluation appears. This is not a fatal flaw, but it is load-bearing for the claim of robust IR.\n\nWho is this for? People building query-rewriting systems in production and researchers working on RL for IR. It deserves a serious referee; with error bars, cross-retriever evaluation, a tempered SOTA claim, and code release, this could become a solid reference point.","headline":"A practical, honest RL-for-query-rewriting paper whose core efficiency result is compelling but whose SOTA claim is overstated and whose evaluation is retriever-bound; worth a serious referee.","tokens_in":14313,"tokens_out":2236,"would_cite":true,"duration_ms":22336,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAGE claims that injecting five expert-crafted rewriting strategies plus two reward-shaping schemes into GRPO yields state-of-the-art NDCG@10 on HotpotQA and NFCorpus with fewer than 160 tokens per rewrite.","keywords":["query rewriting","dense retrieval","reinforcement learning","reward shaping","GRPO","strategy-guided generation","retrieval-augmented generation","NDCG@10"],"falsifier":"Evaluate SAGE's trained rewrites with a retriever that was trained on a corpus disjoint from HotpotQA and NFCorpus, and compare NDCG@10 against the original queries; if the advantage vanishes or reverses, the reported gains came from benchmark familiarity rather than rewriting quality.","tokens_in":13239,"feed_emoji":"🔎","tokens_out":6301,"duration_ms":59271,"temperature":0.7,"pith_summary":"The paper claims that query rewriting for dense retrieval improves substantially when a language model is guided by a small set of five hand-written strategies—semantic expansion, entity disambiguation, sub-question decomposition, concise rewriting, and neutralized claim reformulation—rather than left to free-form RL exploration. It introduces SAGE, an RL framework built on GRPO that makes each rewrite carry an explicit strategy choice, and two reward-shaping schemes (Strategic Credit Shaping and Contrastive Reward Shaping) that turn raw NDCG@10 scores into sharper learning signals. On HotpotQA and NFCorpus, SAGE variants reach state-of-the-art NDCG@10 while using far fewer tokens per rewrite than baselines, competitive with much larger proprietary models. The paper also shows that without forced exploration, RL collapses into a reward-hacking policy that simply copies the original query, so it penalizes identical rewrites and prompts proactive exploration. A sympathetic reader would take away that strategy-guided RL with shaped rewards offers a scalable, cheap, more interpretable path to better retrieval.","feed_headline":"Strategy-guided RL sets retrieval records under 160 tokens","feed_subtitle":"A 4B-parameter engine with five hand-written strategies beats baselines and rivals big proprietary models on NDCG@10.","key_machinery":"SAGE (Strategy-Adaptive Generation Engine) is the central object: an RL framework that converts query rewriting into a two-part action—rewritten query plus a strategy ID—sampled with GRPO and rewarded by NDCG@10 after shaping. Two shaping mechanisms carry the argument: SCS (Strategic Credit Shaping), which ranks strategies by their mean reward in a batch and scales each rollout's reward by the inverse rank, and CRS (Contrastive Reward Shaping), which centers each reward at a batch baseline so the agent optimizes relative advantage. A third component, forced exploration, penalizes rewrites identical to the input query and replaces conservative prompt instructions with proactive ones. The five strategy primitives define the action space and are what make credit assignment meaningful.","core_discovery":"On the paper's own terms, the central discovery is that injecting a concise, human-specified strategy space into an on-policy RL loop is enough to make a 4B-parameter model outperform specialized RL baselines and rival much larger proprietary models at dense query rewriting. SAGE frames rewriting as the joint generation of a rewritten query and a strategy ID (1–5), and the reward is the retrieval NDCG@10 after shaping. SCS divides a rollout batch into strategy groups and scales each individual reward by the inverse rank of its group's mean performance, so the agent learns which strategies pay off; CRS subtracts a running baseline such as the batch median, forcing rewrites to beat typical performance rather than merely score positively. Because modern retrievers already score well on the original query, the authors find the agent otherwise converges to copying the input—a reward hack—and they counter it with an exploration penalty and a proactive prompt. The result is state-of-the-art NDCG@10 (0.6955 on HotpotQA, 0.4035 on NFCorpus) and an emergent compression of reasoning to roughly 66–154 tokens per rewrite.","pith_inferences":["Inference beyond the paper: SCS and CRS are generic reward transformations, so they should transfer to any RL task where each rollout carries a discrete type label, such as tool selection or SQL operation choice.","Inference: Because the paper itself warns that many powerful retrievers may have been trained on these benchmarks, the clean validation is to evaluate SAGE rewrites against a retriever trained on disjoint data; if the margin shrinks, the reported numbers partly reflect benchmark familiarity rather than rewrite quality.","Inference: The five strategies are human-designed; a natural extension is to mine strategy definitions automatically from failed retrievals, which would remove the manual-analysis bottleneck the paper lists as a limitation.","Inference: The emergent token compression suggests the strategy ID acts as a reasoning shortcut; probing whether the stated strategy genuinely causes the rewrite or merely rationalizes it would clarify how interpretable SAGE really is."],"forward_implications":["Strategy-guided RL transfers to any dense-retrieval setting where query intent is underspecified and the retriever is sensitive to phrasing.","The SCS and CRS recipes are model-agnostic and can be bolted onto existing GRPO or similar pipelines without new supervision.","The emergent token compression means the agent can operate at under 160 tokens per rewrite, lowering inference latency and cost at scale.","Without an explicit penalty against copying the input, RL on strong retrievers collapses to a trivial do-nothing policy, so future RL-based rewriting systems should include an anti-copying term.","Prompting alone with the five strategies establishes a strong performance upper bound, making strategy-guided prompting a practical baseline even without RL."],"supporting_citations":[{"why":"Supplies the DMQR-RAG strategies and conservative prompting baseline that SGP and SAGE must outperform.","marker":"(Li et al., 2024)"},{"why":"Provides the DeepRetrieval RL baseline that SAGE reproduces and beats.","marker":"(Jiang et al., 2025)"},{"why":"Defines GRPO, the on-policy optimization algorithm SAGE is built on.","marker":"(Shao et al., 2024)"},{"why":"Provides BGE-base-en-v1.5, the retriever used for NDCG@10 rewards and evaluation.","marker":"(Xiao et al., 2023)"},{"why":"Supplies the HotpotQA dataset, one of the two main RL evaluation sets.","marker":"(Yang et al., 2018)"},{"why":"Supplies the NFCorpus dataset, the other main RL evaluation set.","marker":"(Boteva et al., 2016)"},{"why":"Supplies the FEVER dataset used for the strategy prompting scaling-law analysis.","marker":"(Thorne et al., 2018)"},{"why":"Supplies the SciFact dataset used for the strategy prompting scaling-law analysis.","marker":"(Wadden et al., 2020)"},{"why":"Provides the Contriever retriever used to confirm SGP's advantage is not retriever-specific.","marker":"(Izacard et al., 2021)"}],"fun_headline_variants":["Strategy-guided RL rewrites queries to new retrieval SOTA","SAGE uses strategy rewards to hit retrieval SOTA","4B model with strategy-guided RL rivals big proprietary engines","SAGE's strategy-guided RL yields SOTA retrieval","Strategy-guided RL: new SOTA in query rewriting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that the retrieval score used for training honestly reflects rewrite quality, rather than being inflated because the retriever has already seen the benchmark queries—a risk the paper itself raises—and that five hand-written rewriting strategies are enough.","fun_headline_variants_meta":{"raw":{"variants":["Strategy-guided RL rewrites queries to new retrieval SOTA","SAGE uses strategy rewards to hit retrieval SOTA","4B model with strategy-guided RL rivals big proprietary engines","SAGE's strategy-guided RL yields SOTA retrieval","Strategy-guided RL: new SOTA in query rewriting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2960,"prompt_tokens":991,"completion_tokens":1969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":1891}},"tokens_in":607,"tokens_out":1969,"duration_ms":14297,"temperature":1.0,"reasoning_tokens":1891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:24:15.086801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate SAGE's trained rewrites with a retriever that was trained on a corpus disjoint from HotpotQA and NFCorpus, and compare NDCG@10 against the original queries; if the advantage vanishes or reverses, the reported gains came from benchmark familiarity rather than rewriting quality.","supporting_citations":[{"cited_title":"We experiment with rollout numbers of 16 and","cited_arxiv_id":null,"evidence_quote":"Defines GRPO, the on-policy optimization algorithm SAGE is built on."}],"review_version":2}