{"id":"aaabc332-eb37-4812-a8db-a8607d9271ec","arxiv_id":"2411.14574","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"SRSA, a router that picks between direct, parallel, and planning searches for each query, improves informativeness and completeness on a new contextual-query benchmark while using fewer LLM inference steps than a ReAct agent.","lead":"This paper introduces SRSA, a search agent that routes each user query to one of three search strategies: direct, parallel, or planning. It also presents a new contextual query dataset, CQED, and reports that SRSA beats a simple search agent and a ReAct agent on LLM-judged quality metrics.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central quality claim depends entirely on an unvalidated GPT-4o-mini judge: no agreement statistics are reported, and the 'manual evaluation' of 8 data sets is too vague to rule out length/format bias.","rationale":"The reader's weakest_assumption identifies the most load-bearing concern: every headline quality result is a comparison of LLM-judge scores, and the paper provides no evidence that those scores are valid. The manual check is described in one vague sentence and lacks the statistical grounding needed to rule out systematic bias. This concern dominates because it undermines not only the SRSA-versus-baseline result but also the ReAct degeneration finding and the router analysis in Table II, all of which are central to the paper's claims. Even if one were to accept the judge, the missing router-isolation ablation and the absence of any cost measurement are additional serious gaps, but they are secondary to the question of whether the outcome variable itself is trustworthy. Since the reader's verdict is REJECT and this stress-test reinforces the same weakness, the verdict should remain unchanged.","tokens_in":11739,"tokens_out":5209,"duration_ms":55281,"concrete_test":"Run a blinded human evaluation on a random sample of at least 50 CQED items, ideally all 182: three human annotators score the simple-search, ReAct, and SRSA outputs on the same 0-5 rubrics, with agent identity hidden and answer order randomized. Compute inter-annotator agreement (e.g., Krippendorff's alpha) and the Spearman correlation between human scores and GPT-4o-mini scores for each metric. Additionally, run a length-control: take simple-search answers and append length-matched, topically redundant sentences; if GPT-4o-mini's informativeness/completeness scores rise substantially for padded answers, the judge is confounded by length/format. If human rankings reproduce SRSA > simple and > ReAct on informativeness and completeness, the concern is resolved; if not, the reported t-tests in Section IV-C do not support the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All comparative conclusions in Section IV-C—the t-tests against simple search (t=14.98, p<0.01 for informativeness), the ReAct degeneration finding in Table I, and the router analyses in Table II—are computed from scores produced by a single LLM judge, GPT-4o-mini, whose validity is never established. The paper reports only 'we manually evaluated 8 data sets' (Section IV-B.3), with no number of items, no inter-annotator agreement, no correlation between human and judge scores, and no check for position or length bias. The judge sees the three agent outputs simultaneously and scores 'informativeness' and 'completeness,' rubrics that naturally reward longer, more structured answers. SRSA's planning strategy deliberately emits full first-step search results plus per-step summarizations, so it is the most likely to benefit from a length/format confound. The fact that novelty and actionability show no SRSA advantage makes the possibility of rubric-specific bias concrete rather than hypothetical. If the judge is biased, the central assertion that SRSA 'generates more comprehensive and informative responses' simply does not follow, regardless of the significance of the t-tests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SRSA, a strategy-router search agent that classifies each contextual query into one of three search strategies (direct, parallel, planning) and then generates the final answer from the retrieved supporting documents. The authors introduce a new dataset, CQED, of realistic, context-rich user queries, and evaluate SRSA against a single-round simple search agent and a ReAct-based agent using GPT-4o-mini as an LLM judge on informativeness, completeness, novelty, and actionability. The reported results, based mainly on Mistral-7B, claim significant gains in informativeness and completeness over both baselines, and a cost-efficiency advantage from routing easy queries to cheaper strategies.","tokens_in":11880,"tokens_out":4273,"duration_ms":39802,"significance":"If the empirical claims were properly supported, the idea of routing queries to different search strategies to balance quality and cost would be practically valuable for real-world chatbot deployments, and the CQED dataset would fill a genuine gap in evaluation resources. The paper also contributes a clear description of a planning search strategy with a compression/filtering step aimed at avoiding the degeneration seen with ReAct. However, the current evidence is not sufficient to establish these contributions because the evaluation rests on an unvalidated LLM judge, the cost claim is never measured, and the router's effectiveness is inferred from a null result rather than a direct ablation.","major_comments":[{"comment":"All comparative conclusions—the t-tests in Section IV-C.3 (t=14.98, p<0.01 for informativeness), the ReAct degeneration finding in Table I, and the strategy comparisons in Table II—depend entirely on quality scores assigned by a single LLM judge, GPT-4o-mini, whose validity is never established. The paper reports only 'we manually evaluated 8 data sets' with no details on the number of items, the evaluation procedure, inter-annotator agreement, or correlation with the LLM judge's scores. Because the judge sees all three outputs simultaneously and the rubrics for informativeness and completeness reward longer, more structured answers, the reported superiority of SRSA may be an artifact of answer format rather than genuine quality. The authors should provide a human-annotation study with agreement statistics, plus a length- or position-bias analysis, to validate the judge before the headline quality claims can be accepted.","section":"§IV-B.3, §IV-C"},{"comment":"The claim that the strategy router is effective is supported only by a null result: after routing, there are few significant differences between the strategies. The paper states in the same section that the authors 'aim to demonstrate that without the search router, when all questions are processed through a single strategy, there are significant differences in the quality of query answers,' but this no-router comparison is never reported. Without this comparison, the observed null result is equally consistent with the router being unnecessary, because all strategies may perform similarly regardless of routing. The authors should add a direct ablation comparing quality when all queries are forced through each strategy against the router-assigned strategy, and ideally also measure router accuracy against ground-truth labels.","section":"§IV-C.4, Table II"},{"comment":"The paper is titled and framed as 'cost-efficient' and claims to balance quality and computational cost, yet no cost measurements are reported anywhere. There are no counts of LLM calls, token usage, inference time, or API costs for SRSA versus the baselines. The router's cost-saving claim is only supported by the qualitative statement that simple queries are routed to the cheaper direct strategy. The authors should report quantitative cost metrics (e.g., average LLM calls per query, total tokens, wall-clock time) for each agent across the CQED dataset.","section":"Abstract, Title, §IV-C.4, §V"},{"comment":"The successful full implementation of SRSA is demonstrated only with Mistral-7B-Instruct-v0.3; the paper acknowledges that Gemma and Llama were unable to follow the required formatted output and therefore defaulted to the direct strategy. While this is noted as a limitation, the abstract and conclusion make general claims that SRSA 'provides an approach' that 'effectively and efficiently parses complex user queries.' Given that three of the core modules (router, parallel, planning) rely heavily on formatted instruction following, the paper should either restrict its claims to models with strong instruction-following capabilities or include results from at least one additional model that can execute the full workflow.","section":"§IV-C, §V (Limitation)"},{"comment":"The statistical analysis assumes that the automatic evaluation scores follow a truncated normal distribution, but no justification is given for this assumption, and the scores are discrete integers on a 0–5 scale. Additionally, the paper performs multiple pairwise t-tests across four metrics and three strategy comparisons without any correction for multiple comparisons, which inflates the risk of false positives. The authors should justify the t-test assumption or use a non-parametric test, and report adjusted p-values or a clear multiple-comparison strategy.","section":"§IV-B.3"}],"minor_comments":[{"comment":"The sentence reporting t-statistics of 14.9827 and 4.8846 does not specify which comparisons these correspond to (SRSA vs. simple search, or SRSA vs. ReAct, and for which metric each). The text should be explicit.","section":"§IV-C.3"},{"comment":"The caption says 'After searching the router'; this should be 'After routing by the router' or 'After routing.'","section":"Table II caption"},{"comment":"The phrase 'we manually evaluated 8 data sets' is ambiguous and likely means '8 data points' or '8 examples.' The number of manually evaluated items and the selection procedure should be stated clearly.","section":"§IV-B.3"},{"comment":"The dataset construction paragraph says 'a total of 182 answers,' but it should clarify whether this is 182 queries, 182 reference answers, or both. Also, the statement that warm-up queries are removed should report the final dataset size.","section":"§IV-A.2"},{"comment":"Reference [21] is cited for Interleaving Retrieval with Chain-of-Thought (IR-CoT), but [21] is Query2doc; the IR-CoT reference appears later as [10]. The citation numbering should be corrected.","section":"§II-C.1"},{"comment":"The notation is confusing because the letter S is used both for the chosen strategy (S, sug(S)) and for the search function S(·). Using two different symbols would improve readability.","section":"§III-B"},{"comment":"The bar charts in Figure 7 are not described with exact numbers in the text; reporting the mean scores for each strategy would make the figure self-contained and easier to verify.","section":"§IV-C.4"}],"recommendation":"major_revision","confidential_remarks":"The core idea of routing queries to different search strategies is interesting and the CQED dataset is a useful resource, but the current experimental validation has load-bearing gaps: the LLM judge is unvalidated, cost is not measured, and the router ablation is missing. These issues are addressable with additional experiments and analysis, so I do not view them as grounds for rejection at this stage. However, the authors should be asked to provide the missing validation and measurements before the paper can be considered for publication. I would also encourage the editor to check the reproducibility of the anonymous code link, as it was not evaluated in this review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper brings a new dataset (CQED) and a plausible cost-aware router over three known search strategies, but the evidence for the headline claims is too thin. The evaluation rests on an unvalidated GPT-4o-mini judge, no cost numbers, and an incomplete router ablation. I'd send it to review, but it needs major revision.\n\nWhat is genuinely useful: CQED, with 182 scenario-based queries built from Reddit posts, fills a real gap. Most RAG benchmarks are factual QA; conversational queries with implicit sub-questions are underserved. The three strategies (direct, parallel, planning) are each known, but making the router choose among them is a sensible system contribution, and the design is described clearly. The observation that a vanilla ReAct agent degrades on these long contextual queries, while not new in spirit, is a useful data point.\n\nThe soft spots are real and appear exactly where the reader's report and the stress-test note say. The judge is the whole ballgame: every t-test in Section IV-C flows from GPT-4o-mini's scores, and the paper's manual check ('we manually evaluated 8 data sets') gives no agreement statistics, no correlation, no length-bias check. Given that SRSA's planning strategy emits longer, structured references, rubric-specific bias is a concrete risk. The cost-efficiency claim is asserted but never measured—there are no token counts, latencies, or dollar figures. The router ablation is missing: the paper says it wants to show quality differences without the router, but Table II only compares strategies after routing; a null result there could equally mean routing is irrelevant. Only Mistral ran the full framework, so the headline is a single-model result. And the ReAct baseline omits summarization, so the degeneration finding is not a controlled comparison.\n\nCredit where due: the Limitations section is honest about several of these (one model, few baselines, subjective dataset eval), and the prose is straightforward. The logic of the router is not circular in itself; the circularity is in the interpretation of Table II.\n\nVerdict: this deserves a serious referee, not a desk reject. The dataset and the system design warrant discussion, and the evaluation gaps are fixable. I'd send it out with a request for a real human-evaluation study, cost measurements, and the missing ablation. In current form the central claims don't hold, but the paper has a useful core.","headline":"Useful new dataset and a plausible cost-aware router, but the evaluation is too weak to support the headline quality and cost claims.","tokens_in":12473,"tokens_out":2496,"would_cite":true,"duration_ms":22237,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A strategy router that matches each user query to one of three search plans—rewrite, parallel, or stepwise—can produce more informative and complete answers than a single search or a naive ReAct loop, without fine-tuning the LLM.","keywords":["strategy router","search agent","retrieval-augmented generation","large language models","query rewriting","ReAct","contextual queries","LLM evaluation"],"falsifier":"Run the same CQED comparison with human raters (or a second independent LLM judge) scoring the answers blind to agent identity, and check whether SRSA's advantage over the simple search agent on informativeness and completeness persists. If it shrinks or disappears when answer length and structure are controlled for, the reported gains are an artifact of the judge rather than of the routing.","tokens_in":11452,"feed_emoji":"🔍","tokens_out":8036,"duration_ms":60936,"temperature":0.7,"pith_summary":"Real chatbot queries are often long, personal, and tangled: they mix several implicit sub-questions and depend on context a search engine cannot see. The paper proposes a Strategy-Router Search Agent (SRSA) that first classifies each query into one of three search strategies—rewrite-and-search, parallel sub-question search, or stepwise planning with per-step filtering—and then feeds the resulting reference passages into a frozen LLM to write the final answer. The central claim is that this routing fixes a degeneration seen in earlier iterative agents: a ReAct-style agent that just thinks-acts-observes repeatedly performs worse than a single-round search on these contextual queries, while SRSA's planning strategy compresses and filters each search round and outperforms both baselines on informativeness and completeness. If true, it means better answers for realistic human-chatbot interactions can be obtained without fine-tuning, and at lower cost because simple queries are sent to cheap strategies.","feed_headline":"Strategy router lifts search agents on long, contextual queries","feed_subtitle":"Router sends easy queries to cheap searches, hard ones to stepwise plans, improving answers without fine-tuning.","key_machinery":"The central mechanism is the strategy-router module and the three parameter-free search strategies it selects among. The router is a single LLM call that, given the user query (with a time module), outputs both a strategy label $\\mathcal{S} \\in \\{D, P, R\\}$ and a set of search suggestions $\\mathrm{sug}(\\mathcal{S})$. Direct search ($D$) rewrites the query and searches once; parallel search ($P$) generates several sub-questions, searches them concurrently, and concatenates the results; planning search ($R$) runs a loop that compresses and filters the previous round's results, evaluates whether they are relevant, rewrites the query if they are not, and stops when the accumulated memory suffices. The final answer is generated by a RAG prompt from the original query plus the strategy's curated reference passage. This design is what lets the system balance cost and quality: easy queries go to $D$, multi-concept queries to $P$, and sequential, context-dependent queries to $R$.","core_discovery":"On its own terms, the paper's core discovery is that a strategy router—a single LLM prompt that outputs a strategy label plus search suggestions—can allocate queries among three search strategies so that the system as a whole is both more informative and more complete than either a single-search agent or a ReAct-based agent, while spending less compute on easy queries. The experimental result on the Mistral model shows SRSA significantly beats the ReAct agent and the simple search agent on informativeness (t=14.98, p<0.01) and completeness (t=4.88, p<0.01), with no significant loss on novelty or actionability. The paper also demonstrates that a naive ReAct agent \"degenerates\" on contextual queries, underperforming even the simple search agent, and attributes this to irrelevant search results accumulating in the iterative history. The router's planning strategy addresses this by summarizing and filtering each iteration's results before deciding the next search. In addition, the paper contributes the Contextual Query Enhancement Dataset (CQED), built from Reddit threads, to test such contextual queries.","pith_inferences":["The routing principle likely extends beyond search to other tool-using agents: instead of always running the same expensive loop, an agent could classify a task's complexity first and dispatch to a cheaper or more elaborate tool sequence accordingly.","A testable extension is to replace the LLM judge with human raters or a different judge model; if the SRSA advantage shrinks, the observed gains may be partly an artifact of judge preferences for longer, structured answers.","The degeneration of ReAct on contextual queries suggests a broader caution about iterative reasoning agents: without filtering, errors and irrelevance compound over turns, which may also affect multi-step planning agents in other domains.","The router's cost savings could be quantified more precisely by measuring token usage or response latency per strategy, not just quality scores."],"forward_implications":["A straightforward ReAct-style search loop can be worse than a single search on context-rich queries, so search agents should filter or compress intermediate results rather than accumulate all of them.","Query rephrasing alone, before a single search, improves informativeness and completeness on lengthy queries, even for models that cannot follow complex routing prompts.","A frozen LLM with prompt-based routing can match or beat fine-tuned search agents on contextual queries, which lowers the cost of deployment.","The CQED dataset provides a way to measure whether search agents understand user context, complementing existing QA datasets that assume clean factual questions."],"supporting_citations":[{"why":"Supplies the ReAct baseline's iteration limit and a state-of-the-art metacognitive search agent comparison.","marker":"[3]"},{"why":"Defines the thought-action-observation baseline whose degeneration SRSA is designed to fix.","marker":"[7]"},{"why":"Explains how accumulated irrelevant search results degrade final answers, motivating the planning strategy's filtering.","marker":"[8]"},{"why":"The closest prior approach to planning search; SRSA adapts it with summarization and selective reference documents.","marker":"[10]"},{"why":"Underpins the rephrasing module, which alone accounts for significant gains on lengthy queries.","marker":"[20]"},{"why":"The common search engine used across all agents, making the comparisons fair.","marker":"[35]"},{"why":"Justifies using an LLM judge for the four quality metrics.","marker":"[39]"}],"fun_headline_variants":["Strategy router makes search agents smarter on contextual queries","Route queries smartly: SRSA cuts cost, boosts answer quality","Contextual query routing improves LLM search without fine-tuning","Smart query routing: easy queries cheap, hard ones stepwise","Strategy router beats ReAct on contextual queries, cuts compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The head-to-head comparisons assume the LLM judge's scores for informativeness, completeness, novelty, and actionability are unbiased and meaningful; the paper reports only a vague manual check of 8 data sets, with no inter-rater agreement or evidence that the judge is not favoring longer, more structured answers.","fun_headline_variants_meta":{"raw":{"variants":["Strategy router makes search agents smarter on contextual queries","Route queries smartly: SRSA cuts cost, boosts answer quality","Contextual query routing improves LLM search without fine-tuning","Smart query routing: easy queries cheap, hard ones stepwise","Strategy router beats ReAct on contextual queries, cuts compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000914,"raw_usage":{"total_tokens":3949,"prompt_tokens":991,"completion_tokens":2958,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":2876}},"tokens_in":607,"tokens_out":2958,"duration_ms":20649,"temperature":1.0,"reasoning_tokens":2876,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:07:42.413609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same CQED comparison with human raters (or a second independent LLM judge) scoring the answers blind to agent identity, and check whether SRSA's advantage over the simple search agent on informativeness and completeness persists. If it shrinks or disappears when answer length and structure are controlled for, the reported gains are an artifact of the judge rather than of the routing.","supporting_citations":[{"cited_title":"Metacognitive retrieval- augmented large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the ReAct baseline's iteration limit and a state-of-the-art metacognitive search agent comparison."},{"cited_title":"Large language models can be easily distracted by irrelevant context,","cited_arxiv_id":null,"evidence_quote":"Explains how accumulated irrelevant search results degrade final answers, motivating the planning strategy's filtering."},{"cited_title":"Tavily Search API,","cited_arxiv_id":null,"evidence_quote":"The common search engine used across all agents, making the comparisons fair."}],"review_version":1}