{"id":"000185d0-8b69-4a55-bc7e-0c142e8fa718","arxiv_id":"2601.14224","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Listwise reranking of the top 10–50 retrieved documents delivers comparable or better deep-search accuracy than increasing reasoning effort, at substantially lower effective token cost on BrowseComp-Plus.","lead":"This paper measures whether re-sorting the documents an AI search agent retrieves — before it reasons over them — buys accuracy more cheaply than asking the agent to think longer. On a fixed web corpus, a moderate reranking step matches or beats much heavier reasoning at a fraction of the effective token cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy is scored by the same oss-120b model that generates the answers; the paper never tests an independent judge, so the central reranking-vs-reasoning cost comparison may rest on a biased metric. Table 2's CIs (e.g., ±0.08) also appear to be within-run, not across-query.","rationale":"I read the paper's central claim as a cost-effectiveness Pareto argument: for a given model size, moderate reranking depth achieves accuracy comparable to higher reasoning effort at much lower token cost. The paper's Table 4 raw token counts make the cost side of this claim robust to α/β: for oss-120b, med+d=20 uses 210.6M raw tokens vs 1040.8M for high+d=0, so even with α=1, β=1 the cost ratio is ~5x. Thus the reader's identified weakest assumption (ETC parameters) is not the most load-bearing. The accuracy side is less secure. The judge is the same model family (oss-120b) as the systems being compared, and no independent judge or human evaluation is reported. This is a direct threat to the primary outcome, and it is absent from the Limitations section (which does note same-family search/reranking but not judging). The implausibly small CIs in Table 2 compound this by giving a false sense of precision. Because the conclusions depend on accuracy deltas (e.g., the +10.6 point gain from adding d=20 to medium) and comparability (50.91 vs 52.00), a judge-induced shift could alter the Pareto ranking. A concrete test with an independent judge model would resolve this. I therefore maintain the reader's CONDITIONAL verdict, but for a different primary reason.","tokens_in":14012,"tokens_out":14587,"duration_ms":144084,"concrete_test":"Run the BrowseComp-Plus evaluation for 100 randomly sampled queries (or the full set if compute allows) with an independent judge model from a different model family (e.g., a strong open-weight model or a different commercial API) using the same judging prompt; then compare the accuracy deltas between the key configurations (e.g., oss-120b medium+d=20 vs high+d=0, and oss-120b low+d=50 vs med+d=0). If the deltas change by more than 5 points or the Pareto ordering flips, the central cost-effectiveness claim is not robust to judge choice. Additionally, recompute the 95% CIs in Table 2 using query-level bootstrap over the 830 queries (or McNemar's paired test) to determine if the reported accuracy differences are within noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The primary outcome is LLM-as-a-judge accuracy, computed by prompting gpt-oss-120b with the BrowseComp-Plus judge prompt (§2, Figure 5). The same model family is used as the search agent and reranker, so every reported accuracy number comes from a judge that is also the system under evaluation. Self-evaluation bias in LLM judges is a known phenomenon; even with a constrained 'extract final answer and compare' prompt, judges are sensitive to answer style and length. In the key comparisons (e.g., oss-120b medium+d=20 at 50.91% vs. high+d=0 at 52.00%, Table 2), the high-reasoning traces produce much longer outputs, which can systematically shift the judge's accept/reject decisions. If the judge is lenient toward verbose outputs, the high-reasoning accuracy is inflated and the paper's conclusion is conservative; if the judge makes more extraction errors on long outputs, high-reasoning accuracy is deflated and the conclusion that reranking is cost-effective could be an artifact. The limitation section acknowledges same-family search and reranking but omits this judge bias entirely. Additionally, Table 2's CIs (e.g., 14.87 ±0.08) are far too tight for 830 queries (a 95% CI for a proportion near 0.15 would be about ±2.4 points); the only way to get ±0.08 is to compute dispersion across the five repeated judge runs, which does not reflect query-level uncertainty. The paper gives no CI methodology, so the accuracy deltas underpinning 'comparable accuracy' and 'larger gains' lack a stated uncertainty.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how to allocate test-time computation in a deep research agent on the BrowseComp-Plus benchmark. It introduces the Effective Token Cost (ETC) metric (Eq. 1), a weighted sum of non-cached input, cached input, and output tokens with a cached-input discount α and output premium β, and compares gpt-oss-20b/120b search agents under low/medium/high reasoning with and without a listwise reranking stage at depths d∈{10,20,50}. The central empirical claim is that reranking consistently improves retrieval and end-to-end accuracy, and that moderate reranking (d≤20) often yields accuracy comparable to -- or better than -- increasing the search agent's reasoning effort, at substantially lower ETC. The appendix provides raw token counts, hardware configurations, prompts, and a link to code.","tokens_in":14437,"tokens_out":7337,"duration_ms":80287,"significance":"The question is timely, and the experimental design has real strengths: the fixed human-verified corpus removes web-API confounds; Table 4 gives full raw token usage; Tables 6 and 7 (Figs. 6-7) sweep the ETC hyperparameters; and the monotonic improvements in Table 2 across model sizes, reasoning efforts, and reranking depths are consistent and suggestive. If the accuracy comparisons survive a more rigorous evaluation protocol, the paper would give practitioners concrete, actionable guidance on where to allocate test-time budget in deep search. The main weaknesses are that the accuracy metric is produced by an LLM judge from the same model family as the systems under test, and that the reported confidence intervals are not query-level; both concerns are load-bearing for the quantitative 'comparable accuracy at lower cost' claim. The ETC framework is a useful analytic lens, but its 'realistic' status is asserted rather than calibrated to any concrete pricing or throughput regime.","major_comments":[{"comment":"Accuracy is computed by prompting gpt-oss-120b with the BrowseComp-Plus judging prompt. Since both search and reranking are also gpt-oss-20b/120b, every reported accuracy number comes from a judge from the same model family as the system under evaluation. The Limitations section acknowledges same-family search and reranking but does not mention the judge. This issue is load-bearing for the central comparison: for example, the key contrast in Table 2 between rows 2e and 2f (oss-120b medium+d=20 at 50.91% vs. high+d=0 at 52.00%) compares two gpt-oss-120b outputs judged by gpt-oss-120b, and the outputs differ substantially in length and style across reasoning efforts. Please provide at least one independent judge (a different model family or a human-verified subset) or a sensitivity analysis showing that judge decisions are stable with respect to output length, verbosity, and answer formatt","section":"§2 Evaluation Metrics; Figure 5; Table 2"},{"comment":"The reported 95% confidence intervals are not consistent with query-level uncertainty. For example, an accuracy of 14.87% on 830 queries has a binomial standard error of about 1.2 points, giving a 95% CI of roughly ±2.4 points, not ±0.08. The intervals appear to be computed over the five repeated judge runs mentioned in §2, which captures judge stochasticity rather than sampling over queries. The paper never states the CI methodology. This makes it impossible to assess whether accuracy differences such as 52.00 vs. 50.91 (rows 2f/2e) or 44.15 vs. 42.17 (rows 2b/0c) are meaningful. Please provide query-level bootstrap confidence intervals (or a variance decomposition separating query and judge variance) for all accuracy and calibration numbers in Table 2, and state the method explicitly.","section":"Table 2; §2 Evaluation Metrics"},{"comment":"The ETC metric is a weighted sum whose parameters α∈{0.1,0.3,0.5} and β∈{3,5,7} are configurable proxies, but no evidence is given that these ranges correspond to actual deployed prices or vLLM throughput ratios. The abstract and conclusion state that reranking achieves 'substantially lower cost' without hedging. The sensitivity analysis in Figs. 6-7 is helpful and shows qualitative stability, but it does not establish the external validity of the cost numbers. Please either calibrate α and β to a concrete pricing/throughput regime, or explicitly state that the cost-effectiveness conclusion is conditional on the swept ranges and should be read as a qualitative trend, not as measured dollar or latency savings.","section":"§3 Effective Token Cost; Eq. (1); abstract/conclusion"}],"minor_comments":[{"comment":"The abstract gives the code URL as github.com/sahel-sh/DeepHone while the full text gives github.com/texttron/BrowseComp-Plus.git. Please unify.","section":"Abstract vs. full text"},{"comment":"The header 'Inp. (Cash.)' appears to be a typo for 'Cach.' (cached tokens).","section":"Table 6 header"},{"comment":"The caption says 'per ten million tokens' but the x-axis is labeled in billions; align the wording with the units actually plotted.","section":"Figure 2 caption/axis"},{"comment":"The abstract's phrase 'moderate reranking often yields larger gains than increasing search-time reasoning' is ambiguous. Read as raw accuracy gains, it is not supported by Table 2 (e.g., oss-120b low d=20 gains 6.85 points over d=0, whereas increasing reasoning low→medium at d=0 gains 12.68 points). The paper's actual claim is a cost-effectiveness claim. Please rephrase to say 'larger gains per unit of effective token cost' or similar.","section":"Abstract; §4 Deep Research Token Usage"},{"comment":"The statement that 'varying α does not noticeably impact the observed trends' is made for Figure 1 and then again for Figure 2, but the full sweeps are only in the appendix. A one-sentence explanation of why the trends are invariant (input tokens are mostly non-cached in reranking, and mostly cached in search but with different fractions) would help the reader trust the abbreviated main-text plots.","section":"§4 Reranking Token Usage"}],"recommendation":"major_revision","confidential_remarks":"The paper is close to being acceptable, but the quantitative headline is not yet supported. The two blocking issues are statistical (non-query-level CIs in Table 2) and evaluator validity (gpt-oss-120b judging gpt-oss-120b). Both are fixable with additional analysis or small experiments, so I do not recommend rejection. The ETC calibration issue is more a framing matter than a fatal flaw, given the explicit sensitivity sweeps. I would like the editor to see the paper after revision with an independent judge or clear judge-stability analysis, and with proper query-level uncertainty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fairly solid empirical study, but two things should worry you before you rely on the headline: the accuracy numbers come from a judge that is the same model family as the system under test, and the reported CIs are not query-level and are far too tight. The paper's main finding—moderate listwise reranking often buys more accuracy per token than extra search-time reasoning—is supported by the monotonic pattern in Table 2 and by raw token tables, so the qualitative conclusion is probably right. But the quantitative \"substantially lower cost\" claim rests on a metric whose weights are chosen, not measured, and on accuracy differences that may be within noise.\n\nWhat's new: a systematic measurement of reranking depth × reasoning effort on BrowseComp-Plus, plus the ETC metric. ETC is a simple linear combo of non-cached/cached input and output tokens, with α and β swept widely; the qualitative trends are stable across sweeps, which is good. The paper also ships extensive token counts and is honest about limitations, though the limitations section stops short of discussing judge bias.\n\nThe soft spots, in order:\n1. CI methodology. Table 2 reports ±0.08 for 14.87% on 830 queries. A binomial proportion near 0.15 has a 95% CI of about ±2.4 points. The tiny CIs must be dispersion across the five repeated judge calls, which tells you about judge stability, not query-level uncertainty. No methodology is given. This matters because key claims like \"comparable accuracy\" (e.g., 50.91 vs 52.00) are within that plausible noise.\n2. Same-model judge. Accuracy is scored by gpt-oss-120b, same family as the search agent and reranker. The judge prompt is the standard BrowseComp-Plus one, so it's not arbitrary, but self-evaluation bias is real and could go either way. If the judge is lenient toward verbose high-reasoning outputs, the conclusion is conservative; if it makes more extraction errors on long outputs, the conclusion could be partly artifact. The paper should test an independent judge.\n3. Reproducibility. The abstract points to an unrelated repo (github.com/texttron/BrowseComp-Plus); no link to the actual code. Also no commit hash.\n4. Minor: α/β are not measured from real deployments, but the sweep makes this a minor issue.\n\nBottom line: For a reader building deep-search agents, the qualitative allocation rule—add a moderate-depth reranker before increasing reasoning budget—is worth taking seriously. The paper needs revision but deserves a serious referee, not a desk reject.","headline":"Useful empirical study with a plausible qualitative finding, but the CI reporting and same-model judge need to be fixed before the cost-effectiveness claim is quantitatively reliable.","tokens_in":14886,"tokens_out":3003,"would_cite":true,"duration_ms":31417,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Moderate reranking of retrieved candidates yields larger accuracy gains per token than increasing a deep search agent's reasoning effort, and at substantially lower cost.","keywords":["reranking","test-time computation","deep search agents","token cost","efficiency","listwise ranking","reasoning budget","LLM agents"],"falsifier":"A direct test would be to run the same deep-search pipeline on a live-web-search benchmark (rather than a fixed corpus) and measure accuracy against ETC with the same α and β; if reranking no longer delivers comparable accuracy at lower ETC—for example because web results are noisier and need more reasoning to verify—the paper's claim would fail. A second, cheaper test is to use the paper's published token counts and vary α and β outside the ranges tested (e.g., β=1 or α=1) to see if the ordering of configurations flips.","tokens_in":13923,"feed_emoji":"🔀","tokens_out":4181,"duration_ms":39643,"temperature":0.7,"pith_summary":"The paper claims that in deep research agents, adding a listwise reranking stage over the top 10–50 retrieved documents is a more token-efficient way to improve accuracy than increasing the search agent's reasoning effort from low to high. Comparing two open-weights model families on a fixed-corpus deep search benchmark, it finds that medium reasoning combined with deeper reranking reaches accuracy comparable to high reasoning with shallow or no reranking, at a much lower effective token cost. The central evidence is a new metric, effective token cost (ETC), which weights cached input, uncached input, and generated output tokens by plausible discount and premium factors. If the claim holds, the practical lever for efficiency in deep search pipelines is the retrieval-reranking stage, not simply more search-time reasoning.","feed_headline":"Reranking beats extra reasoning at lower token cost","feed_subtitle":"Medium reasoning plus a reranker matches high-reasoning accuracy for a fraction of the effective token cost.","key_machinery":"The effective token cost (ETC) metric, ETC = Input_nc + α·Input_c + β·Output_t, where Input_nc and Input_c are non-cached and cached input tokens, Output_t is generated output tokens, α (0.1–0.5) is the caching discount, and β (3–7) is the output premium. This single scalar lets the paper compare configurations across hardware and API pricing regimes by converting token counts into a cost-like number. The listwise reranking stage itself—a model that rank-orders the top d retrieved candidates and returns the top 5—is the mechanism that shifts the cost-effectiveness frontier. The paper uses the same underlying model family for search and reranking, which isolates the effect of reranking depth","core_discovery":"The paper's central claim is that reranking before reasoning is the lever that matters: for a fixed model, increasing reranking depth from none to 10–50 candidates reliably improves both retrieval recall and end-to-end answer accuracy, and this improvement is cheaper per unit of accuracy than increasing the agent's reasoning effort. Specifically, medium reasoning effort paired with deeper reranking matches the accuracy of high reasoning effort with shallow or no reranking while consuming substantially fewer effective tokens. The authors support this with controlled experiments on a deep-search benchmark with a fixed, human-verified corpus, using the same two model families for both search an","pith_inferences":["If the ETC ratios hold in real deployments, a system designer could lower operating cost by replacing a high-reasoning agent with a medium-reasoning agent plus a reranker, without sacrificing accuracy—an effect likely to widen as rerankers get cheaper relative to reasoning.","The paper's fixed top-d design suggests an extension: a learned relevance assessor that dynamically selects a variable-size subset of evidence might beat any fixed-d reranking, since it could cut redundant context before it reaches the search agent.","Because the experiments keep the full message history and rely on automatic truncation, adding explicit history compression could shift the tradeoff even further toward reranking, as reranking input tokens are the dominant uncached cost.","The same ETC accounting could be applied to other agent designs, such as iterative retrieval with feedback or multi-agent debate, to identify where token budgets are best spent."],"forward_implications":["Reranking with depth d=10 already yields most of the accuracy gain; increasing d from 20 to 50 gives diminishing returns, especially under medium and high reasoning.","A smaller model with high reasoning effort can outperform a larger model with low reasoning effort, but the same accuracy is reachable more cheaply by pairing medium reasoning with reranking.","Reranking reduces calibration error in addition to improving accuracy, and this benefit shrinks as d grows.","Because most input tokens are uncached and most output tokens go to reasoning, the ETC tradeoffs are stable across the exact choice of α and β in the tested ranges.","The paper's results suggest that search-time reasoning budgets should be allocated after accounting for the reranking stage; a fixed budget goes further when part of it is spent on retrieval quality."],"fun_headline_variants":["Rerank before reasoning for cheaper gains","Reranking beats extra reasoning at lower cost","Rerank more, reason less for accuracy","Deep search: reranking outperforms reasoning","Reranking depth trumps reasoning effort"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The conclusion that reranking is cheaper than more reasoning rests on the ETC weightings, specifically on the assumption that cached input tokens cost roughly 10–50% of uncached input tokens and that generated output tokens cost 3–7 times as much per token; if real prices or throughput ratios differ, the regime where reranking wins could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Rerank before reasoning for cheaper gains","Reranking beats extra reasoning at lower cost","Rerank more, reason less for accuracy","Deep search: reranking outperforms reasoning","Reranking depth trumps reasoning effort"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":2927,"prompt_tokens":629,"completion_tokens":2298,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":373,"completion_tokens_details":{"reasoning_tokens":2230}},"tokens_in":373,"tokens_out":2298,"duration_ms":18153,"temperature":1.0,"reasoning_tokens":2230,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:16:37.614487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run the same deep-search pipeline on a live-web-search benchmark (rather than a fixed corpus) and measure accuracy against ETC with the same α and β; if reranking no longer delivers comparable accuracy at lower ETC—for example because web results are noisier and need more reasoning to verify—the paper's claim would fail. A second, cheaper test is to use the paper's published token counts and vary α and β outside the ranges tested (e.g., β=1 or α=1) to see if the ordering of configurations flips.","supporting_citations":[],"review_version":1}