Pith. sign in

REVIEW 3 major objections 6 minor 25 references

When Should LLMs Search? Counterfactual Supervision for Search Routing

T0 review · 3 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Comparing no-search and forced-search outcomes for the same question yields a usable oracle that trains LLMs to call search only when it improves task success.

desk verdict Clean first-action routing paper: counterfactual N/S oracle + SFT/DPO gives real macro-F1 gains on two models; main soft spot is unvalidated KUQ judges, not the framing. read the letter →

arxiv 2607.05752 v1 pith:E6ZKHPAV submitted 2026-07-07 cs.CL cs.AI

Minho Kim This is my paper · ORCID
classification cs.CLcs.AI
keywords searchroutingretrieval-augmentedgenerationcounterfactualsupervisiontoolusewhennottocallpreferenceoptimizationknown-unknownquestionsselective
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Search tools help language models answer questions their parameters cannot, but search is not always the right first move: it can be unnecessary, noisy, or worse than correcting a false premise, asking for clarification, or abstaining. This paper treats the decision as instance-level search routing—whether to answer without the tool or to issue a search call—and builds supervision without human labels of “search usefulness.” For each question it runs a no-search trace and a forced-search trace, scores both with task-specific success criteria, and labels the case NO SEARCH, SEARCH, or UNSOLVED. On the eligible labels it trains routing policies with supervised fine-tuning and preference optimization, lifting routing macro-F1 well above a prompt-only selective-search baseline for two small models. Residual dual-failure cases are kept as diagnostics, showing that many remaining errors sit outside the first-action choice.

What carries the argument

Counterfactual routing oracle: for each question, compare binary task success under a no-search execution and a forced first-turn search execution; assign NO SEARCH if no-search already succeeds, SEARCH if search recovers a no-search failure, and UNSOLVED if both fail (excluded from routing training and accuracy).

What would settle it

Hold the same models, prompt, and Brave search interface fixed, but re-label a large held-out set with human task-success judgments (or with a multi-hop search protocol): if the counterfactual oracle then disagrees with human necessity labels on a large fraction of cases, or if SFT/preference training on the new labels fails to beat the prompt-only selective-search baseline on routing macro-F1, the central claim is falsified.

Watch

Extended reading notes

Core claim

An outcome-based oracle built from paired no-search and forced-search traces for the same question is both a fair evaluation target and a learning signal for instance-level search routing. Training on that oracle with supervised fine-tuning and preference optimization raises routing macro-F1 on held-out oracle-eligible examples from 0.7082 to 0.8235 for Gemma E2B and from 0.7053 to 0.8365 for Qwen3.5-4B, correcting model-specific under-search and over-search rather than merely changing global search rate.

Load-bearing premise

The automated judges and the single first-turn forced-search setup correctly decide whether search was needed; if they systematically mis-score answers or if richer multi-step search would flip many dual-failure labels, the oracle and the reported gains no longer hold.

Editorial extensions

If this is right

  • Search-augmented systems can train the first-action call decision from automatic paired rollouts instead of hand labels of search usefulness.
  • Routing training should correct under-search and over-search jointly and model-specifically, not chase a single global search-call rate.
  • NO SEARCH is a success path that includes correct direct answers, premise correction, clarification, and abstention—not a failure mode.
  • Dual-failure (UNSOLVED) cases should stay outside binary routing supervision and be diagnosed for capacity, retrieval budget, evidence use, and later pipeline stages.
  • The same outcome-oracle template can evaluate free-policy selective-search prompts against observed recoveries rather than against dataset-level assumptions that every PopQA item needs search or every boundary item forbids it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If paired counterfactual traces are cheap enough, the same supervision recipe could extend to other optional tools (calculator, code interpreter) where “when not to call” matters as much as correct arguments.
  • As base models grow, the NO SEARCH / SEARCH boundary will shift with parametric knowledge, so routing adapters may need periodic re-oracle collection rather than a fixed label set.
  • Production systems could use the UNSOLVED diagnostic probes (larger model, expanded retrieval, post-training policy) as a cheap triage queue for hard failures instead of forcing a binary route.
  • Preference optimization’s modest further gains after SFT suggest the remaining headroom may sit more in query formulation and evidence use than in the first-action bit alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper frames search use in tool-augmented LLMs as an instance-level first-action routing problem (NO SEARCH vs SEARCH), where NO SEARCH covers direct answering, premise correction, clarification, and abstention. Supervision is derived from paired counterfactual traces: no-search N(q) and forced-search S(q). Task-specific binary success induces an oracle over NO SEARCH, SEARCH, and UNSOLVED; UNSOLVED is excluded from routing training/metrics and retained for diagnostics. On PopQA and KUQ false-assumption/ambiguous questions, SFT and DPO-style preference optimization raise free-policy routing macro-F1 on held-out oracle-eligible examples from 0.7082 to 0.8235 (Gemma E2B) and from 0.7053 to 0.8365 (Qwen3.5-4B) relative to a fixed selective-search prompt baseline, with model-specific error-direction repairs and exclusive-rescue analysis of hard dual-failure cases.

Significance. If the result holds under a validated success criterion, the contribution is a practical and well-scoped protocol for learning when to call search from observed outcomes rather than from a separate usefulness classifier or heuristic. Strengths include: (i) an explicit model-specific, outcome-based oracle rather than dataset-level search labels; (ii) controlled comparison of Ppre, SFT, and preference optimization under a fixed free-policy prompt and search interface; (iii) directional over-/under-search metrics that expose different failure modes across model families; and (iv) a diagnostic treatment of UNSOLVED that avoids collapsing heterogeneous pipeline failures into binary routing targets. The work is relevant to adaptive retrieval, tool-use, and over-search literature, and the counterfactual construction is reusable beyond the two models studied.

major comments (3)
  1. The central claim treats the counterfactual oracle as a usable proxy for search need, but y_N and y_S for KUQ rest on unvalidated gpt-5.4 resolution judges that map free-form correction/clarification/abstention to SUCCESS/FAIL (Appendix A.1, Tables A.1–A.2, prompts C.7–C.8). These labels determine both oracle eligibility and the FA Acc./Amb. Acc. columns in Table 2. Without human agreement, inter-judge reliability, or sensitivity analysis on KUQ labels, the reported routing gains on search-boundary subsets—and the interpretation of the oracle as reflecting true task success—are not yet load-bearing. PopQA’s alias-aware factuality judge is more standard; the gap is concentrated in the regime that motivates the paper.
  2. Section 4 and Table 2 evaluate agreement between free-policy first actions and the oracle, not whether post-training free-policy rollouts improve end-to-end task success relative to Ppre (or relative to always-search / never-search). Because free-policy search trajectories need not match successful forced-search S(q) traces, and free-policy no-search need not match N(q), routing macro-F1 gains do not by themselves establish improved task success—the quantity used to define the oracle in Section 2. Reporting free-policy success rates (overall and by oracle bucket) on the same test split is needed to close the loop on RQ2 as framed in the introduction.
  3. Section 2 and Appendix A.2 define SEARCH benefit via a single forced first-turn search call under a fixed Brave budget, with invalid first calls dropped and max-tool-round failures counted as y_S=0. This is a reasonable controlled protocol, but it systematically understates search utility when multi-hop refinement or larger evidence budgets would flip N0S1/N0S0 labels (the R_TR probe in Section 5 already shows some hard cases are budget-limited). The paper should either (a) quantify how often richer search would re-label oracle buckets on a subsample, or (b) more sharply limit claims to “search under this first-call protocol,” so that residual UNSOLVED and under-search rates are not over-interpreted as intrinsic search need.
minor comments (6)
  1. Figure 1 and Figure 2 are described but the manuscript text does not fully specify visual content for readers of the text-only version; ensure captions are self-contained.
  2. Table 2 reports identical SFT macro-F1 (0.8207) for both models; a brief note on whether this is coincidence or a shared checkpoint-selection effect would help.
  3. Appendix A.4: conditional-error denominators (e.g., Gemma test NO SEARCH=306, SEARCH=502) are useful; consider promoting a short version into the main experimental setup for readability.
  4. Related Work could more explicitly contrast the outcome-based oracle with confidence/self-knowledge triggers (Self-RAG, Adaptive-RAG, SKR) in a single comparison table of supervision source and decision granularity.
  5. Typos/formatting: “F AGEN Workshop” spacing in the header; “google/gemma-4-E2B-it” model naming is fine but should be consistent with “Gemma E2B” in tables.
  6. The free-policy prompt card (Figure C.5) is central to Ppre; quoting the key selective-search instruction in the main text (1–2 sentences) would reduce dependence on the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: counterfactual oracle is an observed label source; SFT/PO gains are ordinary held-out supervised agreement, not a result forced by construction.

full rationale

The paper’s load-bearing chain is methodological and empirical, not a first-principles derivation that collapses into its inputs. For each question it collects independent controlled traces N(q) and S(q), maps final responses to binary task success via fixed judges, and defines the oracle o(q) from the four outcome cells (NO SEARCH / SEARCH / UNSOLVED). Training then imitates oracle-consistent first actions on train-split eligible rows; free-policy evaluation measures agreement of a separate selective-search rollout P(q) with the same oracle on held-out eligible rows. That is standard supervised/preference learning against an externally constructed label, not a fitted parameter re-presented as a prediction, nor a definition of X in terms of Y. The reported macro-F1 lifts (0.7082→0.8235 Gemma; 0.7053→0.8365 Qwen) are empirical policy changes under a fixed prompt and search interface, not identities forced by the label construction. There is no self-citation uniqueness theorem, no ansatz smuggled from prior author work, and no renaming of a known closed-form result. Residual UNSOLVED diagnostics and model-specific oracle buckets further treat dual failures as outside the binary routing target rather than folding them back into the claim. Concerns about judge validity or forced-search protocol adequacy are correctness/assumption risks, not circular reductions. Score 0 with empty steps is therefore the warranted finding.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim rests on standard supervised/preference learning plus domain assumptions about LLM judges and a fixed search interface; the main invented construct is the three-way outcome oracle itself. Hyperparameters are ordinary training knobs, not free parameters fitted to produce the headline metric.

free parameters (3)
  • LoRA rank / alpha / dropout = r=16, α=32, dropout=0.05
    r=16, alpha=32, dropout=0.05 chosen for PEFT; not fitted to the routing metric but still a design choice that affects capacity.
  • DPO β and NLL α = model-specific (Table A.8)
    Preference-optimization temperature and regularization strength selected per model (β∈{0.03,0.05}, α∈{0.05,0.1}); checkpoint chosen on dev routing.
  • Search evidence budget = 10 results / 5 URLs / 2048 tokens / 4 rounds
    Brave result count, max URLs, evidence tokens, tool rounds fixed a priori; expanded only in the RTR diagnostic.
assumptions (4)
  • domain assumption Task-specific LLM judges (PopQA alias-aware SimpleQA-style; KUQ resolution JSON) correctly map free-form responses to binary success.
    All oracle labels and metrics are defined through these judges (Appendix A); no human agreement study is reported.
  • domain assumption A single forced first-turn search call under the fixed Brave interface is a sufficient counterfactual for whether search can recover a no-search failure.
    SEARCH labels require y_N=0 and y_S=1 under this protocol; multi-hop or alternative backends are out of scope (Section 7).
  • ad hoc to paper First-action routing (NO SEARCH vs SEARCH) is the decision of interest; post-search query refinement and synthesis are left unoptimized.
    Explicit methodological choice in Sections 2 and 7 that scopes both training and evaluation.
  • standard math Standard DPO + chosen-response NLL regularization improves preference for the oracle-consistent first action.
    Uses the established Rafailov et al. / Pang et al. objective without modification of the math.
invented entities (2)
  • Counterfactual search-routing oracle (NO SEARCH / SEARCH / UNSOLVED)
    purpose: Convert paired no-search and forced-search binary outcomes into instance-level supervision and evaluation labels.
    Defined in Section 2 from y_N and y_S; not a physical entity but a new labeling construct central to the method.
  • Hard UNSOLVED subset with exclusive-rescue probes (M_N, M_S, R_TR, P_POST)
    purpose: Diagnose heterogeneous residual failures without forcing a binary routing label.
    Constructed in Section 5 / Appendix B; probes are operational diagnostics, not independently validated causal factors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Should LLMs Search? Counterfactual Supervision for Search Routing." pith.science (2026). https://pith.science/paper/E6ZKHPAV

@misc{pith2026260705752,
  author       = {Pith},
  title        = {Pith review of: When Should LLMs Search? Counterfactual Supervision for Search Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E6ZKHPAV}},
  note         = {Machine review of arXiv:2607.05752}
}
read the original abstract

Search-augmented language models can use external evidence to compensate for limitations in parametric knowledge, but search is not uniformly beneficial: models may call search for questions they can already answer, or rely on noisy evidence when correction, clarification, or abstention would be more appropriate. We formulate this as an instance-level search-routing problem: deciding whether search is needed to improve task success relative to a no-search execution. To derive supervision, we compare no-search and forced-search outcomes for the same question and construct an oracle over NO SEARCH, SEARCH, and UNSOLVED based on task-specific success. Using this oracle as both an evaluation criterion and a learning signal, we train search-routing policies with supervised fine-tuning and preference optimization, improving routing macro-F1 on oracle-eligible examples from 0.7082 to 0.8235 for Gemma E2B and from 0.7053 to 0.8365 for Qwen3.5-4B. Further analysis shows that the learned policies reduce model-specific routing failures: Gemma primarily learns no-search restraint, while Qwen further reduces missed search; residual UNSOLVED cases reveal heterogeneous bottlenecks involving model capacity, retrieval budget, evidence use, and policy behavior.

Figures

Figures reproduced from arXiv: 2607.05752 by the authors.

Figure 1
Figure 1. Motivation for instance-level search routing. Search can help, be unnecessary, or mislead, motivating instance-level routing. et al., 2023). However, the availability of a search tool does not imply that search should be used for every question. For questions the model can already answer, responding without search may be cheaper and less vulnerable to noisy retrieved evidence. For questions with false premises or mi… view at source ↗
Figure 2
Figure 2. Counterfactual search-routing pipeline. (A) We collect controlled no-search and forced-search traces, together with a free￾policy trace collected under a selective-search prompt. (B) Paired no-search and forced-search outcomes induce a routing oracle over NO SEARCH, SEARCH, and UNSOLVED. (C) The oracle is used both for routing supervision and for free-policy routing evaluation. 2. Method We formalize search routing … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages

  1. [1]

    International Conference on Learning Representations , author =

    work page

  2. [2]

    2025 , pages =

    Lu, Jiarui and Holleis, Thomas and Zhang, Yizhe and Aumayer, Bernhard and Nan, Feng and Bai, Haoping and Ma, Shuang and Ma, Shen and Li, Mengyu and Yin, Guoli and Wang, Zirui and Pang, Ruoming , editor =. 2025 , pages =. doi:10.18653/v1/2025.findings-naacl.65 , booktitle =

    work page doi:10.18653/v1/2025.findings-naacl.65 2025

  3. [3]

    Iterative

    Pang, Richard Yuanzhe and Yuan, Weizhe and Cho, Kyunghyun and He, He and Sukhbaatar, Sainbayar and Weston, Jason , editor =. Iterative. 2024 , pages =. doi:10.52202/079017-3702 , booktitle =

    work page doi:10.52202/079017-3702 2024

  4. [4]

    and Ermon, Stefano and Finn, Chelsea , editor =

    Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Manning, Christopher D. and Ermon, Stefano and Finn, Chelsea , editor =. Direct. 2023 , pages =. doi:10.52202/075280-2338 , booktitle =

    work page doi:10.52202/075280-2338 2023

  5. [5]

    2026 , howpublished =

    work page 2026

  6. [6]

    arXiv.org , author =

    Measuring short-form factuality in large language models , url =. arXiv.org , author =

    work page

  7. [7]

    Krishna, Satyapriya and Krishna, Kalpesh and Mohananey, Anhad and Schwarcz, Steven and Stambler, Adam and Upadhyay, Shyam and Faruqui, Manaal , year =. Fact,. doi:10.18653/v1/2025.naacl-long.243 , booktitle =

    work page doi:10.18653/v1/2025.naacl-long.243 2025

  8. [8]

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , year =. G-. doi:10.18653/v1/2023.emnlp-main.153 , booktitle =

    work page doi:10.18653/v1/2023.emnlp-main.153 2023

Show all 25 references
  1. [9]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , year =. Judging. Advances in Neural Information Pro...

  2. [10]

    International Conference on Learning Representations , volume =

    Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and Zhao, Sihan and Hong, Lauren and Tian, Runchu and Xie, Ruobing and Zhou, Jie and Gerstein, Mark and Li, Dahai and Liu, Zhiyu...

    2024

  3. [11]

    Toolformer:

    Schick, Timo and Dwivedi-Yu, Jane and Dess. Toolformer:. 2023 , pages =. doi:10.52202/075280-2997 , booktitle =

    2023 doi

  4. [12]

    The Twelfth International Conference on Learning Representations , author =

  5. [13]

    Journal of Machine Learning Research , author =

    Atlas:. Journal of Machine Learning Research , author =. 2023 , pages =

    2023

  6. [14]

    Retrieval-

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-. Advances in Neural Information Processing Systems , volume =. 2020 , pages =

    2020

  7. [15]

    doi:10.18653/v1/2024.emnlp-main.637 , booktitle =

    Zhang, Yuxiang and Chen, Jing and Wang, Junjie and Liu, Yaxin and Yang, Cheng and Shi, Chufan and Zhu, Xinyu and Lin, Zihao and Wan, Hanwen and Yang, Yujiu and Sakai, Tetsuya and Feng, Tian and Yamana, Hayato , year =. doi:10.18653/v1/2024.emnlp-main.637 , booktitle =

    2024 doi

  8. [16]

    doi:10.18653/v1/2023.emnlp-main.187 , booktitle =

    Li, Minghao and Zhao, Yingxiu and Yu, Bowen and Song, Feifan and Li, Hangyu and Yu, Haiyang and Li, Zhoujun and Huang, Fei and Li, Yongbin , year =. doi:10.18653/v1/2023.emnlp-main.187 , booktitle =

    2023 doi

  9. [17]

    Jiang, Zhengbao and Xu, Frank and Gao, Luyu and Sun, Zhiqing and Liu, Qian and Dwivedi-Yu, Jane and Yang, Yiming and Callan, Jamie and Neubig, Graham , year =. Active. doi:10.18653/v1/2023.emnlp-main.495 , booktitle =

    2023 doi

  10. [18]

    Adaptive-

    Jeong, Soyeong and Baek, Jinheon and Cho, Sukmin and Hwang, Sung Ju and Park, Jong , year =. Adaptive-. doi:10.18653/v1/2024.naacl-long.389 , booktitle =

    2024 doi

  11. [19]

    Wang, Yile and Li, Peng and Sun, Maosong and Liu, Yang , year =. Self-. doi:10.18653/v1/2023.findings-emnlp.691 , booktitle =

    2023 doi

  12. [20]

    Mallen, Alex and Asai, Akari and Zhong, Victor and Das, Rajarshi and Khashabi, Daniel and Hajishirzi, Hannaneh , year =. When. doi:10.18653/v1/2023.acl-long.546 , booktitle =

    2023 doi

  13. [21]

    Knowledge of

    Amayuelas, Alfonso and Wong, Kyle and Pan, Liangming and Chen, Wenhu and Wang, William Yang , year =. Knowledge of. doi:10.18653/v1/2024.findings-acl.383 , booktitle =

    2024 doi

  14. [22]

    Xie, Roy and Gopinath, Deepak and Qiu, David and Lin, Dong and Sun, Haitian and Potdar, Saloni and Dhingra, Bhuwan , year =. Over-. doi:10.18653/v1/2026.eacl-long.361 , booktitle =

    2026 doi

  15. [23]

    doi:10.18653/v1/2025.naacl-long.174 , booktitle =

    Ross, Hayley and Mahabaleshwarkar, Ameya Sunil and Suhara, Yoshi , year =. doi:10.18653/v1/2025.naacl-long.174 , booktitle =

    2025 doi

  16. [24]

    and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E

    Patil, Shishir G. and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , month =. The. Proceedings of the 42nd. 2025 , pages =

    2025

  17. [25]

    Cheng, Qinyuan and Sun, Tianxiang and Liu, Xiangyang and Zhang, Wenwei and Yin, Zhangyue and Li, Shimin and Li, Linyang and He, Zhengfu and Chen, Kai and Qiu, Xipeng , month =. Can. Proceedings of the 41st. 2024 , pages =

    2024

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.