Pith. sign in

REVIEW 2 major objections 5 minor 31 references

Equal Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents

T0 review · 2 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Search APIs that look interchangeable by answer accuracy actually hand agents different decision surfaces that change fetch cost, exploration, and contradiction risk.

desk verdict Accuracy parity is real under a frozen agent; the useful result is that search providers still create different pre-fetch evidence economies and fetch regimes. read the letter →

arxiv 2607.10198 v1 pith:UMVVAQ5M submitted 2026-07-11 cs.CL

Sriram Selvam , Anneswa Ghosh This is my paper
classification cs.CL
keywords searchAPIstool-usingagentsdecisionsurfacesprogressivedisclosureretrieval-augmentedgenerationpre-fetchevidencecontradiction-to-goldratioagentevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that commercial web search APIs should not be judged mainly by final answer accuracy for tool-using agents. Because agents usually see ranked titles, URLs, and snippets first and only then decide whether to open pages, the pre-fetch result list is a decision surface: it shapes whether the agent answers immediately, searches again, or spends tokens on fetches. Holding one agent, prompt, tools, and fetch backend fixed while swapping only the search provider, the authors find nearly identical audited correctness on a hard 100-question set, yet sharply different evidence economies. One provider surfaces more gold answers in snippets, another concentrates support at rank 1, and a third is tied to broader page exploration; a new surface contradiction-to-gold ratio also swings widely. A sympathetic reader should care because provider choice then becomes a retrieval-budget and policy choice, not a simple recall or leaderboard swap.

What carries the argument

The decision surface: the ranked snippets, URLs, and metadata visible before any page fetch. The paper measures it with a per-URL oracle that labels every element the agent saw, splits pre-fetch support from post-fetch discovery, assigns each query to a four-way action partition (SMART, MISSED, BLIND, NO-OP), and computes a surface contradiction-to-gold URL ratio over snippet-only rows.

What would settle it

Rerun the same 100 questions with a second answer model or a deliberately different fetch policy while still freezing the provider adapters and oracle; if pre-fetch support, rank-1 concentration, contradiction ratios, and decision-cell shares collapse to the same profile across providers, the claim that equal accuracy hides distinct decision surfaces fails.

Watch

Extended reading notes

Core claim

Under a frozen agent and an audited semantic-match correctness label, three commercial search providers reach nearly the same accuracy (25, 25, and 26 of 100 hard questions), but their pre-fetch surfaces differ sharply: gold-answer-rich snippets, rank-1 concentration of supporting URLs, and broader exploration regimes, with surface contradiction-to-gold ratios from 0.92 to 2.59. Equal accuracy therefore masks unequal evidence economies, so a search API is better understood as a decision surface than as a static ranked-list retriever.

Load-bearing premise

That the provider-linked fetch, support, and exploration patterns seen under one fixed agent policy and one judge will still describe how other agents and prompts behave on the same surfaces.

Editorial extensions

If this is right

  • Provider selection should be paired with a provider-aware fetch policy rather than a single universal top-k heuristic.
  • Agent evaluation needs decision-surface metrics—pre-fetch support, rank concentration, contradiction contamination, and fetch budget—not only final accuracy or gold-URL hit rate.
  • Practitioners can score candidate search APIs from traces by judging visible URLs and classifying snippet-rich, rank-concentrated, or exploration-heavy surfaces.
  • Agent-ready search products should optimize actionability: calibrated snippets, low contradiction-to-gold contamination, and rankings aligned with common fetch policies.
  • Because providers solve overlapping but different question sets, routing or multi-provider strategies can matter more than crowning a single winner.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Agent scaffolding that hard-codes “always fetch top-1” or “never fetch if the snippet looks answerable” will systematically favor some providers and punish others even when accuracy looks tied.
  • Cost and latency SLOs for production agents may move more from swapping models than from swapping search providers, once surface-aware fetch policies are tuned.
  • A natural next stress test is whether the same surface signatures appear on fresher or multi-hop tasks where contradiction density and rank instability are higher.
  • If providers begin optimizing for agent actionability, classical IR leaderboards may diverge further from what progressive-disclosure agents actually experience.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that commercial search APIs for tool-using agents should be evaluated as decision surfaces—the ranked URLs, titles, snippets, and metadata that shape whether an agent answers, re-searches, or spends tokens on page fetches—rather than solely by final answer accuracy. Under a frozen GPT-5.4 agent, shared tools and Jina fetch backend, and 100 SEALQA-HARD questions, Brave, Tavily, and Firecrawl yield near-parity semantic correctness (25, 25, 26/100) while differing sharply in pre-fetch gold support (30 vs 16 vs 16), rank-1 concentration among gold-supporting pre-fetch rows (13% vs 50% vs 13%), surface contradiction-to-gold ratio rc:g (0.92–2.59), and decision-partition mass (SMART/MISSED/BLIND/NO-OP). A Kimi-K2.6 per-URL oracle (6,869 valid judgments; 94% human agreement on clear cases) separates pre-fetch from post-fetch support and grounds the structural claims.

Significance. If the result holds under the stated protocol, it is a useful and timely contribution for agent systems research: it shows that interchangeable-looking answer accuracy can hide different retrieval-budget, contamination, and policy regimes, and it supplies concrete, trace-computable diagnostics (pre-fetch support, rc:g, decision partition) that practitioners can use without treating providers as a pure leaderboard. Strengths include a carefully controlled freeze of model/prompt/tools/fetch backend, an audited semantic_match answer label, a large per-URL oracle with human validation, paired-bootstrap intervals that keep correctness differences including zero while structural contrasts stay positive, and a released reproducibility package (code, configs, sampling, evaluation scripts, aggregates) with clear non-redistribution boundaries for raw provider content. The single-agent/observational scope is explicit and does not overclaim universality.

major comments (2)
  1. §5.1–5.3 and §7: The central claim is scoped to one frozen GPT-5.4 policy and one Kimi-K2.6 judge, and the paper correctly labels regimes as provider-associated rather than causal. That scope is load-bearing for transfer: SMART/MISSED/BLIND/NO-OP mass and fetch appetite could shift under a different answer model or fetch policy. The manuscript should either (a) add a small second-agent or policy-sensitivity check on a subset of queries, or (b) strengthen the abstract/conclusion wording so that “provider choice is a retrieval-budget and policy decision” is always read as under a fixed agent policy, not as a provider ranking independent of policy. Without one of these, the practical recommendation in §6 is slightly stronger than the evidence boundary.
  2. Table 4 and §7 (snippet-surface asymmetry): Brave’s pre-fetch support advantage is partly volume-driven (more provider-native extra snippets). The length-normalized rows help, but the paper still treats Brave’s richer surface as a product difference while aggregating extra snippets into the generic snippet channel. For the decision-surface claim, this is acceptable only if the main text more clearly separates “more pre-fetch text” from “higher per-token evidence density” when stating Brave’s gold-answer-rich-snippet regime; otherwise readers may over-attribute the 30 vs 16 support gap to ranking quality alone.
minor comments (5)
  1. Table 3 rank-1 row: the denominator is gold-supporting pre-fetch rows (101/34/30), not queries; a one-line reminder in the caption would prevent misreading the 13%/50%/13% figures as query-level rates.
  2. Figure 3 cell counts are small (e.g., SMART 3/3); Appendix F Wilson intervals are good, but the main-text discussion of SMART rarity should briefly flag small-cell uncertainty.
  3. §3 run window (2026-05-17) and live-API non-bit-identical reruns are stated; a short note near Table 2 that absolute counts are time-stamped snapshots would help practitioners interpreting the numbers.
  4. Related Work could more explicitly contrast decision-surface metrics with classical NDCG/BEIR and with over-searching cost metrics, so the complementary evaluation target is sharper for IR readers.
  5. Minor consistency: abstract says “semantic match” while body uses semantic_match; unify the label name.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical provider comparison with independently audited correctness and descriptive (not definitional) evidence metrics.

full rationale

This paper is an observational comparative study, not a first-principles derivation. The load-bearing claim—near-parity semantic correctness (25/25/26) alongside divergent pre-fetch support, rank-1 concentration, contradiction-to-gold ratio, and SMART/MISSED/BLIND/NO-OP mass—is obtained by freezing the agent/prompt/tools/fetch backend and varying only the search provider, then measuring with two separately constructed instruments: (1) an audited semantic_match answer label and (2) a per-URL Kimi oracle whose gold-support and contradiction fields are defined relative to the gold answer and visible document text, not relative to the final accuracy claim. The surface contradiction-to-gold ratio is explicitly independent of the model’s final answer. Decision cells are joins of pre-fetch support labels with observed fetch actions and are labeled descriptive, not causal. No parameter is fitted to a subset and re-presented as a prediction; no uniqueness theorem or ansatz is imported via self-citation; authors do not rename a known IR result as a forced derivation. Residual single-agent/judge transfer risk is a stated limitation, not circular construction. Score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 4 invented entities

The central claim rests on experimental design choices and domain assumptions about progressive-disclosure agents, not on free physical constants or new particles. Load-bearing invented constructs are the decision-surface framing and the derived support/partition metrics; free parameters are protocol knobs (sample size, iteration budget, provider configs). Reliability of the Kimi oracle and transfer beyond one agent are the main domain assumptions.

free parameters (3)
  • SEALQA-HARD sample size and seed = 100 queries, seed 20260509
    Deterministic stratified sample of 100/254 questions with seed 20260509; cell counts and bootstrap widths depend on this choice.
  • Agent loop budget and tool limits = 10 iterations, ≤10 results/search, temp 0
    Max 10 iterations, up to 10 results per search, temperature 0; fetch appetite and decision-cell mass depend on these fixed knobs.
  • Provider request configurations = fixed YAML configs in Appendix C
    Brave US/en moderate safe search; Tavily basic/general raw content off; Firecrawl web-only markdown scrape off—product-surface differences are partly configuration choices.
assumptions (4)
  • domain assumption Progressive disclosure (snippets first, optional page fetch) is the relevant agent retrieval architecture for evaluating commercial search APIs.
    Stated in Introduction and Experimental Protocol; without it, full-page-first or provider-side extract pipelines would change the decision surface.
  • domain assumption Kimi-K2.6 per-URL labels with the fixed JSON schema are a usable audit instrument for gold support, contradiction, and garbage.
    Supported by 94% agreement on clear human judgments (Table 1) but remains a single-judge assumption (Limitations).
  • domain assumption semantic_match (audited) is the appropriate correctness label, preserving exact matches while accepting harmless formatting/naming variants.
    Headline CORRECT metric; legacy token F1 is reported but not primary.
  • domain assumption Standard IR and agent evaluation practice (NDCG-style ranking, answer accuracy) is incomplete for progressive-disclosure agents.
    Related Work and Discussion; motivates decision-surface metrics.
invented entities (4)
  • Decision surface (commercial search API as pre-fetch action state) independent evidence
    purpose: Reframe ranked snippets/URLs/metadata as the evidence state that drives answer / re-search / fetch decisions.
    Core formalization in Contributions and Introduction; not a physical entity but a new evaluative object.
  • SMART / MISSED / BLIND / NO-OP decision partition independent evidence
    purpose: Classify each provider-query pair by pre-fetch gold support crossed with whether the agent fetched a supporting URL.
    Defined in §4 from oracle labels joined to traces; descriptive, not causal.
  • Surface contradiction-to-gold ratio rc:g independent evidence
    purpose: Quantify pre-fetch contamination as contradicting snippet-only rows over gold-supporting snippet-only rows.
    Introduced in §4; ranges 0.92–2.59 across providers in Table 3.
  • Pre-fetch surface support vs post-fetch discovered support independent evidence
    purpose: Separate gold answerability visible before fetch from support found only after page open.
    Defined in §4 Support split; drives the main evidence-economy contrast.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Equal Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents." pith.science (2026). https://pith.science/paper/UMVVAQ5M

@misc{pith2026260710198,
  author       = {Pith},
  title        = {Pith review of: Equal Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UMVVAQ5M}},
  note         = {Machine review of arXiv:2607.10198}
}
read the original abstract

Search APIs are the fundamental retrieval layer for many agents and are often their most frequently used tool. Traditional search APIs provide URLs, titles, and snippets that preview website contents. Because full-page retrieval is token-intensive, agent retrieval architectures increasingly use progressive disclosure: the agent first sees snippets and then chooses whether to fetch full pages. In such systems, search API performance is often evaluated primarily by answer accuracy. We argue that a commercial search API is better understood as a decision surface: the ranked snippets, URLs, and metadata that determine whether an agent answers immediately, searches again, or spends tokens opening pages. We test this claim with one frozen GPT-5.4 agent, two tools (search_web and fetch_page), and 100 questions from SEALQA-HARD, varying only the search provider (Brave, Tavily, Firecrawl). A Kimi-K2.6 oracle labels every content element visible to the agent (URL, title, snippet, and fetched page, when fetched), producing 6,869 valid per-URL judgments. We use an audited correct-answer label, semantic match, which preserves exact matches while accepting harmless formatting and naming variants. Under this measure, the providers remain close (25, 25, 26 / 100), but their evidence economies differ sharply: Brave offers gold-answer-rich snippets, Tavily concentrates gold-supporting URLs at rank 1, and Firecrawl is associated with broader exploration under this fixed agent policy. We also introduce a surface contradiction-to-gold URL ratio, which varies from 0.92 to 2.59. Provider choice is therefore a retrieval-budget and policy decision, not merely a recall decision.

Figures

Figures reproduced from arXiv: 2607.10198 by the authors.

Figure 1
Figure 1. Execution and judging pipeline. The agent, prompt, tools, and fetch backend are fixed; only the search [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Provider profiles under semantic correctness. Similar final accuracy hides different pre-fetch support, rank [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Decision partition by provider. Each cell reports queries / semantic-correct answers after joining pre-fetch [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Correct-answer overlap across providers. Provider choice changes which questions are solved even when [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 4 linked inside Pith

  1. [1]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and Kuttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rocktaschel, Tim and Riedel, Sebastian and Kiela, Douwe , booktitle =. Retrieval-Augmented Generation for Knowledge-Intensive

  2. [2]

    Guu, Kelvin and Lee, Kenton and Tung, Zora and Pasupat, Panupong and Chang, Ming-Wei , booktitle =

  3. [3]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , year =

    Dense Passage Retrieval for Open-Domain Question Answering , author =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , year =

    2020

  4. [4]

    Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics , pages =

    Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering , author =. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics , pages =

  5. [5]

    Izacard, Gautier and Lewis, Patrick and Lomeli, Maria and Hosseini, Lucas and Petroni, Fabio and Schick, Timo and Dwivedi-Yu, Jane and Joulin, Armand and Riedel, Sebastian and Grave, Edouard , journal =

  6. [6]

    ACM Transactions on Information Systems , volume =

    Cumulated Gain-Based Evaluation of IR Techniques , author =. ACM Transactions on Information Systems , volume =

  7. [7]

    Introduction to Information Retrieval , author =

  8. [8]

    Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , year =

    Thakur, Nandan and Reimers, Nils and Ruckl. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , year =

Show all 31 references
  1. [9]

    Nakano, Reiichiro and Hilton, Jacob and Balaji, Suchir and Wu, Jeff and Ouyang, Long and Kim, Christina and Hesse, Christopher and Jain, Shantanu and Kosaraju, Vineet and Saunders, William and Jiang, Xu and Cobbe, Karl and Eloundou, Tyna and Krueger, Gretchen and Button, Kevin...

  2. [10]

    arXiv preprint arXiv:2210.03350 , year =

    Measuring and Narrowing the Compositionality Gap in Language Models , author =. arXiv preprint arXiv:2210.03350 , year =

    arXiv

  3. [11]

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =

  4. [12]

    arXiv preprint arXiv:2212.10509 , year =

    Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions , author =. arXiv preprint arXiv:2212.10509 , year =

    arXiv

  5. [13]

    Advances in Neural Information Processing Systems , year =

    Toolformer: Language Models Can Teach Themselves to Use Tools , author =. Advances in Neural Information Processing Systems , year =

  6. [14]

    Asai, Akari and Wu, Zeqiu and Wang, Yizhong and Sil, Avirup and Hajishirzi, Hannaneh , booktitle =. Self-

  7. [15]

    Vu, Tu and Iyyer, Mohit and Wang, Xuezhi and Constant, Noah and Wei, Jerry and Wei, Jason and Tar, Chris and Sung, Yun-Hsuan and Zhou, Denny and Le, Quoc and Luong, Thang , journal =

  8. [16]

    arXiv preprint arXiv:2601.05503 , year =

    Over-Searching in Search-Augmented Large Language Models , author =. arXiv preprint arXiv:2601.05503 , year =

  9. [17]

    Mialon, Gregoire and Fourrier, Clementine and Swift, Craig and Wolf, Thomas and LeCun, Yann and Scialom, Thomas , booktitle =

  10. [18]

    Zhou, Shuyan and Xu, Frank F. and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , booktitle =

  11. [19]

    Wei, Jason and Sun, Zhiqing and Papay, Spencer and McKinney, Scott and Han, Jeffrey and Fulford, Isa and Chung, Hyung Won and Passos, Alex Tachard and Fedus, William and Glaese, Amelia , journal =

  12. [20]

    Pham, Thinh and Nguyen, Nguyen and Zunjare, Pratibha and Chen, Weiyuan and Tseng, Yu-Min and Vu, Tu , journal =

  13. [21]

    Es, Shahul and James, Jithin and Espinosa-Anke, Luis and Schockaert, Steven , booktitle =

  14. [22]

    Saad-Falcon, Jon and Khattab, Omar and Potts, Christopher and Zaharia, Matei , journal =

  15. [23]

    arXiv preprint arXiv:2304.09848 , year =

    Evaluating Verifiability in Generative Search Engines , author =. arXiv preprint arXiv:2304.09848 , year =

    arXiv

  16. [24]

    arXiv preprint arXiv:2305.06311 , year =

    Automatic Evaluation of Attribution by Large Language Models , author =. arXiv preprint arXiv:2305.06311 , year =

    arXiv

  17. [25]

    Transactions of the Association for Computational Linguistics , volume =

    Lost in the Middle: How Language Models Use Long Contexts , author =. Transactions of the Association for Computational Linguistics , volume =

  18. [26]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , journal =. Judging

  19. [27]

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , journal =

  20. [28]

    2026 , howpublished =

    Brave Search API Reference , author =. 2026 , howpublished =

    2026

  21. [29]

    2026 , howpublished =

    Tavily Search API Documentation , author =. 2026 , howpublished =

    2026

  22. [30]

    2026 , howpublished =

    Firecrawl Search API Documentation , author =. 2026 , howpublished =

    2026

  23. [31]

    2026 , howpublished =

    Jina Reader: URL to LLM-Friendly Input , author =. 2026 , howpublished =

    2026

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.