{"id":"4bdca9f2-7e36-4ab8-b704-443c3d135196","arxiv_id":"2508.06600","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"BrowseComp-Plus is a fixed, human-verified document corpus with supporting and negative documents that allows controlled, reproducible evaluation of deep-research agents and their retrievers.","lead":"This paper introduces BrowseComp-Plus, a fixed collection of 100,000 documents and 830 verified questions for testing deep-research AI agents that combine language models with search. Instead of using live, changing web search, it lets researchers compare agents and retrieval systems on the same stable corpus.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Relevance labels are non-exhaustive: unjudged corpus docs are scored as non-relevant (§4.8.4), so retrieval/citation metrics are lower bounds and agent/retriever rankings can be distorted by unlabeled relevant evidence.","rationale":"The reader's weakest assumption matches my main concern. The benchmark's usefulness as a component-level testbed depends on qrels being close to exhaustive, and the construction does not demonstrate this. §3.2.1 uses o3 to mine evidence, and §3.2.2 limits additional searching to about 20 minutes, so completeness is bounded by one model's retrieval behavior. §4.8.4's Fineweb experiment is valuable and shows the corpus is robust to a 10x negative expansion for the tested ranking conclusions, but it does not test the actual threat: positive evidence that exists in the 100k corpus and was never labeled. If such evidence is surfaced by a retriever, it is scored as a false positive, which depresses that retriever's recall/nDCG and could change relative rankings. The oracle result (93.49%) verifies answerability of the labeled evidence, not exhaustiveness. The paper's other strengths—400 hours of annotation, span-level verification, the Fineweb ablation, and the reproducible Pyserini/Tevatron setup—support the benchmark's value, but they do not remove the incompleteness risk. I therefore keep the reader's CONDITIONAL verdict: the concern is real, the paper partially mitigates it, but the fairness/transparency headline should be qualified until a pooling study is done. The concrete test above would settle whether the concern actually lands.","tokens_in":24779,"tokens_out":5365,"duration_ms":66840,"concrete_test":"Select a stratified random sample of 50–100 queries. For each, take the union of top-100 documents from BM25, Qwen3-Embedding-8B, and ReasonIR-8B (plus, optionally, documents retrieved by o3/GPT-5 agents) that are not currently in the evidence or gold qrels. Have the original annotators apply the §3.2.2 span-labeling protocol to judge whether any of these unjudged documents contain clue-supporting evidence or the gold answer. Merge newly found positives into the qrels and recompute Table 2 Recall@5/100/1000, nDCG@10, and Table 3 citation precision/recall, plus the Table 1 accuracy ranking under the updated evidence set. If recall rises by more than a few points or any ranking inversion occurs, the non-exhaustive qrels materially distort component evaluation; if no meaningful change appears, the incompleteness concern is bounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BrowseComp-Plus supports independent, transparent component evaluation via human-verified evidence and gold documents. That requires the qrels to be sufficiently complete that unjudged documents can safely be treated as non-relevant. The construction does not establish this. §3.2.1 mines evidence with o3 (a model under evaluation), and §3.2.2 annotators verify only o3-surfaced documents plus whatever they find in 'at least 20 minutes' of additional searching. §4.8.4 then explicitly scores all unjudged Fineweb documents as non-relevant: 'these additional documents are unjudged, they are treated as non-relevant under standard TREC-style evaluation.' The same policy is implicit for unjudged documents in the 100k corpus. A relevant document that o3 did not surface and the annotator did not find is therefore mislabeled as negative. Consequences: Recall@k, nDCG@10, and citation recall are lower bounds; a retriever or agent that finds genuinely relevant but unlabeled evidence is penalized, so Table 2 and Table 3 rankings can shift once qrels are completed. The Fineweb ablation does not close this gap: it tests one negative expansion and re-measures with the same incomplete qrels; the paper itself concedes top-ranked unjudged documents 'may be false negatives that still provide useful evidence.' This is the standard Cranfield incompleteness problem, but it is a premise here, not a verified property, and it bears directly on the fairness/transparency claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BrowseComp-Plus, a fixed-corpus benchmark derived from BrowseComp for evaluating deep-research agents. It filters the original 1,266 queries to 830, attaches human-verified evidence and gold documents to each query, adds mined hard negatives, and assembles a 100,195-document corpus. The authors then evaluate a range of closed- and open-source LLM agents paired with BM25, Qwen3-Embedding, and ReasonIR retrievers, reporting end-to-end accuracy, recall, search calls, citation metrics, and standalone retriever effectiveness. The central claim is that this carefully curated fixed corpus enables fair, reproducible, and component-level evaluation of retrieval and LLM reasoning in deep-research systems.","tokens_in":25042,"tokens_out":3721,"duration_ms":42507,"significance":"If the construction is sound, BrowseComp-Plus fills a real gap: current BrowseComp-style evaluation depends on opaque, dynamic web search APIs, making it hard to disentangle retriever quality from agent reasoning. The paper's strengths are substantial: a fixed public corpus, 400+ hours of human annotation, over 80% inter-annotator agreement, a Fineweb-based robustness check, and an unusually broad set of agent/retriever pairings. The oracle-retrieval and reasoning-effort ablations are informative, and the observation that retriever quality affects accuracy and search-call efficiency is a useful empirical contribution. The main risk is that the evidence/qrel set is non-exhaustive, so several headline numerical claims are lower bounds whose ranking-level validity has not been demonstrated.","major_comments":[{"comment":"The relevance labels are non-exhaustive. Evidence documents are collected from o3's web search plus 'at least 20 minutes' of annotator searching, and §4.8.4 explicitly scores unjudged Fineweb documents as non-relevant. The same policy is implicit for unjudged documents in the 100k corpus. Consequently, Recall@k, nDCG@10, and citation recall in Tables 2 and 3 are lower bounds, and a retriever or agent that finds genuinely relevant but unlabeled evidence is penalized. Because this bears directly on the 'fair and transparent' claim, the paper should provide a sensitivity analysis—e.g., pooling top-ranked unjudged documents from each retriever, having annotators judge them, and recomputing metrics—or at least quantify the possible impact of incomplete qrels. This is the standard Cranfield incompleteness issue, but it is currently a premise rather than a verified property.","section":"§3.2.2 / §4.8.4"},{"comment":"The evidence set was generated using OpenAI o3, and o3 is later evaluated as a deep-research agent on that same corpus. Human verification ensures the documents are relevant, but the selection of which documents enter the corpus is o3-dependent. This can bias the benchmark in o3's favor and makes the retriever qrels a function of one system's search behavior. A concrete check would be to construct a held-out subset whose evidence is collected by a different pipeline (or from externally available evidence), and verify that retriever rankings and agent insights are stable. Without such a check, the 'fair comparison' claim is weakened.","section":"§3.2.1 / Table 1"},{"comment":"Of 1,266 BrowseComp queries, only 830 survive the pipeline: 124 lack evidence, 137 have unscrapable URLs, and 175 fail verification or are excluded for ambiguity. The paper does not analyze whether this subset is representative of BrowseComp or whether conclusions about retrieval difficulty and agent ranking transfer to excluded queries. Since BrowseComp-Plus is proposed as a benchmark for deep-research agents, not merely a diagnostic subset, the authors should report characteristics of excluded queries (category distribution, difficulty proxies, answer types) and show that the main results are stable under alternative inclusion criteria.","section":"§3.2.1–§3.2.2 / Appendix C"}],"minor_comments":[{"comment":"Typo: 'retrieval-agumented generation' should be 'retrieval-augmented generation'.","section":"Abstract / §2.1"},{"comment":"Table 6 reports BM25 Original Recall@1000 as 13.6%, while Table 2 reports 13.7% for the same setting. The source of the small discrepancy should be clarified or corrected.","section":"Table 6 vs Table 2"},{"comment":"The phrase 'Qwen3-8B surpassing ReasonIR-8B' should read 'Qwen3-Embedding-8B' for consistency with the rest of the paper.","section":"§4.7"},{"comment":"Model-name capitalization is inconsistent (e.g., 'GPT-5' vs 'gpt-5', 'SearchR1' vs 'Search-R1'). Please unify.","section":"Throughout"},{"comment":"The axes and markers in Figure 1 are not fully described in the caption; please define what each point and color represents and state which retriever/LLM combinations are shown.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious and useful benchmark contribution with unusually careful construction. The main concern is shared with the reader's report: the qrels are incomplete, and the paper treats unjudged documents as non-relevant without demonstrating that this does not affect the reported rankings. I do not think this requires rejection, but it needs to be addressed explicitly, ideally with a pooling-style sensitivity analysis. The o3-seeded evidence collection is a secondary concern that should be discussed or empirically bounded. If the authors can provide such evidence, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid benchmark-resource paper. It ships a new, fixed corpus for BrowseComp with human-verified evidence and gold documents, hard negatives, and qrels — exactly the kind of artifact that lets people measure retrieval and agent components separately. The construction is careful: o3 mining plus human verification, 14 annotators, 400+ hours, >80% agreement, and a Fineweb robustness check. I'd put it in the same category as HotpotQA for multi-hop QA: not a field re-organization, but a stable testbed that the community will use.\n\nThe soft spots are real but not fatal. The stress-test note lands: relevance judgments are non-exhaustive. Annotators only verified documents o3 surfaced plus whatever they found in ~20 minutes of searching, and Section 4.8.4 explicitly scores unjudged Fineweb documents as non-relevant. That means Recall@k, nDCG, and citation recall are lower bounds, and the rankings in Tables 2–3 could shift if the qrels were filled in. The paper concedes this for Fineweb ('may be false negatives') but doesn't address it for the 100k corpus itself. Also, 436 of 1,266 BrowseComp queries were filtered out, and there's no evidence that rankings on the remaining 830 transfer. The o3-as-evidence-miner circularity is mild — human verification grounds it — but it's worth noting. And I don't see a repo or commit hash in the text, so reproducibility is not yet independently checkable.\n\nI don't think any of this invalidates the central contribution. The benchmark is what it claims to be: a controlled, reusable corpus for component-level evaluation. The 'fair and transparent' framing is slightly oversold until the qrels are completed or their completeness is argued more carefully.\n\nWho's this for? Anyone building or evaluating deep-research agents, and IR folks who care about retriever evaluation on reasoning-intensive queries. It deserves a serious referee. My recommendation: engage with it — send it to review, but ask the authors to address the qrel completeness question and the filtered-query selection bias.","headline":"A genuinely useful fixed-corpus benchmark for deep-research evaluation, with caveats about incomplete qrels and filtered queries.","tokens_in":25696,"tokens_out":2082,"would_cite":true,"duration_ms":22126,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed, human-verified corpus of 830 BrowseComp queries lets researchers measure retrieval and reasoning separately in deep-research agents, and the measurements show the retriever often decides the score.","keywords":["BrowseComp-Plus","deep-research agents","benchmark","retrieval evaluation","Cranfield paradigm","hard negatives","reproducibility","tool-use reasoning"],"falsifier":"Judge a random sample of corpus documents that are currently unlabeled for their query — including the Fineweb-augmented runs' top-ranked \"false negatives\" — with fresh annotators: if a meaningful share contain verifiable answer evidence, the completeness premise fails and the published Recall and nDCG figures are underestimates. A second check: rebuild the evidence labels with an independent discovery agent (not the one used to mine the original set) and see whether the relative ranking of retrievers at Recall@1000 survives the new label set.","tokens_in":24591,"feed_emoji":"🔍","tokens_out":12007,"duration_ms":114631,"temperature":0.7,"pith_summary":"This paper argues that current evaluations of deep-research agents — LLMs that search iteratively and reason over results — conflate two distinct abilities: finding evidence and reasoning over it. BrowseComp-Plus is its answer: a fixed, human-verified corpus for 830 BrowseComp queries, with evidence documents, answer-containing gold documents, and mined hard negatives per query, so each component can be scored independently and runs are reproducible without a live web API. The paper demonstrates the payoff by pairing seven agent models with several retrievers: better retrieval lifts accuracy sharply (gpt-5 from 55.9% to 70.1%; gpt-4.1 from 14.6% to 35.4%) while cutting search calls, and oracle retrieval reaches 93.5%, locating the main bottleneck in retrieval rather than reasoning. If the benchmark is right, component-level diagnosis of deep-research systems becomes tractable and cheap.","feed_headline":"Swap the retriever: GPT-5 accuracy jumps to 70%","feed_subtitle":"A fixed 100k-document corpus lets researchers score retrieval and reasoning separately in deep-research agents.","key_machinery":"The load-bearing object is the per-query, human-verified evidence set: for each of 830 queries, on average 6.1 evidence documents, 2.9 gold documents (which contain the answer in implicit or variant form, not just as an exact string), and about 76 mined hard negatives, all inside one fixed 100,195-document corpus. This object carries the argument by turning the retriever into a measurable component: standalone retrieval is scored with Recall@k and nDCG under the Cranfield paradigm, and any accuracy difference between agent runs becomes attributable to the retriever, the reasoning model, or their interaction, rather than to a drifting web corpus.","core_discovery":"BrowseComp-Plus converts 830 of BrowseComp's 1,266 queries into a closed evaluation: a deduplicated, fixed corpus of 100,195 documents in which each query carries human-verified evidence documents, gold documents containing the final answer semantically (not just as a substring), and mined hard negatives. The paper's central claim is that this makes retrieval a controlled variable — retrieval alone can be scored under the Cranfield paradigm with Recall@k and nDCG, and end-to-end agent runs become reproducible because no live web API is involved. The demonstration is that retriever choice, more than model identity, separates agent scores: gpt-4.1 rises from 14.58% (BM25) to 35.42% (Qwen3-Embe","pith_inferences":["Editorial extension: the oracle-gap decomposition (93.49% oracle vs. 14.58% with BM25 for gpt-4.1) quantifies headroom per system; the same decomposition could be applied to other multi-hop QA benchmarks if comparable human-verified evidence labels were produced.","Editorial extension: the paper leaves open whether retriever quality shapes reinforcement-learning dynamics; a testable follow-up would train Search-R1-style agents against BM25 versus a dense retriever and compare training curves and cross-retriever generalization.","Editorial extension: because unlabeled corpus documents count as non-relevant, the published Recall and nDCG figures are lower bounds; pooling and judging the top-ranked documents of many retrievers would produce a second-generation label set that measures the current evidence set's completeness.","Editorial extension: the citation metrics (coverage, precision, and recall against labeled evidence) form a cheap, corpus-grounded faithfulness proxy that other fixed-corpus agent benchmarks could adopt as a standard reporting block."],"forward_implications":["Retriever choice becomes a first-order lever: replacing BM25 with Qwen3-Embedding-8B more than doubles accuracy for several agents (gpt-4.1: 14.58% to 35.42%; Sonnet 4: 14.34% to 36.75%) and raises gpt-5 from 55.90% to 70.12%.","Retrieval, not reasoning, is the measured bottleneck: the strongest retriever recalls only 14.5% of evidence documents at k=5, whereas oracle retrieval gets gpt-4.1 to 93.49%, so closing the retrieval gap is the largest available gain.","The open-source deficit is in interleaved tool use, not question answering: Qwen3-32B scores 83.25% when handed all positive documents versus 10.36% end-to-end.","Reasoning effort trades compute against coverage: gpt-oss-120B with Qwen3-Embedding-8B rises from 24.94% (low effort) to 42.89% (high), while search calls and cost rise with effort.","Tenfold corpus growth (adding Fineweb documents) leaves retriever and agent rankings intact, supporting the compact 100k corpus as a fast, valid experimental testbed."],"supporting_citations":[{"why":"Supplies the 1,266 BrowseComp question-answer pairs, 830 of which BrowseComp-Plus carries into its fixed corpus.","marker":"[1]"},{"why":"Provides the Cranfield evaluation tradition the benchmark instantiates for standalone retrieval scoring with Recall@k and nDCG.","marker":"[6]"},{"why":"The Qwen3-Embedding dense retriever whose effectiveness drives the main accuracy comparisons across agents.","marker":"[26]"},{"why":"ReasonIR, the reasoning-intensive dense retriever used as the state-of-the-art comparison at the 8B scale.","marker":"[22]"},{"why":"BM25, the lexical baseline whose contrast with dense retrievers demonstrates retrieval quality's impact on accuracy.","marker":"[35]"},{"why":"Search-R1, the reinforcement-learned open-source search agent used as the principal open-source baseline.","marker":"[12]"},{"why":"The text-extraction tool used to parse scraped web pages into the corpus documents.","marker":"[31]"},{"why":"Work on corpus subsampling that justifies a small, negative-rich corpus as a valid setting for retrieval evaluation.","marker":"[32]"}],"fun_headline_variants":["Swap retriever: GPT-5 deep-research accuracy hits 70%","Fixed corpus benchmark: GPT-5 with better retriever reaches 70%","Retriever choice matters: GPT-5 improves from 56% to 70%","Closed corpus benchmark reveals retriever impact on deep-research","BrowseComp-Plus: fairer deep-research eval with controlled corpus"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the human-verified evidence set is complete enough for every unlabeled corpus document to be treated as non-relevant: annotators only verified pages surfaced by one web-enabled model plus about 20 minutes of extra searching, and Section 4.8.4 explicitly scores unjudged added documents as non-relevant — so if relevant evidence sits unlabeled in the corpus, the reported Recall@k, nDCG, and citation metrics are lower bounds and agent rankings cou","fun_headline_variants_meta":{"raw":{"variants":["Swap retriever: GPT-5 deep-research accuracy hits 70%","Fixed corpus benchmark: GPT-5 with better retriever reaches 70%","Retriever choice matters: GPT-5 improves from 56% to 70%","Closed corpus benchmark reveals retriever impact on deep-research","BrowseComp-Plus: fairer deep-research eval with controlled corpus"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000805,"raw_usage":{"total_tokens":3423,"prompt_tokens":842,"completion_tokens":2581,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":2479}},"tokens_in":586,"tokens_out":2581,"duration_ms":19983,"temperature":1.0,"reasoning_tokens":2479,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:40:25.115713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Judge a random sample of corpus documents that are currently unlabeled for their query — including the Fineweb-augmented runs' top-ranked \"false negatives\" — with fresh annotators: if a meaningful share contain verifiable answer evidence, the completeness premise fails and the published Recall and nDCG figures are underestimates. A second check: rebuild the evidence labels with an independent discovery agent (not the one used to mine the original set) and see whether the relative ranking of retrievers at Recall@1000 survives the new label set.","supporting_citations":[{"cited_title":"Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford","cited_arxiv_id":null,"evidence_quote":"BM25, the lexical baseline whose contrast with dense retrievers demonstrates retrieval quality's impact on accuracy."},{"cited_title":"Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction","cited_arxiv_id":null,"evidence_quote":"The text-extraction tool used to parse scraped web pages into the corpus documents."},{"cited_title":"Corpus subsampling: Estimating the effectiveness of neural retrieval models on large corpora","cited_arxiv_id":null,"evidence_quote":"Work on corpus subsampling that justifies a small, negative-rich corpus as a valid setting for retrieval evaluation."}],"review_version":1}