{"id":"04918c66-ceba-4e61-add4-0b52443d07d6","arxiv_id":"2505.15872","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"InfoDeepSeek is a 245-question benchmark that measures how well AI agents seek information on the live web, with new metrics for answer accuracy, evidence quality, and compactness.","lead":"Researchers built InfoDeepSeek, a set of 245 hard questions for testing how AI agents find and use up-to-date information from the live web. It shows that even the best AI models answer only about 22% of these questions correctly, pointing to where agentic search needs improvement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IA@k/ACC conflate evidence quality with parametric memory; retrieval-interference results show substantial memorized answers, so the headline 'information quality remains low' is not cleanly isolated.","rationale":"I read the paper as a benchmark-and-evaluation contribution whose central claims are: (1) InfoDeepSeek is the first benchmark/evaluation framework for agentic information seeking in live web environments, and (2) current LLM agents achieve low accuracy and low information quality. Claim (1) is supported by the artifact and the evaluation protocol; I do not have a decisive objection to it. Claim (2), however, depends on metrics that cannot distinguish whether low scores reflect poor evidence seeking or the answer generator's memory and inference limitations. The paper already contains the key evidence for this concern: the retrieval-interference analysis in Section 6.3 and Appendix D.4 shows that a substantial fraction of questions are answered correctly without retrieval by the very models under test, and that retrieval can flip correct answers to incorrect ones. Because no no-retrieval ACC is reported in Table 3, the reader cannot tell how much of ACC and IA@k is due to information seeking versus parametric knowledge. The reader identified the LLM-judge pipeline as the weakest assumption; that is a real and important concern, but the paper's own 99.29% judge agreement with human evaluation, even if internally validated, is at least a direct check. The parametric-contamination issue is more load-bearing because it is not addressed anywhere in the paper and is evidenced by the paper's own interference results. I would still accept the paper conditionally, with the added condition that a no-retrieval control and subset analyses be reported, because the benchmark itself remains useful and the empirical findings can be corrected without changing the core artifact. Hence the verdict is unchanged from the reader's CONDITIONAL recommendation.","tokens_in":29213,"tokens_out":5973,"duration_ms":60110,"concrete_test":"Run a no-retrieval control for each evaluated LLM: call φ(q, ∅) with empty evidence set and empty observations, using the same generation prompt, and report no-retrieval ACC. Then recompute IA@k, EEU, and IC restricted to the subset of questions that the model cannot answer without retrieval. If IA@k on the non-memorized subset is materially below the reported values, the claim that evidence quality is low is confounded with parametric knowledge; if IA@k is unchanged, the concern is refuted. Independently, report judge agreement stratified by this memorized/non-memorized split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that agents retrieve low-quality evidence (IA@5 21.63% for Gemini-2.5-Pro) rests on Section 5.1 metrics: IA@k = (1/|Q|) Σ I(φ(q,C_{1:k}) = y_q), where φ is the same LLM used for retrieval, augmentation, and generation. Because φ has parametric knowledge, IA@k actually measures 'top-k evidence plus memorized knowledge is sufficient', not evidence quality in isolation. The paper's own retrieval-interference analysis (Section 6.3, Figure 5a, Appendix D.4) reports that 40-80% of the questions a model answers correctly without retrieval are answered incorrectly after retrieval, using the number of no-retrieval-correct questions as the denominator. This is internal evidence that a material subset of benchmark items is answerable from memory. Without a published no-retrieval ACC for each model in Table 3, the reported ACC and IA@k numbers cannot separate retrieval quality from the generator's prior knowledge or generation ability. A model with strong priors can score IA@k high even when evidence is useless, and a weak generator can score low even when evidence is sufficient. Since EEU and IC are ratios built on ACC and IA@k, all three information-quality metrics inherit the same contamination. This does not invalidate the benchmark as a challenging task suite, but it weakens the specific empirical conclusion that information seeking, rather than parametric knowledge or generation, is the bottleneck.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces InfoDeepSeek, a 245-question benchmark for agentic information seeking in live web environments. Questions are constructed to satisfy determinacy, difficulty, and diversity, with a difficulty filter that removes questions solvable by GPT-4o or DeepSeek-R1 in a single-turn search. The authors propose an Agentic RAG framework and four metrics: answer accuracy (ACC), information accuracy (IA@k), effective evidence utilization (EEU), and information compactness (IC), together with an LLM-judge protocol that is validated against human annotations (99.29% agreement). Experiments across ten LLM configurations, four search engines, and several ablations report low absolute accuracy (best ACC 22.45% for Gemini-2.5-Pro), large gains when switching from DuckDuckGo to Google/Yahoo, attribute-level differences, test-time scaling benefits, retrieval interference, and language effects.","tokens_in":29578,"tokens_out":7881,"duration_ms":70970,"significance":"If the results hold, the benchmark is a useful community resource for evaluating agentic RAG in dynamic settings. The construction pipeline is careful: fact-grounded reverse drafting, anchor-knowledge expansion, two-verifier plus adjudicator validation, and an explicit difficulty filter. The auto-eval protocol is validated against human judgments, with a clear improvement from 95.57% to 99.29% after the separate false-premise prompt strategy. The detailed appendices and project page make the protocol largely reproducible. The most robust empirical findings are likely that current agents score low on these deliberately hard questions, that search engine choice has a large effect (Table 4), and that retrieval interference is prevalent. The claims that need qualification are the interpretation of IA@k as pure evidence quality and the attribution of the language effect, as detailed in the major comments.","major_comments":[{"comment":"IA@k is defined as the accuracy of ϕ(q, C1:k), where ϕ is the same LLM used for retrieval, augmentation, and generation. Because ϕ has parametric knowledge, IA@k measures whether the top-k evidence plus the model's memory is sufficient, not the quality of the evidence in isolation. This is not a hypothetical concern: Section 6.3 reports interference rates of 40–80%, meaning a substantial fraction of benchmark items are answerable without retrieval. Thus the low IA@k values in Table 3 cannot be cleanly interpreted as 'information quality remains low', and EEU and IC inherit the same contamination since they are derived from IA@k. Please report the no-retrieval ACC for every model in Table 3, and report IA@k restricted to questions that are not answerable from memory (or otherwise condition on no-retrieval correctness), so that retrieval quality and parametric knowledge can be separated.","section":"Section 5.1 (IA@k) and Section 6.3 (Figure 5a)"},{"comment":"The paper motivates its dynamic evaluation by saying that a 'gold evidence set in advance' is infeasible, yet IC = (nq or n+b)/|Sq| uses Sq, a human-annotated standard set of source webpages, as a gold reference. This is a static gold set and is in tension with the paper's central claim of evaluating information seeking without pre-defined gold documents. Please either justify the use of Sq for compactness (for example, as a stable answer-support set rather than a retrieval ranking) or redefine IC without it; otherwise the claim of a 'dynamic evaluation w/o golden doc set' in Figure 1 is not fully accurate.","section":"Section 5.1 (IC definition)"},{"comment":"Attribute-level analyses rest on small subsets (False Premise n=27, Freshness n=48) but no confidence intervals or significance tests are reported. For example, in Table 7 the false-premise accuracy difference between Llama-4-Maverick (20.00%) and Qwen3-32B w/o think (7.69%) is based on 27 questions and is within the range of sampling noise. Add bootstrap confidence intervals (or exact binomial intervals) for all attribute-level and search-engine comparisons, and temper conclusions that rely on small denominators.","section":"Table 7 / Figure 3"},{"comment":"EEU is defined as max_k IA@k / ACC and is described as indicating suboptimal evidence selection when below 1. However, several models in Table 3 have EEU clearly above 1 (DeepSeek-R1 1.189, Gemini-2.5-Flash 1.171), and the paper does not explain what EEU > 1 means. Since ACC uses all retrieved observations O while IA@k uses the curated top-k set, an EEU above 1 likely reflects beneficial filtering of noisy observations; the current one-sided interpretation ('EEU significantly below 1 suggests... suboptimal') is incomplete and should be revised, with explicit discussion of the >1 cases.","section":"Section 5.1 (EEU)"},{"comment":"The predominant-language condition uses a different prompt that explicitly instructs the agent to switch search languages, whereas the Chinese and English conditions use a prompt that does not contain this instruction. Consequently, the observed superiority of 'Predominant Language' in Table 15 may be caused by the extra meta-instruction rather than by the language of retrieval. Please add an ablation in which the language-switching instruction is included in the English and Chinese conditions, or otherwise equate prompt content across conditions.","section":"Section 6.2, Figure 5(b), Appendix D.5"},{"comment":"The difficulty filter discards any question that GPT-4o and DeepSeek-R1 can answer with a single-turn search, so the low absolute ACC values in Table 3 are partly a consequence of the benchmark construction and should not be presented as a surprising empirical discovery about LLM information-seeking ability in general. I do not regard this as circular, because multi-turn agents are still free to succeed, but the paper should report single-turn-search baselines for the evaluated models and frame the ACC numbers as 'performance on questions that are deliberately hard for single-turn search'. This would make the comparison transparent and address the concern that the headline accuracy is an artifact of filtering.","section":"Section 4.2 (Difficulty filtering)"}],"minor_comments":[{"comment":"The legend states 'Symbol ✗ signifies the lack of this attribute, while symbol ✗ means it is not explicitly considered', but both symbols are rendered identically; one of them should be a different glyph (for example, '—').","section":"Table 1"},{"comment":"The text says 'A higher RA@k implies better evidence relevance'; this should read 'IA@k'.","section":"Section 5.1"},{"comment":"The word 'infromation' appears in the sentence 'This highlights the importance of search engine quality in supporting effective agentic infromation seeking'; it should be 'information'.","section":"Section 6.2"},{"comment":"The auto-eval agreement of 99.29% is reported overall; since the separate-prompt strategy was specifically designed for false-premise questions and only 27 such questions exist, please report agreement separately for false-premise and non-false-premise questions, with confidence intervals and the human inter-annotator agreement on that sample.","section":"Appendix C.2"},{"comment":"Interference-rate denominators are not reported. For example, Qwen3-32B w/o think is listed at 100.00%, which could correspond to a very small number of no-retrieval-correct questions; include the counts and exact binomial intervals for all interference rates.","section":"Section 6.3 / Tables 11–14"},{"comment":"The figure legend labels one series 'DeepSeek-V1', while the text and the rest of the paper refer to DeepSeek-V3; the label should be corrected.","section":"Figure 5(a)"},{"comment":"Because the benchmark runs against live web search engines, results will drift over time; please add query timestamps, search-engine version/date information, and an archival release of agent trajectories to support reproducibility.","section":"General reproducibility"}],"recommendation":"major_revision","confidential_remarks":"This is a benchmark-and-measurement paper rather than a new algorithm, which is within scope for an applied IR venue. The construction and validation are solid enough that I would not recommend rejection. The main issues are interpretational: IA@k conflates retrieval quality with parametric knowledge, IC reintroduces a gold source set, and several secondary findings lack uncertainty quantification. The reviewer concern about circularity of the difficulty filter is, in my view, mostly a framing and baseline-reporting issue, not a fatal flaw. One editorial point: the abstract's 'first evaluation framework' claim should be checked carefully against BrowseComp and other live-web agent benchmarks before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. This paper gives the community something concrete: a 245-question dataset of hard, multi-hop, long-tail, false-premise questions built with a careful manual pipeline, plus a metrics suite (IA@k, EEU, IC) aimed at dynamic web environments where a static gold-document set doesn't exist. The construction methodology is sound and unusually well documented, the auto-eval is validated against humans at 99.29%, and the controlled experiments across search engines, step budgets, and languages produce genuinely useful observations—especially the search-engine quality effects and the retrieval-interference phenomenon. I also read the deliberate difficulty filter as a feature, not a flaw; it guarantees the benchmark is hard, which is what a challenge set is for.\n\nThe main soft spot is the one the stress-test flags, and I think it lands. IA@k is computed by feeding the top-k evidence set to the same LLM and asking whether it produces the right answer. That conflates evidence quality with parametric knowledge and generation ability. A model with strong priors can score well even when the evidence is useless, and a weak generator can score poorly when the evidence is sufficient. The paper's own interference numbers confirm that a non-trivial subset of questions are answerable from memory, but no-retrieval ACC is reported per model only inside the interference denominator, not as a Table 3 column. So the headline \"information quality remains low\" is not cleanly established. This weakens the interpretation that retrieval is the bottleneck, but it doesn't invalidate the benchmark as a task suite.\n\nTwo smaller issues. First, several comparative claims rest on tiny subsets—27 false-premise items—with no confidence intervals or error bars. Second, the IC metric reintroduces a human-annotated source set S_q, which sits in tension with the \"no gold set\" framing; it's a fine compactness measure, but the rhetoric should be walked back. The \"first\" claim is plausible within RAG, though BrowseComp and related web-agent benchmarks deserve a more careful positioning.\n\nNone of this is fatal. The artifact is reusable, the paper is honest about its limitations, and the empirical findings are suggestive rather than definitive. I'd send it to serious peer review with a request for code/data release, confidence intervals, a per-model no-retrieval baseline, and a reframing of what IA@k measures. I'd cite it in my own work if I were working on agentic RAG evaluation.","headline":"InfoDeepSeek delivers a genuinely useful benchmark and metrics suite for agentic RAG, but the headline information-quality finding is only partly supported because the metrics don't isolate evidence quality from parametric memory.","tokens_in":30097,"tokens_out":1699,"would_cite":true,"duration_ms":17967,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces InfoDeepSeek, a 245-question live-web benchmark for agentic information seeking, and reports that the strongest tested agent, Gemini-2.5-Pro, reaches only 22.45% accuracy while search engine quality outweighs model…","keywords":["agentic RAG","information seeking","live-web benchmark","retrieval-augmented generation","LLM evaluation","dynamic evaluation","retrieval interference","multilingual question answering"],"falsifier":"Have two independent human raters score the agent outputs for all 245 questions using their own instructions rather than the paper's evaluation prompts, then compare each judgment to the LLM-judge majority verdict; if agreement on the false-premise subset falls well below the reported 99.29%, or if the judge systematically accepts answers that only say 'the premise is wrong' without naming the correct error, then ACC, IA@k, EEU, and IC would shift and the benchmark's headline numbers would need revision.","tokens_in":2031,"feed_emoji":"🔎","tokens_out":3399,"duration_ms":100377,"temperature":0.7,"pith_summary":"This paper argues that existing RAG benchmarks cannot measure what agentic RAG systems actually do—multi-turn planning, searching, and evidence gathering on the live, changing web—because they are built on static corpora with fixed gold documents and simple queries. To fill that gap, it presents InfoDeepSeek, a set of 245 manually constructed, triple-validated questions that are determinate, hard for LLMs even with one search, and diverse across six difficulty attributes, 14 domains, and 19 languages. It also proposes an evaluation protocol for dynamic settings that needs no pre-defined gold documents, with four metrics: answer accuracy, information accuracy at top-k, effective evidence utilization, and information compactness. The central empirical claim is that even the strongest LLM agents answer only about a fifth of these queries correctly, and that the quality of the underlying search engine matters more than the choice of reasoning model.","feed_headline":"AI search agents get only 22% of InfoDeepSeek's 245 hard queries right","feed_subtitle":"New live-web benchmark measures multi-step research; search-engine quality, not model smarts, drives success.","key_machinery":"The load-bearing machinery is the agentic information-seeking loop itself—plan, act through search, browse, time, or termination tools, observe, reflect, and continue up to T steps—followed by an augmentation stage that ranks and condenses retrieved content into an evidence set. Around this loop the paper builds the evaluation scaffold, and because a dynamic web has no gold document set, the metrics judge outcomes instead. The central metric decomposition is ACC (whether the final answer matches ground truth), IA@$k$ (whether the top-$k$ ranked evidence alone yields the correct answer), $\\mathrm{EEU} = \\max_{1\\le k\\le n} \\mathrm{IA}@k / \\mathrm{ACC}$ (whether the agent selected evidence worse than the best available subset), and IC (evidence-set length relative to a human-annotated source set, penalized when the answer fails). This decomposition is what lets the paper attribute failures to retrieval, evidence selection, or generation rather than treating agentic RAG as a single black box.","core_discovery":"InfoDeepSeek is presented as the first benchmark and evaluation framework for agentic information seeking in real-world, dynamic web environments. The paper's central claim is that evaluating agents on open-ended web search requires replacing fixed gold document sets with outcome-based metrics that ask whether the retrieved evidence, the selected evidence set, and the final answer are sufficient and compact. On this yardstick the reported results are stark: Gemini-2.5-Pro, the best model tested, scores 22.45% on ACC and 21.63% on IA@5, most models score below 15%, and most effective-evidence-utilization values sit at or below 1, meaning agents commonly fail to extract the useful evidence they already retrieved. The paper also reports that search engine choice changes accuracy more than model choice—DeepSeek-V3 with Google reaches 28.57% versus 8.98% with DuckDuckGo—and that retrieval interference, where web content makes a model abandon a correct answer it knew from parametric memory, occurs at rates from roughly 40% to 100% depending on the model.","pith_inferences":["A consequence the authors leave implicit: because the difficulty filter is tied to GPT-4o and DeepSeek-R1 at one point in time, some InfoDeepSeek questions will become single-turn answerable as those models and search indexes improve, so keeping the benchmark hard will require versioned re-filtering of the question set.","The EEU and IC metrics are generic enough to apply to any evidence-producing agent, including non-RAG tool use and human search sessions, as diagnostics of whether the selected evidence is better than the best available subset and whether it is compact.","A design the paper does not test is multi-engine fusion: since search engine quality dominates and different engines excel on different question attributes, an agent that queries several engines and reranks the union should outperform any single-engine configuration.","A second untested design follows from the interference numbers: a confidence-gated agent that answers from memory first and searches only when uncertain should reduce retrieval interference and outperform the current always-search pipeline on the questions the model initially knows."],"forward_implications":["If the benchmark's difficulty filter does its job, any system that scores well must engage in multi-turn retrieval and evidence selection; parametric memory or a single search will not suffice.","Search engine choice becomes a first-order experimental variable: switching from DuckDuckGo to Google lifts DeepSeek-V3's accuracy from 8.98% to 28.57%, a larger gain than swapping between many LLM choices.","Allocating more retrieval steps—from 1 up to 20—roughly doubles or triples accuracy for the tested models, establishing test-time compute for search as a real scaling axis.","Retrieval interference is widespread, so robust agents will need mechanisms that protect correct internal knowledge from being overridden by noisy or tangential web content.","Language-aware prompting to search in a query's predominant language improves accuracy over English-only or Chinese-only search, particularly for multilingual long-tail questions."],"supporting_citations":[{"why":"NQ is the static single-hop QA benchmark whose simplicity and fixed corpus the paper contrasts with its own hard, dynamic questions.","marker":"[20]"},{"why":"MultiHop-RAG supplies the multi-hop static benchmark line that InfoDeepSeek extends to live web environments without gold document sets.","marker":"[43]"},{"why":"FreshLLM grounds the freshness and real-world question line, which InfoDeepSeek pushes further by adding dynamic evaluation and time-sensitive attributes.","marker":"[48]"},{"why":"BrowseComp is the prior hard browsing benchmark with human difficulty filtering that motivates the paper's automatic difficulty-filtering approach.","marker":"[51]"},{"why":"CRAG provides both a static benchmark comparison and the LLM-evaluator majority-vote method the paper adapts for automatic answer judging.","marker":"[58]"},{"why":"BrowseComp-ZH grounds the multilingual browsing benchmark line that InfoDeepSeek extends with predominant-language annotations.","marker":"[64]"},{"why":"GPT-4o is one of the two LLMs used in the single-turn difficulty filter that discards questions answerable by parametric knowledge or one search.","marker":"[17]"},{"why":"DeepSeek-R1 is the other difficulty-filter model and provides the reasoning-model comparison in the agent evaluations.","marker":"[10]"},{"why":"DeepSeek-V3 is one of the two primary LLM judges whose majority verdict with Gemini-2.0-Flash underlies the ACC and IA@k numbers.","marker":"[3]"},{"why":"Gemini-2.0-Flash is the second primary judge in the agreement pipeline.","marker":"[6]"}],"fun_headline_variants":["Best AI search agent scores 22% on InfoDeepSeek live-web benchmark","Search engine quality beats model choice in agentic RAG benchmark","Web content makes AI agents drop correct answers up to 100%","Most AI search agents score below 15% on InfoDeepSeek live benchmark"],"cache_read_input_tokens":32128,"weakest_assumption_plain":"The load-bearing premise is that the automatic judge—two large language models scoring answers plus a third model as tiebreaker—correctly decides when an agent's answer matches the ground truth, especially on the 27 questions built on false premises, and the evidence for that correctness is the authors' own annotation set.","fun_headline_variants_meta":{"raw":{"variants":["Best AI search agent scores 22% on InfoDeepSeek live-web benchmark","Search engine quality beats model choice in agentic RAG benchmark","Web content makes AI agents drop correct answers up to 100%","Most AI search agents score below 15% on InfoDeepSeek live benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2745,"prompt_tokens":988,"completion_tokens":1757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":1677}},"tokens_in":604,"tokens_out":1757,"duration_ms":11238,"temperature":1.0,"reasoning_tokens":1677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:13:54.863521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent human raters score the agent outputs for all 245 questions using their own instructions rather than the paper's evaluation prompts, then compare each judgment to the LLM-judge majority verdict; if agreement on the false-premise subset falls well below the reported 99.29%, or if the judge systematically accepts answers that only say 'the premise is wrong' without naming the correct error, then ACC, IA@k, EEU, and IC would shift and the benchmark's headline numbers would need revision.","supporting_citations":[],"review_version":1}