{"id":"76376a60-7e2d-415a-afcd-22cc062e6d1c","arxiv_id":"2607.13920","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A controlled synthetic-retrieval stress test shows that AI search agents differ greatly in handling untrustworthy, irrelevant, or false documents, and that HotpotQA is partly answerable from parametric knowledge alone.","lead":"DeepStress replaces the search tool of AI 'search agents' with a controllable simulator that feeds them deliberately untrustworthy, irrelevant, or false documents, and measures how they react. The paper shows agents differ sharply—some keep answering confidently from bad evidence while others abstain or search until they run out of budget—and introduces a reliability-aware score for ranking them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic label fidelity is the load-bearing assumption: factuality validation is weak and relevance is unvalidated, so axis-specific robustness differences may be generation artifacts.","rationale":"I read the paper in good faith: it proposes a controlled simulation environment, runs a large 33,600-trace experiment, and carefully separates outcome categories. The central argument is plausible, and the direct-inference HotpotQA finding is an independent, falsifiable result that strengthens the paper. However, the entire empirical contribution rests on the validity of the synthetic labels. The reader's weakest_assumption identifies exactly this point, and the manuscript's own Appendix A.1 and Table 3 provide evidence that the assumption is not fully secured. The factuality axis has only 85.5% agreement with an asymmetric confusion pattern, and relevance is explicitly not human-validated. Because the conclusion that 'agents exhibit substantial differences' across axes is a comparison of conditions defined by these labels, a systematic label-generation confound would undermine the headline claim. This does not mean the paper is wrong; it means the current evidence is conditionally supportive. I agree with the reader's CONDITIONAL verdict and would not move it: the concern is concrete but addressable through the proposed validation check. I do not see a more load-bearing objection in the metric design or the single-document top_k=1 simplification, since the paper acknowledges those limitations and they do not threaten the internal validity of the comparison in the same way label infidelity does.","tokens_in":31940,"tokens_out":4508,"duration_ms":53016,"concrete_test":"Annotate a fresh random sample (e.g., 100 documents per axis per dataset) of R=0 documents against the original agent question, and F=1/F=0 documents against gold supporting evidence, using the actual agent query shown to the generator. Compute label agreement and confusion matrices. Then re-run the headline analysis (at minimum the Figure 1 and Table 1 comparisons) restricted to documents that pass human-confirmed labels, or reweight by inverse label accuracy. If the outcome distributions or system rankings change materially—especially if R=0 documents are frequently judged relevant to the original question, or if F=1 documents are frequently judged non-factual—the axis-specific robustness claims are confounded and require revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that agents differ substantially in handling untrustworthy, irrelevant, and non-factual documents—requires that the simulator actually realizes the intended T/R/F labels without correlated confounding. The manuscript's own sanity check is insufficient for this load. Section 3.3/Table 2 reports factuality agreement of only 85.5% (Cohen's κ=0.64), and Table 3 shows the errors are asymmetric: 13 of 69 human judgments on intended-factual documents were scored non-factual, while intended non-factual documents were almost always recognized as such. Thus the F=100% control condition is contaminated with documents that annotators perceived as non-factual, and the F=0% condition may consist of obviously exaggerated falsehoods rather than realistic unreliable evidence. Even more directly, Appendix A.1 states that in the human evaluation 'relevance is always annotated as True,' so the relevance axis—which produces some of the largest behavioral differences in Figure 1b—has no human validation at all. R=0 documents are generated by substituting an unrelated question's support context; nothing in the paper verifies that these documents do not accidentally contain answer-relevant content or that they are not systematically distinguishable from relevant documents by surface cues. Trustworthiness is better validated (96.4% agreement), but low-trust documents also differ in URL, style, and instructions, so trustworthiness effects are entangled with prompt-level style. Since all downstream results (Figure 1, Table 1, RAS/CoPRAS) are conditioned on these labels, any systematic generation cue could make the reported 'substantial differences' reflect artifacts of the generation prompt rather than agent robustness to unreliable information. This is a correctness risk, not an internal inconsistency, and it is exactly the kind of concern that can be resolved with targeted validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeepStress, a controlled simulation environment for stress-testing search agents. The framework intercepts an agent's search tool and replaces retrieval with dynamically generated synthetic documents whose trustworthiness, relevance, and factuality are sampled from experimenter-controlled probabilities. The authors evaluate twelve systems (six GPT-based configurations, five open-weight search/RL agents, plus a no-reasoning DR-Tulu ablation) on HotpotQA and BrowseComp-Plus under seven scenarios (0%/50%/100% along each axis while the other axes are fixed at 100%), totaling 33,600 traces. They introduce two aggregate metrics, RAS and CoPRAS, which give partial credit to abstention and tool-budget exhaustion under unreliable evidence. The main empirical claims are: (i) agents differ substantially in how they handle degraded documents, with GPT systems tending to abstain or over-search, open-weight systems producing confident errors, and DR-Tulu showing format brittleness; (ii) RAS and CoPRAS reveal reliability-aware behavior that raw accuracy and TPC miss; and (iii) HotpotQA is partly answerable from parametric knowledge, as shown by a direct-inference baseline, so it should not be used as a pure search-agent benchmark without such a baseline.","tokens_in":32232,"tokens_out":7955,"duration_ms":84598,"significance":"If the central claims hold, DeepStress is a useful contribution: it provides a reproducible, controlled laboratory for measuring how search agents react to evidence-quality failures, a dimension that standard final-answer benchmarks under-sample. The paper has clear strengths: it releases (or promises to release) the environment code; it evaluates a broad and diverse set of systems; it uses dynamic per-query document generation rather than a static corpus; it includes a direct-inference baseline that cleanly demonstrates parametric-knowledge leakage on HotpotQA; and its outcome taxonomy goes beyond binary accuracy. The proposed RAS/CoPRAS metrics are transparently specified and their design choices are discussed. However, the validity of the entire axis-specific analysis rests on the fidelity of the synthetic labels. The human sanity check is only partially convincing: factuality agreement is modest and asymmetric, and relevance — the axis producing the largest behavioral differences — is never human-validated. Because these are load-bearing concerns rather than cosmetic ones, the manuscript needs substantial revision before the empirical conclusions can be accepted.","major_comments":[{"comment":"Factuality is the central degradation axis for the dataset-contrast claims in §5.2, yet the human validation is weak and asymmetric. Table 2 reports only 85.5% raw agreement with the intended factuality label and Cohen's κ=0.64. Table 3 shows the error pattern: 13 of 69 documents intended as factual were judged non-factual by annotators, while intended non-factual documents were almost always recognized as such. Thus the F=100% control condition is contaminated with documents that annotators read as false, and the F=0% condition may consist of obviously exaggerated falsehoods (see the Appendix A.3 example) rather than realistic misleading evidence. This directly threatens the claim that agents differ specifically in handling non-factual documents. Please quantify the effect of annotator-disputed documents, report per-condition results excluding them, or strengthen the generation protocol","section":"§3.3, Appendix A.1, Tables 2–3"},{"comment":"The relevance axis has no human validation at all: Appendix A.1 states that 'relevance is always annotated as True.' This is especially problematic because Figure 1b shows the largest behavioral differences on the relevance axis (e.g., Search-R1 incorrect answers rising to 79.2% at R=0). R=0 documents are generated by replacing the target question and support context with an unrelated question, and the document title is sampled from that unrelated context. Nothing verifies that these documents are actually perceived as irrelevant, that they contain no overlapping entities or answer-relevant content, or that they are not systematically identifiable by surface cues such as title mismatches or formatting oddities. Because the R=0 condition is defined by construction rather than validated, the measured relevance-specific effects could be artifacts of the swapping procedure. Please add a huma","section":"Appendix A.1; §3.2"},{"comment":"RAS's reliability definition is internally inconsistent with its motivation. Eq. (4) defines ρ = π_T × π_R × π_F, so any axis at 0% makes ρ=0 and gives full RAS credit to abstention and tool-budget exhaustion. But the text preceding Eq. (5) justifies partial credit only for conditions where 'relevance or factuality is set to 0%' — i.e., where the answer cannot be recovered from the documents. A document that is untrustworthy in style but factually correct and relevant is still answerable; yet in the T=0, R=F=100% scenario, a system that refuses to answer solely because the source looks unreliable receives RAS=1. This inflates the 0% column of Table 1 and favors GPT systems that abstain or over-search on stylistic grounds. Please either restrict partial credit to R/F degradation or provide a substantive argument that low trustworthiness alone makes evidence unusable; the current choice co","section":"§3.4, Eqs. (4)–(5); Table 1"}],"minor_comments":[{"comment":"The sentence 'In 2, we report raw accuracy...' is missing the table number; it should refer to Table 2. Also, the claim of 'strong agreement' for factuality is overstated given the asymmetric confusion matrix in Table 3.","section":"Appendix A.1"},{"comment":"The phrase 'This phenomenon occurs rarely in realistic benchmarks' has an unclear antecedent ('this phenomenon' most likely refers to poor-quality evidence, but the sentence reads as if it refers to dramatic failure). Please rephrase for clarity.","section":"Abstract / §1"},{"comment":"The figure repeats the 100% condition three times and uses small fonts and dense legends; consider showing the reference condition once per panel and using larger labels. Also, the paper does not report confidence intervals or significance tests for the 200-question per-system scenario cells; a brief note on variability would strengthen the claims of 'substantial differences.'","section":"Figure 1"},{"comment":"The final-answer classification relies entirely on GPT-4.1 as an LLM judge with no human validation of the judge itself. Given that the outcome taxonomy (especially the abstention category) is central to RAS, a small human agreement study for the judge would increase confidence.","section":"§3.4 / Appendix E"}],"recommendation":"major_revision","confidential_remarks":"This is a promising and useful stress-testing framework, and I believe the core idea is sound. My recommendation rests on the synthetic-label fidelity issue: the factuality validation is too weak, the relevance axis is unvalidated, and the RAS definition rewards abstention under low trustworthiness in a way that is not fully justified. These are fixable within the scope of the manuscript — by adding relevance validation, analyzing factuality disputes, and revising or reinterpreting the RAS definition. I would be happy to revisit after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a genuinely useful contribution to search-agent evaluation, and the main risk is not the framing or the metrics but the under-validated synthetic labels.\n\nWhat's new: DeepStress replaces the retrieval module with a dynamic synthetic environment that controls document trustworthiness, relevance, and factuality per agent query. That is a real step beyond static corpora like BrowseComp-Plus, because it adapts to the actual queries agents issue. RAS and CoPRAS are sensible additions that credit abstention under unreliable evidence, and the paper shows they change system rankings in meaningful ways. The direct-inference baseline is also a nice touch, and the HotpotQA parametric-memory warning is important: several systems answer 50-60% of HotpotQA questions with no documents at all, which should make people rethink that benchmark.\n\nThe evaluation itself is solid for a preprint: 12 systems, 33,600 traces, two datasets with complementary properties. The behavioral differences are plausible and mostly consistent: GPT models abstain or hit the tool budget, open-weight models produce confident errors, and irrelevant documents are easier to reject than relevant-but-false ones. Those findings are worth having even if the exact numbers shift with better validation.\n\nNow the soft spots. The load-bearing assumption is that the simulator actually produces documents with the intended labels, and the evidence for that is thinner than the claims require. Trustworthiness is well validated (96.4% agreement), but factuality agreement is only 85.5% with errors skewed in one direction, and relevance is not human-validated at all — the appendix simply says annotators were asked to judge relevance as always true. That means the R=0 condition, which produces some of the largest behavioral differences, is essentially unverified. The LLM-as-a-judge for final answers is also used without any human check, and the reported numbers have no error bars. None of this is fatal, but it means the axis-specific robustness comparisons should be treated as provisional until the labels and judge are properly validated.\n\nThe paper is honest about its own limitations, and I don't see circularity or a hidden fitting target. The code and data are deferred, which is irritating but common for under-review work.\n\nWho this is for: anyone working on search agents, RAG robustness, or agent evaluation. It deserves a serious referee; the requested revisions should focus on validating the relevance and factuality conditions, validating the judge, and releasing the environment.","headline":"Useful stress-test rig and two metrics worth taking seriously; the relevance and factuality label validations are too thin to support the sharpest axis-specific claims, but the core findings are solid enough to referee.","tokens_in":32826,"tokens_out":1682,"would_cite":true,"duration_ms":20491,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that reliability-aware evaluation, not final-answer accuracy, reveals how search agents handle degraded evidence, and that a controlled synthetic retrieval environment can measure those differences.","keywords":["search agents","stress testing","synthetic document generation","reliability-aware evaluation","knowledge conflicts","retrieval robustness","multi-hop question answering","abstention behavior"],"falsifier":"Audit the generated corpus in a blind design: ask annotators to label documents for relevance and factuality without seeing the intended label, and rerun the 0%-relevance and 0%-factuality conditions with a second generator and a second prompt wording; if the agent outcome distributions shift across generators, the stress test is measuring generator artifacts rather than a stable agent property.","tokens_in":31785,"feed_emoji":"🔎","tokens_out":9996,"duration_ms":98541,"temperature":0.7,"pith_summary":"DeepStress claims that the ability to handle unreliable search evidence is not captured by final-answer accuracy, and that a laboratory environment which replaces retrieval with controllable synthetic documents can expose how agents really differ. The authors intercept every search call and return one generated document whose trustworthiness, relevance, and factuality are sampled from experimenter-chosen probabilities. Across two question-answering benchmarks and twelve agents, each degraded axis produces its own failure signature—some agents ignore untrustworthy source cues, some over-search until budget exhaustion, some break output format, and some confidently answer from relevant-but-false documents. To make these differences comparable, the paper proposes a reliability-aware score (RAS) that gives partial credit for abstaining or stopping under unreliable evidence, and a cost-normalized variant (CoPRAS) that re-ranks systems relative to raw accuracy or efficiency. If the approach is sound, search-agent evaluation should move beyond accuracy to report reliability-aware outcomes, and benchmarks whose questions are answerable from memory need a no-search baseline.","feed_headline":"Degrade the evidence and search agents fail in very different ways","feed_subtitle":"A controlled simulator credits abstention over guessing and reorders rankings under unreliable evidence.","key_machinery":"The central object is the search-call interceptor plus scenario sampler. Each agent search is routed to a simulator that draws T, R, F labels from scenario probabilities, then builds a document (grounding factual ones in gold evidence, sampling known low-credibility domains for untrustworthy ones, and swapping in an unrelated question for irrelevant ones). The evaluative machinery is the reliability-aware score: RAS_i = 1 for correct answers, 1−ρ for abstention or budget exhaustion with ρ = π_T×π_R×π_F, and 0 for incorrect or malformed answers; CoPRAS divides total cost by total RAS. This scoring treats 'not answering under unreliable evidence' as partially correct, which separates reliabili","core_discovery":"Search agents do not have one robustness trait but several: under controlled degradation of trustworthiness, relevance, or factuality, twelve agents diverge into distinct failure profiles—some ignore untrustworthy source cues, some fail mainly on irrelevant documents, some answer confidently from relevant-but-false documents, and others abstain or exhaust their tool budget. Each search call draws a label per axis from scenario probabilities, and a generator produces a document realizing those labels. The paper's reliability-aware score gives full credit to correct answers, partial credit to refusal when evidence is unreliable, and zero to incorrect or malformed answers; its cost-normalized v","pith_inferences":["Editorial inference: the simulator could double as a training adversary—an RL objective maximizing RAS under degraded scenarios would directly reward abstention and source-checking, not just final accuracy.","Editorial inference: the one-document-per-call design leaves open whether the same failure signatures hold when a search returns a mix of reliable and contradictory documents; testing multi-document returns is the natural next step.","Editorial inference: because the factuality manipulation showed label noise (13 of 69 intended-factual documents were judged non-factual), the measured contrasts are likely a lower bound; a generator with tighter label adherence would sharpen the behavioral differences.","Editorial inference: agents that verbally flag degradation without changing their answer suggest future process metrics should score detection-and-action, not detection alone."],"forward_implications":["Failure taxonomies must be reported alongside accuracy: abstention, budget exhaustion, missing answer tags, and incorrect answers move in opposite directions under different degradation axes.","RAS and CoPRAS change system rankings at degraded quality, so reliability-aware and cost-efficiency evaluations can disagree with raw tokens-per-correct answers.","No-search baselines are required for question sets whose answers are memorizable; otherwise, correct answers under completely false evidence are misread as evidence robustness.","Contradictory-evidence conditions (50% factuality) provide a direct probe of an agent's ability to aggregate consistent evidence across multiple searches.","Unreliable evidence raises search cost, on average roughly doubling the cost-normalized metric between fully reliable and fully degraded documents, making robustness an efficiency issue as well as an accuracy issue."],"fun_headline_variants":["Search agents fail differently under poor evidence","Twelve agents, four failure profiles under degraded evidence","Robustness isn't one trait: stress test reveals distinct failure modes","Controlled evidence degradation reorders search agent rankings","Untrustworthy, irrelevant, false: agents fail along three axes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that synthetic documents generated under a given degradation label differ from clean documents only along that axis; because the paper's own sanity check finds imperfect factuality label agreement and no human validation of the relevance manipulation, the measured behavioral contrasts could partly be artifacts of generation instead of agent robustness.","fun_headline_variants_meta":{"raw":{"variants":["Search agents fail differently under poor evidence","Twelve agents, four failure profiles under degraded evidence","Robustness isn't one trait: stress test reveals distinct failure modes","Controlled evidence degradation reorders search agent rankings","Untrustworthy, irrelevant, false: agents fail along three axes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2430,"prompt_tokens":620,"completion_tokens":1810,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":364,"completion_tokens_details":{"reasoning_tokens":1731}},"tokens_in":364,"tokens_out":1810,"duration_ms":12978,"temperature":1.0,"reasoning_tokens":1731,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:18:22.947308+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the generated corpus in a blind design: ask annotators to label documents for relevance and factuality without seeing the intended label, and rerun the 0%-relevance and 0%-factuality conditions with a second generator and a second prompt wording; if the agent outcome distributions shift across generators, the stress test is measuring generator artifacts rather than a stable agent property.","supporting_citations":[],"review_version":1}