{"id":"d6d22f0b-58ad-4ca6-b957-6cc4a7727456","arxiv_id":"2608.05212","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SearchAuditBench and SearchAuditor let LLM auditors localize, attribute, and repair failures in long-horizon search agents, reaching a 32.3% end-to-end pass rate with GPT-5.5 versus 26.6% for the strongest baseline.","lead":"This paper builds a benchmark of 1,243 failed long-horizon search-agent runs, each labeled with the wrong step, root cause, and a repair, and proposes SearchAuditor, an LLM framework that localizes, attributes, and repairs such failures. It matters because automating failure diagnosis could reduce human debugging cost for web-search agents and reveals that most failures are evidence-processing errors, not retrieval misses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SearchAuditBench gold labels come from one annotator per case with no inter-annotator agreement, and SearchAuditor's prompts are built from the same taxonomy; the claimed FPS lead may partly reflect labeler-specific alignment rather than general auditing ability.","rationale":"The reader's weakest assumption is exactly the reliability of the expert annotations, and I agree that this is the load-bearing point. I considered two other candidates and set them aside: missing significance tests (the main FPS gap is roughly 3 standard errors, so it is probably not pure noise) and rubric-grader noise (real, but the paper at least reports a blind validation, and the external recovery experiment does not use rubrics). The decisive issue is that every benchmark metric is scored against single-annotator gold labels while the proposed method is built from the same annotation guide; that makes the 32.3% versus 26.6% headline hard to interpret as a general result. The LiveBrowseComp experiment partially rescues the practical utility claim because it relies only on final-answer correctness and shows SearchAuditor repairs outperform unguided retry, generic hint, All-at-Once, and AgentRx on fix rate. I therefore would not reject the paper; the correct disposition remains CONDITIONAL, pending independent annotation and artifact release, which is exactly the reader's verdict. The proposed check would settle whether the benchmark ranking survives consensus labeling.","tokens_in":29448,"tokens_out":8047,"duration_ms":89128,"concrete_test":"Release SearchAuditBench and have two independent annotators re-label a stratified random 200-case subset using the Appendix B guideline; report root-cause Cohen's kappa, CS-Strict agreement, and CS-Loose agreement. Then recompute the Table 2 comparison between SearchAuditor and All-at-Once on the subset restricted to consensus gold labels. If root-cause kappa is below 0.6, CS-Loose agreement is below 80%, or the FPS lead shrinks below about 3 points, the claimed 'consistently outperforms' is not robust to label noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central benchmark claim — SearchAuditor reaches 32.26% FPS vs 26.55% for the strongest baseline in Table 2 — is measured entirely against expert labels whose reliability is not established. Section 3.3 assigns every case to a single annotator, with cases split disjointly across four annotators, and no inter-annotator agreement is reported. The critical-step definition is counterfactual ('correcting k* would remove this mechanism and make a correct final answer attainable under competent subsequent search', Section 3.2), and the tolerance span is constructed by the same annotator, so localization agreement can encode one person's judgment. Furthermore, SearchAuditor's prompts in Appendix G are written from the same taxonomy and the same 'earliest decisive error' rule as the annotation guideline in Appendix B, and its repair synthesizer is instructed to emit rubric-shaped directives. This creates a real alignment risk: the method may score higher because it reproduces the authors' labeling conventions, not because it audits better in general. The rubric grader adds noise: the blind validation in Appendix D reports case-level kappa 0.650 and a 21% false-failure rate, with grader errors not broken down by method, so the 5.7-point FPS gap could shift under a different grader. The LiveBrowseComp recovery experiment in Table 4 is genuinely annotation-free and supports the practical repair claim, but it does not validate the benchmark-based superiority over all baselines. The headline benchmark claims therefore hinge on label reliability that is currently unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents two contributions. First, SearchAuditBench, a benchmark of 1,243 expert-annotated failed deep-search trajectories (averaging 73.1 messages and 65.1K tokens) collected from eight open-weight models running five deep-search benchmarks under a unified scaffolding; each trajectory is labeled with a critical error step and tolerance span, a root cause from a six-way taxonomy, and a reference repair with atomic rubrics. Second, SearchAuditor, a prompt-based auditing pipeline consisting of three parallel audit branches (holistic, backward constraint, forward timeline), an evidence-grounded adjudicator, and a diagnosis-conditioned repair synthesizer. The paper reports that SearchAuditor outperforms all baselines (All-at-Once, Step-by-Step, Binary Search, AgentRx) on every metric across three frontier backbones (GPT-5.5, Gemini-3.1-Pro, Claude-Opus-4.8), raising the end-to-end fully-passed score from 26.55% to 32.26% with GPT-5.5. It also reports that resuming failed LiveBrowseComp runs with SearchAuditor's repairs lifts Kimi-K2.6 accuracy from 34.03% to 45.07%, that compute-matched ablations support the multi-perspective design, and that the corpus reveals systematic failure patterns such as a negative correlation between exploration budget and Search Coverage Gap share.","tokens_in":29708,"tokens_out":21090,"duration_ms":196422,"significance":"If the claims hold, the work is a solid contribution to agent failure auditing. Strengths include: a large reusable annotated corpus for an under-studied task; a parameter-free prompted method with full prompts in the appendix; a frozen-trajectory protocol that makes auditor evaluation reproducible and independent of live-web drift; well-designed compute-matched ablations (Table 3); and, most importantly, a held-out LiveBrowseComp recovery experiment whose generic-hint control cleanly isolates the value of the repair directive. The corpus-level findings (25% answer-in-hand rate, r=-0.88 between exploration budget and Search Coverage Gap share) are falsifiable claims other groups can test. The principal weakness is that all benchmark-relative numbers in Table 2 are anchored to gold labels authored by the same team, single-annotator, without inter-annotator agreement, and scored by an LLM rubric grader with a 21% case-level false-failure rate (Appendix D). The LiveBrowseComp result, being annotation-free, survives these concerns, but it validates the repair/utility claim rather than the benchmark-relative superiority claim.","major_comments":[{"comment":"The benchmark's gold labels are produced by a single annotator per case, with the 1,243 cases split disjointly across four annotators, and no inter-annotator agreement is reported. This is load-bearing because all Table 2 metrics are computed against these labels, and the labels themselves are highly judgment-dependent: the critical step is defined counterfactually ('correcting k* would remove this mechanism and make a correct final answer attainable under competent subsequent search', Section 3.2), and the tolerance span is constructed by the same annotator who chooses k*. Without an agreement measurement, the benchmark's reliability is unestablished and the 5.7-point FPS gap in Table 2 cannot be separated from annotator idiosyncrasy. Please re-annotate a random subset (at least 100 instances) with all four annotators and report per-field agreement (exact k*, within-span match, and root-cause kappa), and report Table 2 metrics by annotator partition to confirm the gains are not driven by a single annotator's cases.","section":"Section 3.3; Tables 2 and 6"},{"comment":"There is a genuine alignment risk between the annotation conventions and SearchAuditor's prompts. The gold rule labels 'the earliest decision step at which the annotated failure mechanism becomes established' (Section 3.2), and the same 'earliest decisive error' rule and the same six-way taxonomy definitions are embedded in the stage-1 and stage-2 prompts of Appendix G; the repair synthesizer is additionally instructed to emit 2-4 rubric-shaped atomic directives, mirroring the format of the gold rubrics it is graded against. SearchAuditor is therefore likely to reproduce the authors' labeling conventions, and part of its Table 2 advantage may reflect that alignment rather than general auditing skill. This is not strict circularity (the method never sees the gold labels) and it does not affect the annotation-free LiveBrowseComp experiment, but the benchmark-relative claim needs a robustness check: (i) the IAA measurement requested above; (ii) evaluation of SearchAuditor and the baselines against independently produced labels on a subset; and (iii) a per-annotator breakdown of the Table 2 metrics.","section":"Sections 3.3 and 4; Appendices B and G; Table 2"},{"comment":"The rubric-grading procedure is load-bearing for the FPS headline. The blind validation reports case-level kappa of 0.650 and a 21% false-failure rate (43 of 159 grader failures accepted by the human reference, Table 8), and the statement that the shared grader 'does not affect the relative comparisons' is asserted, not tested: the FP/FF rates are not broken down by method, backbone, or repair format. Because SearchAuditor's repairs are deliberately format-matched to the rubric style while the baselines' repairs are not, an LLM grader could systematically favor one family of outputs. Please report per-method agreement on the 200-case blind set, or recompute back-to-back FPS on that set with the human case outcomes, and ideally confirm the ranking with a second independent grader model.","section":"Appendix D; Table 2"}],"minor_comments":[{"comment":"No significance tests or confidence intervals accompany the Table 2 comparisons or the Table 4 fix-rate differences; with N=1,243 and paired failed-run cohorts, McNemar tests would be appropriate.","section":"Section 5.2; Table 2 and 4"},{"comment":"Table 3's 300-instance subset has no reported sampling seed, and the Full-variant scores (CS-Strict 49.00, Rep@Diag 79.18) deviate from the full-benchmark SearchAuditor scores (44.89 and 84.78, Table 2) in opposite directions; the paper should report the seed and briefly reconcile the subset with the full-benchmark results.","section":"Table 3"},{"comment":"Figure 4 reports Pearson r=-0.88 on eight points; a p-value or bootstrap interval should be reported, and the analysis should state whether the correlation uses model-level aggregates (n=8).","section":"Figure 4"},{"comment":"The pre-screening model is Claude Opus 4.6, from the same family as the evaluated Claude-Opus-4.8 backbone, so the claimed generator separation from the evaluated auditors is partial; the paper should report how often annotators accepted, corrected, or discarded the LLM proposals.","section":"Appendix B.2"},{"comment":"The headline recovery gain rests on small counts (37 vs 20 fixed runs out of 213 for Kimi-K2.6); a test and confidence intervals for these proportions should be reported.","section":"Table 4"},{"comment":"Appendix B.1 reports that 1,674 failed runs were reduced to 1,243 by the D1-D5 discard criteria, but the per-criterion discard counts are not given; reporting them would help readers assess selection bias in the root-cause distribution of Table 6.","section":"Appendix B.1"},{"comment":"Table 1's header has a garbled column layout ('Scale' and 'Avg. trace length' appear in sequence); the table should state explicitly what each column measures.","section":"Table 1"},{"comment":"The sizes of the 'bounded window' and 'evidence window' used by the adjudicator and repair synthesizer are not specified; state them, since they determine how much context each stage sees.","section":"Section 4, Stage 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well within the journal's scope and the experimental design is generally careful (compute-matched ablations, a generic-hint control in the recovery experiment, disclosed discard criteria). The decisive issue for the benchmark contribution is the absence of inter-annotator agreement and the alignment between author-defined labels and author-designed prompts; both are addressable with additional analyses rather than being fundamental flaws. I would recommend the editor treat the LiveBrowseComp recovery experiment as the strongest evidence for practical utility and request the label-reliability analyses before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: search-agent failure diagnosis is a real gap, and this is the first expert-annotated corpus aimed at it. The 1,243-instance benchmark is substantial: long, evidence-heavy trajectories from eight open-weight models across five benchmarks, with critical step, six-way root cause, and repair rubrics. The failure anatomy is interesting (45.9% late errors, 27.2% candidate mismanagement, answer-in-hand in 25% of cases), and the strong negative correlation between exploration budget and Search Coverage Gap is a believable finding.\n\nThe strongest evidence is the LiveBrowseComp resumption experiment. It is annotation-free, uses a held-out benchmark, and gives the agent a repair at the predicted critical step. SearchAuditor repairs roughly double unguided retry and beat a generic hint that shares the same localization, which supports the claim that the repair content, not just the step, matters. That is the part I would defend.\n\nThe soft spot is exactly where the stress-test note points. Gold labels are produced by the four authors, one per case, with no inter-annotator agreement. The critical-step definition is counterfactual and the tolerance span comes from the same annotator. The SearchAuditor prompts reuse the same taxonomy and the same 'earliest decisive error' rule as the annotation guideline, so part of the measured advantage may be reproducing the authors' conventions. Also, the rubric grader has case-level kappa 0.65 and a 21% false-failure rate. Because the same grader grades every method, relative comparisons degrade less than absolute numbers, but the 5.7-point FPS gap should be read with that noise in mind. I would like to see IAA on a sample, released data and code, and confidence intervals or significance tests on the main table. None of this kills the paper: the external recovery experiment gives independent support that the method has real utility, and the qualitative pattern across three backbones is consistent. But the headline numbers are provisional until the artifacts and reliability evidence appear.\n\nBottom line: this deserves a serious referee. The benchmark fills a real gap, the task design is careful, and the external validation is uncommon. Require artifact release and inter-annotator agreement evidence before accepting.","headline":"Solid benchmark and a genuinely useful recovery result; the headline gap over baselines is plausible but rests on author-produced labels with no inter-annotator agreement, so treat the exact numbers as provisional.","tokens_in":30264,"tokens_out":1899,"would_cite":true,"duration_ms":21442,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a multi-perspective auditor can find the exact step where a long-horizon search agent fails, explain why, and repair it, reaching a 32.3% end-to-end pass rate and recovering 17.4% of failed runs on a held-out…","keywords":["search agents","trajectory auditing","failure attribution","root cause analysis","repair directives","LLM benchmarks","long-horizon tasks","deep search"],"falsifier":"Re-annotate a random sample of about 200 SearchAuditBench trajectories with at least two independent annotators who are blind to the original labels, and measure agreement on the critical step (within the tolerance span) and on the root cause; if agreement falls well below the levels the benchmark implicitly assumes (for example, Cohen's kappa below 0.5 on root causes, or fewer than 60% of step choices inside the original tolerance spans), then the gold labels are unstable and the reported 32.3% end-to-end pass rate would not survive a change of annotator.","tokens_in":29236,"feed_emoji":"🔍","tokens_out":6935,"duration_ms":62843,"temperature":0.7,"pith_summary":"The paper seeks to show that large-language-model auditors can take a long, noisy, failed web-search trajectory and pinpoint the single decision step where the search went wrong, classify the root cause, and write a repair directive that, if injected, helps the same agent succeed. To test this, it builds SearchAuditBench, 1,243 failed trajectories from eight open-weight models on five deep-search benchmarks, each annotated with a critical step, a tolerance span, a root cause from a six-way taxonomy, and a repair with atomic rubrics. It then proposes SearchAuditor, a three-branch auditing framework whose reports are adjudicated against trajectory evidence before a repair is synthesized. On the benchmark, SearchAuditor achieves a 32.3% end-to-end pass rate, ahead of all baselines, and its repairs let Kimi-K2.6 recover 17.4% of its failed runs on LiveBrowseComp, lifting accuracy from 34.0% to 45.1%. The sympathetic reading is that automated auditing is becoming practical enough to cut human debugging effort, while the low absolute ceiling shows how far it has to go.","feed_headline":"Audit-guided repairs rescue 17% of failed deep-search runs","feed_subtitle":"Three-perspective auditor beats all baselines, fully passing 32.3% of 1,243 expert-labeled audit cases.","key_machinery":"The mechanism that carries the argument is the decoupling of diagnosis proposal from adjudication. Three parallel audit branches—a holistic read, a backward constraint audit that checks each query constraint against trajectory evidence, and a forward timeline audit that walks decisions chronologically—each propose a critical step and a root cause. A single adjudicator then re-decides the diagnosis using deterministic disagreement summaries, a global outline of all decision steps, and verbatim evidence windows around each proposed step, following rules that treat agreement as evidence rather than a vote and prefer the earliest supported step. Only after the diagnosis is fixed does a repair synthesizer write two to four atomic, process-level directives that are executable from the critical step onward. This three-stage pipeline is what the ablations identify as the source of the gains: replacing the specialized branches with extra holistic copies drops the end-to-end fully-passed score by 4.3 points, replacing the LLM adjudicator with voting drops it by 6.3 points, and removing the outline and evidence windows drops it by 4.3 points.","core_discovery":"The central claim is that failures of deep-search agents can be audited automatically: an LLM can identify the earliest assistant decision that fixes the failure path, assign it one of six root causes (Candidate Mismanagement, Search Coverage Gap, Constraint Neglect, Unverified Source Reliance, Entity–Relation Misbinding, Unsupported Answer), and write a case-specific process repair. The paper also claims that doing this well requires more than a single pass over the trace: three complementary audit perspectives (holistic, backward constraint tracing, forward timeline) generate candidate diagnoses, and an adjudicator re-decides the diagnosis over bounded evidence windows plus a global outline, which beats both majority voting and any single perspective. On 1,243 expert-annotated failed trajectories, this reaches 44.9% exact critical-step localization and 32.3% end-to-end fully passed audits with a GPT-5.5 backbone, and the improvement holds across Gemini and Claude backbones. The paper further claims the repairs are practically useful: resuming a failed run from the predicted critical step with the repair directive appended fixes 17.4% of Kimi-K2.6 failures and 10.3% of Quest-35B failures on LiveBrowseComp, roughly doubling unguided retry, and outperforming a generic hint that shares the same localization. Finally, the corpus shows that most failures (77.2%) are not retrieval failures but errors in how the agent used evidence it already had, and that nearly half of all tool calls and generated tokens occur after the critical error has already been committed.","pith_inferences":["A practical near-term use may be human-in-the-loop auditing: the auditor proposes a diagnosis and a repair, a human confirms before the run is resumed, and that would sidestep the roughly 68% of audits that currently fail end-to-end.","The benchmark's design could transfer to other tool-using agents (code, CLI, memory) by replacing the six-way search taxonomy with a domain-specific one and keeping the three-branch proposal-plus-adjudication architecture.","The resumption-asymmetry observation suggests a testable extension: truncating the trajectory prefix more aggressively (earlier than the predicted critical step) might let agents escape flawed reasoning better than resuming exactly at the critical step.","The 32.3% ceiling may understate practical value, because repair rubrics require all atomic directives to pass; a graded or partial-credit repair score could show larger recoverable fractions."],"forward_implications":["If the benchmark's pass rates are accepted, automated auditors are now competitive enough to serve as a first-pass failure diagnosis tool for deep-search agents, cutting the human inspection burden on 73-message, 65K-token trajectories.","Repair-guided resumption, where a failed run is restarted from the predicted critical step with the repair appended, approximately doubles the recovery rate of unguided retry on both tested models.","Because 77.2% of failures are attributed to how evidence was processed rather than whether it was retrieved, improving candidate management and constraint checking may matter more for search agents than expanding their search budgets.","Since 47.1% of generated tokens and 45.4% of tool calls occur after the critical error is established, reliable online auditing could reclaim nearly half the compute spent by failing search agents.","The failure-mode shift with exploration budget (Pearson r = −0.88 between Search Coverage Gap share and median tool calls) implies that simply scaling test-time search shifts, rather than removes, the binding failure mode."],"supporting_citations":[{"why":"Source benchmark BrowseComp supplies queries whose answers require long-horizon browsing; it is one of the five trajectory sources.","marker":"Wei et al. 2025"},{"why":"BrowseComp-ZH provides Chinese-language deep-search queries, broadening trajectory diversity.","marker":"Zhou et al. 2025"},{"why":"DeepSearchQA contributes query sets that target comprehensiveness gaps in deep research agents.","marker":"Gupta et al. 2026"},{"why":"SealQA (Seal-0) supplies search-augmented reasoning queries to the trajectory pool.","marker":"Pham et al. 2026"},{"why":"xBench-DeepSearch provides professional-aligned real-world tasks that produce failed trajectories.","marker":"Chen et al. 2025"},{"why":"The unified scaffold (search and visit tools) under which all eight source agents run isolates model behavior from tooling differences.","marker":"Tongyi-DeepResearch-Team et al. 2026"},{"why":"AgentRx, the closest prior trajectory-diagnosis framework, serves as the baseline that SearchAuditor must beat and whose constraint-synthesis approach is adapted.","marker":"Barke et al. 2026"},{"why":"Who&When attribution strategies (All-at-Once, Step-by-Step, Binary Search) are adapted as direct-prompting baselines for the full audit output.","marker":"Zhang et al. 2025b"},{"why":"LiveBrowseComp, the held-out benchmark where repair-guided resumption is tested, provides the annotation-free validation of diagnostic utility.","marker":"Fan et al. 2026"}],"fun_headline_variants":["SearchAuditor: AI that finds and fixes deep-search agent errors","Three-perspective audit lifts search-agent pass rate to 32.3%","Auditing deep-search failures: 77% are reasoning, not retrieval","SearchAgent repairs double unguided retry success","Auto-audit rescues 17% of failed long-horizon search runs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold labels that define a correct audit—the critical step, its tolerance span, the root cause, and the repair rubrics—are trustworthy, even though each trajectory was annotated by a single person with no reported agreement check, so noisy or biased labels would make every pass rate uninterpretable.","fun_headline_variants_meta":{"raw":{"variants":["SearchAuditor: AI that finds and fixes deep-search agent errors","Three-perspective audit lifts search-agent pass rate to 32.3%","Auditing deep-search failures: 77% are reasoning, not retrieval","SearchAgent repairs double unguided retry success","Auto-audit rescues 17% of failed long-horizon search runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1847,"prompt_tokens":1121,"completion_tokens":726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":737,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":737,"tokens_out":726,"duration_ms":7306,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:03:46.752235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of about 200 SearchAuditBench trajectories with at least two independent annotators who are blind to the original labels, and measure agreement on the critical step (within the tolerance span) and on the root cause; if agreement falls well below the levels the benchmark implicitly assumes (for example, Cohen's kappa below 0.5 on root causes, or fewer than 60% of step choices inside the original tolerance spans), then the gold labels are unstable and the reported 32.3% end-to-end pass rate would not survive a change of annotator.","supporting_citations":[],"review_version":1}