Pith. sign in

REVIEW 5 major objections 7 minor 2 cited by

A new benchmark replays expert tool calls on real advertising queries and finds that even top LLM agents struggle badly on the hardest multi-step tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:14 UTC pith:2JSAFHME

load-bearing objection A useful benchmark concept undermined by a self-judging evaluator and an abstract that disagrees with its own Table 2. the 5 major comments →

arxiv 2602.14257 v2 pith:2JSAFHME submitted 2026-02-15 cs.CL cs.AIcs.IRcs.LG

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

Lingxiang Hu , Yiding Sun , Tianle Xia , Wenwei Li , Ming Xu , Lan Xu , Siying Wang , Wei Xu
show 1 more author
Jie Jiang
This is my paper
classification cs.CL cs.AIcs.IRcs.LG
keywords LLM agentsadvertising analytics benchmarktrajectory-aware evaluationdynamic ground truthPass@ktrajectory coveragetool-use agentsReAct framework
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces AD-Bench, a benchmark for LLM agents that work in advertising and marketing analytics. It is built from 2,000 real user requests on a production ad platform, cleaned to 823 validated instances and a 100-instance subset with unique expert trajectories. Its two design ideas are a dynamic ground-truth pipeline, in which expert tool-call trajectories are replayed and re-executed so answers track the live environment, and a trajectory-aware evaluation that scores both answer correctness (Pass@k) and coverage of the expert trajectory. Experiments indicate that current models, whether measured by the abstract's Claude-Opus-4.7 numbers or the main table's Gemini-3-Pro numbers, are reliable on simple retrieval tasks but lose roughly 20–30 percentage points on the hardest tier, so complex industrial advertising analytics remains an open capability gap.

Core claim

The central claim is that a benchmark assembled from real customer requests, with ground truth regenerated by replaying expert tool-call trajectories, offers a sustainable and business-realistic way to evaluate advertising LLM agents, and that trajectory coverage is a meaningful, predictive signal of end-to-end success. On this benchmark, state-of-the-art agents approach saturation on easy tasks (Pass@3 around 87–96%) but drop to roughly 60–70% Pass@3 on the hardest tasks, with trajectory coverage falling to around 70% or lower for the best model. The paper therefore argues that multi-round, multi-tool orchestration, not basic retrieval, is the current bottleneck.

What carries the argument

The core mechanism is the replayable ground-truth trajectory: human experts solve real queries with a 9-tool advertising toolkit (account listing, daily/hourly data, knowledge search, calculator, and so on), and their tool-call sequences are re-executed at evaluation time to regenerate answers. Evaluation uses Pass@k for answer correctness and a trajectory-coverage metric that checks whether the expert's tool sequence appears as a subsequence of the agent's execution. Difficulty tiers L1–L3 stratify tasks by trajectory complexity, and a 100-query subset with unique trajectories supports the dual answer-and-trajectory evaluation.

Load-bearing premise

All reported Pass@k and trajectory-coverage numbers assume the Gemini-3-Pro LLM judge reliably distinguishes correct from wrong answers and covered from uncovered trajectories; the paper reports no human-agreement check, and coverage matching ignores parameter correctness.

What would settle it

Take a random sample of 100 scored trajectories from Table 2, rescore them with a second judge (a different LLM or human raters), and compare per-item judgments and the resulting Pass@k rankings. If inter-rater agreement is low or rankings change materially, the benchmark's headline conclusions about model ordering and the L3 gap are not robust.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If AD-Bench is a fair proxy for real ad-analytics work, current LLM agents are production-ready for simple retrieval queries but not for long-horizon, multi-tool judgment tasks.
  • Because trajectory coverage correlates with Pass@1 (r≈0.69 overall, ≈0.80 on L3), measuring coverage can flag planning failures even when the final answer looks plausible.
  • The dynamic ground-truth pipeline means benchmark scores do not silently decay as platform data and rules change, so repeated evaluation over time is meaningful.
  • The error analysis predicts that progress will come mainly from reducing planning and parameter errors, which dominate failures across all models.
  • The reported performance stratification gives model developers a concrete target: closing the 20–30 point L1-to-L3 gap.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same replay-and-coverage recipe could transfer to other fast-changing domains (finance operations, e-commerce analytics, customer support) where static QA benchmarks go stale quickly.
  • Because trajectory coverage uses subsequence matching without checking parameters, an agent could get credit for the right tool order while passing wrong date ranges or filters; a stricter parameter-aware coverage metric would likely widen the apparent gap.
  • The paper's abstract and main table name different best models with different numbers; a reader should treat the exact leaderboard figures as provisional until the discrepancy is resolved, even though the qualitative L3 drop appears in both.
  • A cheap testable extension is to use coverage as a dense reward for training or as a stopping rule for sampling: on L3, coverage is the strongest predictor of success, so boosting coverage may directly boost Pass@k.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces AD-Bench, a benchmark for LLM agents in advertising analytics, built from 2,000 real user requests that are cleaned and deduplicated into 823 instances. Human experts provide reference answers and tool-call trajectories; a dynamic ground-truth pipeline replays these trajectories to regenerate answers under the current environment. The evaluation jointly measures answer correctness (Pass@k) and trajectory coverage, with instances stratified into three difficulty levels (L1–L3). Experiments on ten open-source and proprietary models show a large performance drop on L3 tasks, which the authors attribute to planning, parameter, and dependency failures. The paper also includes prompt templates, an error analysis, and case studies.

Significance. If the evaluation pipeline is valid, AD-Bench addresses a real need: existing agent benchmarks are often static or simulated, while advertising analytics requires multi-round tool use and ground truth that changes as data and platform rules evolve. The dynamic ground-truth replay idea is valuable, and the trajectory-aware evaluation goes beyond final-answer accuracy. The paper ships the full prompt templates and tool descriptions, which aids reproducibility. However, the manuscript's central empirical claims currently rest on an unvalidated, self-referential LLM judge and are internally inconsistent at the headline level. The benchmark's significance therefore depends on whether these issues can be resolved with additional validation and clarification.

major comments (5)
  1. [Abstract vs. §3.1/Table 2] The abstract in the submission header reports 'the best model, Claude-Opus-4.7, attains Pass@1 = 76.9%' and L3 Pass@1 = 61.4%, but the body's Table 2 and §3.1 evaluate Gemini-3-Pro, GPT-5.1, HY-2.0, o3, GLM-4.7, DeepSeek-V3, Kimi-K2, and Qwen3 models, and never mention Claude-Opus-4.7. Table 2 names Gemini-3-Pro as the best overall (Pass@3 = 83.0), and Figure 1 shows GPT-5.1 as best on L3. This is a direct contradiction in the headline result and must be resolved; as written, the reader cannot tell which model is being claimed as state of the art.
  2. [§3.1, Figure 9] Gemini-3-Pro is used as the LLM judge for outcome correctness and trajectory coverage, and the same Gemini-3-Pro is one of the evaluated models in Table 2. The judge is asked to emit binary labels for 'result_correct' and 'trajectory_covered' (Figure 9), and no human agreement, inter-annotator reliability, or independent judge comparison is reported. This is a circularity risk: if the judge is biased toward its own outputs, all Pass@1/Pass@3 and coverage numbers are ungrounded. Please report human agreement on a sample (e.g., Cohen's kappa) and/or use a judge that is not among the evaluated models.
  3. [§2.4 vs. Figure 9] The text defines trajectory coverage as subsequence matching ('the standard trajectory appears as a sub-trajectory within the actual execution trajectory'), but the actual evaluation prompt in Figure 9 asks the judge to decide 'trajectory_covered' subjectively. These are different procedures. Additionally, it is unclear whether the reported Pass@k and coverage numbers are computed over all 823 instances or only the 100 'unique-trajectory' instances described in §2.3. Please specify the evaluation set for each reported number and, if subsequence matching is used, give the matching algorithm and state whether parameter correctness is checked.
  4. [§3.3, Table 3] The text states 'each failed case contains about 1.4–1.9 error types on average,' but Table 3's own numbers contradict this. For example, for Gemini-3-Pro the six error columns sum to 89 against #Total = 128 (ratio ≈ 0.70); for GPT-5.1 the sum is 123 against #Total = 234 (ratio ≈ 0.53). No model reaches even 1.0. Moreover, #Total values (e.g., GPT-5.1 = 234) are not consistent with a Pass@1 of 64.7% on 823 instances (~290 failures) or on the 100-instance subset (~35 failures). Please clarify how #Total is defined, and correct the average-error claim or the table.
  5. [§2.1 dynamic ground-truth pipeline] The paper claims that replaying expert trajectories regenerates ground-truth answers consistent with the current environment, mitigating obsolescence. However, no evidence is provided that replayed answers remain valid or match human-verified answers after environment changes. Since this is a core design claim, please report a stability/validation check (e.g., replay the same queries at two different time points and have human experts verify the changed answers).
minor comments (7)
  1. [§2.4] Pass@3 is defined as 'whether at least one of three independent runs is judged correct' but no sampling temperature or number of samples per run is reported. Please specify the decoding parameters.
  2. [Figure 1] The caption says 'the best model achieving only 69% Pass@3', and the figure appears to show GPT-5.1 at 69%. This should be labeled consistently with Table 2, where Gemini-3-Pro (not GPT-5.1) is best overall but GPT-5.1 is best on L3.
  3. [Table 1] The L1/L2/L3 ratios (24%/47%/29%) are not tied to instance counts. Please specify the number of instances per tier for both the 823 set and the 100-instance unique-trajectory subset.
  4. [References] The model name 'GLM-4.7' appears in Table 2 and §3.1, but the bibliography entry (Zeng et al., 2025a) is for 'GLM-4.5'. Please align the model version and citation.
  5. [Appendix B] Typo: 'shwon' should be 'shown' in the tool inventory description.
  6. [Related Work] Typo: 'Deep Researh' should be 'Deep Research' in §4.1.
  7. [§2.4 and Figure 9] The text says exact parameter matching is infeasible, but Figure 9's judge prompt includes 'has_error_parameters'. Please clarify the relationship between the automated trajectory-coverage metric and the judge's parameter-error flag.

Circularity Check

0 steps flagged

No significant circularity; AD-Bench's benchmark construction is self-contained, though judge validity and internal consistency remain risks.

full rationale

The paper's load-bearing claims are empirical benchmark measurements, not derived quantities. Ground-truth answers are produced by replaying expert trajectories (§2.1), and the metrics Pass@k and trajectory coverage are defined as independent measurement protocols (§2.4). No parameter is fitted to a subset and then renamed as a prediction. The observed correlation between trajectory coverage and Pass@1 (§3.2) is partly a design consequence, because the reference answer is generated from the same standard trajectory used to define coverage, but the paper does not define either metric in terms of the other, so this is not a circular reduction. The only apparent self-citation (Xia et al., 2025, in the introduction) is a background reference and is not load-bearing. Using Gemini-3-Pro as the judge (§3.1) while also evaluating Gemini-3-Pro (Table 2), with no reported human agreement, is a real measurement-validity risk, and the abstract/Table 2 discrepancy over the best model is a consistency error; however, the judge is given ground-truth answers and expected trajectories (Figure 9), so the scores are not definitionally equivalent to the judge's preferences. These issues belong to correctness risk, not circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The benchmark introduces no numerical free parameters or invented scientific entities. It relies on expert judgment, LLM-judge validity, and representativeness assumptions that are not independently verified.

axioms (4)
  • domain assumption Expert-labeled trajectories are correct, and replaying them on a changing environment produces valid current ground truth.
    §2.1: the system 'dynamically synchronizes with the online environment and the domain knowledge base before each evaluation round.' No validation of replayed outputs is described.
  • domain assumption Gemini-3-Pro, used as the LLM judge, accurately evaluates result_correct and trajectory_covered.
    §3.1 and Figure 9: the judge is used to determine both correctness and coverage, with no human agreement or calibration reported.
  • domain assumption The 823 cleaned instances and the curated 100 unique-trajectory subset are representative of the real long-tail distribution of advertising analytics requests.
    §2.3: the selection and exclusion criteria are described qualitatively; no quantitative representativeness analysis is given.
  • domain assumption Trajectory coverage, defined as the standard trajectory appearing as a sub-trajectory of the actual execution, captures execution quality.
    §2.4: exact parameter matching is deemed infeasible, so subsequence matching is used; this may miss wrong parameter values even when tool names align.

pith-pipeline@v1.3.0-alltime-deepseek · 13571 in / 12741 out tokens · 124322 ms · 2026-08-02T23:14:34.649505+00:00 · methodology

0 comments
read the original abstract

While Large Language Model (LLM) agents have made remarkable progress on complex reasoning, evaluating them in real-world environments remains an open problem. Existing benchmarks are largely confined to idealized simulations and fail to capture specialized domains such as advertising and marketing analytics, where tasks require multi-round interaction with professional tools and where ground-truth answers quickly become obsolete as data and platform rules evolve. To address this, we propose AD-Bench, a benchmark built from real user marketing-analysis requests on a production advertising platform. AD-Bench introduces two key designs: (i) a dynamic ground-truth pipeline that replays expert tool-call trajectories to regenerate answers consistent with the current environment, mitigating answer obsolescence; and (ii) a trajectory-aware evaluation that jointly measures end-to-end answer correctness (Pass@k) and trajectory coverage. Requests are stratified into three difficulty levels (L1-L3) to probe multi-round, multi-tool collaboration. Experiments show that the best model, Claude-Opus-4.7, attains Pass@1 = 76.9% and Pass@3 = 80.4% with 82.7% trajectory coverage overall, yet drops sharply on L3 to Pass@1 = 61.4% and Pass@3 = 65.1%, revealing that even state-of-the-art agents have substantial gaps in complex advertising analytics.

Figures

Figures reproduced from arXiv: 2602.14257 by Jie Jiang, Lan Xu, Lingxiang Hu, Ming Xu, Siying Wang, Tianle Xia, Wei Xu, Wenwei Li, Yiding Sun.

Figure 1
Figure 1. Figure 1: Model performance comparison on hard queries (L3). Even state-of-the-art models struggle with complex multi-step reasoning tasks, with the best model achieving only 69% Pass@3 accuracy. 2 AD-Bench 2.1 Benchmark Construction The construction of AD-Bench relies on a real￾world Online Advertising Environment, as illus￾trated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: System overview of AD-Bench. The online advertising environment (left) yields human-validated ground￾truth trajectories, while the offline evaluation environment (right) evaluates LLM agents and uses LLM-as-a-judge to assess answer correctness and trajectory coverage. Level Ratio Definition Example Query L1 24% Retrieve a metric for a fixed scope “Get the total cost across all accounts yesterday.” L2 47% C… view at source ↗
Figure 3
Figure 3. Figure 3: Correlation between Pass@1 and Trajectory [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Trajectory coverage across difficulty levels (L1, L2, L3), illustrating the distribution of execution patterns [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A typical tool invocation tree. Solid arrows [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Agent system prompt for the advertising marketing assistant. The agent decomposes user queries into [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template for report generation. The LLM analyst synthesizes tool-returned data into a precise, [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt template for Stage 1 error analysis (Part 1: Tool Definitions). Lists all available tools, their [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt template for Stage 1 error analysis (Part 2: Dependency Graph, Error Patterns, and Evaluation [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Overview comparison of three representative cases across difficulty levels. L1 requires direct retrieval; L2 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Error analysis of the L3 case. Each subplot pairs the correct trajectory (left) with one error pattern (right). [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. NaiAD: Initiate Data-Driven Research for LLM Advertising

    cs.LG 2026-05 unverdicted novelty 7.0

    NaiAD is a new dataset and framework for LLM-native advertising that uses decoupled generation and calibrated scoring to identify four semantic strategies for balancing user and commercial utilities.

  2. GraSP: Graph-Structured Skill Compositions for LLM Agents

    cs.CL 2026-04 unverdicted novelty 7.0

    GraSP introduces executable skill graphs that improve LLM agent rewards by up to 19 points and reduce steps by up to 41% over ReAct, Reflexion, ExpeL, and flat-skill baselines across ALFWorld, ScienceWorld, WebShop, a...

Reference graph

Works this paper leans on

19 extracted references · 6 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Output Requirements

    Analyze the user’s request and conversation history, plan the data needed at the current stage, and execute tool calls accordingly. Output Requirements

  2. [2]

    Round to two decimal places (e.g., 0.11 and 0.1091 are the same)

  3. [3]

    ‘#’ comments are prohibited in INPUT

    Multiple queries can nest multiple<tool>blocks. ‘#’ comments are prohibited in INPUT

  4. [4]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, and 1 others

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, and 1 others. 2024a. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345. 7 Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang...

    Pith/arXiv arXiv 2025

  5. [5]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. τ-bench: A benchmark for tool- agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reas...

    Pith/arXiv arXiv 2024

  6. [6]

    All authorized accounts must be obtained from get_user_account_list with full account_id_list

  7. [7]

    Calculator results must useprint()

  8. [8]

    Output the following tags: <plan>PLAN: </plan>What objective information the current step expects to collect

    Responses must include planning and tool calls. Output the following tags: <plan>PLAN: </plan>What objective information the current step expects to collect. <tools></tools>List of tool calls. 2.<plan>format: <plan> PLAN: Describe the current step goal and what information to collect. </plan> 3.<tools>format (strict): <tools> <tool> TOOL_NAME: tool_name I...

  9. [9]

    2.<plan>specifies what information to collect;<tools>implements the plan

    Answer based on the user’s account data, providing customized solutions. 2.<plan>specifies what information to collect;<tools>implements the plan

  10. [11]

    For unavailable data/parameters, usesummarize_resultsto indicate inability to retrieve

  11. [12]

    If no specific account is mentioned, analyze all accounts with permissions

  12. [15]

    Last week

    “Last week” refers to last Monday through last Sunday. Available Tools •search– Knowledge retrieval (query: str, max 20 chars) •daily_data_by_group_and_field– Daily-granularity report data (user_id,begin,end,group_by_type,fields,account_id_list,page_size, . . . ) •hourly_data_by_group_and_field– Hourly-granularity report for a specific date (user_id,date,...

  13. [16]

    Output data directly without scientific notation or comma-separated formats

  14. [18]

    Analyze and answer based on the user’s question, not indiscriminate summarization

  15. [19]

    Cost (CNY)

    For knowledge-based Q&A, provide a brief explanation of 10–20 words. Input User Question:{query} Data Context: <context> {context} </context> Figure 7: Prompt template for report generation. The LLM analyst synthesizes tool-returned data into a precise, user-facing answer. Tool Description get_user_account_list Retrieve the list of authorized advertising ...

  16. [2020]

    Niklas Höpner, Leon Eshuijs, Dimitrios Alivanistos, Giacomo Zamprogno, and Ilaria Tiddi

    Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300. Niklas Höpner, Leon Eshuijs, Dimitrios Alivanistos, Giacomo Zamprogno, and Ilaria Tiddi. 2025. Au- tomatic evaluation metrics for artificially generated scientific research.arXiv preprint arXiv:2503.05712. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vla...

    Pith/arXiv arXiv 2009

  17. [2023]

    8 A Prompt Templates This appendix presents the complete prompt tem- plates used in AD-Bench for the agent system, re- port generation, and two-stage error analysis

    Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854. 8 A Prompt Templates This appendix presents the complete prompt tem- plates used in AD-Bench for the agent system, re- port generation, and two-stage error analysis. Each prompt is shown verbatim (translated to English) to ensure full reproducibility. A.1...

    Pith/arXiv arXiv

  18. [2024]

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Representa- tions. OpenAI Team. 2025. GPT-5. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hu...

    Pith/arXiv arXiv 2023

  19. [2025]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman

    Ai idea bench 2025: Ai research idea genera- tion benchmark.Preprint, arXiv:2504.14191. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First conference on language modeling. Zhengliang Shi, Yiqun Chen, H...

    Pith/arXiv arXiv 2025