Pith. sign in

REVIEW 3 major objections 3 minor 25 references

WorkSurface-Bench shows that choosing the correct knowledge surface—document, table, or graph—is necessary but not sufficient for enterprise agents to answer correctly; even with the required surfaces handed to them, models still answer 24.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:30 UTC pith:6ZOCQKI4

load-bearing objection Solid, verifiable benchmark that makes a real point about routing vs. answering; main caveat is the gold surface labels beyond the 200-task human audit rest on LLM screening, so the headline gap could be slightly overstated but not likely wrong. the 3 major comments →

arxiv 2607.25765 v1 pith:6ZOCQKI4 submitted 2026-07-28 cs.CL cs.DB

WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

Hao Liang , Meiyi Qiang , Sizhe Qiu , Linzhuang Sun , Wentao Zhang This is my paper
classification cs.CL cs.DB
keywords surface routingmulti-surface knowledgeenterprise agentsagent benchmarkdocument-table-graphrouting vs answeringgold-constrained evaluationauditable answers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that 'surface routing'—deciding which knowledge representation a question needs (documents for narrative facts, tables for computation, dependency graphs for file lineage)—is a distinct capability from 'answering' that capability correctly. To test this, it builds a benchmark of 1,151 atomic enterprise tasks with auditable gold answers: table answers come from executed SQL, document answers from verified text spans, graph answers from source dependency annotations. Under the gold-constrained setting, where agents are told exactly which surfaces are required, they achieve 98.7–99.8 Route F1 yet only 56.1–75.3% Answer accuracy. The paper argues this near-perfect routing is necessary but insufficient: the residual error concentrates in evidence acquisition and downstream computation/synthesis. A sympathetic reader should care because it means improving enterprise agents requires fixing post-routing reasoning, not just better routers.

Core claim

The central discovery is that routing and answering are separable failures that one end-to-end score would hide. On WorkSurface-Bench, gold-constrained agents—given the names of the required surfaces and only those tools—reach Route F1 of 98.7–99.8 across four backbones, yet Answer accuracy remains between 56.1% and 75.3%, leaving 24.7–43.9% of tasks wrong despite perfect or near-perfect surface selection. The authors also show, via matched interventions, that surface hints (naming the required surfaces while all tools remain available) improve Answer for three of four models, whereas removing irrelevant tools primarily improves routing and efficiency, not the final answer.

What carries the argument

The load-bearing object is the benchmark's scoring protocol: four metrics—Route (F1 over the set of surfaces actually used vs. the required set), Evidence (fraction of required artifacts accessed), Answer (exact or tolerance-matched final correctness), and Efficiency (token budget)—are computed per task from the tool trace and final answer, not from a model-generated gold. The gold-constrained condition (S5) is the key experimental mechanism: it gives the agent the required surface names and restricts tools to those surfaces, isolating the post-routing bottleneck; near-perfect Route with much lower Answer demonstrates that surface selection is decoupled from correct answering.

Load-bearing premise

The benchmark assumes that the gold 'required surfaces' label is correct for every one of the 1,151 tasks; only a stratified 200-item sample received human audit (all passing), while the remaining 951 were validated only by an automated at-least-2-of-3 LLM screening pass, so a wrong necessity label would skew Route F1 and the routing-vs-answering conclusion.

What would settle it

Re-audit the 951 tasks that were not human-checked: if a substantial fraction of their required-surface labels are judged non-necessary by human annotators, the near-perfect Route F1 under gold constraints would be artificially high and the central routing-vs-answering gap would be miscalibrated. Alternatively, re-score the 1,151 tasks with a semantically lenient Answer metric (e.g., accepting correct reasoning despite formatting differences); if the gap collapses, the 'distinct capability' claim is partly an artifact of strict Answer scoring.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Agent evaluations should report surface selection as a separate metric; end-to-end accuracy conflates routing failures with reasoning failures.
  • Perfect routing does not guarantee correct answers: even with gold surface constraints, 24.7–43.9% of tasks are answered incorrectly, meaning downstream acquisition and synthesis are the limiting factors.
  • Surface hints are not uniformly beneficial across models: they improved Answer for three of four tested backbones, with the gains model-dependent.
  • Removing irrelevant tools after a hint improves routing and efficiency, but not consistently answer accuracy, so over-restricting tools is not a fix for reasoning errors.
  • The benchmark's auditable gold answers (executed SQL, verified spans, source graph paths) allow failures to be traced to selection, acquisition, or computation stages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the routing/answer gap generalizes, then research on enterprise agents should shift from router architectures to evidence-verification and multi-surface composition—e.g., models that check whether retrieved evidence actually supports the answer.
  • The benchmark's scoring suggests a concrete testable extension: applying self-consistency or verification passes on the gold-constrained condition might close part of the Answer gap without additional surface selection, which would confirm that the bottleneck is post-routing.
  • The near-zero closed-book answerability (≤2.4%) suggests the benchmark is unlikely to be solvable by memorization, but it does not rule out item-level contamination; a held-out version of the benchmark would be a stronger test.
  • Because the benchmark's difficulty is uneven by surface composition, a fair model comparison needs per-surface reporting; aggregating masks the fact that cross-surface tasks are the hardest for most models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. WorkSurface-Bench introduces a benchmark for evaluating enterprise agents on 'surface routing' — selecting among document, table, and graph knowledge surfaces — separately from end-task answer correctness. The benchmark contains 1,151 atomic tasks derived from Workspace-Bench-Lite workspaces, with gold answers anchored to executed DuckDB queries, verified document spans, and source dependency graph paths. Four model backbones are evaluated under six agent settings (no-tool, always-RAG, naive-router, ReAct-all, gold-constrained, gold-hint/all) yielding 27,624 protocol-error-free trajectories. The central empirical finding is that gold-constrained agents achieve 98.7–99.8 Route F1 while Answer remains 56.1–75.3%, and matched interventions show that surface hints and tool restriction affect routing, answering, and efficiency differently. The paper argues that surface routing and answering are distinct capabilities, and that near-perfect routing is far from sufficient for correct task completion.

Significance. If the central claim holds, the benchmark fills a real gap: existing RAG, tool-use, and agent benchmarks do not score surface selection separately from final answer correctness. The paper's strengths are substantial. The gold-answer construction is unusually auditable: all 453 table-bearing golds are reproduced by executable DuckDB queries, all 601 RAG-bearing golds point to verified spans, and all 600 graph-bearing golds are validated against source dependency annotations. The matched S4/S5/S6 intervention design is a principled way to separate surface information from tool restriction. The release of construction pipeline, scoring code, and agent harness, together with per-trajectory token counts and retry status, supports reproducibility. The decomposed Route/Evidence/Answer/Efficiency protocol is a useful diagnostic contribution for the enterprise-agent evaluation community, provided the gold surface labels are trustworthy at scale.

major comments (3)
  1. [§3.4, §3.5, A.4; used in §5.1 and Table 4]
  2. [Appendix C and Table 4]
  3. [§3.4 and §5.1]
minor comments (3)
  1. [Table 4]
  2. [§3.2]
  3. [Appendix A.6 / §5.6]

Circularity Check

0 steps flagged

No significant circularity: the central routing-vs-answering result is an empirical contrast over proof-carrying golds and controlled interventions, not a fitted or self-referential quantity.

full rationale

The benchmark's gold answers are anchored to executed DuckDB queries, verified text spans, and source graph paths (§3.3–3.4, A.3), and Answer is scored by exact/relative match with no model-based judge (§4). The headline claim—98.7–99.8 Route F1 with Answer 56.1–75.3 under S5—is a controlled intervention, not a prediction: S5 exposes only the gold surfaces, and the paper explicitly calls these conditions 'controls, not evidence of autonomous routing ability' (§6). Route is computed from surfaces actually used rather than assigned perfect by construction (App. B), and Answer is computed independently of Route, so there is no equation-level reduction of the claimed result to its inputs. The most closely related concern—GPT-5.5 both screens candidates (§3.4) and is evaluated (§5.1)—is a possible selection bias, but final acceptance requires a second independent auditor, the golds are proof-carrying, and no formula links screening judgments to the Answer score, so it is not a circular step under the definitional/fitted-input tests. Similarly, if some gold-surface labels are under-inclusive, S5's Route could remain high while Answer is depressed, but that would be a label-validity error rather than a derivation that reduces to its own inputs. The limited human audit of 200/1,151 tasks (§3.5, A.4) is a validity limitation, not a circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No fitted physical parameters; the benchmark's validity rests on inherited source correctness and on scaling a 200-item human audit to 1,151 tasks. Scoring tolerances and aggregate weights are hand-set policy choices, not fitted to data.

free parameters (3)
  • Answer numeric tolerance = 5% relative error; exact for integers with absolute value ≤ 100
    Hand-set scoring rule. The exact Answer percentages in Table 4 depend on this tolerance, though the qualitative routing-answer gap is likely robust to modest changes.
  • Aggregate weights (alpha, beta, gamma, delta) = 0.35, 0.30, 0.25, 0.10
    Hand-set weighting of Answer, Evidence, Route, and Efficiency in the aggregate. The paper explicitly treats component scores as primary outcomes, so this choice does not drive the central claim.
  • Efficiency budget multiplier = 2× the token count of the canonical GPT-4o-mini S5 trace
    Hand-set normalization for the Efficiency score. This affects the secondary Efficiency metric and aggregate, not the central routing-vs-answering result.
axioms (4)
  • domain assumption Workspace-Bench-Lite source tasks, rubrics, and dependency graphs are correct and expert-validated.
    The benchmark inherits all ground-truth content from WSB-Lite; the paper does not independently verify the source corpus, only freezes it.
  • domain assumption The three-surface projection (RAG documents, DuckDB views, file-dependency graph) captures the knowledge forms needed for the tasks.
    Surface-necessity labels depend on this projection; only a 200-item sample is human-audited for necessity.
  • domain assumption LLM screening judgments (GPT-5.5, DeepSeek, Gemini) are reliable for the 951 tasks not human-audited.
    The final 1,151-task set requires a strict pass from at least two of three LLM auditors; human audit covers only the 200-item sample.
  • domain assumption The human majority audit on 200 stratified items generalizes to the full 1,151-task set.
    No human audit is performed on the remaining 951 tasks, yet the full set is released with gold labels.

pith-pipeline@v1.3.0-alltime-deepseek · 12348 in / 10172 out tokens · 97191 ms · 2026-08-01T01:30:53.476085+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing." pith.science (2026). https://pith.science/paper/6ZOCQKI4

@misc{pith2026260725765,
  author       = {Pith},
  title        = {Pith review of: WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ZOCQKI4}},
  note         = {Machine review of arXiv:2607.25765}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Enterprise agents often need to integrate heterogeneous knowledge sources: documents for narrative facts, tables for computation, and dependency graphs for file relationships. Existing benchmarks typically evaluate retrieval or tool use without distinguishing whether an agent first selects the appropriate knowledge sources. We introduce WorkSurface-Bench, a benchmark for evaluating this capability as surface routing. It contains 1,151 atomic tasks derived from persona-scoped Workspace-Bench-Lite workspaces, spanning document, table, graph, and cross-surface questions. Its reference answers are auditable: table answers are reproduced through executed DuckDB queries, document answers are grounded in verified text spans, and graph answers are traced to source dependency annotations. We evaluate four model backbones across six controlled agent settings, yielding 27,624 protocol-error-free trajectories. Under gold-constrained tool access, agents achieve 98.7-99.8 Route F1, while Answer remains only 56.1-75.3 percent, showing that correct surface selection is necessary but insufficient for task completion. Matched interventions further show that surface hints improve Answer for three of four models, whereas removing irrelevant tools primarily improves routing and efficiency. In an independent three-annotator audit, all 200 sampled tasks pass all six quality criteria by majority vote, with 192 receiving unanimous judgments on every criterion. We release the dataset, construction pipeline, scoring code, and agent harness at https://github.com/haolpku/WorkSurface-Bench.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 13 linked inside Pith

  1. [1]

    Hybridqa: A dataset of multi-hop question answering over tabular and textual data

    Wenhu Chen et al. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In Findings of EMNLP, 2020

    2020

  2. [2]

    Open question answering over tables and text

    Wenhu Chen et al. Open question answering over tables and text. InICLR, 2021

    2021

  3. [3]

    T-eval: Evaluating the tool utilization capability of large language models step by step.arXiv preprint arXiv:2312.14033, 2023

    Zehui Chen et al. T-eval: Evaluating the tool utilization capability of large language models step by step.arXiv preprint arXiv:2312.14033, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Benchmarking deep search over heterogeneous enterprise data.arXiv preprint arXiv:2506.23139, 2025

    Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Kung-Hsiang Huang, Caiming Xiong, and Chien- Sheng Wu. Benchmarking deep search over heterogeneous enterprise data.arXiv preprint arXiv:2506.23139, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    DeepSeek V4 Preview Release

    DeepSeek-AI. DeepSeek V4 Preview Release. DeepSeek API documentation, 2026. URLhttps://api-docs. deepseek.com/news/news260424/

    2026

  6. [6]

    Workarena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718, 2024

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, et al. Workarena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Ragbench: Explainable benchmark for retrieval-augmented generation systems.arXiv preprint arXiv:2407.11005, 2024

    Robert Friel et al. Ragbench: Explainable benchmark for retrieval-augmented generation systems.arXiv preprint arXiv:2407.11005, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    2021

  9. [9]

    Gemini 3.1 Pro Preview

    Google. Gemini 3.1 Pro Preview. Gemini API model documentation, 2026. URL https://ai.google.dev/ gemini-api/docs/models/gemini-3.1-pro-preview

    2026

  10. [10]

    Metatool benchmark for large language models: Deciding whether to use tools and which to use

    Yue Huang et al. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    T2-ragbench: Text-and-table benchmark for evaluating retrieval-augmented generation

    Enes Kutay Isgorur, Jan Strich, Maximilian Trescher, Martin Semmann, and Chris Biemann. T2-ragbench: Text-and-table benchmark for evaluating retrieval-augmented generation. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, 2026

    2026

  12. [12]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls

    Jinyang Li et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. In NeurIPS, 2023

    2023

  13. [13]

    Api-bank: A comprehensive benchmark for tool-augmented llms

    Minghao Li et al. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Agentboard: An analytical evaluation board of multi-turn llm agents

    Chang Ma, Junlei Zhang, Zhihao Zhu, et al. Agentboard: An analytical evaluation board of multi-turn llm agents. arXiv preprint arXiv:2401.13178, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    GPT-4o mini: Advancing cost-efficient intelligence

    OpenAI. GPT-4o mini: Advancing cost-efficient intelligence. OpenAI model release, 2024. URL https: //openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  16. [16]

    GPT-5.5 System Card

    OpenAI. GPT-5.5 System Card. OpenAI system card, 2026. URL https://openai.com/index/ gpt-5-5-system-card/. OpenDCAI Technical Report 11

    2026

  17. [17]

    Toolllm: Facilitating large language models to master 16000+ real-world apis

    Yujia Qin, Shihao Liang, Yining Ye, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Multimodalqa: Complex question answering over text, tables and images

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images. In International Conference on Learning Representations, 2021

    2021

  19. [19]

    Workspace-bench 1.0: Benchmarking ai agents on workspace tasks with large-scale file dependencies, 2026

    Zirui Tang, Xuanhe Zhou, et al. Workspace-bench 1.0: Benchmarking ai agents on workspace tasks with large-scale file dependencies, 2026

    2026

  20. [20]

    Appworld: A controllable world of apps and people for benchmarking interactive coding agents.arXiv preprint arXiv:2407.18901, 2024

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, et al. Appworld: A controllable world of apps and people for benchmarking interactive coding agents.arXiv preprint arXiv:2407.18901, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Stark: Benchmarking llm retrieval on semi-structured knowledge bases

    Shirley Wu et al. Stark: Benchmarking llm retrieval on semi-structured knowledge bases. arXiv preprint arXiv:2404.13207, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    mmRAG: A modular benchmark for retrieval- augmented generation over text, tables, and knowledge graphs.arXiv preprint arXiv:2505.11180, 2025

    Chuan Xu, Qiaosheng Chen, Yutong Feng, and Gong Cheng. mmRAG: A modular benchmark for retrieval- augmented generation over text, tables, and knowledge graphs.arXiv preprint arXiv:2505.11180, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Theagentcompany: Benchmarking llm agents on consequential real world tasks.arXiv preprint arXiv:2412.14161, 2024

    Frank F Xu et al. Theagentcompany: Benchmarking llm agents on consequential real world tasks.arXiv preprint arXiv:2412.14161, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    For workspace task 107: what is the total sales amount across the Asia Pacific re- gion?

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, ...

    2018