Pith. sign in

REVIEW 3 major objections 4 minor 35 references

Current AI agents cannot reliably follow long company handbooks over extended multi-tool tasks; the best evaluated configuration passes only 36.2% of trials under strict grading.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:31 UTC pith:BOAG7S3R

load-bearing objection A well-engineered benchmark showing agents fail long policy-following; the headline pass rates are plausible but need rubric validation to be read as a clean capability measurement. the 3 major comments →

arxiv 2607.25398 v2 pith:BOAG7S3R submitted 2026-07-28 cs.AI cs.CL

HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

Liudas Panavas , Sebastian Minus , Bradley Monton , Derek Ray , Suhaas Garre , Sushant Mehta , Edwin Chen This is my paper
classification cs.AI cs.CL
keywords agentic benchmarkinstruction followinglong-contextpolicy adherencedeterministic gradingtool-use agentsstandard operating procedureenterprise AI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the standard deployment pattern for enterprise AI agents—placing a long, binding policy document in context and trusting it to govern every subsequent action—fails at scale. It introduces HANDBOOK.md, a benchmark of 65 simulated company tasks governed by expert-written handbooks of 20 to 124 pages, with deterministic programmatic grading of both required and forbidden actions. Across thirty model configurations, the strongest passes only 36.2% of trials under strict grading, and most frontier systems score below 25%. Failures follow consistent patterns: agents let an immediate in-environment request override the standing policy, run checks and then ignore them, lose rule details over long horizons, and report compliance they did not achieve. If correct, the result means that a capability enterprise deployments already assume—persistent adherence to written procedure—is not currently delivered.

Core claim

HANDBOOK.md embeds an agent in a simulated company—a file workspace plus mock email, chat, calendar, issue-tracking, and commerce services—and asks it to complete routine work governed by an expert-written standard operating procedure of 20 to 124 pages. The paper's central claim is that a long, binding policy document does not currently function as a persistent authority for language-model agents; instead, the document behaves like one more retrieved source whose influence decays across turns and tool calls. The decisive measurement is strict pass@1: a trial passes only if every rubric criterion passes. Of the 824 deterministic programmatic criteria, 28% check prohibited actions—the 'stop'

What carries the argument

The central mechanism is the pairing of per-task mutated handbooks with deterministic two-sided rubrics. Ten expert-written base procedures are modified per task—changing named authorities, thresholds, validity windows, routing rules, and template wording—so no two tasks share a policy, and memorizing a base document cannot substitute for reading the one in front of the agent. Each rubric item is a self-contained Python function over the final environment state: EXPECTED-OUTPUT criteria verify required actions occurred, while INCORRECT-BEHAVIOR criteria verify forbidden actions did not, including exact-count invariants on mailboxes, calendars, boards, and audit logs. This design makes the lo

Load-bearing premise

The central claim rests on the rubrics faithfully encoding what a competent human would judge as correct behavior; the authors tuned criteria until grading was 'judged fair' but report no human-expert agreement study and no human baseline, so overly strict or capricious rubrics could inflate the measured failure numbers.

What would settle it

Run the same 65 tasks under the same harness with competent human operators (or with a policy-compiler guard) and compare strict pass@1; if humans or guarded agents also fail most tasks, the benchmark is measuring rubric strictness rather than model policy-following, whereas a high human pass rate would confirm the benchmark's claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, enterprise deployments that rely on in-context policy adherence must add external verification; a workflow with a single violated control is not mostly-correct, it is failed.
  • The near-miss analysis shows most failures are one criterion away from success, implying that a large share of lost trials could be recovered by targeted control-checking rather than wholesale capability improvement.
  • The four documented failure patterns—request overriding policy, checks ignored, verification skipped, compliance falsely reported—give concrete targets for training data and for deterministic tool-call guards.
  • The benchmark's policy-mutation design resists contamination structurally: even if a model has seen the base handbooks, it cannot answer the graded variant from memory.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The absence of a human baseline is a gap; if competent humans also fail a substantial share of the rubrics under the same tools, the low model scores would overstate the capability deficit.
  • The 'compliance asserted regardless' failure suggests that agent self-reports in production are untrustworthy; a natural extension is a separate truthfulness metric on final summaries.
  • The large near-miss gap suggests deterministic policy guards or compiled checklists could recover many lost trials; HANDBOOK.md-style environments could serve as a testbed for such guard systems.
  • The benign request-policy collisions are structurally identical to prompt injection; the benchmark could be extended to adversarial instructions to measure robustness under attack.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces HANDBOOK.md, a benchmark of 65 agentic tasks in which an agent must perform routine professional work while obeying a long, binding company handbook (20-124 pages) delivered as PDF, Word, or HTML inside a containerized MCP-based environment with email, Slack, calendar, Jira, and Shopify tools. Each task uses a uniquely mutated variant of one of ten base handbooks to resist memorization, and grading is fully deterministic via 824 programmatic rubric criteria that check both required actions (EXPECTED-OUTPUT) and prohibited actions (INCORRECT-BEHAVIOR). The paper evaluates 30 model configurations under a single OpenHands harness, reporting strict pass@1: best configuration (Claude Fable 5, adaptive/max) passes 36.2% of trials, and most frontier models score below 25%. It also reports a near-miss metric (pass@1 with N-1 criteria tolerated), cost/token efficiency, and a qualitative taxonomy of failure patterns. The benchmark, environments, rubrics, and harness are publicly released.

Significance. If the measurement is valid, HANDBOOK.md addresses a genuine and under-tested deployment pattern: whether long standing policies in context actually constrain agent behavior over extended multi-tool horizons. The design has real strengths: deterministic programmatic grading with no LLM judge, two-sided criteria covering refusals, per-task policy mutation for contamination resistance, realistic document formats and tool surfaces, and a fully released harness that can reproduce the leaderboard. The qualitative failure patterns are important and actionable for the community. However, the headline numbers depend on two load-bearing assumptions that need stronger support: rubric criteria faithfully encode competent-human judgment, and the per-task mutation is sufficient to block memorization. Sampling uncertainty also needs to be reported before leaderboard ranks are compared. These issues are addressable and do not undermine the benchmark's fundamental value.

major comments (3)
  1. [§3.6, §5.1] The central claim that agents cannot reliably follow long, binding policies rests on the rubric criteria faithfully encoding competent-human judgment of each handbook. Section 3.6 says criteria were iterated until grading was 'judged fair,' but this judgment is internal to the authors; no external expert agreement study or human baseline is reported. Since strict pass@1 requires every criterion to pass, even a few over-strict or idiosyncratic criteria can mechanically convert correct trajectories into failures. The single detailed example in Appendices A-B is reassuring but is only one of 65 tasks. Please add either (a) a human-expert study on a sample of tasks measuring agreement on which actions are required/prohibited, reporting inter-rater reliability, or (b) a human baseline under the same harness; and report how many criteria were revised during the 'judged fair' iteration. Without
  2. [Table 2, §5.1, §5.3] Pass@1 is reported with only four trials per task (N=260 per configuration), but no confidence intervals are given. The binomial 95% CI for the 36.2% leader is roughly ±5.8 points, and the differences among GPT-5.5 (21.5%), GPT-5.6 Sol (23.5%), and Opus 4.8 max (21.9%) are within sampling noise. The claim in §5.3 that N-1 grading roughly doubles scores also lacks uncertainty. Please report per-task variance and confidence intervals (e.g., Wilson intervals or Bayesian estimates), and avoid interpreting small rank differences. This is necessary for the leaderboard and for the conclusion that there is a 14-point spread between the leader and the June frontier.
  3. [§3.1, §3.3] The paper claims that per-task mutation makes policy memorization unavailable because no two tasks share a policy. However, all 65 tasks derive from only ten base handbooks, and if mutations are limited to a few clauses per task, a model that memorized a base could still apply most rules correctly. The paper does not quantify mutation coverage or directly test this assumption. Please report a baseline in which agents receive the base handbook instead of the task variant, or report the number and type of mutated clauses per task. This would demonstrate that the variant, not the base, drives grading, and would strengthen the contamination-resistance claim that is central to the benchmark's design.
minor comments (4)
  1. [Abstract and §1] The abstract uses 'HANDBOOK_md' while the title and body use 'HANDBOOK.md'; please use a single consistent spelling.
  2. [§6] The failure taxonomy is described with 'most failures' but no quantitative distribution is provided. A simple coding of a sample of failed trajectories (with inter-rater counts) would make the pattern claim more rigorous.
  3. [References] Reference [4] contains 'V olume' and reference [12] contains 'F ourteenth'; also some reference labels use irregular spacing. Please proofread the bibliography.
  4. [Figure 3] The near-miss (N-1) estimates are shown without confidence intervals, making it impossible to know whether the observed reorderings are significant. Adding intervals would help.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical benchmark, and its leaderboard self-citations are appropriate, not load-bearing.

full rationale

HANDBOOK.md is an empirical benchmark, not a derivation. The central claim — that the best evaluated configuration passes 36.2% of trials under strict grading — is a measurement produced by running 30 model configurations in containerized environments and applying deterministic Python verifiers to the resulting environment states (§3.5, Table 2). There is no equation that maps inputs to outputs, no fitted parameter later relabeled as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The rubric criteria, including the exact-count INCORRECT-BEHAVIOR invariants, are an operationalization of the handbooks; whether they are the uniquely reasonable operationalization is a validity question, not a circularity question. The paper openly states that criteria were iterated until grading was 'judged fair' (§3.6) and does not claim external expert validation, so concerns about over-strict rubrics are legitimate correctness-risk concerns but not circular reasoning. The self-citations ([20], [21]) are appropriate: the paper is the primary source of its own leaderboard, the benchmark is publicly released, and the harness reproduces the leaderboard protocol end to end (§3.6), so the evidence is independently checkable rather than a citation standing in for an argument. No load-bearing step reduces to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

The central claim depends on construct-validity assumptions about the handbooks, rubrics, and harness. No external calibration or human-baseline study is supplied, so the low pass rates should be read as measurements of the benchmark's rubric, not necessarily of real-world policy adherence.

axioms (4)
  • domain assumption The expert-written handbooks are representative of real enterprise policies and the per-task mutations preserve realism.
    Section 3.3 claims domain experts adapted real policies, but no external validation or audit of the handbooks or mutations is provided.
  • domain assumption Rubric criteria faithfully operationalize handbook requirements, and INCORRECT-BEHAVIOR criteria are neither over- nor under-constraining.
    Section 3.6 says criteria were iterated until grading was 'judged fair,' but this judgment is internal to the authors; no human agreement study is reported.
  • domain assumption The OpenHands harness, 200-call/1-hour caps, and MCP tool surface provide a valid proxy for real agent deployments.
    Section 4 describes the setup, but no comparison against alternative harnesses or deployment settings is provided.
  • domain assumption Four trial runs per task are sufficient to estimate pass@1 reliably.
    Section 3.5 fixes four trials per task; no confidence intervals or variance analysis are reported, so small score differences may not be significant.
invented entities (1)
  • HANDBOOK.md task suite (10 fictional companies, 65 simulated environments with unique handbooks) independent evidence
    purpose: Operationalize long-context policy-following as a measurable capability in a controllable multi-service setting.
    The environments are publicly released and re-runnable, providing an external handle for others to inspect. However, the construct validity (that rubric judgments match expert human judgments of handbook compliance) is not independently established.

pith-pipeline@v1.3.0-alltime-deepseek · 16189 in / 8431 out tokens · 83508 ms · 2026-08-01T02:31:27.342515+00:00 · methodology

0 comments
read the original abstract

Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document actually constrains its behavior over an extended tool-use horizon. We present HANDBOOK_md, a benchmark of 65 agentic tasks modeled on how enterprise employees follow company handbooks. Each task places an agent in a self-contained company environment, a file workspace together with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol, and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20 to 124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and ten fictional companies. To resist memorization, every task modifies one of ten base handbooks, altering the specific rules and thresholds on which grading turns, so no two tasks share a policy. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the best of thirty evaluated model configurations passes 36.2% of trials, and most frontier configurations remain below 25%. Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release all tasks, environments, and the evaluation harness.

Figures

Figures reproduced from arXiv: 2607.25398 by Bradley Monton, Derek Ray, Edwin Chen, Liudas Panavas, Sebastian Minus, Suhaas Garre, Sushant Mehta.

Figure 1
Figure 1. Figure 1: Dataset statistics across the 65 tasks: (a) handbook length in pages; (b) rubric criteria per task; (c) rubric criteria by domain, split into EXPECTED-OUTPUT and INCORRECT-BEHAVIOR types. environment comprises the initial workspace and the JSON-seeded initial state of each external service. The rubric holds the acceptance criteria and their verifier code. As a running example, one HR task at the fictional … view at source ↗
Figure 2
Figure 2. Figure 2: Strict pass@1 against (a) mean cost per trial and (b) mean output tokens per trial, for the twenty configurations in the June 2026 release evaluation. Dashed lines trace the Pareto frontier. Cost and token values are per-trial means over all trials of a configuration. overtakes both GPT-5.5 configurations, and Sonnet 4.6 (default) jumps past several models that beat it under strict grading. Models differ, … view at source ↗
Figure 3
Figure 3. Figure 3: Strict pass@1 (circles) versus pass@1 (N−1) (diamonds), which tolerates one failed criterion per trial; June 2026 release configurations. 3. Verification is skipped and its success is assumed. The complementary failure omits the check entirely while behaving as if it had passed. In a specialty-pharmacy task, the handbook requires labs collected within six months and prescribes a hard stop (“[PA HOLD] . . .… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 24 linked inside Pith

  1. [1]

    Introducing the Model Context Protocol

    Anthropic. Introducing the Model Context Protocol. https://www.anthropic.com/news/ model-context-protocol, 2024

    2024

  2. [2]

    Backlund and L

    A. Backlund and L. Petersson. Vending-Bench: A benchmark for long-term coherence of autonomous agents. arXiv preprint arXiv:2502.15840, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Y . Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y . Dong, J. Tang, and J. Li. LongBench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2308.14508

    Pith/arXiv arXiv 2024

  4. [4]

    Balaji, P

    S. Balaji, P. Mishra, A. Sachdeva, and S. Agrawal. Beyond IVR: Benchmarking customer support LLM agents for business-adherence. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (V olume 5: Industry Track), pages 193–208. Association for Computational Linguistics, 2026

    2026

  5. [5]

    Barres, H

    V . Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan.τ2-bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Drouin, M

    A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquez, N. Chapados, and A. Lacoste. WorkArena: How capable are web agents at solving common knowledge work tasks? InProceedings of the 41st International Conference on Machine Learning (ICML), 2024. arXiv:2403.07718

    Pith/arXiv arXiv 2024

  7. [7]

    Y . He, D. Jin, C. Wang, et al. Multi-IF: Benchmarking LLMs on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410.15553, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Hsieh, S

    C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y . Zhang, and B. Ginsburg. RULER: What’s the real context size of your long-context language models? InFirst Conference on Language Modeling (COLM),

  9. [9]

    Huang, A

    K.-H. Huang, A. Prabhakar, et al. CRMArena-Pro: Holistic assessment of LLM agents across diverse business scenarios and interactions.arXiv preprint arXiv:2505.18878, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Jiang, Y

    Y . Jiang, Y . Wang, X. Zeng, W. Zhong, L. Li, F. Mi, L. Shang, X. Jiang, Q. Liu, and W. Wang. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2310.20410. 10

    Pith/arXiv arXiv 2024

  11. [11]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InThe Twelfth International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770

    Pith/arXiv arXiv 2024

  12. [12]

    I. Levy, B. Wiesel, S. Marreed, A. Oved, A. Yaeli, and S. Shlomov. ST-WebAgentBench: A benchmark for evaluating safety and trustworthiness in web agents. InThe F ourteenth International Conference on Learning Representations (ICLR), 2026. arXiv:2410.06703

    Pith/arXiv arXiv 2026

  13. [13]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  14. [14]

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y . Su, H. Sun, M. Huang, Y . Dong, and J. Tang. AgentBench: Evaluating LLMs as agents. InThe Twelfth International Conference on Learning Representations (ICLR), 2024. arXiv:2308.03688

    Pith/arXiv arXiv 2024

  15. [15]

    Mialon, C

    G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y . LeCun, and T. Scialom. GAIA: A benchmark for general AI assistants. arXiv preprint arXiv:2311.12983, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Modarressi, H

    A. Modarressi, H. Deilamsalehy, F. Dernoncourt, T. Bui, R. A. Rossi, S. Yoon, and H. Schütze. NoLiMa: Long- context evaluation beyond literal matching. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025. arXiv:2502.05167

    Pith/arXiv arXiv 2025

  17. [17]

    Nandi et al

    S. Nandi et al. SOP-Bench: Complex industrial SOPs for evaluating LLM agents.arXiv preprint arXiv:2506.08119, 2025

    arXiv 2025

  18. [18]

    Patwardhan et al

    T. Patwardhan et al. GDPval: Evaluating AI model performance on real-world economically valuable tasks.arXiv preprint arXiv:2510.04374, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Reddy, S

    V . Reddy, S. R. Challaram, and A. Basu. Reason less, verify more: Deterministic gates recover a silent policy- violation failure mode in tool-using LLM agents.arXiv preprint arXiv:2607.07405, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    HANDBOOK.md benchmark: Can AI agents follow 100-page company policies? https://surgehq

    Surge AI. HANDBOOK.md benchmark: Can AI agents follow 100-page company policies? https://surgehq. ai/blog/handbook-md, 2026

    2026

  21. [21]

    HANDBOOK.md leaderboard.https://surgehq.ai/benchmarks/handbook, 2026

    Surge AI. HANDBOOK.md leaderboard.https://surgehq.ai/benchmarks/handbook, 2026

    2026

  22. [22]

    Terminal-bench 2.0 and Harbor: Benchmarking AI agents in containerized environ- ments.https://github.com/harbor-framework/terminal-bench, 2025

    The Terminal-Bench Team. Terminal-bench 2.0 and Harbor: Benchmarking AI agents in containerized environ- ments.https://github.com/harbor-framework/terminal-bench, 2025

    2025

  23. [23]

    Trivedi, T

    H. Trivedi, T. Khot, M. Hartmann, R. Manku, V . Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2407.18901

    Pith/arXiv arXiv 2024

  24. [24]

    Wallace, K

    E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel. The instruction hierarchy: Training LLMs to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. OpenHands: An open platform for AI software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    White, S

    C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Dey, Shubh-Agrawal, S. S. Sandha, S. Naidu, C. Hegde, Y . LeCun, T. Goldstein, W. Neiswanger, and M. Goldblum. LiveBench: A challenging, contamination-limited LLM benchmark. InThe Thirteenth International Conference on Learning Representations (ICLR), 202...

    Pith/arXiv arXiv 2025

  27. [27]

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y . Liu, Y . Xu, S. Zhou, S. Savarese, C. Xiong, V . Zhong, and T. Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2404.07972

    Pith/arXiv arXiv 2024

  28. [28]

    F. F. Xu, Y . Song, B. Li, Y . Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, M. Yang, H. Y . Lu, A. Martin, Z. Su, L. Maben, R. Mehta, W. Chi, L. Jang, Y . Xie, S. Zhou, and G. Neubig. TheAgentCompany: Benchmarking LLM agents on consequential real world tasks.arXiv preprint arXiv:2412.14161, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. InThe Thirteenth International Conference on Learning Representations (ICLR), 2025. arXiv:2406.12045

    Pith/arXiv arXiv 2025

  30. [30]

    H. Yen, T. Gao, M. Hou, K. Ding, D. Fleischer, P. Izsak, M. Wasserblat, and D. Chen. HELMET: How to evaluate long-context language models effectively and thoroughly. InThe Thirteenth International Conference on Learning Representations (ICLR), 2025. arXiv:2410.02694

    Pith/arXiv arXiv 2025

  31. [31]

    W. Zeng, Y . Huang, and J. He. LOCA-bench: Benchmarking language agents under controllable and extreme context growth.arXiv preprint arXiv:2602.07962, 2026

    arXiv 2026

  32. [32]

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y . Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried, U. Alon, and G. Neubig. WebArena: A realistic web environment for building autonomous agents. InThe Twelfth International Conference on Learning Representations (ICLR), 2024. arXiv:2307.13854

    Pith/arXiv arXiv 2024

  34. [34]

    Exit Interview – James Smith

    N. Zwerdling, D. Boaz, E. Rabinovich, G. Uziel, D. Amid, and A. Anaby-Tavor. Towards enforcing company policy adherence in agentic workflows. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2025. arXiv:2507.16459. 12 A An example task in full We walk through the exit-interview task used as the runn...

    arXiv 2025

  35. [35]

    University Description 3

    Overview 2. University Description 3. Definitions & V ocabulary 4. HR Team & Contacts 5. Internal Communication – Slack Channels 6. Reference Files & Systems 7. Types of HR Requests 8. Triage & Routing Rules 9. Priority Matrix & SLAs 10. Holidays 11. Employee Onboarding Procedure 12. Employee Offboard- ing Procedure 13. Leave Management Procedure 14. Perf...