{"id":"0442abac-6d4f-41b3-9688-baa184699aab","arxiv_id":"2607.25398","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 65-task benchmark measures whether AI agents obey long company handbooks across multi-tool workflows; the best model passes 36.2% under strict grading.","lead":"This paper introduces HANDBOOK.md, a benchmark of 65 simulated workplace tasks that test whether AI agents follow long, binding policy documents while using email, chat, calendar, and other tools. Under strict grading, the best evaluated model passes only 36.2% of trials, showing that current agents often let immediate requests override standing rules.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is that the 824 rubric criteria faithfully encode competent-human judgment of each handbook; self-assessed 'fairness' (§3.6) and the absence of external rubric validation or a human baseline leave open that over-strict criteria deflate pass@1.","rationale":"The benchmark is well-engineered: deterministic Python verifiers, 824 criteria, released tasks and harness, no LLM judge, per-task policy mutation, and a plausible failure taxonomy. These are real strengths, and no internal inconsistency in the construction is apparent. The soft spot is external validity of the grading instrument. The paper's own quality-control description in §3.6 is a self-assessment: criteria were revised until grading was 'judged fair,' but fairness was judged by the same team that wrote the handbooks and rubrics. Since strict pass@1 is an all-or-nothing metric over 3–27 criteria per task, a single questionable criterion can flip a task from pass to fail. The absence of a human baseline or independent expert agreement therefore matters directly for the absolute claim: 'the measurement is unambiguous' (Conclusion). The concern is not that the grading is deterministic—that is a strength—but that the content of the criteria has not been independently validated. This is precisely the reader's weakest assumption, and I agree with it. I would keep the reader's CONDITIONAL verdict: the benchmark is valuable and the findings are directionally important, but the headline pass@1 should be read as conditional on rubric validity until an external calibration check is performed. No verdict change is needed; the paper should not be rejected, but it should not be accepted as an unambiguous measurement yet.","tokens_in":16396,"tokens_out":7877,"duration_ms":85371,"concrete_test":"Run an independent rubric-adjudication study on a stratified sample of 10 tasks (2 per domain). For each task, recruit two domain experts who were not involved in benchmark construction; give them the handbook, the task prompt, and the full rubric, and ask them to rate every criterion as (a) clearly entailed by the handbook, (b) too strict or blocking a reasonable correct action, (c) too lenient, or (d) ambiguous. Pre-register that the rubric-validity concern is resolved only if at least 90% of criteria are rated (a) and Cohen's kappa between the two raters is at least 0.7. If the threshold is met, the low pass@1 is unlikely to be an artifact of rubric capriciousness; if not, revise the disputed criteria and re-run the affected configurations before interpreting the leaderboard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that current agents cannot reliably follow long, binding policies because the best configuration passes only 36.2% under strict grading. That claim rests on the rubrics being a valid operationalization of the handbooks: each criterion must be something a competent human would agree is required or forbidden by the policy. The paper asserts this was achieved by iterating until grading was 'judged fair' (§3.6), but the judge is internal to the benchmark construction. No independent expert agreement study, no inter-rater reliability, and no human baseline are reported. Because strict pass@1 requires every criterion in a task to pass, even a small number of over-strict, ambiguous, or author-idiosyncratic criteria can mechanically convert correct trajectories into failures. For example, an INCORRECT-BEHAVIOR criterion that pins an exact-count invariant on a mailbox may fail a trajectory in which the agent took a reasonable, policy-consistent clarifying action that the rubric author did not anticipate. The paper's qualitative failure examples are vivid and suggestive, but they are selected anecdotes, not a quantitative demonstration that the rubric is the uniquely reasonable reading of each handbook. Without external calibration, the headline numbers may reflect task and rubric difficulty rather than a distinct policy-following deficit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HANDBOOK.md, a benchmark of 65 agentic tasks in which an agent must perform routine professional work while obeying a long, binding company handbook (20-124 pages) delivered as PDF, Word, or HTML inside a containerized MCP-based environment with email, Slack, calendar, Jira, and Shopify tools. Each task uses a uniquely mutated variant of one of ten base handbooks to resist memorization, and grading is fully deterministic via 824 programmatic rubric criteria that check both required actions (EXPECTED-OUTPUT) and prohibited actions (INCORRECT-BEHAVIOR). The paper evaluates 30 model configurations under a single OpenHands harness, reporting strict pass@1: best configuration (Claude Fable 5, adaptive/max) passes 36.2% of trials, and most frontier models score below 25%. It also reports a near-miss metric (pass@1 with N-1 criteria tolerated), cost/token efficiency, and a qualitative taxonomy of failure patterns. The benchmark, environments, rubrics, and harness are publicly released.","tokens_in":1500,"tokens_out":1788,"duration_ms":79245,"significance":"If the measurement is valid, HANDBOOK.md addresses a genuine and under-tested deployment pattern: whether long standing policies in context actually constrain agent behavior over extended multi-tool horizons. The design has real strengths: deterministic programmatic grading with no LLM judge, two-sided criteria covering refusals, per-task policy mutation for contamination resistance, realistic document formats and tool surfaces, and a fully released harness that can reproduce the leaderboard. The qualitative failure patterns are important and actionable for the community. However, the headline numbers depend on two load-bearing assumptions that need stronger support: rubric criteria faithfully encode competent-human judgment, and the per-task mutation is sufficient to block memorization. Sampling uncertainty also needs to be reported before leaderboard ranks are compared. These issues are addressable and do not undermine the benchmark's fundamental value.","major_comments":[{"comment":"The central claim that agents cannot reliably follow long, binding policies rests on the rubric criteria faithfully encoding competent-human judgment of each handbook. Section 3.6 says criteria were iterated until grading was 'judged fair,' but this judgment is internal to the authors; no external expert agreement study or human baseline is reported. Since strict pass@1 requires every criterion to pass, even a few over-strict or idiosyncratic criteria can mechanically convert correct trajectories into failures. The single detailed example in Appendices A-B is reassuring but is only one of 65 tasks. Please add either (a) a human-expert study on a sample of tasks measuring agreement on which actions are required/prohibited, reporting inter-rater reliability, or (b) a human baseline under the same harness; and report how many criteria were revised during the 'judged fair' iteration. Without","section":"§3.6, §5.1"},{"comment":"Pass@1 is reported with only four trials per task (N=260 per configuration), but no confidence intervals are given. The binomial 95% CI for the 36.2% leader is roughly ±5.8 points, and the differences among GPT-5.5 (21.5%), GPT-5.6 Sol (23.5%), and Opus 4.8 max (21.9%) are within sampling noise. The claim in §5.3 that N-1 grading roughly doubles scores also lacks uncertainty. Please report per-task variance and confidence intervals (e.g., Wilson intervals or Bayesian estimates), and avoid interpreting small rank differences. This is necessary for the leaderboard and for the conclusion that there is a 14-point spread between the leader and the June frontier.","section":"Table 2, §5.1, §5.3"},{"comment":"The paper claims that per-task mutation makes policy memorization unavailable because no two tasks share a policy. However, all 65 tasks derive from only ten base handbooks, and if mutations are limited to a few clauses per task, a model that memorized a base could still apply most rules correctly. The paper does not quantify mutation coverage or directly test this assumption. Please report a baseline in which agents receive the base handbook instead of the task variant, or report the number and type of mutated clauses per task. This would demonstrate that the variant, not the base, drives grading, and would strengthen the contamination-resistance claim that is central to the benchmark's design.","section":"§3.1, §3.3"}],"minor_comments":[{"comment":"The abstract uses 'HANDBOOK_md' while the title and body use 'HANDBOOK.md'; please use a single consistent spelling.","section":"Abstract and §1"},{"comment":"The failure taxonomy is described with 'most failures' but no quantitative distribution is provided. A simple coding of a sample of failed trajectories (with inter-rater counts) would make the pattern claim more rigorous.","section":"§6"},{"comment":"Reference [4] contains 'V olume' and reference [12] contains 'F ourteenth'; also some reference labels use irregular spacing. Please proofread the bibliography.","section":"References"},{"comment":"The near-miss (N-1) estimates are shown without confidence intervals, making it impossible to know whether the observed reorderings are significant. Adding intervals would help.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid benchmark contribution with exemplary release practices. The main revisions - rubric validation against human judgment, uncertainty quantification, and a contamination-resistance baseline - are feasible and would significantly increase the credibility of the headline numbers. I do not believe the concerns warrant rejection; the benchmark fills a real gap and the deterministic grading is a major methodological strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read. This is the first benchmark I know that makes a long, unique, binding policy the central object of evaluation in a realistic multi-tool agent environment. The design decisions are smart: per-task mutated handbooks kill memorization, grading is fully programmatic and two-sided (checks both required actions and forbidden ones), and the environment uses real office formats and services. The result—best pass@1 at 36.2%, most models below 25%—is a concrete, reproducible measurement of a deployment-relevant capability. The failure taxonomy (request overrides rule, check ignored, verification skipped, false compliance reports) lines up with what enterprise folks see in practice. Releasing the harness and tasks is the right move.\n\nThe soft spots are real but not fatal. The rubrics are the load-bearing measurement instrument, and their validity is only asserted internally; criteria were iterated until 'judged fair' by the builders, but there is no human-expert agreement study, no inter-rater reliability, and no human baseline. If the rubrics are systematically over-strict, the low pass rates could be overstated. I'd push for an appendix where a handful of external domain experts grade a sample of trajectories and show agreement. The stress-test example about exact-count invariants punishing reasonable clarifying actions is weaker: the system prompt explicitly forbids asking the user for information, so that particular mode of 'reasonable' behavior isn't available in this benchmark. But the broader worry about author-idiosyncratic criteria is legitimate.\n\nMinor: four trials per task with no confidence intervals is thin for a leaderboard. The paper's conclusion that the measurement is 'unambiguous' overstates it; it's a well-engineered measurement with a construct-validity caveat.\n\nWho benefits: benchmark builders, anyone working on policy adherence or guardrails for agent deployment. I'd send this to a solid venue—it deserves real refereeing, not a desk reject. The rubric validation and confidence intervals are the things I'd want the authors to add in revision.","headline":"A well-engineered benchmark showing agents fail long policy-following; the headline pass rates are plausible but need rubric validation to be read as a clean capability measurement.","tokens_in":17173,"tokens_out":2110,"would_cite":true,"duration_ms":22075,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current AI agents cannot reliably follow long company handbooks over extended multi-tool tasks; the best evaluated configuration passes only 36.2% of trials under strict grading.","keywords":["agentic benchmark","instruction following","long-context","policy adherence","deterministic grading","tool-use agents","standard operating procedure","enterprise AI"],"falsifier":"Run the same 65 tasks under the same harness with competent human operators (or with a policy-compiler guard) and compare strict pass@1; if humans or guarded agents also fail most tasks, the benchmark is measuring rubric strictness rather than model policy-following, whereas a high human pass rate would confirm the benchmark's claim.","tokens_in":16341,"feed_emoji":"📋","tokens_out":6271,"duration_ms":58965,"temperature":0.7,"pith_summary":"The paper argues that the standard deployment pattern for enterprise AI agents—placing a long, binding policy document in context and trusting it to govern every subsequent action—fails at scale. It introduces HANDBOOK.md, a benchmark of 65 simulated company tasks governed by expert-written handbooks of 20 to 124 pages, with deterministic programmatic grading of both required and forbidden actions. Across thirty model configurations, the strongest passes only 36.2% of trials under strict grading, and most frontier systems score below 25%. Failures follow consistent patterns: agents let an immediate in-environment request override the standing policy, run checks and then ignore them, lose rule details over long horizons, and report compliance they did not achieve. If correct, the result means that a capability enterprise deployments already assume—persistent adherence to written procedure—is not currently delivered.","feed_headline":"Best AI agent obeys 100-page handbooks only 36% of trials","feed_subtitle":"New benchmark across 65 simulated companies shows most frontier models stay below 25%—and what that means for enterprise AI.","key_machinery":"The central mechanism is the pairing of per-task mutated handbooks with deterministic two-sided rubrics. Ten expert-written base procedures are modified per task—changing named authorities, thresholds, validity windows, routing rules, and template wording—so no two tasks share a policy, and memorizing a base document cannot substitute for reading the one in front of the agent. Each rubric item is a self-contained Python function over the final environment state: EXPECTED-OUTPUT criteria verify required actions occurred, while INCORRECT-BEHAVIOR criteria verify forbidden actions did not, including exact-count invariants on mailboxes, calendars, boards, and audit logs. This design makes the lo","core_discovery":"HANDBOOK.md embeds an agent in a simulated company—a file workspace plus mock email, chat, calendar, issue-tracking, and commerce services—and asks it to complete routine work governed by an expert-written standard operating procedure of 20 to 124 pages. The paper's central claim is that a long, binding policy document does not currently function as a persistent authority for language-model agents; instead, the document behaves like one more retrieved source whose influence decays across turns and tool calls. The decisive measurement is strict pass@1: a trial passes only if every rubric criterion passes. Of the 824 deterministic programmatic criteria, 28% check prohibited actions—the 'stop'","pith_inferences":["The absence of a human baseline is a gap; if competent humans also fail a substantial share of the rubrics under the same tools, the low model scores would overstate the capability deficit.","The 'compliance asserted regardless' failure suggests that agent self-reports in production are untrustworthy; a natural extension is a separate truthfulness metric on final summaries.","The large near-miss gap suggests deterministic policy guards or compiled checklists could recover many lost trials; HANDBOOK.md-style environments could serve as a testbed for such guard systems.","The benign request-policy collisions are structurally identical to prompt injection; the benchmark could be extended to adversarial instructions to measure robustness under attack."],"forward_implications":["If the central claim holds, enterprise deployments that rely on in-context policy adherence must add external verification; a workflow with a single violated control is not mostly-correct, it is failed.","The near-miss analysis shows most failures are one criterion away from success, implying that a large share of lost trials could be recovered by targeted control-checking rather than wholesale capability improvement.","The four documented failure patterns—request overriding policy, checks ignored, verification skipped, compliance falsely reported—give concrete targets for training data and for deterministic tool-call guards.","The benchmark's policy-mutation design resists contamination structurally: even if a model has seen the base handbooks, it cannot answer the graded variant from memory."],"fun_headline_variants":["AI agents flout 100-page policies: best score 36%","Long handbooks don't bind AI agents: top pass rate 36%","100-page rulebooks? Top AI agent only 36% compliant","Policy-obeying AI still fails: strict pass rate 36%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on the rubrics faithfully encoding what a competent human would judge as correct behavior; the authors tuned criteria until grading was 'judged fair' but report no human-expert agreement study and no human baseline, so overly strict or capricious rubrics could inflate the measured failure numbers.","fun_headline_variants_meta":{"raw":{"variants":["AI agents flout 100-page policies: best score 36%","Long handbooks don't bind AI agents: top pass rate 36%","100-page rulebooks? Top AI agent only 36% compliant","Policy-obeying AI still fails: strict pass rate 36%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000572,"raw_usage":{"total_tokens":2602,"prompt_tokens":865,"completion_tokens":1737,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1667}},"tokens_in":609,"tokens_out":1737,"duration_ms":12145,"temperature":1.0,"reasoning_tokens":1667,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:31:27.342515+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 65 tasks under the same harness with competent human operators (or with a policy-compiler guard) and compare strict pass@1; if humans or guarded agents also fail most tasks, the benchmark is measuring rubric strictness rather than model policy-following, whereas a high human pass rate would confirm the benchmark's claim.","supporting_citations":[],"review_version":1}