REVIEW 3 major objections 4 minor
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read Current AI agents cannot reliably follow long company handbooks over extended multi-tool tasks; the best evaluated configuration passes only 36.2% of trials under strict grading.
desk verdict A well-engineered benchmark showing agents fail long policy-following; the headline pass rates are plausible but need rubric validation to be read as a clean capability measurement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the pairing of per-task mutated handbooks with deterministic two-sided rubrics. Ten expert-written base procedures are modified per task—changing named authorities, thresholds, validity windows, routing rules, and template wording—so no two tasks share a policy, and memorizing a base document cannot substitute for reading the one in front of the agent. Each rubric item is a self-contained Python function over the final environment state: EXPECTED-OUTPUT criteria verify required actions occurred, while INCORRECT-BEHAVIOR criteria verify forbidden actions did not, including exact-count invariants on mailboxes, calendars, boards, and audit logs. This design makes the lo
What would settle it
Run the same 65 tasks under the same harness with competent human operators (or with a policy-compiler guard) and compare strict pass@1; if humans or guarded agents also fail most tasks, the benchmark is measuring rubric strictness rather than model policy-following, whereas a high human pass rate would confirm the benchmark's claim.
Extended reading notes
Core claim
HANDBOOK.md embeds an agent in a simulated company—a file workspace plus mock email, chat, calendar, issue-tracking, and commerce services—and asks it to complete routine work governed by an expert-written standard operating procedure of 20 to 124 pages. The paper's central claim is that a long, binding policy document does not currently function as a persistent authority for language-model agents; instead, the document behaves like one more retrieved source whose influence decays across turns and tool calls. The decisive measurement is strict pass@1: a trial passes only if every rubric criterion passes. Of the 824 deterministic programmatic criteria, 28% check prohibited actions—the 'stop'
Load-bearing premise
The central claim rests on the rubrics faithfully encoding what a competent human would judge as correct behavior; the authors tuned criteria until grading was 'judged fair' but report no human-expert agreement study and no human baseline, so overly strict or capricious rubrics could inflate the measured failure numbers.
Editorial extensions
If this is right
- If the central claim holds, enterprise deployments that rely on in-context policy adherence must add external verification; a workflow with a single violated control is not mostly-correct, it is failed.
- The near-miss analysis shows most failures are one criterion away from success, implying that a large share of lost trials could be recovered by targeted control-checking rather than wholesale capability improvement.
- The four documented failure patterns—request overriding policy, checks ignored, verification skipped, compliance falsely reported—give concrete targets for training data and for deterministic tool-call guards.
- The benchmark's policy-mutation design resists contamination structurally: even if a model has seen the base handbooks, it cannot answer the graded variant from memory.
Reading between the lines
- The absence of a human baseline is a gap; if competent humans also fail a substantial share of the rubrics under the same tools, the low model scores would overstate the capability deficit.
- The 'compliance asserted regardless' failure suggests that agent self-reports in production are untrustworthy; a natural extension is a separate truthfulness metric on final summaries.
- The large near-miss gap suggests deterministic policy guards or compiled checklists could recover many lost trials; HANDBOOK.md-style environments could serve as a testbed for such guard systems.
- The benign request-policy collisions are structurally identical to prompt injection; the benchmark could be extended to adversarial instructions to measure robustness under attack.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces HANDBOOK.md, a benchmark of 65 agentic tasks in which an agent must perform routine professional work while obeying a long, binding company handbook (20-124 pages) delivered as PDF, Word, or HTML inside a containerized MCP-based environment with email, Slack, calendar, Jira, and Shopify tools. Each task uses a uniquely mutated variant of one of ten base handbooks to resist memorization, and grading is fully deterministic via 824 programmatic rubric criteria that check both required actions (EXPECTED-OUTPUT) and prohibited actions (INCORRECT-BEHAVIOR). The paper evaluates 30 model configurations under a single OpenHands harness, reporting strict pass@1: best configuration (Claude Fable 5, adaptive/max) passes 36.2% of trials, and most frontier models score below 25%. It also reports a near-miss metric (pass@1 with N-1 criteria tolerated), cost/token efficiency, and a qualitative taxonomy of failure patterns. The benchmark, environments, rubrics, and harness are publicly released.
Significance. If the measurement is valid, HANDBOOK.md addresses a genuine and under-tested deployment pattern: whether long standing policies in context actually constrain agent behavior over extended multi-tool horizons. The design has real strengths: deterministic programmatic grading with no LLM judge, two-sided criteria covering refusals, per-task policy mutation for contamination resistance, realistic document formats and tool surfaces, and a fully released harness that can reproduce the leaderboard. The qualitative failure patterns are important and actionable for the community. However, the headline numbers depend on two load-bearing assumptions that need stronger support: rubric criteria faithfully encode competent-human judgment, and the per-task mutation is sufficient to block memorization. Sampling uncertainty also needs to be reported before leaderboard ranks are compared. These issues are addressable and do not undermine the benchmark's fundamental value.
major comments (3)
- [§3.6, §5.1] The central claim that agents cannot reliably follow long, binding policies rests on the rubric criteria faithfully encoding competent-human judgment of each handbook. Section 3.6 says criteria were iterated until grading was 'judged fair,' but this judgment is internal to the authors; no external expert agreement study or human baseline is reported. Since strict pass@1 requires every criterion to pass, even a few over-strict or idiosyncratic criteria can mechanically convert correct trajectories into failures. The single detailed example in Appendices A-B is reassuring but is only one of 65 tasks. Please add either (a) a human-expert study on a sample of tasks measuring agreement on which actions are required/prohibited, reporting inter-rater reliability, or (b) a human baseline under the same harness; and report how many criteria were revised during the 'judged fair' iteration. Without
- [Table 2, §5.1, §5.3] Pass@1 is reported with only four trials per task (N=260 per configuration), but no confidence intervals are given. The binomial 95% CI for the 36.2% leader is roughly ±5.8 points, and the differences among GPT-5.5 (21.5%), GPT-5.6 Sol (23.5%), and Opus 4.8 max (21.9%) are within sampling noise. The claim in §5.3 that N-1 grading roughly doubles scores also lacks uncertainty. Please report per-task variance and confidence intervals (e.g., Wilson intervals or Bayesian estimates), and avoid interpreting small rank differences. This is necessary for the leaderboard and for the conclusion that there is a 14-point spread between the leader and the June frontier.
- [§3.1, §3.3] The paper claims that per-task mutation makes policy memorization unavailable because no two tasks share a policy. However, all 65 tasks derive from only ten base handbooks, and if mutations are limited to a few clauses per task, a model that memorized a base could still apply most rules correctly. The paper does not quantify mutation coverage or directly test this assumption. Please report a baseline in which agents receive the base handbook instead of the task variant, or report the number and type of mutated clauses per task. This would demonstrate that the variant, not the base, drives grading, and would strengthen the contamination-resistance claim that is central to the benchmark's design.
minor comments (4)
- [Abstract and §1] The abstract uses 'HANDBOOK_md' while the title and body use 'HANDBOOK.md'; please use a single consistent spelling.
- [§6] The failure taxonomy is described with 'most failures' but no quantitative distribution is provided. A simple coding of a sample of failed trajectories (with inter-rater counts) would make the pattern claim more rigorous.
- [References] Reference [4] contains 'V olume' and reference [12] contains 'F ourteenth'; also some reference labels use irregular spacing. Please proofread the bibliography.
- [Figure 3] The near-miss (N-1) estimates are shown without confidence intervals, making it impossible to know whether the observed reorderings are significant. Adding intervals would help.
Circularity Check
No significant circularity: the paper is an empirical benchmark, and its leaderboard self-citations are appropriate, not load-bearing.
full rationale
HANDBOOK.md is an empirical benchmark, not a derivation. The central claim — that the best evaluated configuration passes 36.2% of trials under strict grading — is a measurement produced by running 30 model configurations in containerized environments and applying deterministic Python verifiers to the resulting environment states (§3.5, Table 2). There is no equation that maps inputs to outputs, no fitted parameter later relabeled as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The rubric criteria, including the exact-count INCORRECT-BEHAVIOR invariants, are an operationalization of the handbooks; whether they are the uniquely reasonable operationalization is a validity question, not a circularity question. The paper openly states that criteria were iterated until grading was 'judged fair' (§3.6) and does not claim external expert validation, so concerns about over-strict rubrics are legitimate correctness-risk concerns but not circular reasoning. The self-citations ([20], [21]) are appropriate: the paper is the primary source of its own leaderboard, the benchmark is publicly released, and the harness reproduces the leaderboard protocol end to end (§3.6), so the evidence is independently checkable rather than a citation standing in for an argument. No load-bearing step reduces to its own inputs.
Assumptions & free parameters
assumptions (4)
- domain assumption The expert-written handbooks are representative of real enterprise policies and the per-task mutations preserve realism.
- domain assumption Rubric criteria faithfully operationalize handbook requirements, and INCORRECT-BEHAVIOR criteria are neither over- nor under-constraining.
- domain assumption The OpenHands harness, 200-call/1-hour caps, and MCP tool surface provide a valid proxy for real agent deployments.
- domain assumption Four trial runs per task are sufficient to estimate pass@1 reliably.
invented entities (1)
-
HANDBOOK.md task suite (10 fictional companies, 65 simulated environments with unique handbooks)
independent evidence
Cite this review
Pith. "Pith review of HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following." pith.science (2026). https://pith.science/paper/BOAG7S3R
@misc{pith2026260725398,
author = {Pith},
title = {Pith review of: HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following},
year = {2026},
howpublished = {\url{https://pith.science/paper/BOAG7S3R}},
note = {Machine review of arXiv:2607.25398}
}
read the original abstract
Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let that document govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document constrains its behavior over an extended tool-use horizon. We present HANDBOOK_md, a benchmark of 65 agentic tasks modeled on how employees follow company handbooks. Each task places an agent in a self-contained company environment (a file workspace with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol) and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20-124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and 10 fictional companies. To resist memorization, every task modifies one of 10 base handbooks, altering the specific rules and thresholds on which grading depends, so no two tasks share the same set of policies. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the strongest evaluated model passes 36.2% of trials, and most frontier models remain below 25%. Failures follow consistent patterns: agents let a plausible but unauthorized in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release the tasks, environments, and evaluation harness.
Figures
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.