APEX-Accounting

Austin Bennett; Bertie Vidgen; Brendan Foody; Charis Ching; Felix Mercier; Hayley Popiel; Jasmin Kern; Julien Benchek; Rene Sultan; Ryan Stevens

REVIEW 3 major objections 5 minor 12 references

APEX-Accounting

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read APEX-Accounting shows that the best frontier AI agent meets only 56.4% of expert accounting rubric criteria, and no model can fully solve any task on all eight attempts.

desk verdict A serious, well-built benchmark with a plausible central finding: frontier agents are nowhere near reliable on month-end close work; the main caveat is the difficulty filter, which the authors acknowledge but never quantify. read the letter →

arxiv 2607.27189 v2 pith:7ZYPOSCV submitted 2026-07-29 cs.CL cs.AIcs.HC

Julien Benchek , Austin Bennett , Jasmin Kern , Ryan Stevens , Rene Sultan , Charis Ching , Hayley Popiel , Vaibhav Mittal

show 3 more authors

Felix Mercier Brendan Foody Bertie Vidgen

This is my paper

classification cs.CLcs.AIcs.HC

keywords accountingbenchmarkfrontierAIagentsmonth-endcloserubricgradingreasoningfailuresPass^8consistencySimpson'sparadoxagentharness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

APEX-Accounting is a private benchmark of 160 expert-authored accounting tasks set in ten fictional company worlds frozen at month-end close, covering reconciliation, data entry, variance analysis, and schedules and accruals. The paper claims that current frontier AI agents can find and read the right files but fail when the work demands sustained accounting judgment: the best model meets only 56.4% of rubric criteria on average, and no model produces a fully correct answer on all eight attempts for any task (best Pass^8 = 2.6%). The consequence, if true, is that an agent that is usually correct but not reliably so cannot close the books unsupervised. A failure taxonomy shows the bottleneck is reasoning — substituting the wrong basis or authorization logic and dropping correct intermediate results — rather than tool use or harness design, which points to where progress must come from.

What carries the argument

The load-bearing measurement object is the expert-authored rubric criterion: each task is graded by 10-20 binary, outcome-based Pass/Fail criteria, which makes partial credit possible. Mean Criteria@3 turns those criteria into a leaderboard metric that separates models that nearly finish a task from models that fail outright, while Pass^8 — the same criteria requiring a perfect score on all eight runs — provides the consistency floor that carries the paper's conclusion. A validated LM judge (97.1% agreement with human expert labels on 1,687 criteria) applies the rubric, and a seven-level failure taxonomy classifies low-scoring trajectories.

What would settle it

Rescore all 160 held-out tasks' trajectories with human expert graders and confirm the reported relative ordering and the low Pass^8; or, run a model from a family that was not part of the difficulty-filtering trio on the full 599-task candidate pool and check whether the retained worlds remain hardest. If a new-family model scores dramatically higher on the same 160 tasks under the same harness, the 'cannot close the books' claim would be falsified for that generation.

Watch

Extended reading notes

Core claim

The paper introduces APEX-Accounting: 160 close-cycle bookkeeping tasks across 10 synthetic company 'worlds,' each with its own accounting system, spreadsheets, PDFs, and other files. Every task was authored and solved by a 42-person expert panel, who also wrote 13.7 binary, outcome-based rubric criteria per task on average. Evaluated nine frontier models on a while-loop-with-tools agent harness, the paper reports that the best model reaches 56.4% Mean Criteria@3 (percentage of rubric criteria met, averaged over three runs), Pass@8 of 21.5% (passing a task at least once across eight attempts) for the strongest model, and a maximum Pass^8 of 2.6% (fully correct on all eight attempts). Since 5

Load-bearing premise

The ten worlds are assumed to be a fair sample of real close-cycle accounting work, but they were selected precisely because three frontier models scored worst on them, so the absolute scores (and the 'cannot close the books' conclusion) may be an artifact of that difficulty filter.

Editorial extensions

If this is right

If correct, organizations cannot yet deploy frontier agents to close the books without human review; the gap between Pass@8 (21.5%) and Pass^8 (2.6%) defines the reliability problem.
The failure analysis implies that improving harnesses or tooling will yield little: a purpose-built ReAct-style harness with subagent delegation shifted scores by only +1.2 percentage points on average, and no annotated failure involved tool use.
Raising the per-task budget from $1 to $50 helps mostly token-expensive models (the leader gains +43.4 points) while barely helping already-cheap models (+4.7 points), so cost caps, not model capability, explain much of the low-budget spread.
The paper concludes that progress will come mainly from model-side improvements: training that instills the discipline to carry results through, surface contradictions in source documents, and refuse to post entries the evidence does not support.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the benchmark was filtered to keep the worlds where three frontier models scored worst, the absolute figures are a lower bound on typical month-end close work; a random sample of accounting tasks would likely show higher scores, so the 'cannot close the books' conclusion is about the hardest slice, not the average.
The Simpson's-paradox finding suggests a practical deployment rule: budget models by task difficulty rather than a flat dollar cap, since within a fixed harness harder tasks consume more tokens yet still score lower.
The near-universal reasoning-failure profile, if it generalizes across document-heavy knowledge work, implies that agent frameworks have sharply diminishing returns relative to base-model reasoning fidelity.
A testable extension: re-run the benchmark with a future model family not used in the difficulty filter; if it scores dramatically higher on the same retained tasks, the absolute scores (and the 'unsupervised close' conclusion) would need to be re-calibrated for that generation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a genuinely new benchmark and the measurements look careful. It is the first large-scale, expert-authored eval of month-end close work that I know of, and the paper does several things right. The tasks are authored and solved by accountants with real experience, the rubrics are outcome-based and expert-reviewed, and there is an independent re-solve check that surfaced only one divergent task. The judge selection is sound: DeepSeek-v4-Flash was validated against 1,687 majority-vote human labels at 97.1% agreement, and the self-preference stress test places its own solver mid-pack, which is the right kind of evidence. The failure taxonomy is a real contribution, and the Simpson's-paradox cost-budget analysis is original and clearly explained. The multiple-testing correction on the pairwise comparisons is more than most benchmark papers do.

The central qualitative finding holds up: even the best model scores 56.4% Mean Criteria@3, and no model exceeds 2.6% Pass^8. The consistency gap is so large that the conclusion — these agents cannot yet close the books unsupervised — is robust even if individual numbers shift. I would not fight that conclusion.

The soft spots are real but mostly addressable. The biggest is the difficulty filter in Section B.1. The benchmark worlds were selected as the hardest among 599 candidates under three frontier models. That means the absolute scores are conditional on deliberately hard tasks, not a random sample of close work. The paper acknowledges this but only defends the relative ordering, not the absolute levels. Because the eval set is closed, nobody can check the magnitude externally. A reader should treat 56.4% as a performance floor on the hardest selected worlds, not an estimate for typical month-end close work. Second, the primary metric Mean Criteria@3 uses three runs chosen lexicographically by task ID, asserted to be 'essentially random' without evidence. That is a small statistical hole, easily fixed by reporting all eight runs or a random subsample. Third, the dollar-to-token conversion assumes a fixed 90:10 input/output split; a sensitivity analysis would be nice but this is minor.

Who is this for? Anyone building agent evals for document-heavy knowledge work, and anyone tracking frontier model reliability in finance and accounting. It deserves a serious referee. The authors should be asked to quantify the selection effect or release scores on unselected worlds, and to fix the run-selection justification. The closed eval set is a limitation, but the public dev set and the methodology are enough to evaluate the claims.

I would send this to review and would cite it. Bring it to reading group.

Referee Report

3 major / 5 minor

Summary. The paper introduces APEX-Accounting, a private benchmark of 160 expert-authored accounting tasks across 10 synthetic company worlds, and evaluates nine frontier models under a tool-using agent harness. It reports Mean Criteria@3 (best: Claude-Fable-5, 56.4%), Pass@8 (highest 21.5%), and Pass^8 (highest 2.6%), together with token-budget and harness ablations and a failure-mode analysis. The authors conclude that current agents are 'usually correct but not reliably so' and cannot close the books unsupervised.

Significance. If the measurements are taken at face value, APEX-Accounting provides a realistic, expert-authored, outcome-based benchmark for a previously under-tested domain. The evaluation design is unusually careful: a judge validated against 1,687 majority-vote human labels (97.1% accuracy) with a self-preference stress test; 8 runs per model with task-bootstrap CIs; Benjamini–Hochberg correction across 36 pairwise comparisons; paired cost ablations; a public dev set; and golden responses scoring 100%. The failure taxonomy is a useful contribution. The headline result — that no model reliably completes close-cycle bookkeeping — is plausible and important, but the selection and measurement issues below need to be addressed before the broad 'cannot close the books' claim is fully supported.

major comments (3)

[B.1, 6, 7] The eval set is not a representative sample of close-cycle work: Section B.1 states that of 599 authored tasks, the 11 worlds with the lowest mean scores under Claude-Opus-4.8, GPT-5.5, and Gemini-3.1-Pro Preview were retained, and the 10 hardest became the benchmark. Section 6 acknowledges this but defends only relative ordering. The headline conclusion in Section 7 ('cannot yet close the books unsupervised') rests on absolute levels (56.4% Mean Criteria@3, 2.6% Pass^8, 58% unsolved). These levels are depressed by construction for the models used to filter. Please either restrict the conclusion to the deliberately hardest subset, or quantify the selection effect by scoring a representative sample of unselected/typical tasks (e.g., the full 599-task pool or a random subset) and reporting the resulting absolute scores.
[3.5] Mean Criteria@3 is computed from 3 of 8 runs 'chosen lexicographically based on task ID strings, which essentially make up a random sample.' This is not a formal random sample, and no evidence is given that lexicographic selection is exchangeable with random draws. If run IDs or task IDs are correlated with run order or difficulty, the primary leaderboard metric is biased. Please justify the selection (e.g., by comparing the 3-run mean to the 8-run mean on a subset) or replace with a seeded random subset.
[3.4, 4.3] Pass^8 is an absolute threshold metric: a task passes only if every rubric criterion is judged Met. The deployed judge has 97.1% accuracy and recall 97.4% on the Met class. For a task with ~14 criteria, the probability that all criteria are correctly classified Met is roughly 0.974^14 ≈ 0.69, so even a strong judge can substantially undercount true passes. The paper reports judge error only as an average and argues it is uniform across models; that argument addresses relative ranking, not the absolute Pass^8 numbers featured in the abstract and conclusion. Please report a sensitivity analysis for Pass^8/Pass@8 (e.g., ensemble judge labels, or applying the confusion matrix to bound the error), or soften the absolute consistency claim.

minor comments (5)

[Table 6] The rows 'Mean (w/ dev set)' and 'Mean (w/o dev set)' show negative numbers (-207, -219, -191, -192) in the Total/Tasks columns; these appear to be typographical artifacts. Please correct.
[3.5 / Figure 3] The notation Pass^k is used for k=1..8, but the definition says 'each of its 8 attempts' regardless of k. Define Pass^k explicitly for general k, including the superscript convention.
[5] Failure-category labels are assigned by an LM judge, but no validation of these labels is reported. A human-sample agreement check (or at least a note on the judge's reliability for taxonomy labeling) would strengthen the failure analysis.
[4.4 / Appendix I] The 90:10 input:output split assumption for dollar-to-token conversion is a modeling choice; actual mixes vary across models and tasks. A sensitivity check using observed per-task mixes would make the cost ablation more robust.
[6 / Table 12] The text says 'two adjacent-rank pairs are statistically indistinguishable' and Table 12 identifies Muse-Spark-1.1 vs GPT-5.6-Sol and GLM-5.2 vs Grok-4.5. It would be helpful to state explicitly that the exact ordering at these boundaries should not be over-read.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark results are independent measurements; the acknowledged difficulty-filter limitation is selection bias, not circularity.

full rationale

The paper's central results are direct measurements on a fixed, expert-authored benchmark, not derivations from fitted parameters or from the authors' prior results. The rubric criteria are outcome-based and were written by accounting experts before any model outputs existed; golden responses score 100%, and the grader (DeepSeek-v4-Flash) is not among the evaluated solver models and was validated against 1,687 expert labels with 97.1% agreement, with a self-preference stress test showing its own solver lands mid-pack. The only self-citation (Stevens 2026, Section 3.2) is used to state what the harness excludes, not to justify any result. The difficulty filter (Section B.1, acknowledged in Section 6) is a genuine selection-bias limitation because it may depress absolute scores and weaken the generalization "cannot yet close the books unsupervised," but it does not make the reported scores equal to the filter inputs by construction; the evaluated models could in principle score anywhere on the selected tasks. This is a validity/representativeness concern for the correctness pass, not circularity. Hence no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three layers of validation: expert rubrics/goldens, an LM judge checked against human labels, and the task-difficulty filter. The first two are well supported by the paper's own experiments; the third is a hand-chosen selection rule that the authors admit can bias absolute scores. No fitted physical or learned parameters appear; the hand-chosen evaluation configurations (token/step caps, 90:10 cost split, lexicographic run selection) are the closest analog to free parameters.

free parameters (2)

Dollar-to-token conversion ratio = 90:10 input:output split; higher cost tier
Section I: every dollar budget is converted to a token allowance via tokens(B)=B/((0.9p+0.1c)/1e6). This hand-chosen ratio changes the effective token caps per model and is load-bearing for the cost-ablation and Simpson's-paradox results.
Mean Criteria@3 run selection = 3 of 8 runs, chosen lexicographically by task ID
Section 3.5: the primary metric averages only 3 of 8 runs, selected by lexicographic task-ID order, asserted to be 'essentially random' without a randomness check.

assumptions (4)

domain assumption Expert-authored rubrics and golden answers define correct outputs
Section 2.3: tasks graded only on final output vs. expert rubrics; validated by 92.8% annotator agreement, Fleiss κ=0.857, and independent re-solve of 20 tasks (1.1% criteria defect).
domain assumption DeepSeek-v4-Flash judge grades match expert majority with uniform error across solvers
Sections 3.4/E: 97.1% accuracy, F1=0.970 on n=1,687 human labels; Table 10 shows per-solver agreement 94.7–98.4%; a 2.9% error rate is assumed not to affect relative rankings.
ad hoc to paper The 10 retained worlds are representative of real accounting work
Section B.1: worlds were selected as the 10 hardest among 11 qualifying worlds based on low scores of three filtering models; Section 6 acknowledges the risk that filtered model families have depressed scores and that tax/audit/consolidation/human-in-the-loop tasks are excluded.
ad hoc to paper Lexicographic choice of 3 runs is equivalent to random sampling
Section 3.5: 'chosen lexicographically based on task ID strings, which essentially make up a random sample' — no distributional argument or citation is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of APEX-Accounting." pith.science (2026). https://pith.science/paper/7ZYPOSCV

@misc{pith2026260727189,
  author       = {Pith},
  title        = {Pith review of: APEX-Accounting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZYPOSCV}},
  note         = {Machine review of arXiv:2607.27189}
}

abstract

We introduce APEX-Accounting, a benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants. Tasks include reconciling accounts, accruing expenses, posting transactions, and producing reports. The private eval set comprises 160 tasks, split across 10 worlds. Each world contains an accounting system, as well as spreadsheets, PDFs, and other files. Every task was authored and solved by experts in accounting and bookkeeping, who also wrote grading rubrics. Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%. No model scores more than 2.6% Pass^8 (GPT-5.6-Sol (Max+Pro)) and the highest Pass@8 is 21.5% (Muse-Spark-1.1 (xHigh)). We experiment with increasing the token budget from $1 to $50 and observe an instance of Simpson's paradox: scores increase as the token budget increases but within a given budget-constrained harness, scores are lower on tasks where the model spends more tokens. As APEX-Accounting is a closed benchmark, leaderboard evals can be run for any frontier model on request.

Figures

Figures reproduced from arXiv: 2607.27189 by the authors.

**Figure 2.** Ramp Harness schematic. Fable-5, GPT-5.6-Sol, and Gemini-3.1-Pro, where we compared unlimited token spend against the 5 million token cap. For all three models, the 5 million token limit scores were slightly higher, supporting this design choice. The harness reports remaining tokens at every step, and a finite, visible budget pushes the model to consolidate intermediate results and commit to an answer, whereas unl… view at source ↗

**Figure 3.** Pass@k versus Pass^k by model, k = 1 through 8, capability ceiling versus consistency floor. Claude-Fable-5 (Max) and Muse-Spark-1.1 (xHigh) scores are bolded for visibility. computational cost.8 Beyond the leaderboard, we report Pass@8 as an indicative ceiling on current frontier agent capabilities. It measures whether a model passes a task at least once across 8 attempts. We also report Pass^8 (Yao et al., 2025),… view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: Panel A. For five models, the Mean Criteria@3 versus the per-task dollar budget cap (set at $1, $5, $10 and $50). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Per-task Mean Criteria@3 versus realized spend per run (USD), by model, at each dollar budget ( [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 5.** Figure 5: Per-task Mean Criteria@3 versus realized spend per run (USD), by model, at each dollar budget ( [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Task-level change in Mean Criteria@3 relative to the $50 baseline, versus realized spend per run, by model. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Mean Criteria@3 by model under the Loop Har [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 7.** Figure 7: Mean Criteria@3 by model under the Loop Har [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

**Figure 8.** Figure 8: L1 failure-mode counts for the three best-performing models, over annotated failures ( [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 8.** Figure 8: L1 failure-mode counts for the three best-performing models, over annotated failures ( [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

**Figure 9.** Figure 9: L2 subclass counts within Reasoning failure for the three best-performing models. Non-numeric reasoning [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

**Figure 10.** Figure 10: Gross headroom for each model pair: Gross headroom counts tasks where the row model achieved a perfect [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

**Figure 11.** Figure 11: Our world construction and audit process consisted of expert and synthetic generation with multiple rounds [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

**Figure 12.** Figure 12: System-prompt excerpt disclosing the converted [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

**Figure 13.** Figure 13: Mean subagents spawned per task by model [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

**Figure 14.** Figure 14: Failure-mode breakdown for the three best-performing models ( [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]

Review history (3 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 1 linked inside Pith

[1]

An automated hygiene pass scanned for blatant contradictions and data-hygiene problems
[2]

Multiple experts manually reviewed all files for realism and consistency
[3]

A validation script confirmed the accounting soft- ware data was schema-compatible with the eval- uation environment
[4]

At the task level, each submission passed through one layer of expert review (with revision cycles as needed) and an automated QC pass

Three to five experts each completed a handful of tasks against the world files, surfacing contra- dictions and missing data by having profession- als actually do the work. At the task level, each submission passed through one layer of expert review (with revision cycles as needed) and an automated QC pass. Tasks failing automated QC were audited by the p...

2026
[5]

• Planning failure.The agent fails to plan, or to course-correct, a viable course of action for com- pleting the request

Planning and reflection failure The agent fails to appropriately plan, monitor, or adjust its process while completing the request. • Planning failure.The agent fails to plan, or to course-correct, a viable course of action for com- pleting the request. • Failure to complete task.The agent enters a doom loop, repeating the same or functionally similar ste...
[6]

• No information gathering.The agent does not attempt to search or retrieve any files at any point in the task, relying on parametric knowledge to complete the request

Information-gathering failure The agent fails to find the correct information, or all of the information, required to successfully complete the request. • No information gathering.The agent does not attempt to search or retrieve any files at any point in the task, relying on parametric knowledge to complete the request. • Incomplete search.The agent searc...
[7]

not,” “except,

Instruction-following and prompt understanding failure The agent fails to understand or apply some or all of the instructions and constraints in the prompt. • Ignoring the requirements.The agent ignores one or more of the prompt’s requirements, constraints, or dimensions, such as a time period, business segment, or entity. • Failure to follow implicit ins...
[8]

Reasoning failure The agent has the correct information available but applies faulty logic, inference, or calculation to it. • Numeric reasoning failure.The agent makes an incorrect calculation or applies faulty quantitative logic to numeric data; the underlying values may be correct and complete, but the arithmetic or quantitative method applied to them ...

Show all 12 references

[9]

• No tool used.The agent does not identify a tool to use to complete an action, relying solely on parametric knowledge

Tool use failure The agent fails to use any tools, identifies the correct tool but does not use it, uses an incorrect tool, or uses the correct tool incorrectly. • No tool used.The agent does not identify a tool to use to complete an action, relying solely on parametric knowle...
[11]

• Final output mistake.The agent correctly com- pletes all reasoning and analysis but misstates the results in its output

Communication and presentation failure The agent makes a style, grammar, formatting, or other communication error. • Final output mistake.The agent correctly com- pletes all reasoning and analysis but misstates the results in its output. • Tone failure.The agent fails to adapt...
[12]

N LM Usage We used LMs to assist with drafting and refinement of this paper

Other valid failure The agent fails in a way that is not captured by any other failure label. N LM Usage We used LMs to assist with drafting and refinement of this paper. 26
[2025]

Automating Financial Statement Audits with Large Language Models

Finmaster: A holistic benchmark for mastering full- pipeline financial workflows with llms. National Center for O*NET Development. 2025. 13-2011.00 — accountants and auditors. O*NET Online. Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkin...

2025 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

An automated hygiene pass scanned for blatant contradictions and data-hygiene problems

[2] [2]

Multiple experts manually reviewed all files for realism and consistency

[3] [3]

A validation script confirmed the accounting soft- ware data was schema-compatible with the eval- uation environment

[4] [4]

At the task level, each submission passed through one layer of expert review (with revision cycles as needed) and an automated QC pass

Three to five experts each completed a handful of tasks against the world files, surfacing contra- dictions and missing data by having profession- als actually do the work. At the task level, each submission passed through one layer of expert review (with revision cycles as needed) and an automated QC pass. Tasks failing automated QC were audited by the p...

2026

[5] [5]

• Planning failure.The agent fails to plan, or to course-correct, a viable course of action for com- pleting the request

Planning and reflection failure The agent fails to appropriately plan, monitor, or adjust its process while completing the request. • Planning failure.The agent fails to plan, or to course-correct, a viable course of action for com- pleting the request. • Failure to complete task.The agent enters a doom loop, repeating the same or functionally similar ste...

[6] [6]

• No information gathering.The agent does not attempt to search or retrieve any files at any point in the task, relying on parametric knowledge to complete the request

Information-gathering failure The agent fails to find the correct information, or all of the information, required to successfully complete the request. • No information gathering.The agent does not attempt to search or retrieve any files at any point in the task, relying on parametric knowledge to complete the request. • Incomplete search.The agent searc...

[7] [7]

not,” “except,

Instruction-following and prompt understanding failure The agent fails to understand or apply some or all of the instructions and constraints in the prompt. • Ignoring the requirements.The agent ignores one or more of the prompt’s requirements, constraints, or dimensions, such as a time period, business segment, or entity. • Failure to follow implicit ins...

[8] [8]

Reasoning failure The agent has the correct information available but applies faulty logic, inference, or calculation to it. • Numeric reasoning failure.The agent makes an incorrect calculation or applies faulty quantitative logic to numeric data; the underlying values may be correct and complete, but the arithmetic or quantitative method applied to them ...