pith. sign in

arxiv: 2607.01245 · v1 · pith:LIWMYURYnew · submitted 2026-05-29 · 💻 cs.CL · cs.AI· cs.CY· cs.IR· cs.LG

Office Comprehension Benchmark

Firoz Shaik , Mateus Pican\c{c}o Lima Gomes , Tanvir Aumi , Jingci Wang , Milos Milunovic , Filip Basara , Ivana Jovanovic , Vishwas Suryanarayanan
show 12 more authors
Neha Nandan Kenkare Weiyao Xie Zhipeng Han Zheng Zhang Waleed Shahid Jay Rathi Russell Scherer Thong Q. Nguyen Michael Bentley Tamara Stankovic Rasika Chakravarthy Vishal Chowdhary
This is my paper

Pith reviewed 2026-07-04 00:19 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CYcs.IRcs.LG
keywords Office Comprehension BenchmarkLLM evaluationdocument understandingWord Excel PowerPointDomain Q&AFile Fidelityatomic claimsLLM judges
0
0 comments X

The pith

Even the strongest frontier LLMs reach only 59.3 percent on expert-level reasoning over real office documents.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces the Office Comprehension Bench to test LLMs jointly on Word, Excel, and PowerPoint files in their native formats. One track measures structural and visual perception of elements such as tables, charts, formulas, and speaker notes. The second track measures multi-step reasoning and synthesis across industry documents in twelve professional domains. Reference answers are broken into atomic claims that an ensemble of LLM judges scores independently. Results show top models top out near 59 percent on the reasoning track, with little gain from deeper thinking in the same tier.

Core claim

The Office Comprehension Bench is the first public benchmark for LLM comprehension of Word, Excel, and PowerPoint over native file formats. It consists of File Fidelity Q&A for structural and visual perception of office artifacts and Domain Q&A for expert-level reasoning grounded in real-world documents across 12 professional domains. Each reference answer is decomposed into atomic, binary-gradable claims scored independently by an ensemble of LLM judges. Even the strongest frontier system reaches only about 59.3 percent on Domain Q&A.

What carries the argument

Office Comprehension Bench with its two tracks and atomic-claim scoring by an LLM-judge ensemble

If this is right

  • Increasing thinking depth within a model tier does not move Domain Q&A performance materially.
  • Moving to a higher product tier yields only modest gains on domain reasoning.
  • Current systems still have large gaps on multi-step synthesis across native office artifacts.
  • The public dataset, tooling, and leaderboard enable standardized tracking of progress on office file comprehension.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Specialized parsing or embedding layers for binary office formats may be needed to close the remaining gap.
  • The atomic-claim decomposition method could transfer to benchmarks for other structured document types.
  • The modest tier gains suggest that raw scale or inference budget alone will not solve the comprehension limits observed.

Load-bearing premise

The ensemble of LLM judges produces reliable, unbiased scores when evaluating model responses against independently scored atomic claims derived from reference answers.

What would settle it

A new model scoring well above 59.3 percent on Domain Q&A under the same atomic-claim evaluation, or human raters showing large systematic disagreement with the LLM judge ensemble on the same set of responses.

Figures

Figures reproduced from arXiv: 2607.01245 by Filip Basara, Firoz Shaik, Ivana Jovanovic, Jay Rathi, Jingci Wang, Mateus Pican\c{c}o Lima Gomes, Michael Bentley, Milos Milunovic, Neha Nandan Kenkare, Rasika Chakravarthy, Russell Scherer, Tamara Stankovic, Tanvir Aumi, Thong Q. Nguyen, Vishal Chowdhary, Vishwas Suryanarayanan, Waleed Shahid, Weiyao Xie, Zheng Zhang, Zhipeng Han.

Figure 2
Figure 2. Figure 2: File Fidelity Q&A accuracy on OCB. Bars are assertion-level accuracy for each model. Dashed lines show single-rater human-annotator accuracy on the same metric, under the same single-pass setup as the LLM systems; annotators were vetted for general office literacy rather than specialized domain expertise. See Appendix O for coverage and methodology. 2.5 pp, and their 95% scrape-level confidence in￾tervals … view at source ↗
Figure 1
Figure 1. Figure 1: Domain Q&A Accuracy on OCB. where ns = 3 and t0.025, 2 = 4.303. This is a conservative interval reflecting only 3 independent response samples per model. Discussion. Two findings drive how we report results elsewhere. Response-sampling noise dom￾inates judge noise for every model (Scrape Var is roughly 1.6×–3.1× Eval Var), consistent with Madaan et al. (2024), so evaluation budget is bet￾ter spent on multi… view at source ↗
Figure 3
Figure 3. Figure 3: Domain Q&A accuracy by industry [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Claude Opus 4.7: assertion accuracy by thinking mode [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: GPT-5.5: asser￾tion accuracy across six thinking modes [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Source document excerpt for the Domain Q&A worked example. • Calculates Base Year (2024) NOPAT as ≈$293M (range: $278M–$308M). • Identifies the interest rate basis as IQVIA’s weighted average interest rate (≈4.69%). • Models interest expense on new debt as (IQVIA weighted average rate) × average debt balance; computes ≈$141M. • [. . . approximately 43 additional atomic as￾sertions] This illustrates why Dom… view at source ↗
Figure 10
Figure 10. Figure 10: Source slide for the File Fidelity worked [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Distribution of Domain Q&A queries across [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Pairwise per-assertion vote breakdown across [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Leave-one-out judge ablation The analysis yields three main observations: • GPT-5.4 Thinking is the most stabilising judge. Removing it leaves Claude and Gemini tied on 22.0% of assertions and gives the weakest pair￾level agreement (κ = 0.579) — roughly double the tie rate observed when either Claude or Gem￾ini is removed, consistent with GPT-5.4’s cen￾tral pass rate (53.6%) sitting between Claude’s (63.5… view at source ↗
Figure 14
Figure 14. Figure 14: Weighted Domain Q&A accuracy. Equiva￾lent to [PITH_FULL_IMAGE:figures/full_fig_p027_14.png] view at source ↗
Figure 16
Figure 16. Figure 16: Weighted Domain Q&A accuracy by indus￾try. Equivalent to [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 15
Figure 15. Figure 15: Weighted Domain Q&A accuracy by file type. Equivalent to [PITH_FULL_IMAGE:figures/full_fig_p028_15.png] view at source ↗
Figure 17
Figure 17. Figure 17: Word File Fidelity accuracy by artifact type. [PITH_FULL_IMAGE:figures/full_fig_p030_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Word File Fidelity accuracy by document size. 0%, Highlight 8%, Metadata 0%). The divergence on structural artifacts is the dominant source of the overall Word performance gap between Claude Opus 4.7 and the other two systems. Accuracy by file size. Documents are bucketed by page count: Small (≤10 pages), Medium (11– 30 pages), and Long (>30 pages) [PITH_FULL_IMAGE:figures/full_fig_p030_18.png] view at source ↗
Figure 21
Figure 21. Figure 21: PowerPoint File Fidelity accuracy by artifact [PITH_FULL_IMAGE:figures/full_fig_p031_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: PowerPoint File Fidelity accuracy by deck [PITH_FULL_IMAGE:figures/full_fig_p031_22.png] view at source ↗
read the original abstract

We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats (.docx, .xlsx, .pptx) and their variants. OCB consists of two tracks. File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges. Domain Q&A tests expert-level reasoning grounded in real-world industry documents across 12 professional domains, with queries requiring multi-step analysis and synthesis across documents. Each reference answer is decomposed into atomic, binary-gradable claims, and an ensemble of LLM judges scores responses against each claim independently. Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A; increasing thinking depth within a tier does not move performance materially, while moving to a higher product tier yields modest gains. We release the dataset, evaluation tooling, judge prompt, and a public leaderboard.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper introduces the Office Comprehension Benchmark (OCB), the first public benchmark for LLM evaluation on native Word, Excel, and PowerPoint files (.docx, .xlsx, .pptx). It defines two tracks—File Fidelity Q&A for structural and visual perception of document elements and Domain Q&A for expert-level multi-step reasoning across 12 professional domains—where reference answers are decomposed into atomic binary claims scored independently by an ensemble of LLM judges. The central empirical claim is that even the strongest frontier system reaches only 59.3% on Domain Q&A in default mode, with negligible gains from increased thinking depth within a tier and only modest gains from higher product tiers. The dataset, evaluation tooling, judge prompt, and leaderboard are released.

Significance. If the judge pipeline proves reliable, OCB would be a valuable addition to the field by exposing concrete limitations in current LLMs' handling of complex, multi-document office reasoning tasks. The explicit release of the full dataset, tooling, judge prompt, and public leaderboard is a clear strength that supports reproducibility and follow-on work.

major comments (1)
  1. [Abstract / Evaluation pipeline description] The 59.3% Domain Q&A result and the claims about reasoning depth versus tier effects are produced by decomposing references into atomic claims and scoring via an LLM judge ensemble. No human calibration data, inter-judge agreement statistics, or ablation on judge model choice or prompt variants are reported, making the headline numbers and downstream conclusions load-bearing on an unvalidated component of the evaluation pipeline.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting a key aspect of our evaluation pipeline. We address the major comment below and will incorporate the requested validations in the revised manuscript.

read point-by-point responses

  1. Referee: [Abstract / Evaluation pipeline description] The 59.3% Domain Q&A result and the claims about reasoning depth versus tier effects are produced by decomposing references into atomic claims and scoring via an LLM judge ensemble. No human calibration data, inter-judge agreement statistics, or ablation on judge model choice or prompt variants are reported, making the headline numbers and downstream conclusions load-bearing on an unvalidated component of the evaluation pipeline.

    Authors: We agree that the LLM judge ensemble is a load-bearing component of the reported results and that the submitted manuscript lacks explicit validation of this pipeline. In the revised version we will add a dedicated subsection under Evaluation Methodology that reports: (1) human calibration results on a stratified sample of 250 atomic claims (with inter-annotator agreement and agreement with the judge ensemble), (2) pairwise and ensemble-level agreement statistics (percentage agreement and Fleiss’ kappa), and (3) ablation tables comparing judge performance across model families and prompt variants. These additions will directly support the 59.3 % Domain Q&A figure and the reasoning-depth versus tier conclusions. We have already begun the human annotation effort and expect to complete it within the revision window. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark with independent evaluation pipeline

full rationale

The paper introduces an empirical benchmark (OCB) and reports performance numbers obtained by decomposing reference answers into atomic claims and scoring via an LLM judge ensemble. No mathematical derivations, equations, fitted parameters, or predictions appear in the provided text. No self-citations are invoked as load-bearing premises, and the central claims (e.g., 59.3% Domain Q&A) are direct empirical outputs rather than reductions by construction to the paper's own inputs. The work is self-contained as a benchmark release against external model evaluations.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The benchmark relies on standard LLM-as-judge evaluation practices and atomic claim decomposition without introducing new free parameters, unstated axioms, or postulated entities beyond the benchmark itself.

pith-pipeline@v0.9.1-grok · 5812 in / 1086 out tokens · 27005 ms · 2026-07-04T00:19:08.061679+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages · 1 internal anchor

  1. [1]

    Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball

    LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models.Ad- vances in neural information processing systems, 36:44123– 44279. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball

    work page

  2. [2]

    FinanceBench: A New Benchmark for Financial Question Answering

    Cuad: An expert-annotated nlp dataset for legal con- tract review.Advances in Neural Information Processing Systems. Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Fi- nancebench: A new benchmark for financial question an- swering.arXiv preprint arXiv:2311.11944. Yuta Koreeda and Christopher D Manning. 20...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

    Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. 2020. PlotQA: Reasoning over scientific plots. InProceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision (WACV), pages 1527...

    work page arXiv 2020

  4. [4]

    Hierarchical multimodal transformers for multi-page docvqa.Pattern Recognition, 144:109834. Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, and 1 others. 2023. Document understanding dataset and evaluation (DUDE). InProceedings of the...

    work page arXiv 2023

  5. [5]

    TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. InPro- ceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3277–3287. Fengbin Zhu, Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenj...

    work page arXiv 2024

  6. [6]

    Model revenue, EBITDA, D&A, EBIT, tax, net income, capex, de- preciation, integration costs (cash vs

    Calculate HealthDataCo’s standalone Free Cash Flow for 2024, then build a projection of Free Cash Flow pro forma with synergies for 2025–2027. Model revenue, EBITDA, D&A, EBIT, tax, net income, capex, de- preciation, integration costs (cash vs. non- cash), working capital (using AR, AP, in- ventory), and free cash flow. Present free cash flow as a simple ...

    work page 2024

  7. [7]

    Model recursive interest expense (interest based on average debt balance)

    Layer in debt funding: $3B new debt at IQVIA’s weighted average interest rate. Model recursive interest expense (interest based on average debt balance). Show im- pact on net income. Use IQVIA’s FY2024 effective tax rate for the interest tax shield; keep the 25% tax rate for HealthDataCo operating taxes

    work page

  8. [8]

    Which model has the highest presentation quality score for Amazon_Toys_and_Games with NMF recommendation model and what is its score?

    Compute ROIC (2027) as NOPAT (2027) divided by average invested capital across 2026–2027. Define invested capital as $6,000M plus the working-capital balance tied to revenue synergies (cumulative). Ex- clude goodwill and any debt tax shield. Cal- culate EV A (2027) using WACC = 8.0% and the same average invested capital used for ROIC.” Source file.IQVIA H...

    work page 2027

  9. [9]

    Interpret the context of the <assertion> based on the <question> provided

    Read the <question>, <ai_response> and <assertion> provided thoroughly. Interpret the context of the <assertion> based on the <question> provided. Make sure you understand what needs the ground truth from the assertion that needs to be evaluated. ,→ ,→ ,→ ,→

    work page

  10. [10]

    Analyze the <ai_response> and locate evidence relevant to the <assertion>, either to support or refute it.,→

    work page

  11. [11]

    Analyze the <question> and locate the relevant context that qualifies the <assertion> in order to clarify its requirement. ,→ ,→

    work page

  12. [12]

    response should contain at least X out of Y conditions

    Evaluate the <assertion> based on the evidence you identified.,→ - Consider partial fulfillment: if the <ai_response> meets some but not all criteria, requirements or expectations in the <assertion>, check if the <assertion> allows for partial correctness (e.g., "response should contain at least X out of Y conditions", "response should X OR Y"). If it doe...

    work page

  13. [13]

    Compose your reasoning using the following structure xml-like structure:,→ <assertion> Restate the <assertion> verbatim as provided </assertion> <interpretation> Interpretation of the assertion in the context of the <question> asked as a single statement.,→ </interpretation> <evidence> [Quoted evidence from the <question> and <ai_response> that supports y...

    work page

  14. [14]

    reasoning

    Fill in the final output as a valid JSON object (without markdown code blocks) with your reasoning and score. Only valid JSON is allowed in the output. ,→ ,→ </reasoning_instructions> <output> { "reasoning": "Structured explanation as described above",,→ "score": "0 if assert is FALSE, 1 if assert is TRUE" } </output> <examples> <example_1> <input> <quest...

    work page 2021

  15. [15]

    Sources span SEC filings, slide decks, spread- sheets, and operational documents

    Source curation.Domain practitioners select or compile authoritative source documents that a qualified professional would consult. Sources span SEC filings, slide decks, spread- sheets, and operational documents

    work page

  16. [16]

    Each Q&A is au- thored by 2-3 domain experts

    Question and answer authoring.Experts draft realistic prompts grounded in the source documents and produce reference answers that a qualified practitioner would recognize as correct and complete. Each Q&A is au- thored by 2-3 domain experts

    work page

  17. [17]

    Rubric construction.Each Q&A pair is paired with a rubric composed of atomic as- sertions - single, verifiable claims that a cor- rect response should satisfy. We use multi- ple authoring approaches: rubrics authored independently of the reference answer (captur- ing what any correct response should demon- strate), rubrics derived from the reference an- s...

    work page

  18. [18]

    Model evaluation against these rubrics is not part of the construction pipeline; it is the downstream evaluation procedure described in §4

    Quality assurance.A separate reviewer, dis- tinct from the original prompt and rubric au- thors, verifies prompt construction, rubric de- sign, and consistency across items. Model evaluation against these rubrics is not part of the construction pipeline; it is the downstream evaluation procedure described in §4. Scoring.We reportassertion-level accuracyas...

    work page

  19. [19]

    The LLM output a structured flag set (e.g., {compound: true, ambiguous: false, ...}) and a brief justification per flag

    Automated triage.Every candidate asser- tion was scored by a frontier LLM along the failure-mode dimensions enumerated below. The LLM output a structured flag set (e.g., {compound: true, ambiguous: false, ...}) and a brief justification per flag. We did not act on these flags directly; they served only to prioritize human review

    work page

  20. [20]

    Expert decision.A domain expert (the rubric author or a peer reviewer) examined each flagged assertion and selected one of:keep (false positive),modify(rewrite for clarity or specificity),split(decompose a compound assertion into multiple atomic ones),merge (collapse parallel assertions into one general claim), ordelete(remove off-topic or unverifi- able ...

    work page

  21. [21]

    Rubrics typically required two to three loop iterations to stabilize

    Cross-check.An independent reviewer re- ran the automated triage on the revised rubric and resolved any remaining flags. Rubrics typically required two to three loop iterations to stabilize. E.2 Failure mode taxonomy with examples Examples below are drawn from real rubrics across our domain authoring collections. Identifying de- tails have been preserved ...

    work page

  22. [22]

    Skip anything,→ that requires inference

    Identify facts explicitly stated in the document (timelines, named,→ entities, roles, system features, numeric values). Skip anything,→ that requires inference

    work page

  23. [23]

    Generate up to {{QUESTIONS_PER_CHUNK}} high-quality questions.,→ Prioritize questions that: - Are unambiguous with clear scope - Are answerable from the provided text alone - Target high-value content: KPIs, financial figures, technical,→ specifications, research findings, actionable insights - Expect atomic, extractable data points (numbers, dates, names),→

    work page

  24. [24]

    qas": []}. </restrictions> <output> {

    For each question, draft three candidate answers: - Detailed: thorough explanation with context - TL;DR: concise factual one-liner - Balanced: key fact plus brief rationale Reconcile via majority vote. Final answer is short (one sentence,→ or a few words). </task> <restrictions> - Use only information explicitly stated in the document. - Do not speculate ...

    work page

  25. [25]

    Fig. 3",

    atomic answers tied to specific visual content. Each pair is emitted in two question forms (with and without an explicit figure reference) to support both retrieval-style and reading-comprehension- style framings downstream. <role> You generate vision-grounded QA pairs for images extracted from,→ documents. </role> <input> - IMAGE_INPUT: base-64 PNG of on...

    work page

  26. [26]

    ______",

    Subject text must be meaningful. If a fact references placeholder,→ content ("______", "..."), describe the element by purpose ("the,→ signature line", "the location number field") rather than quoting,→ the placeholder

    work page

  27. [27]

    red" not

    Color answers must be human-readable names, not hex codes ("red" not "#FF0000")

    work page

  28. [28]

    Skip low-value facts: CUSTOM_PROPERTY entries (SharePoint metadata,,→ ContentTypeId) and trivial hyperlinks where the URL matches the,→ display text

    work page

  29. [29]

    CORE_PROPERTY metadata (author, title, created date) and hyperlinks where the URL meaningfully differs from display text,→ are in scope

    work page

  30. [30]

    What is the largest font size used,→ in the document?

    Comparative or aggregate questions are encouraged when multiple,→ related facts support them ("What is the largest font size used,→ in the document?"). </quality_rules> <output> { "qas": [ { "question": "...", "answer": "...", "question_type": "<one of the types above>", "fact_ids": ["FACT_ID_001", "..."] } ] } </output> <input> <|startofcontext|> {{fidel...

    work page 2024