pith. sign in

arxiv: 2606.31154 · v1 · pith:WHKJJLPVnew · submitted 2026-06-30 · 💻 cs.LG · cs.AI

PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

Apurva Gandhi , Vishwas Suryanarayanan , Raja Hasnain Anwar , Firoz Shaik , Shubhang Desai , Thong Q. Nguyen , Muhammad Taqi Raza , Vishal Chowdhary
show 1 more author
Graham Neubig
This is my paper

Pith reviewed 2026-07-01 06:51 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords benchmarkcomputer-use agentsPowerPointrubric evaluationmultimodal taskspresentation creationAI agent evaluationpartial credit scoring
0
0 comments X

The pith

A benchmark of 120 PowerPoint tasks shows frontier agents reach only 45 percent full success.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces PPT-Eval to measure how well computer-use agents perform on realistic slide creation and editing work. It develops task-specific rubrics that assign partial credit for intermediate steps, penalize unneeded changes, and supply natural language feedback, reaching 0.77 correlation with human ratings. Results indicate that even leading models achieve just 45 percent complete success and 57 percent average partial score across content and editing scenarios. If accurate, this means current agents cannot yet handle the multimodal and open-ended demands of common presentation software reliably.

Core claim

We introduce PPT-Eval, a benchmark of 120 PowerPoint tasks across 12 files covering content creation and presentation editing at varying difficulties. Task-specific rubrics award partial credit for intermediate steps, penalize unnecessary changes and poor aesthetics, and provide natural language feedback, achieving a Kendall's tau-b correlation of 0.77 with human judgments. Existing frontier agents still struggle, with strong models like Claude-4.5-Opus reaching only 45 percent success rate and 57 percent average partial score.

What carries the argument

Task-specific rubrics that evaluate partial progress on multimodal PowerPoint tasks with many valid solutions.

If this is right

  • Binary success metrics miss the partial progress agents typically make on these tasks.
  • Rubric design must explicitly address aesthetics, unnecessary edits, and multiple solution paths.
  • PowerPoint serves as a practical testbed because slide work combines text, layout, and visual decisions.
  • Natural language feedback from rubrics can support iterative agent improvement.
  • Performance gaps on the benchmark point to needed advances in handling complex office software interfaces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Rubric methods developed here could transfer to evaluating agents on other productivity tools with open-ended outputs.
  • Strong benchmark performance would likely predict better results in real professional settings that require presentations.
  • Extending the partial-credit approach might improve evaluation in any domain where agents rarely complete tasks perfectly on first try.

Load-bearing premise

The task-specific rubrics provide a robust and generalizable way to score complex PowerPoint outputs that admit many valid solutions.

What would settle it

A direct comparison in which human judges rate a large set of agent-produced slides and the scores diverge substantially from the rubric ratings on the same outputs.

Figures

Figures reproduced from arXiv: 2606.31154 by Apurva Gandhi, Firoz Shaik, Graham Neubig, Muhammad Taqi Raza, Raja Hasnain Anwar, Shubhang Desai, Thong Q. Nguyen, Vishal Chowdhary, Vishwas Suryanarayanan.

Figure 1
Figure 1. Figure 1: Illustration of task-solving in the PPT-EVAL Benchmark. Given a task, an agent interacts with a file in a sandboxed instance of PowerPoint Online. Interaction can take full advantage of PowerPoint’s feature-set using the GUI. PPT-EVAL provides rubrics that can score different degrees of task completion and provide both nuanced partial credit and natural language feedback. PPT-EVAL comprises 120 tasks sourc… view at source ↗
Figure 2
Figure 2. Figure 2: Task setup and evaluation workflow. benchmark framework remains compatible with both ap￾proaches: evaluation depends solely on the original file and the agent’s modified version—not on the sequence of actions taken. This design makes PPT-EVAL method-agnostic. 3.2. The PPT-EVAL Environment [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Representative slides from the 12 benchmark files, spanning topics (e.g., medicine, CS, accounting, history) and visual styles (text-heavy to graphics-heavy) to support heterogeneous task types. shows the percentage of slides relevant to PPT-EVAL tasks containing various elements (e.g., images, tables, anima￾tions, non-standard layouts, etc). The files span topics in￾cluding Medicine, Computer Science, Acc… view at source ↗
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of the 120 tasks by difficulty (easy/medi￾um/hard) and high-level intent categories (e.g., design & layout, image operations, table/data visualization). Zoom to see intent categories. draws inspiration from the RubricTree framework of Mind2Web 2 (Gou et al., 2025), while introducing several key modifications to better suit the PowerPoint domain. 3.5.1. TREE-STRUCTURED RUBRICS Each task is repr… view at source ↗
Figure 6
Figure 6. Figure 6: Example rubric tree and scores for the “add vCPU4” task from [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Breakdown of model success rates with respect to task categories representing PowerPoint structures (Images, Shapes, Tables, Text, etc.) relevant to the task [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of types of manual task (left) and rubric (right) refinement performed by Human Annotators. A.4. Rubric Draft Generation For each task in PPT-EVAL, we first prompt an LLM (e.g., Claude-4-Sonnet or GPT-4.1) to generate a draft rubric that decomposes the task into a tree of evaluation criteria. The rubric specifies both critical requirements and non-critical aspects, ensuring that core goals are… view at source ↗
Figure 9
Figure 9. Figure 9: Example of perfect human-rubric agreement: BMI table insertion task completion 26 [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Example of human-rubric agreement on significant progress: Prostomium text box task 0.5 1 0.5 1.0 Add Appropriate Emojis to Social Class Types on Slide 8 Emojis Present on Slide 8 Emoji Appropriateness No unintended changes Rubric-based scoring critical node original modified 0 Emoji Positioning [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Example of VLM hallucination and subjective disagreement: emoji placement task 27 [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Example of partial completion granularity disagreement: Image replacement task B.4. Implications for Rubric Reliability These meta-evaluation examples demonstrate that while our automated rubric system achieves reasonable alignment with human judgment, there are still sources of disagreement that persist. Some of these are a consequence of LLMs’ subjectiveness. But subjective tasks are unavoidable in prod… view at source ↗
Figure 13
Figure 13. Figure 13: Per-cluster success and failure breakdown across models. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: CUA’s successful trajectory for the task ‘Crop the architectural diagram image on slide 8 to show only the top two rows of buildings’. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_14.png] view at source ↗
Figure 14
Figure 14. Figure 14: CUA’s successful trajectory for the task ‘Crop the architectural diagram image on slide 8 to show only the top two rows of buildings’ (continued). 33 [PITH_FULL_IMAGE:figures/full_fig_p033_14.png] view at source ↗
Figure 14
Figure 14. Figure 14: CUA’s successful trajectory for the task ‘Crop the architectural diagram image on slide 8 to show only the top two rows of buildings’ (continued). 34 [PITH_FULL_IMAGE:figures/full_fig_p034_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: CUA’s failed trajectory for the task ‘Add slide numbers to every slide except the title slide’. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_15.png] view at source ↗
Figure 15
Figure 15. Figure 15: CUA’s failed trajectory for the task ‘Add slide numbers to every slide except the title slide’ (continued). 36 [PITH_FULL_IMAGE:figures/full_fig_p036_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Claude’s successful trajectory for the task "Add slide numbers to all slides except the title slide." 37 [PITH_FULL_IMAGE:figures/full_fig_p037_16.png] view at source ↗
Figure 16
Figure 16. Figure 16: Claude’s successful trajectory for the task "Add slide numbers to all slides except the title slide" (continued). 38 [PITH_FULL_IMAGE:figures/full_fig_p038_16.png] view at source ↗
Figure 16
Figure 16. Figure 16: Claude success steps (continued). 39 [PITH_FULL_IMAGE:figures/full_fig_p039_16.png] view at source ↗
read the original abstract

Creating and editing slides is a rich, multimodal activity that is ubiquitous in professional and educational settings, making it an ideal testbed for real-world computer-use agents. Microsoft PowerPoint is among the most widely adopted and feature-rich environments for presentation creation. We introduce PPT-Eval, a benchmark of 120 PowerPoint tasks across 12 files that cover both content creation and presentation editing scenarios, organized by difficulty. A central challenge in this domain is evaluation: tasks are complex, multimodal, and often admit many valid solutions. Moreover, today's agents frequently make only partial progress, which binary success metrics fail to capture. To address this, we design a robust evaluation framework to help create task-specific rubrics for PowerPoint tasks, taking inspiration from and building on past works for rubric-based evaluation. These rubrics award partial credit for intermediate steps, penalize unnecessary changes and poor aesthetics, and provide natural language feedback. This nuanced approach proves highly effective, achieving a Kendall's {\tau}-b correlation of 0.77 with human judgments. We find that existing frontier agents still struggle with solving PowerPoint tasks, with strong models like Claude-4.5-Opus achieving only a 45% success rate and an average partial score of 57%. The benchmark is located at: https://microsoft.github.io/ppteval.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces PPT-Eval, a benchmark of 120 PowerPoint tasks across 12 files covering content creation and editing scenarios organized by difficulty. It proposes task-specific rubrics that award partial credit for intermediate steps, penalize unnecessary changes and poor aesthetics, and provide natural language feedback; these rubrics achieve a Kendall's τ-b correlation of 0.77 with human judgments. The work reports that frontier agents struggle, with Claude-4.5-Opus reaching only 45% success rate and 57% average partial score. The benchmark is released at https://microsoft.github.io/ppteval.

Significance. If the rubrics prove robust, the benchmark would fill a gap in evaluating computer-use agents on complex, multimodal, open-ended professional tasks where binary success metrics are inadequate and partial progress is common. The reported human correlation and partial-credit design are strengths that could make the resource useful for tracking progress in agent capabilities.

major comments (2)
  1. [Abstract] Abstract: The central claim that the rubrics address the evaluation challenge for tasks that 'admit many valid solutions' rests on the untested assumption that designer-defined criteria systematically cover the solution space without bias. No details are given on rubric construction process, inter-rater validation of rubric completeness, or testing against unanticipated valid solutions; the 0.77 Kendall's τ-b correlation (measured on rubric-scored outputs) would not detect systematic under-scoring of valid but unanticipated paths.
  2. [Abstract] Abstract: The reported 45% success rate and 57% average partial score for Claude-4.5-Opus are presented as evidence that agents struggle, but without the methods section describing task selection criteria, file diversity, or how partial scores aggregate across the 120 tasks, it is impossible to assess whether these numbers support the broader conclusion about frontier model limitations.
minor comments (2)
  1. [Abstract] The abstract mentions 'taking inspiration from and building on past works for rubric-based evaluation' but does not cite the specific prior works; adding these references would improve traceability.
  2. [Abstract] The benchmark URL is provided, but the manuscript should include a brief description of the repository contents (e.g., task files, rubric templates, agent interaction logs) to aid reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We respond point-by-point to the major comments below.

read point-by-point responses

  1. Referee: The central claim that the rubrics address the evaluation challenge for tasks that 'admit many valid solutions' rests on the untested assumption that designer-defined criteria systematically cover the solution space without bias. No details are given on rubric construction process, inter-rater validation of rubric completeness, or testing against unanticipated valid solutions; the 0.77 Kendall's τ-b correlation (measured on rubric-scored outputs) would not detect systematic under-scoring of valid but unanticipated paths.

    Authors: We agree that the manuscript provides no details on the rubric construction process, inter-rater validation, or explicit testing against unanticipated solutions. The correlation with human judgments validates alignment on the evaluated outputs but does not address potential systematic gaps in coverage. We will revise the methods section to describe the rubric design process (including how multiple valid solutions were considered during creation) and explicitly note the limitations of the correlation metric regarding unanticipated paths. revision: yes

  2. Referee: The reported 45% success rate and 57% average partial score for Claude-4.5-Opus are presented as evidence that agents struggle, but without the methods section describing task selection criteria, file diversity, or how partial scores aggregate across the 120 tasks, it is impossible to assess whether these numbers support the broader conclusion about frontier model limitations.

    Authors: We agree the abstract presents the aggregate numbers without referencing supporting details. The manuscript will be revised to ensure the methods section explicitly describes task selection criteria, the 12 files' diversity and difficulty organization, and the aggregation of partial scores (mean across tasks). We will also update the abstract to briefly reference these elements and point to the methods, allowing readers to better evaluate the conclusion. revision: yes

Circularity Check

0 steps flagged

No circularity; benchmark and rubrics validated externally via human correlation

full rationale

The paper introduces PPT-Eval with task-specific rubrics for multimodal PowerPoint tasks and reports a Kendall's τ-b correlation of 0.77 with human judgments as evidence of effectiveness. No equations, derivations, fitted parameters, or predictions appear. Rubrics are stated to take inspiration from prior works, but the reported correlation is measured against independent human judgments rather than reducing to self-referential fitting or self-citation. The work is self-contained against external benchmarks with no load-bearing step that reduces by construction to its inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical free parameters, axioms, or invented entities are introduced; the work is an empirical benchmark creation.

pith-pipeline@v0.9.1-grok · 5809 in / 1213 out tokens · 43356 ms · 2026-07-01T06:51:14.208988+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references

  1. [1]

    Diversity– Try to avoid repetition of same/similar styles of tasks and choose tasks spread across the presentation rather than concentrated on the same slides

  2. [2]

    – Task rephrasing has a large impact on task difficulty and rubric generation

    You may need torephrase certain tasksthat are phrased ambiguously. – Task rephrasing has a large impact on task difficulty and rubric generation

  3. [3]

    NO RELATED TEXT

    Aim to select3 easy, 4 medium, and 3 hardtasks per file: (a)Easy: Simple commanding tasks (∼requiring≤5 steps;≤1 min). e.g., i. Insert a subtitle below the title saying “...” on slide 5. ii. Change the background color of slide 5 to sky blue. (b) Medium: Slightly more complex (∼2–5 min; 5–10 steps), compound tasks where a user will likely start seeing val...

  4. [4]

    A rubric tree consists of nodes that each refer to a particular criterion

  5. [5]

    A criterion can be decomposed into sub-criteria and so on

  6. [6]

    A criterion node can be critical or non-critical

  7. [7]

    A parent node’s score computation depends on whether its children are critical, non-critical, or a mix of both

  8. [8]

    If both critical and non-critical children exist, the parent score is max(0,average(critical)−λ×(1−average(non-critical))), whereλ= ‘%.2f’|format(non_critical_weight)

  9. [9]

    Otherwise (all children critical or all non-critical), the parent score is the average of all children

  10. [10]

    children

    A leaf node’s score is computed using a particular scoring script written for that leaf node. The rubric tree should be as comprehensive as possible, and should be able to evaluate the task in a way that is fair and accurate. The rubric tree should be as concise as possible, and should be able to be easily understood by a human. The rubric tree should be ...

  11. [11]

    Look specifically for:

    Node using VLM: defcompute_score() ->tuple[str,float]: # Compare slide 11 screenshots to detect color changes from yellow to blue original_slide_11 = None modified_slide_11 = None forscreenshotinoriginal_ppt_screenshots: ifscreenshot.slide_number == 11: original_slide_11 = screenshot break forscreenshotinmodified_ppt_screenshots: ifscreenshot.slide_number...

  12. [12]

    Yellow color text in the original image

  13. [13]

    At least some yellow colored text has been changed to blue highlighting', 1.0 elif 'NO' in response: return 'No yellow colored text has been changed to blue highlighting

    Whether any of that yellow color text has been highlighted with blue highlighting in the modified image Respond with: - \'YES\' if you can identify at least some yellow color text that has been 20 PPT-EVAL: A Benchmark for Computer-Use Agents on PowerPoint Tasks changed to blue highlighting - \'NO\' if no yellow color text has been changed to blue highlig...

  14. [14]

    rotation

    Node usingpython-pptx: defcompute_score() ->tuple[str,float]: '''Check that the (only) image on slide 6 is rotated ~45 degrees clockwise.''' frompptximportPresentation SLIDE_IDX = 5# slide numbers are 1-based TARGET_ROTATION = 45 ROTATION_TOLERANCE = 2# degrees # Load modified presentation prs = Presentation(modified_ppt_path) if len(prs.slides) <= SLIDE_...

  15. [15]

    Node usingPPTDiff: defcompute_score(): # Checks that all text on the references slide has 'Fly In' animation from the left. slide_ids =set() forslideinppt_diff.added_slides + [s2for_, s2inppt_diff.modified_slides]: ifslide.titleand'references'inslide.title.lower(): slide_ids.add(slide.slide_id) if notslide_ids: frompptximportPresentation try: pres = Prese...

  16. [16]

    Mind2web 2: This is the scoring strategy introduced in the mind2web 2 paper which gates on critical node success and averages non-critical node scores

  17. [17]

    Default: This is a different scoring strategy where we take a weighted average between critical and non-critical node scores when calculating parent scores. Please try out various levels of task completion progress (no progress, various styles of partial progress, task complete) with both scoring strategies and vote on which method wins for this task/rubr...

  18. [18]

    2.Category 2 (expected score:>0and<0.5): Partial completion but closer to a score 0 than to 1

    Category 1 (expected score: 0): No progress is made towards task completion—this can be the original deck or completely irrelevant changes. 2.Category 2 (expected score:>0and<0.5): Partial completion but closer to a score 0 than to 1. 3.Category 3 (expected score:≥0.5and<1): Partial completion but closer to a score of 1 than to 0. 4.Category 4 (expected s...

  19. [19]

    Human Error: Cases where human evaluators misunderstood task requirements or made inadvertent errors during assessment (no example collected due to self-evident nature)

  20. [20]

    Subjective Task Elements: Disagreements stemming from legitimate differences of opinion on subjective aspects such as color suitability, positioning preferences, or aesthetic choices

  21. [21]

    LLM Hallucination: Instances where the non-deterministic language model component of the rubric scorer generated inaccurate assessments or identified non-existent elements

  22. [22]

    Some Progress

    Partial Completion Granularity: Cases where both human and automated evaluators agreed that partial progress was made, but disagreed on the specific degree of completion (e.g., distinguishing between "Some Progress" and "Significant Progress"). Our evaluation framework employs four completion categories: No Progress (0.0), Some Progress (0.33), Significan...

    2025

  23. [23]

    ** They define the goal andthe exact path you must write your output to

    **Read`TASK.md`and`OUTPUT_INSTRUCTIONS.md`first. ** They define the goal andthe exact path you must write your output to. Follow the output path literally - the grader looksforthat pathandnothingelse

  24. [24]

    ** Copy thefilebefore modifying it

    **Treat`inputs/`asread-only. ** Copy thefilebefore modifying it

  25. [25]

    If thefile ismissingornamed differently, the task scores 0

    **Write your result to`output/<task_id>.<ext>` ** exactlyasspecified. If thefile ismissingornamed differently, the task scores 0

  26. [26]

    On slide N

    **When the outputfileexistsandreflects the requested changes, exit. ** Donotleave background processes running. ## Tools you have - The`pptx`skillin`.claude/skills/pptx/`(SKILL.md, plus OOXML pack/unpack scriptsandpython-pptx workflows). Read`SKILL.md`for guidance on: - text extraction (`python -m markitdown ...`) - unpacking/repacking pptx filesforraw XM...

  27. [27]

    **Re-read`TASK.md` ** and listevery concrete requirement (slide N, targetobject, attribute, exact value, etc.)

  28. [28]

    - For color/font/size changes: read the relevantproperty and assert it equals the requested value

    **Open your outputfile ** with`python-pptx`(orunpack the XMLifthe changeisstyle/layout)andprogrammatically confirm each requirement issatisfied: - For text changes: read the run text on the targeted shapeandcheck it matches exactly (including capitalisationandpunctuation). - For color/font/size changes: read the relevantproperty and assert it equals the r...

  29. [29]

    **Diff against`inputs/<file>.pptx` ** to make sure you did **not** change anything outside the task scope

  30. [30]

    Allow yourself up to 3 verification retries before giving up

    Ifanycheck fails, **fix thefile andre-verify **. Allow yourself up to 3 verification retries before giving up

  31. [31]

    Keep the verification scriptin`sandbox/`orstdout; you dont need to ship it

    Only exit onceallverification checkspass,orafter 3 retries - log what failed so the runisdebuggable. Keep the verification scriptin`sandbox/`orstdout; you dont need to ship it. The harness only reads`output/<task_id>.<ext>`. ## Common failure modes to guard against - Writing the output under the wrong filenameorextension. - Modifying`inputs/`inplace inste...