Pith.

Artificial Intelligence · arXiv:2609.38652 v1 · submitted 29 Sep 2026 · 37 references

AgBench: Agentic AI Benchmarks for Personal AI Devices

Yizhou Han, Di Wu, Dhananjay Saikumar, Blesson Varghese

The paper says

“Agentic AI systems increasingly rely on cloud-hosted large language models for planning, tool use, and iterative execution, raising concerns about API cost and data exposure.”

more of the abstract

Advances in personal AI devices enable agents to execute locally, but limited resources on device may affect task success and performance. Existing benchmarks are inadequate for systematically characterizing these trade-offs across devices, workloads, and deployment architectures. We present AgBench, a benchmark suite and open artifacts for reproducible evaluation of agentic AI on personal devices. Using AgBench, we evaluate local, hybrid, and cloud execution across agentic workloads, examining task success, latency, cloud API cost, and data exposure. Our results, drawn from over 162.07 million data points, show that personal AI devices can complete many agent tasks locally, but local-only execution generally has lower task success and longer completion times than cloud-only execution, especially as concurrency increases. Local-only execution eliminates cloud model API costs and sensitive-information exposure to cloud agents. Hybrid execution can improve task success, but its cloud cost and data exposure depend on how agents divide work and share information. No single architecture performs best across task success, goodput, cloud cost, and data exposure; deployment choices should reflect the intended workload and device capabilities. AgBench is available at https://anonymous.4open.science/r/AgBench-2777.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 2 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Careful, genuinely useful empirical benchmark with an honest but unresolved confound in its headline concurrency claim: send to review and push the authors on the KV-pool and model-pairing choices. This line is wrong

2 major objections · 3 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 37 references say.

The paper says, at §4.1 / Table 12 / §F

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The central claim that local-only success 'falls further behind' cloud-only 'especially as concurrency increases' (b5, b27) rests primarily on the RTX 5090 result (b149): LO drops from 35/60 at C=1 to 12/60 at C=8, with 32 context-limit errors among the failures.

more of objection 1

The paper itself explains that concurrent requests compete for a shared KV-cache pool whose capacity is set to 65,536 tokens in Table 12—equal to the per-agent context window and one eighth of the Max+ 395's 524,288-token pool—and Section F attributes the RTX 5090's frequent context-limit errors precisely to this smaller pool. Because no experiment varies the KV budget, the sharp concurrency collapse is established under exactly one configuration on one device, while the second device shows a much milder drop (31 to 22, b150) with no such errors. I recommend either adding a sensitivity run on the RTX 5090 with a larger shared KV pool (e.g., matching the Max+ 395 budget) and reporting whether the C=8 collapse persists, or explicitly rescoping the 'especially at higher concurrency' conclusion as configuration-dependent; as written, the headline is a configuration-specific artifact risk rather than a demonstrated device property.

The paper says, at §3.3 / §4.1

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The entire LO-versus-CO comparison and the local-versus-cloud generalization use a single local model—Qwen3.8-27B at UD-Q6_K_L 6-bit quantization served by llama.cpp—against a single cloud model, DeepSeek V4 Flash with a 1,000,000-token context window. The success gap, the latency gap, and even the KV-cache pressure that drives the high-concurrency finding could plausibly reflect the difference in model capability and context window between …

more of objection 2

… this pair rather than any property of the devices or of 'personal AI devices' in general, which is how the abstract and conclusion phrase the results. Since the authors do not discuss this as a scope limitation, I ask them to either (a) add a second local model of different scale or deployment precision and/or a second cloud model and check whether the qualitative Q1-Q4 trends survive, or (b) rewrite the generalization statements so that they are explicit that the findings characterize this specific model and backend pairing.

3 minor objections

The paper says, at Table 4

Pith says: minor objection 1 of 3

The mean completion-time comparisons at RTX 5090 C=8 are computed on n=3 tasks and at C=4 on n=6 tasks; with samples this small, the per-architecture ordering (e.g., CO at 0.18 minutes) should carry error bars or an explicit note that these cells are too small to support comparative conclusions.

The paper says, at §E / Table 14

Pith says: minor objection 2 of 3

The stability runs cover only RTX 5090 at C=1 and C=8, and Table 14 shows appreciable run-to-run spread (LO goodput of 4.57 versus 6.40 tasks per hour at C=8); a sentence noting that C=2, C=4, and all Max+ 395 configurations were single-run would help readers calibrate the precision of those cells.

show 1 more

The paper says, at Figure 4

Pith says: minor objection 3 of 3

The failure analysis labels 'verification failure' as a single category, and the fact that HCL records zero context-limit errors while its local component does exhaust context is explained only in the text; a caption note clarifying that the verification-failure category aggregates diverse causes, and that HCL's zero-context-error row does not mean its local component avoids context exhaustion, would prevent misreading.

What would settle it

Re-run the LO architecture on the RTX 5090 at C=1 and C=8 with the shared KV-cache pool raised to its memory-limited maximum, for example by switching Qwen3.8-27B to 4-bit quantization to free VRAM and expanding the pool in Table 12 well beyond 65,536 tokens, holding all other settings fixed. If the 32 context-limit errors at C=8 largely disappear and LO success moves materially above 12/60 (b149), the 'LO collapses under concurrency' claim is a configuration artifact and should be re-scoped; if the errors persist, the device-memory explanation holds.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 2 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 33 s · 22,952 tokens read, 4,520 written
  3. Editor wrote the verdict above 25 s · 24,067 tokens read, 3,774 written
  4. Referee major revision: 2 major objections, 3 minor 44 s · 23,346 tokens read, 6,248 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Here's the one-line: this is a careful, useful systems paper — the first framework I've seen that jointly captures task success, latency, cloud API cost, and data exposure across local, cloud, and two hybrid architectures on real hardware — but its headline concurrency finding leans on a configuration choice the authors flag without resolving. Treat the magnitude claims with caution, not the direction.

What's actually new: AgBench's four-metric by four-architecture sweep on two real devices (discrete RTX 5090 PC vs integrated Ryzen AI Max+ 395), with roughly 115 GB of released traces, pinned software commits, a fixed seed, isolated Docker environments, task-specific verifiers, and repeat runs on the RTX 5090 showing the core trends hold. The methodology is unusually specific about versions and settings. The qualitative results are internally consistent: local-only eliminates cloud cost and exposure but loses task success and speed; hybrid local-led recovers some success while keeping cost and exposure below cloud-only; no architecture dominates on all four metrics. That last point is the paper's real contribution and it survives scrutiny.

Soft spots, ordered. First, the RTX 5090 collapse at concurrency 8 (35 to 12 tasks) is driven by the shared KV-cache pool set to 65,536 tokens — equal to the per-agent context window and an eighth of the Max+ 395's pool. Section F says exactly this: pool exhaustion causes context-limit errors. On the Max+ 395 with the bigger pool, the concurrency drop is roughly three times milder (31 to 22), and there the mechanism is different — timeouts from slower local inference. So the direction holds on both devices, which is evidence in the paper's favor, but the magnitude and the 'limit concurrency' design implication are anchored on one hand-set configuration on one device. Enlarge the pool or use a stronger local model and the headline numbers move. This is acknowledged, not concealed, but the generalization to 'personal AI devices' remains the load-bearing unsettled point.

Second, the model pairing: a single 6-bit quantized 27B local model against one frontier cloud model with a 1M-token context window. The capability gap confounds the device-versus-cloud interpretation. A stronger local model could change large parts of the story.

Third, only the RTX 5090 gets repeat runs, and only at C=1 and C=8; the full 32-configuration matrix runs once, and some high-concurrency completion-time cells have n=3. They report the variance honestly in Section E, but the main narrative underweights it.

Minor: the exposure metric rests on a manually built 527-item sensitive set plus an LLM detector, and the HLL exposure drop partly reflects lower success — which the paper concedes.

Bottom line: this is the benchmark the on-device-agent field needs, the claims are honest, the artifacts are unusually complete, and the flaws are addressable, not fatal. A serious referee should push for a larger KV pool and a second local model before the concurrency claim is stated as device-general. Send it to review.

The strongest objection

The central concurrency finding—that local-only execution falls sharply behind cloud-only at high concurrency—is anchored by one number: on the RTX 5090, LO success drops from 35/60 at C=1 to 12/60 at C=8, driven by 32 context-limit errors (b149, Section 4.1). The paper states those errors arise because 'concurrent requests compete for a shared KV-cache pool, so pool exhaustion can cause a context error even when a request remains below its own sequence limit,' and Section F attributes the 'frequent context-limit errors' on RTX 5090 to 'the smaller shared KV pool,' contrasting with the Max+ 395's 524,288-token pool, where no context-limit errors occur and LO's drop is far milder (31→22, b150). The small pool is itself a configuration choice: Table 12 fixes the shared KV capacity at 65,536 tokens—equal to the per-agent context window and only 1/8 of the Max+ 395's pool—while the 6-bit 27B model consumes most of the device's 32 GB VRAM. The pool is sized to fit that model, so the collapse at C=8 is an artifact of the joint model/quantization/KV-budget decision, not an inherent property of the device or of 'personal AI devices' in general. The repeated runs (R1–R3), which the paper reports as stable (12,12,11), confirm the effect is reproducible under this configuration—but reproducibility of a config-specific artifact does not make the qualitative claim 'LO falls further behind CO at higher concurrency' (b5, b27) robust. On the Max+ 395 the concurrency effect is roughly three times weaker, so the paper's headline concurrency thesis is currently supported by one configuration on one device. The paper honestly acknowledges the small pool in Section F, so this is a matter of interpretation and generalization, not concealment, but it is the most load-bearing unsettled point: if the pool is enlarged, the 35→12 anchor could shift materially.

The reader's weakest_assumption already flags the hand-set 65,536-token RTX 5090 KV pool as the source of the context-limit errors behind the central high-concurrency finding, and my pass converges on the same point as the single most load-bearing concern. I narrowed the focus beyond the reader's two-factor confound (model capability gap plus KV pool) because the model gap inflates the overall LO-versus-CO success and latency differences at every concurrency level, whereas the dramatic collapse with concurrency—the paper's most striking claim, and the one that distinguishes 'device readiness' from mere capability difference—is specifically attributable to the KV-pool budget. The repeated runs (R1–R3) at C=8 on RTX 5090 give 12, 12, 11 successes, so the effect is genuinely reproducible; that reproducibility argues FOR a real config-driven mechanism, not against it. The paper deserves credit for explicitly connecting the small pool to the context-limit errors in Section F and for showing the concurrency effect is much milder on the Max+ 395 with its 8x larger pool, which is exactly the natural experiment that exposes the configuration-dependence of the headline. I did not find internal inconsistency or overclaiming: the 'no single architecture wins' thesis is well supported by the cost/exposure/goodput trade-offs, which are far less config-sensitive. The remaining risk is interpretive scope: 'local-only falls behind at higher concurrency' is presented as a general property (b5, b27) when the decisive evidence is one KV-pool setting on one device. A single config-sensitivity run would settle this, so the reader's CONDITIONAL verdict is appropriate, and my concern does not move it.

The referee's report

If its results hold, AgBench is a genuinely useful first systems-oriented characterization of agentic-AI deployment trade-offs on personal devices, distinguished by the release of execution traces, verifier-grounded success measurement, and careful contention isolation. The paper's discipline—pinned commits, fixed seed, repeat runs, transparent failure analysis, and per-task resource profiles—is a real strength and deserves explicit credit. The main limitation is that the headline concurrency finding is anchored on one configuration choice (the 65,536-token shared KV cache on the RTX 5090) combined with one model pair (a single quantized 27B local model versus a single frontier-capable cloud model), which restricts the generality of the qualitative claims as currently phrased. The contribution is solid as a methodology-plus-case-study; the claims about devices 'in general' need either sensitivity data or explicit rescoping.

Pith's reading of the paper (a machine's summary, not the authors')

AgBench is a benchmark suite of 60 agentic tasks across eight categories, run on two personal AI devices under four deployment architectures — local-only, cloud-only, and two hybrids (cloud-led and local-led) — at concurrency levels 1, 2, 4, and 8. The paper's central claim is that personal devices can complete a large subset of agent tasks locally, but local-only execution has lower task success and longer completion times than cloud-only execution, with the gap widening sharply as concurrency increases: on the RTX 5090, local-only success drops from 35 of 60 tasks at concurrency 1 to 12 at concurrency 8, while cloud-only holds near 48–50. Local-only execution eliminates cloud model API charges and sensitive-information exposure to cloud agents, while hybrid execution recovers some task success at cloud cost and exposure levels that depend on which agent leads and what information is shared. The conclusion is that no single architecture wins across task success, goodput, cloud cost, and data exposure; the right deployment depends on the workload and the device.

What carries the argument. The load-bearing objects are the AgBench task suite and the four deployment architectures. The suite comprises 60 tasks drawn from GAIA, TUA-Bench, and Terminal-Bench 2, spanning local-file analysis, information retrieval, calculation, office productivity, multimedia processing, scientific computing, system operations, and complex terminal problem solving, each with an isolated Docker environment and a task-specific verifier. The four architectures differ in where model inference occurs and who coordinates: local-only and cloud-only run the full agent loop on the device and in the cloud respectively, hybrid cloud-led delegates stages from a cloud planner to a local executor, and hybrid local-led has a local executor consulting a cloud reasoning partner. The four measured outcomes — task success, completion time/goodput, cloud API cost, and the fraction of 527 pre-identified sensitive items appearing in cloud-agent inputs — carry the argument. A shared local model service (Qwen3.8-27B, 6-bit quantized) with a hand-set KV-cache capacity of 65,536 tokens on the RTX 5090 versus 524,288 on the Max+ 395, against cloud inference from DeepSeek V4 Flash, supplies the empirical contrast.

Core claim. Running agentic AI locally is feasible but not free of trade-offs. Across 1,920 task executions and roughly 162 million measurement records, the paper finds that personal AI devices can complete more than half of the 60-task suite locally at low concurrency (35/60 on the RTX 5090, 31/60 on the Max+ 395), and that local-only execution matches cloud-only on some task categories such as calculation and system operations. The decisive finding is concurrency-dependent: local-only success collapses to 12/60 at concurrency 8 on the RTX 5090, driven by context-limit errors as concurrent requests exhaust the shared KV-cache pool, while cloud-only remains near 48–50; on the Max+ 395 the same wedge appears as timeouts from slower local inference. Hybrid architectures recover some of the lost success at low concurrency — both hybrids beat local-only at concurrency 1, completing 39 and 44 tasks respectively — but their advantage is inconsistent at higher concurrency, and cloud-led hybrids can be slower than local-only because the cloud agent delegates extensive work and waits. Fully local execution reports zero cloud API cost and zero sensitive-information exposure across all configurations, whereas cloud-only exposes 76% of the 527 identified sensitive items; hybrids expose 0–22% when the local agent leads, versus 63–76% when the cloud leads. The paper's overall claim is that these four metrics cannot be simultaneously maximized by any one architecture.

The premise it rests on. The entire comparison rests on one quantized 27B local model against one frontier-capability cloud model with a 1,000,000-token context window, together with a hand-set 65,536-token shared KV-cache capacity on the RTX 5090 — so the large capability and context-window gap, more than device constraints alone, plausibly produces the local-versus-cloud success and latency differences, and the small KV pool produces the context-limit errors behind the high-concurrency collapse.

If the paper is right (Pith's inferences, not the authors')

A user who cares about task success and speed should prefer cloud-only execution, which completed the shared task set in 2.07 minutes versus 7.01 minutes for local-only at concurrency 1 on the RTX 5090, and kept success near 48–50 of 60 even at concurrency 8.

A user whose priority is privacy or cost can run local-only: it exposes none of the 527 sensitive items and incurs zero API charges, at the price of lower task success and longer completion times.

Local-led hybrid execution (HLL) is the paper's suggested middle path, recovering some task success while keeping cloud cost and sensitive-information exposure below cloud-only levels, but it does not close the goodput gap with cloud-only.

Concurrency beyond 1 should be the exception rather than the default on personal AI devices, since increasing concurrency improves goodput but degrades task success for every architecture that relies on local inference.

Cloud-led hybrid (HCL) carries a hidden cost: repeated delegation and context-error cycles can drive cloud API spending above cloud-only — unsuccessful HCL executions accounted for 85.2% of its cloud cost at concurrency 8 on the RTX 5090.

The headline numbers compare one quantized 27B local model against a frontier-capability cloud model with a 1,000,000-token context window; a stronger local model or a larger shared KV pool could plausibly shrink or shift the concurrency-driven success gap, so the result should be read as a characterization of this model pair and device settings rather than of local inference in general.

Because the RTX 5090's small 65,536-token KV pool is what produces the context-limit errors behind the central 'local falls behind at high concurrency' finding, the gap may be at least partly a configuration artifact — a testable extension would vary KV-cache capacity independently of concurrency.

Cross-device generalization is not established: the two devices differ in KV pool size (65,536 vs 524,288) and failure mode (context-limit errors vs timeouts), so the paper's trends may reflect hardware configuration as much as hardware capability.

The gating question for practitioners — at what success rate and latency threshold a given workload crosses over from local to cloud — is left open; the benchmark's released execution traces would allow exactly that threshold analysis on other model–device pairs.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Agentic AI systems increasingly rely on cloud-hosted large language models for planning, tool use, and iterative execution, raising concerns about API cost and data exposure. Advances in personal AI devices enable agents to execute locally, but limited resources on device may affect task success and performance. Existing benchmarks are inadequate for systematically characterizing these trade-offs across devices, workloads, and deployment architectures. We present AgBench, a benchmark suite and open artifacts for reproducible evaluation of agentic AI on personal devices. Using AgBench, we evaluate local, hybrid, and cloud execution across agentic workloads, examining task success, latency, cloud API cost, and data exposure. Our results, drawn from over 162.07 million data points, show that personal AI devices can complete many agent tasks locally, but local-only execution generally has lower task success and longer completion times than cloud-only execution, especially as concurrency increases. Local-only execution eliminates cloud model API costs and sensitive-information exposure to cloud agents. Hybrid execution can improve task success, but its cloud cost and data exposure depend on how agents divide work and share information. No single architecture performs best across task success, goodput, cloud cost, and data exposure; deployment choices should reflect the intended workload and device capabilities. AgBench is available at https://anonymous.4open.science/r/AgBench-2777.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-01

The authors' main claim, as the classifier picked it out 0.97

“No single architecture performs best across task success, goodput, cloud cost, and data exposure; deployment choices should reflect the intended workload and device capabilities.”

How the main claim is stated: asserted 0.99

What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.57 · human participants 0.06 · controlled comparison 0.57 · randomized assignment 0.02 · states quantitative result 0.97 · compares with prior results 0.72 · mentions code or data availability 0.97

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Agentic AI systems increasingly rely on cloud-hosted large language models for planning, tool use, and iterative execution, raising concerns about API cost and data exposure. background 1.00 0.04 0.02 asserted 0.98
Advances in personal AI devices enable agents to execute locally, but limited resources on device may affect task success and performance. background 1.00 0.05 0.02 hedged possibility 0.93
Existing benchmarks are inadequate for systematically characterizing these trade-offs across devices, workloads, and deployment architectures. gap 1.00 0.07 0.02 asserted 1.00
We present AgBench, a benchmark suite and open artifacts for reproducible evaluation of agentic AI on personal devices. contribution 1.00 0.91 0.05 asserted 1.00
Using AgBench, we evaluate local, hybrid, and cloud execution across agentic workloads, examining task success, latency, cloud API cost, and data exposure. method 0.84 0.96 0.05 asserted 0.90
Our results, drawn from over 162.07 million data points, show that personal AI devices can complete many agent tasks locally, but local-only execution generally has lower task success and longer completion times than cloud-only execution, especially as concurrency increases. result 1.00 0.97 0.45 asserted 1.00
Local-only execution eliminates cloud model API costs and sensitive-information exposure to cloud agents. result 0.86 0.44 0.07 asserted 1.00
Hybrid execution can improve task success, but its cloud cost and data exposure depend on how agents divide work and share information. result 0.89 0.88 0.08 hedged possibility 0.74
No single architecture performs best across task success, goodput, cloud cost, and data exposure; deployment choices should reflect the intended workload and device capabilities. interpretation 0.95 0.89 0.77 asserted 0.99
AgBench is available at https://anonymous.4open.science/r/AgBench-2777. contribution 0.90 0.28 0.02 asserted 0.94

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-01: 1 request, 7,541 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (12)
Figure 1 of the paper
Figure 1. Overview of AgBench. Agent workloads are executed under controlled combinations of personal AI devices, deployment architectures, and workload concurrency levels. AgBench captures task outcomes, execution and system behavior, cloud model usage, and data sent to cloud for characterizing task success, execution performance, cloud API cost, and data exposure
Figure 2 of the paper
Figure 2. Deployment architectures evaluated by AgBench. Inference setup. For local inference, we run Qwen3.8-27B [22] with UD-Q6_K_L 6-bit quantization [25] using llama.cpp (commit 0b5be7e4) [13], with CUDA 12.8.1 on the RTX 5090 and Vulkan on the Max+ 395. We enable Flash Attention [11], an 8-bit KV cache, and multi-token prediction (MTP) [14] speculative decoding with up to two draft tokens, and allow up to eight concurrent model calls on each device. For cloud inference, we use DeepSeek V4 Flash [12] with high reasoning effort. The local and cloud models use context windows of 65,536 and…
Figure 4 of the paper
Figure 4. Failure analysis. errors are recorded; instead, slower local inference contributes to timeouts for LO, HCL, and HLL (20, 41, and 33 tasks, respectively). The two hybrid architectures differ in how local context errors appear in the results. At 𝐶 = 8 on the RTX 5090, HCL records 44 verification failures but no context-limit errors. In all 44 cases, its local component encounters a context error, but the cloud agent subsequently returns a final response that fails verification. Thus, HCL’s zero recorded context-limit errors do not indicate that its local component avoids context…
Figure 5 of the paper
Figure 5. Goodput (no. of successfully completed tasks / total
Figure 6 of the paper
Figure 6. Breakdown of mean execution time per task.
Figure 7 of the paper
Figure 7. Cloud API cost breakdown
Figure 8 of the paper
Figure 8. Cross metric tradeoffs comparing task success rate
Figure 9 of the paper
Figure 9. Tasks successfully completed by pairs of architec
Figure 10 of the paper
Figure 10. Progress of task execution across devices, deploy
Figure 11 of the paper
Figure 11. Local model-serving throughput. Prefill and de
Figure 12 of the paper
Figure 12. System resource usage over elapsed run time. Rows

Showing 11 of the paper's 12 figures. Figures are the authors', reproduced from the paper.

References (37)
  1. [1] Reyna Abhyankar, Qi Qi, and Yiying Zhang. 2026. OSWorld-Human: Bench- marking the efficiency of computer-use agents.Proceedings of Machine Learning and Systems8 (2026), 482–494 on Pith
  2. [2] Advanced Micro Devices. 2026. Agent Computers. Powering the Future of Agentic AI. Official product webpage. Accessed: 2026-09-10. https://www.amd. com/en/products/processors/consumer/agent-computers.html on Pith
  3. [3] Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. 2024. Agent S: An open agentic framework that uses computers like a human. arXiv:2410.08164 [cs.AI] https://arxiv.org/abs/2410.08164 on PithPith's reviewarXiv
  4. [4] Anthropic. 2026. How Claude Code Works. Claude Code Documentation. Accessed: 2026-09-18. https://code.claude.com/docs/en/how-claude-code-works on Pith
  5. [5] Michael Bolin. 2026. Unrolling the Codex Agent Loop. OpenAI Engineering. Accessed: 2026-09-18. https://openai.com/index/unrolling-the-codex-agent- loop/ on Pith
  6. [6] Hongru Cai, Yongqi Li, Ran Wei, and Wenjie Li. 2026. PalmClaw: A native on-device agent framework for mobile phones. arXiv:2607.13027 [cs.CL] on PithPith's reviewarXiv
  7. [7] Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, and Wei Wang. 2026. From LLM inference to agentic workloads: Characterization and implications for serving systems. arXiv:2608.15127 [cs.OS... on PithPith's reviewarXiv
  8. [8] Shoufa Chen, Luyuan Wang, Xuan Yang, Zhiheng Liu, Yuren Cong, Yuanfeng Ji, Feiyan Zhou, Xiaohui Zhang, Fanny Yang, and Belinda Zeng. 2026. TUA-Bench: A benchmark for general-purpose terminal-use agents. arXiv:2606.28480 [cs.SE] https://arxiv.org/abs/2606.28480 on PithPith's reviewarXiv
  9. [9] Jae-Woo Choi, Hyungmin Kim, Hyobin Ong, Youngwoo Yoon, Minsu Jang, Do- hyung Kim, and Jaehong Kim. 2026. ReAcTree: Hierarchical LLM agent trees with control flow for long-horizon task planning. InProceedings of the 25th In- ternational Conference on Autonomous Agents and Multi... on Pithdoi:10.65109/ucgt7089
  10. [10] Jinha Chung, Byeongjun Shin, Jiin Kim, and Minsoo Rhu. 2026. Agent-X: Full pipeline acceleration of on-device AI agents. InProceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services(University of Cambridge, Cambridge, United Kingdom)(M... on Pithdoi:10.1145/3745756.3809195
  11. [11] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAt- tention: Fast and memory-efficient exact attention with IO-awareness.Advances in Neural Information Processing Systems35 (2022), 16344–16359 on Pith
  12. [12] DeepSeek-AI. 2026. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv:2606.19348 [cs.CL] https://arxiv.org/abs/2606.19348 on PitharXiv
  13. [13] ggml-org. 2026. llama.cpp. GitHub repository. Revision 0b5be7e4. Accessed: 2026-09-18. https://github.com/ggml-org/llama.cpp/tree/ 0b5be7e4a25862bc2777d0c47eae18788a8c963a on Pith
  14. [14] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & faster large language models via multi-token prediction. InProceedings of the 41st International Conference on Machine Learning (ICML’24). JMLR.org, Vienna, Austria, Ar... on Pith
  15. [15] Vincent Hsiao, Mak Roberts, and Leslie Smith. 2026. Procedural knowledge improves agentic LLM workflows. InProceedings of the 25th International Confer- ence on Autonomous Agents and Multiagent Systems(Paphos, Cyprus)(AAMAS ’26). International Foundation for Autonomous Agents ... on Pithdoi:10.65109/vevz5917
  16. [16] Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh, and Ruslan Salakhutdinov. 2026. MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents. arXiv:2606.16748 [cs.LG] https://arxiv.org/abs/2606. 16748 on PitharXiv
  17. [17] Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E Kelly Buchanan, et al. 2026. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv:2601.11868 [cs.SE] https://ar... on PithPith's reviewarXiv
  18. [18] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: A benchmark for general AI assistants. InInternational Conference on Learning Representations, Vol. 2024. 9025–9049 on Pith
  19. [19] Microsoft. 2026. 2026 Work Trend Index Annual Report: Agents, human agency, and the opportunity for every organization. https://www.microsoft.com/en- us/worklab/work-trend-index/agents-human-agency-and-the-opportunity- for-every-organization. Accessed: 2026-09-12 on Pith
  20. [20] NVIDIA. 2026. NVIDIA DGX Spark. Official product webpage. Accessed: 2026-09-10. https://www.nvidia.com/en-us/products/workstations/dgx-spark/ on Pith
show 17 more
  1. [21] OpenAI. n.d.. GPT-5.6 Sol. OpenAI API Documentation. Accessed: 2026-09-21. https://developers.openai.com/api/docs/models/gpt-5.6-sol on Pith
  2. [22] Qwen Team. 2026. Qwen3.8-27B. Hugging Face model card. Accessed: 2026-09-18. https://huggingface.co/Qwen/Qwen3.8-27B on Pith
  3. [23] Corrado Rainone, Davide Belli, Bence Major, and Arash Behboodi. 2026. When cloud agents meet device agents: Lessons from hybrid multi-agent systems. arXiv:2605.30102 [cs.MA] on PithPith's reviewarXiv
  4. [24] Romain Robbes, Théo Matricon, Thomas Degueule, Andre Hora, and Ste- fano Zacchiroli. 2026. Agentic much? Adoption of coding agents on GitHub. arXiv:2601.18341 [cs.SE] doi:10.48550/arXiv.2601.18341 on PithPith's reviewarXivdoi:10.48550/arxiv.2601.18341
  5. [25] Unsloth. 2026. Qwen3.8-27B-GGUF. Hugging Face model repository. Quantiza- tion: UD-Q6_K_L. Accessed: 2026-09-18. https://huggingface.co/unsloth/Qwen3. 8-27B-GGUF/blob/4ca720788d1e01f1bff70c033e0d0028fd02e502/Qwen3.8-27B- UD-Q6_K_L.gguf on Pith
  6. [26] Michael Wang, Yikang Yue, Shaobo Li, Yirui Eric Zhou, Chen Wang, and Jian Huang. 2026. Benchmarking LLM serving systems for agentic AI workloads with XPerf. arXiv:2608.20370 [cs.DC] https://arxiv.org/abs/2608.20370 on PithPith's reviewarXiv
  7. [27] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2025. OpenHands: An open platform for AI software developers as generalist agents. InInternational Conference on Learning Representations, Vol... on Pith
  8. [28] Xingyao Wang, Simon Rosenberg, Juan Michelini, Calvin Smith, Hoang Tran, Engel Nyst, Rohit Malhotra, Xuhui Zhou, Valerie Chen, Robert Brennan, and Graham Neubig. 2026. The OpenHands Software Agent SDK: A composable and extensible foundation for production agents. arXiv:2511.03... on PithPith's reviewarXiv
  9. [29] Di Wu, Jiawei Luo, Yizhou Han, Weisong Shi, and Blesson Varghese. 2026. Agentic Edge AI.IEEE Internet Computing(2026). doi:10.1109/MIC.2026.3724303 on Pithdoi:10.1109/mic.2026.3724303
  10. [30] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652 on Pith
  11. [31] Biao Yi, Xueyu Hu, Yurun Chen, Shengyu Zhang, Hongxia Yang, and Fan Wu on Pith
  12. [32] Mario Zechner. 2026. Pi Coding Agent. GitHub software release. Version 0.84.1. Accessed: 2026-09-18. https://github.com/earendil-works/pi/releases/tag/v0.84.1 on Pith
  13. [33] Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, Liqun Li, Yu Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2025. UFO2: The desktop ... on PithPith's reviewarXiv
  14. [34] Yuxin Zhang, Mengxue Hu, Zheng Lin, Xiaoyi Fan, Fan Xie, Zihan Fang, Jing Yang, Wenjun Zhu, Zhiwen Chen, Chengfei Lv, and Zhe Chen. 2026. Hera: Learning long-horizon coordination for device-cloud collaborative LLM agents. arXiv:2605.24598 [cs.AI] https://arxiv.org/abs/2605.24598 on PithPith's reviewarXiv
  15. [35] Yuxiang Zhou, Jichang Li, Yanhao Zhang, Haonan Lu, and Guanbin Li on Pith
  16. [37] Mobile-Agent-RAG: Driving smart multi-agent coordination with contextual knowledge empowerment for long-horizon mobile automation. arXiv:2511.12254 [cs.AI] https://arxiv.org/abs/2511.12254 9 Supplementary Material AAgBenchTask List Table 8: The 60 tasks inAgBenchand their reso... on PitharXiv
  17. [2026] EcoAgent: An efficient device-cloud collaborative multi-agent frame- work for mobile automation. InProceedings of the AAAI Conference on Artifi- cial Intelligence (AAAI’26/IAAI’26/EAAI’26). AAAI Press, Article 3329, 9 pages. doi:10.1609/aaai.v40i35.40230 on Pithdoi:10.1609/aaai.v40i35.40230

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment