{"id":"762c1be0-56d4-4f93-a2ad-251a1aa1a99e","arxiv_id":"2607.26300","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A local GUI for observing and steering AI agents cut trace-review time by 38% and raised small-model task completion by up to 34pp in a proof-of-concept experiment.","lead":"AgentGUI is a new free, locally hosted dashboard for watching and steering multiple AI agents on long-running tasks. In a small test, people using it found key information in agent traces 38% faster, and a built-in drift checker lifted small-model task completion by up to 34 percentage points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Drift-prevention lift lacks a retry-without-feedback control; observed gains may reflect the extra attempt rather than manager audit content.","rationale":"The reader's weakest assumption and my own concern are the same: the drift-prevention experiment conflates the effect of the manager's audit content with the effect of a second attempt. I considered other weaknesses—the small N=8 user study, post-hoc exclusions, multiple comparisons—but these are transparently acknowledged and do not threaten the central architectural contribution. The drift experiment is the only place where a causal verb ('raises') rests on a missing control condition. The paper is otherwise transparent, open-source, and uses exact permutation tests for the user study, which supports the observability claim. Therefore I do not move the verdict: CONDITIONAL is appropriate until the retry-control experiment is run.","tokens_in":8215,"tokens_out":3624,"duration_ms":37306,"concrete_test":"Run a neutral-retry control arm of the §4.2 experiment: for each of the four model sizes, take N=50 runs that fail the initial programmatic check and resume them with a neutral prompt (e.g., 'Please review your work and try again') in the same harness, without manager audit output. Compare post-retry completion rates to the audit condition's 26/70/78/98%. If neutral retry yields statistically indistinguishable rates, the audit-specific causal claim is unsupported; if it yields materially lower rates, the claim survives. Also report bootstrap CIs for both arms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in §4.2: the automated-steering experiment has no control for the effect of simply giving the agent another run. The protocol states that after a failed initial attempt, the desk is 'prompted to resume by reading the manager's audit output' (§3.3). There is no condition in which a failed desk is resumed with a neutral prompt lacking audit content. Therefore the reported lifts (10→26%, 54→70%, 44→78%, 92→98%) cannot be attributed to the manager's targeted feedback rather than to the extra attempt itself. This is a confound, not a statistical precision issue; even perfect 50-run proportions would not resolve it. The paper's own limitation paragraph does not flag this missing control. Given the abstract's causal phrasing ('automated drift prevention feature raises the task completion rate'), this is the load-bearing point for the drift-prevention half of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents AgentGUI, a locally hosted GUI for observing and steering long-running AI agents across multiple concurrent sessions. It combines trajectory visualizations (activity feed, overview, console, debug log), manual steering (direct message, task editing, profile switching), and an LLM-powered automated manager that audits agent transcripts and artifacts and prompts a stalled agent to resume with an audit report. The evaluation has two parts: (i) a within-participant user study (N=8) comparing AgentGUI with the Hermes Dashboard on information lookup from agent traces, reporting 38% faster identification (p=0.023) and higher accuracy (93% vs 80%, p=0.031); (ii) a proof-of-concept automated-steering experiment on a synthetic patient-chart task (N=50 per model, 0.8B–9B), reporting completion-rate improvements after a single manager audit (10%→26%, 54%→70%, 44%→78%, 92%→98%). The authors release the code and project website.","tokens_in":8411,"tokens_out":7441,"duration_ms":67493,"significance":"If the user-study result holds, AgentGUI is a useful contribution to a real problem—human oversight of long-running agent fleets. The user-study design is careful: within-participant, counterbalanced for interface order and rollout assignment, with exact paired sign-flip permutation tests and per-question-type analyses; the release of an open-source implementation, demo, and programmatic scorer are concrete strengths. The automated-steering experiment is a useful preliminary proof of concept, but as it stands its central causal claim is not supported because the design lacks a retry-without-feedback control. The paper's own limitation section does not mention this confound. The manuscript is appropriate for a systems/UI venue after the steering claim is either re-run with a proper control or reworded as an uncontrolled preliminary observation.","major_comments":[{"comment":"The drift-prevention result is confounded by the extra attempt. The protocol in §3.3 says that when a desk is idle and not solved, the manager leaves an audit report and the session is 'prompted to resume by reading the manager's audit output.' The experiment in §4.2 reports completion 'after one audit,' but there is no condition in which a failed desk is resumed with a neutral prompt that lacks audit content. Thus the reported lifts (10→26, 54→70, 44→78, 92→98) could be due simply to giving the agent a second run (temperature, nondeterminism, or additional wall time), not to the manager's targeted feedback. The paper's limitation paragraph does not flag this missing control, and the abstract states the feature 'raises the task completion rate,' so the causal claim is load-bearing. Please add a control condition (e.g., resume with a neutral 'please try again' without audit content) or, f","section":"Sec. 4.2, Sec. 3.3"},{"comment":"The point estimates are reported without uncertainty or significance tests. With N=50 per model, the 0.8B improvement is 5/50→13/50 and the 9B improvement is 46/50→49/50; exact binomial confidence intervals overlap in both cases, so the 'clean monotonic ordering' claim is not statistically grounded. Report Wilson intervals, exact McNemar-style tests or bootstrap intervals for the before/after per-desk paired outcomes, and avoid claiming a general benefit at every scale based on point estimates alone.","section":"Sec. 4.2, Fig. 6"},{"comment":"The accuracy result rests on N=8 and several post-hoc decisions: one question removed, two answer sets widened, one hardware-flagged question kept primary, and one participant's low baseline score. The paper reports p=0.031 without showing sensitivity to these choices, and the 'sensitivity variant' mentioned in the text is not reported. Please report the timing and accuracy results with and without each adjustment (especially the hardware-flagged question and the widened answer sets). The timing reduction is the more robust outcome; the accuracy claim should be labeled provisional until the sensitivity analysis is provided.","section":"Sec. 4.1, Appendix B"}],"minor_comments":[{"comment":"Grammar: 'The 4B’s slightly lower unaided completion' should be 'The 4B model’s...' or 'The 4B runs’...'.","section":"Sec. 4.2"},{"comment":"Subject-verb agreement: 'User study and proof-of-concept experiment attests' should be 'attest'.","section":"Sec. 5"},{"comment":"'has higher first-shot completion rate' should be 'has a higher...'.","section":"Sec. 4.2"},{"comment":"Please include the exact manager prompt and an example audit report in the appendix or repository, since the intervention content is central to the steering claim. Also, the user-study question sets and raw per-participant data are not included; making them available would improve reproducibility.","section":"Appendix B / reproducibility"}],"recommendation":"major_revision","confidential_remarks":"To the editor only: The manuscript is a solid systems/UI contribution; the user-study timing result is credible and well designed. The blocking issue is the missing retry-control in the automated steering experiment, which makes the abstract's causal claim unsupported. I recommend major revision requiring a control condition or a substantial rewording, plus sensitivity analyses for the user-study accuracy result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: AgentGUI is a real contribution on the systems side, and the user study is more careful than most of these papers. The automated steering experiment, though, has a confound that undermines its headline claim, and the paper doesn't flag it.\n\nThe genuinely new piece is the integration. Agentflow, AgentLens, AutoGen Studio, AGDebugger, ResearStudio all do pieces, but AgentGUI is the first local, open-source tool I know of that combines real-time trajectory visualization, manual intervention, an LLM-manager audit loop, and coordination across Hermes and the Claude SDK. It's released under MIT with source, a demo video, and a Docker-sandboxed design. That alone is worth a look.\n\nThe user study is the strongest part. Eight participants, within-participant, counterbalanced, exact sign-flip permutation tests. The 38% time reduction (90s vs 145s per question, p=0.023) is plausible and consistent across question types. Accuracy went from 80% to 93%; the paper honestly notes one participant's low baseline score could inflate that. The post-hoc exclusions (one question dropped, two answer sets widened) are a yellow flag but they're disclosed. For N=8, the design is about as good as you can do, and the effect sizes are large enough to survive some noise.\n\nThe soft spot is Section 4.2. The drift experiment compares completion before vs. after one manager audit. But the protocol says every failed agent is 'prompted to resume by reading the manager's audit output.' There is no condition where a failed agent is resumed with a neutral 'please try again' prompt. So the 10→26, 54→70, 44→78, 92→98 lifts conflate the audit content with the value of just giving the agent another attempt. That's a confound, not a sample-size issue; more runs won't fix it. The paper's limitation paragraph mentions the experiment is narrow, but doesn't flag this missing control, and the abstract's phrasing ('raises the task completion rate') overstates the evidence. The cost analysis is nice, and the per-model monotonic ordering after audit is suggestive, but the causal claim about drift prevention needs an ablation.\n\nBottom line: this is a paper for agent-systems and HCI people, and the observability half is worth engaging with. The drift-prevention half should be treated as preliminary until the control condition is added. Send it to peer review—it's a serious systems paper with a careful (if small) user study and a fixable flaw. I'd ask the authors for the retry-without-feedback control, full sensitivity results, and error bars on the drift experiment. If those come back consistent, the paper becomes quite useful.","headline":"Useful open-source observability/steering tool with a careful user study; the drift-prevention claim is confounded by the lack of a retry-without-feedback control.","tokens_in":8870,"tokens_out":3299,"would_cite":true,"duration_ms":30315,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgentGUI, a local interface for long-running AI agents, reports 38% faster trace comprehension and up to 34-point completion gains from automated steering.","keywords":["AI agent observability","agent steering","long-running agents","agent trajectory visualization","automated agent auditing","drift prevention","human-in-the-loop","local language models"],"falsifier":"Run the same navigation task with failed desks auto-resumed by a generic prompt—'Please review your work and try again'—instead of the manager's audit. If completion rises by the same amount, the claim that the audit specifically causes the gain is refuted; a larger pre-registered user study that failed to reproduce the 38% lookup-time advantage would similarly test the observability claim.","tokens_in":8117,"feed_emoji":"🖥️","tokens_out":9315,"duration_ms":77776,"temperature":0.7,"pith_summary":"The paper argues that human oversight, not model capability, is the main bottleneck for long-running AI agents, and that a locally hosted interface can relieve it. AgentGUI gives each agent a 'desk' with layered views—activity feed, wall-clock timeline, debug terminal, and code console—so a person can skim a trace instead of reading raw transcripts. In a controlled user study, participants answered questions about agent traces 38% faster (90 vs 145 seconds per question) and more accurately (93% vs 80%) with AgentGUI. The paper also reports that an automated LLM manager, which audits stalled work and prompts the agent to resume with corrective feedback, raised task completion by up to 34 percentage points across small local models, with manager token cost below 1% of the total. If these results hold, long-running agents become practical to supervise, correct, and trust.","feed_headline":"AgentGUI makes long agent runs 38% faster to read","feed_subtitle":"An automated manager also audits stalled agents and lifts small-model completion by up to 34 points.","key_machinery":"The mechanism is the pairing of a layered trajectory display with an automated audit loop. The display makes a trace legible by separating reasoning content, tool requests and responses, terminal output, files, and wall-clock time, so a human can locate key events without reading the full transcript. The audit loop is an LLM 'manager' that turns a task into verifiable criteria, gathers evidence from the agent's transcript and workspace, judges each criterion, and writes a corrective report that the agent is prompted to read before resuming. The audit adds under one percent of total token cost, which is what makes automated steering cheap enough to run repeatedly across a fleet of agents.","core_discovery":"The central claim is that observing and steering an agent should happen in one interface, and that both halves of that interface work. AgentGUI organizes each running agent as a desk whose views separate reasoning, tool calls, files, and elapsed time, and it lets a human interrupt a turn, edit the task, or swap the model profile mid-run. For automated steering, an LLM-powered manager decomposes the task into verifiable criteria, collects evidence from the transcript and workspace, judges each criterion, and leaves an audit report; a session that is not solved resumes by reading that report. The paper reports that this design yields a statistically significant 38% reduction in trace-lookup ti","pith_inferences":["The drift experiment lacks a neutral-resume control, so part of the completion lift may be the value of a second attempt rather than the manager's specific feedback.","The observability effect sizes come from eight graduate students; a replication with a broader, larger sample would show how far they generalize.","The manager's criteria-evidence-judgment loop could be redirected at qualitative outcomes such as answer quality, safety, or faithfulness rather than binary deliverable presence.","A natural extension is to measure not only whether a corrected agent finishes but whether its final artifacts are actually correct, since completion alone can hide superficial repairs."],"forward_implications":["If the observability result generalizes, a person can supervise long-running and multiple concurrent agent sessions without supervision time erasing the time savings from delegation.","The automated manager can be applied where it helps most: small local models, whose first-attempt completion roughly doubled after one audit, at negligible token cost.","Manual steering gives a stalled task a second path: interrupt the current turn, edit the task definition, or hand the task to a more capable model mid-run.","Because the same interface supports local and remote inference and saves and restores full workspaces, private, portable agent workflows become feasible.","The manager's audit decomposes a task into verifiable criteria, which means the same loop can judge completion, not just surface it."],"fun_headline_variants":["AgentGUI: observe and steer AI agents in one window","AgentGUI cuts agent trace reading time by 38%","Steer long-running AI agents live with AgentGUI","AgentGUI: 38% faster trace reading, 34pp more completions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The drift-prevention results assume that the manager's corrective feedback, not simply the chance to run again, causes the completion-rate gains; in the current experiment every failed run resumes by reading the manager's audit, with no plain-retry control.","fun_headline_variants_meta":{"raw":{"variants":["AgentGUI: observe and steer AI agents in one window","AgentGUI cuts agent trace reading time by 38%","Steer long-running AI agents live with AgentGUI","AgentGUI: 38% faster trace reading, 34pp more completions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1592,"prompt_tokens":742,"completion_tokens":850,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":779}},"tokens_in":486,"tokens_out":850,"duration_ms":7502,"temperature":1.0,"reasoning_tokens":779,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:13:03.314920+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same navigation task with failed desks auto-resumed by a generic prompt—'Please review your work and try again'—instead of the manager's audit. If completion rises by the same amount, the claim that the audit specifically causes the gain is refuted; a larger pre-registered user study that failed to reproduce the 38% lookup-time advantage would similarly test the observability claim.","supporting_citations":[],"review_version":1}