{"arxiv_id":"2610.35832","title":"Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy","abstract":"Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated. In this report, we evaluate GPT-6-Astra on EBench as the core of an agent that controls a robot through an agent harness under the same visual and proprioceptive observation protocol as post-trained policies. We study two agent configurations: direct control, in which the agent generates waypoint commands directly, and harness with policy, in which it can also invoke a post-trained VLA or WAM policy for action execution. Under direct control, the agent achieves competitive overall performance relative to post-trained policies. It leads on short-horizon tasks and exhibits the least performance variation across generalization dimensions, whereas it underperforms post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks. In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost. However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control. Overall, our study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design. Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/","primary_cat":"cs.RO","cross_cats":["cs.AI"],"submitted_at":"2026-10-26T17:51:02+00:00","latest_version":1,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":true,"has_source":false,"verdict":"CONDITIONAL","confidence":"HIGH","one_line_summary":"On EBench's 510 simulated manipulation episodes, a GPT-6-Astra agent under direct control reaches 46.7% success, and invoking post-trained VLA or WAM policies through a harness raises it to 53.0-56.0% while cutting cost, though high-precision contact remains unsolved.","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","upvotes":0,"downvotes":0,"pith_score":0,"inbound_count":4,"_links":{"html":"https://pith.science/paper/2610.35832","verdict":"/api/v1/verdicts/2610.35832"}}