Pith.

Robotics · arXiv:2610.35832 v1 · submitted 26 Oct 2026 · 41 references

Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

Yifei Yao, Yude Zou, Guanbo Jiao, Xing Gao, Ning Gao, Yukai Wang, Jiantong Chen, Zihou Zhu

show 7 more authors

Xinyu Li, Ningyu Yan, Shuai Wang, Jianming Ma, Haoyu Guo, Weinan Zhang, Chunhua Shen

The paper says

“Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated.”

more of the abstract

In this report, we evaluate GPT-6-Astra on EBench as the core of an agent that controls a robot through an agent harness under the same visual and proprioceptive observation protocol as post-trained policies. We study two agent configurations: direct control, in which the agent generates waypoint commands directly, and harness with policy, in which it can also invoke a post-trained VLA or WAM policy for action execution. Under direct control, the agent achieves competitive overall performance relative to post-trained policies. It leads on short-horizon tasks and exhibits the least performance variation across generalization dimensions, whereas it underperforms post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks. In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost. However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control. Overall, our study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design. Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/

from the abstract

Pith says

Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A candid and thorough paired evaluation of a frontier MLLM agent on EBench whose qualitative harness-design findings are solid, but whose headline rankings rest on a disclosed-and-unbounded demo leak in 31 of 510 episodes. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 41 references say.

The paper says, at Appendix B.2; Appendix D.3; Section 5.6

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The evaluation premise that the one-shot demonstration measures transfer from a separate episode fails on 31 of 510 episodes, and the effect is disclosed but never bounded. Appendix B.2 states that for these episodes the demonstration is a recording of the test scene itself, with the Frame demonstration recording four of its 20 test scenes, while the prompt in Appendix A.3 tells the agent the demo is reference from a separate recorded episode.

more of objection 1

The contamination is asymmetric: the eight post-trained baselines receive no such exposure, and all three agent configurations share the same demonstrations, so any leak-driven success inflates the agent's overall, short-horizon, generalization-dimension, and harness numbers simultaneously. The potential magnitude is material: Appendix D.3 reports that the demonstration flips Frame from 1/20 to 13/20 successes and from 0.283 to 0.800 score, and the affected headline margins are thin, with 46.7% holding rank 3 by 1.1 points over Qwen-RobotManip, 73.2% leading short-horizon by 7.3 points, and the harness with InternW0-delta surpassing all by only 0.7 points over OpenWAM-alpha. I request that the authors either re-run the 31 leaked episodes with a demonstration genuinely recorded from a separate episode and recompute every headline number, or exclude those episodes from all configurations and report the corrected tables, and that they list the task and instance identifiers of all 31 affected episodes.

The paper says, at Appendix B.2; Section 6.2; Table 2

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The direct-control and harness configurations did not run under identical budgets for two tasks. Appendix B.2 states that the direct-control run evaluated Bottle and Shop at 3,000 physics steps while the harness used the benchmark's 5,000-step protocol, and that under 5,000 steps the harness solves 3 of 20 Shop episodes with pi0.5 and 1 with InternW0-delta, raising the harness overall SR from 52.4% to 53.0% and from 55.8% to 56.0%.

more of objection 2

Two consequences follow. First, the claim that the harness improves over direct control on Shop is a pure budget effect, because no configuration succeeds within 3,000 steps and the reported Shop row in Table 2 compares the 0% direct-control result at 3,000 steps against harness results at 5,000. Second, the 56.0% surpasses-all figure includes a 0.2-point contribution from the larger Shop budget. Please report the matched-budget comparison for the Shop and Bottle rows and state explicitly whether the 53.0 and 56.0 headline figures and the surpasses-all claim survive with matched budgets.

The paper says, at Appendix D.3; Section 5.6

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The only available estimate of demonstration sensitivity is itself confounded, so even an approximate bound on the leak's effect cannot currently be derived. Appendix B.2 discloses that in the D.3 comparison the two conditions were rendered with partly different textures, yet Section 5.6 quotes the 1/20 to 13/20 Frame contrast without this caveat.

more of objection 3

With texture confounded against the demo condition, the 13/20 number conflates the demonstration's content with the rendering difference, and it cannot serve as a clean sensitivity estimate for the 31 leaked episodes. Please either re-run the D.3 comparison with matched textures or mark the 13/20 figure in Section 5.6 as confounded, and state in the main text that the leak's contribution to the headline numbers is therefore unbounded pending the re-analysis requested above.

4 minor objections

The paper says, at Abstract / Contributions

Pith says: minor objection 1 of 4

The contributions list contains the typo 'an GPT-6-Astra-based agent'; it should read 'a GPT-6-Astra-based agent'.

The paper says, at Table 1; Table 3

Pith says: minor objection 2 of 4

Table 3 reports 284 successful episodes under the harness with InternW0-delta, which is 55.7% of 510 episodes, while Table 1 reports 56.0% overall SR. The discrepancy arises because overall SR is averaged equally across the 26 tasks rather than per episode; a one-line footnote stating that the aggregate SR is task-averaged would prevent confusion.

show 2 more

The paper says, at Appendix B.2

Pith says: minor objection 3 of 4

The 56 direct-control episodes interrupted by infrastructure failure were resumed in a new model thread initialized with the recorded action history. Please state whether those episodes are kept in all reported aggregates and, if so, whether any of the six early-termination hold successes fell in this set, since this affects interpretability of the direct-control SR.

The paper says, at Section 5.2 / Table B.4

Pith says: minor objection 4 of 4

When reporting the 6.6-point generalization range as the smallest among the agent and the eight post-trained policies, note that the leaked-demo episodes may be unevenly distributed across the four dimensions; the claim should be re-verified once the 31 leaked episodes are removed or re-run.

What would settle it

From the stored logs, list the 31 leaked episode IDs with their tasks and per-configuration outcomes, and recompute the Table 1 and Table 2 headline SRs plus the 56.0-versus-55.3 surpass margin excluding those episodes; if overall or short-horizon SR shifts by at least one point, or the InternW0-Delta harness falls to 55.3% or below, the ranking and surpass-all claims need qualification. Then re-run the 31 affected episodes (and all Frame episodes) under the same three configurations with the demonstration annotated from a training scene disjoint from the test scenes; if any reported success flips to failure, the scene-recording leak is confirmed as a driver of the headline numbers.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 27 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic partly agrees 2 min 0 s · 52,915 tokens read, 12,213 written
  3. Editor wrote the verdict above 1 min 11 s · 54,035 tokens read, 7,466 written
  4. Referee major revision: 3 major objections, 4 minor 1 min 24 s · 55,606 tokens read, 2,087 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things to know. The qualitative core is good and survives scrutiny: the harness-design failure taxonomy, delegation that ignores policy competence, subgoals that never reach the policy, grasps lost when control returns to the agent, is concrete, quantified, and useful. The headline rankings are more fragile than the writing suggests. The one-shot demonstration is a recording of the test scene for 31 of 510 episodes; B.2 discloses this but never bounds the effect. That matters because the margins are thin.

What is actually new: the harness mechanism follows Harness VLA, and the authors credit it. The contribution is the paired profile, the same GPT-6-Astra core, same observation protocol, direct control versus harness with policy, across all 510 episodes, with per-task failure attribution, token-cost accounting, and a safety screen. The numbers harness builders will use are there: policy step share is basically uncorrelated with policy-alone competence (r=0.31, 0.23), and unmaintained grasps after policy segments (338 and 316 occurrences) disrupt within two calls at roughly double to triple the rate of the agent's own commands. The appendices are unusually candid, B.2 discloses the demo leak, the Bottle/Shop step-limit mismatch, the infrastructure interruptions, and the D.3 texture confound, and the statistics (bootstrap within task, McNemar with Holm correction) fit the paired design.

Soft spots. The demo leak is the load-bearing one. The prompt tells the agent the demo is "reference from a separate recorded episode" (A.3), which is false for those 31 episodes, and the leak is asymmetric: the eight baselines get no demonstration at all. D.3 shows the demo flips Frame from 1/20 to 13/20 successes, and Frame is exactly the task whose demo records test scenes; worse, D.3's two conditions used partly different textures, so even a clean demo-sensitivity estimate is unavailable. Which tasks host the other 27 leaked episodes is never stated. Third rank is held by 1.1 points, the short-horizon lead is 73.2% versus 65.9%, and the "surpasses all" claim is 56.0% versus 55.3%; a handful of leak-dependent successes moves those margins. The step-limit mismatch is genuinely minor by comparison: the paper's own arithmetic bounds it (52.4 to 53.0, 55.8 to 56.0), and Bottle is unsolved at either limit.

Who this is for: anyone building agent harnesses that combine frontier MLLMs with VLAs or WAMs, and anyone evaluating frontier models on manipulation benchmarks. It deserves a serious referee. The revision ask should be specific: re-run the 31 leaked episodes with clean demonstrations, or quantitatively bound their contribution to the headline claims. The qualitative findings can stand either way.

The strongest objection

The load-bearing premise of the whole evaluation is that the single annotated demonstration measures transfer from a separate episode. Appendix B.2 states that for 31 of the 510 test episodes the demonstration is a recording of the test scene itself, with the Frame demonstration recording four of its 20 test scenes, and A.3 explicitly tells the agent the demo is 'reference from a separate recorded episode.' The premise therefore fails on 6% of episodes, and the contamination is asymmetric: the eight post-trained baselines receive no such exposure, so any leak-driven success inflates the agent in all three configurations (direct control and both harnesses share the same demonstrations). The potential magnitude is not academic: Appendix D.3 shows the demonstration flips Frame from 1/20 to 13/20 successes and score from 0.283 to 0.800, and Frame is precisely the task whose demo records test scenes; D.3 is itself confounded because its two conditions were rendered with partly different textures, so even the clean demo-sensitivity estimate is unavailable. The headline margins at stake are thin: 46.7% overall holds third rank by 1.1 points over Qwen-RobotManip; 73.2% leads short-horizon tasks by 7.3 points over OpenWAM-alpha; and the harness with InternW0-Delta at 56.0% 'surpasses all' compared policies by only 0.7 points over OpenWAM-alpha's 55.3%. A handful of leak-dependent successes among the 31 episodes shifts or inverts those margins, and the leaked episodes also violate the 'unseen' claim behind the generalization-dimension results. The step-limit mismatch on Bottle and Shop is secondary because the paper's own arithmetic bounds it precisely (harness SRs are 52.4% and 55.8% without the 5,000-step Shop successes, and Bottle is unsolved at either limit). What remains unresolved is which tasks host the other 27 leaked episodes and whether any of their successes depend on seeing the test scene.

I read this as a careful, large-scale, unusually honest empirical study: the protocol disclosures in Appendix B.2, the paired-episode statistics in B.3, and the call-by-call failure analyses all go well beyond typical practice, and the qualitative claims (failure taxonomy, policy-invocation behavior, unmaintained grasps after control transfer) do not depend on either protocol issue. The central claims are quantitative head-to-head comparisons, and those rest on the premise that the single annotated demonstration is a transfer instrument drawn from a separate training episode. Appendix B.2 states that premise fails for 31 of 510 episodes, and D.3 shows the demonstration can be decisive on a task (Frame, 1/20 to 13/20) that is itself among the leaked tasks. The reader flagged both the demonstration contamination and the step-limit mismatch; I agree on the first and treat the second as secondary, which is why I record partial rather than full agreement. The step-limit issue is real but precisely bounded by the paper's own corrections (the harness SRs are 52.4% and 55.8% without the 5,000-step Shop successes, and no configuration solves either task within 3,000 steps), whereas the demo leak has no stated bound anywhere in the report. The margins at stake make this the single load-bearing concern: the third-rank overall claim holds by 1.1 points over Qwen-RobotManip, the short-horizon lead by 7.3 points over OpenWAM-alpha, and the headline 'surpasses all compared policies' claim by 0.7 points over OpenWAM-alpha's 55.3%. Even a few leak-dependent successes among the 31 episodes move these margins, and the leaked episodes are exactly the ones where the 'unseen' generalization tests are violated, so the generalization-robustness claims are exposed as well. Because direct control and both harness configurations use identical demonstrations, no agent-side number is insulated from the leak. The resolution is cheap and decisive: recompute with the 31 episodes excluded, then re-run them with disjoint-scene demonstrations. Given that the concern is bounded, disclosed, and cleanly testable, the appropriate verdict remains CONDITIONAL; the condition is that the authors quantify the leak's contribution or show that re-running with clean demonstrations leaves the headline margins intact.

The referee's report

If the comparison is valid, this is a useful systematic characterization: it is the first evaluation I am aware of that profiles one frontier MLLM agent against eight post-trained VLA and WAM policies on a common diagnostic benchmark under the same observation protocol, and the episode-paired harness-versus-component analysis is genuinely informative. The strengths are real: the protocol appendix discloses event definitions, interrupted-episode handling, and step limits; the statistics are principled (within-task bootstrap, exact McNemar with Holm correction); and the paper consistently reports the three-way residual gap rather than only aggregate wins. The claims are falsifiable and mostly simulator-computed. The unresolved issue is that the evaluation's load-bearing premise, that the one-shot demonstration is recorded from a separate episode and therefore measures transfer, fails on 31 of 510 episodes, and the size of the resulting contamination is disclosed but never bounded; the headline margins are thin enough that quantification is required before the central claims can be accepted as stated.

Pith's reading of the paper (a machine's summary, not the authors')

This paper tries to establish where a frontier multimodal language model stands as a robot controller with zero robot-specific training, and whether letting it call a trained policy as a tool gets the best of both. On the 510-test-episode EBench benchmark, the GPT-6-Astra agent under direct control reaches 46.7% success and a 0.654 score from RGB and proprioceptive observations plus one annotated demonstration per task, ranking first on short-horizon tasks (73.2%) and showing the smallest success variation across generalization shifts, while trailing trained policies on long-horizon, tabletop, and high-precision contact tasks. When the harness adds a policy-invocation tool, overall success rises to 53.0% with a vision-language-action policy and 56.0% with a world action model, the latter surpassing all eight compared post-trained policies, and token cost per success falls by more than half. The residual gap to the better standalone component is traced to how agent and policy coordinate, not to raw capability, and high-precision contact remains a shared ceiling.

What carries the argument. The carrying mechanism is the agent harness, a triple of prompt, observation interface, and tool set wrapped around an MLLM core that turns the model into a closed-loop policy defined by its tool calls. The paper's argument runs on the contrast between two tool sets: under direct control the agent can only emit end-effector waypoint commands, which the simulator executes open-loop so that the agent observes the scene only after each chunk (median 48 physics steps), whereas the harness-with-policy configuration adds a tool that delegates 1-300 physics steps to a frozen post-trained policy replanning every 10-30 steps, plus a call that re-returns the annotated demonstration. The single-shot demonstration, one training trajectory condensed by a separate model instance into 10-18 annotated keyframes, is the agent's only task-specific input beyond the prompt. The paired-episode protocol, in which every test episode is run under direct control and in each harness configuration under the same observation protocol as the policies, is what lets the paper attribute the residual gap to policy invocation, subgoal passing, and grasp maintenance rather than to scene difficulty.

Core claim. The central discovery is a located capability boundary: a frontier multimodal LLM, used as the core of an agent that controls a robot through a harness, is competitive with, and in several regards stronger than, vision-language-action and world action models post-trained on 6,600 in-domain demonstrations, provided the task stays in its reasoning strengths. Under direct control, GPT-6-Astra at 46.7% SR ranks third of nine compared systems and first among all of them on short-horizon tasks, beats the trained policies on visual-grounding tasks, keeps the narrowest success spread across the four generalization dimensions, and completes a composite bookmark-plus-pen instruction zero-shot that both compared policies fail. Its failures concentrate where physical execution, not reasoning, is decisive: long-horizon tasks (27.3%), tabletop tasks (20.0%), and high-precision contact (11.3%), because it commands open-loop waypoint chunks and sees fresh observations only between them. In the harness with policy, invoking a frozen policy for bounded segments raises overall success to 53.0% or 56.0% SR, cuts token cost per success by more than half, and lets the combination with the stronger policy surpass every post-trained policy compared in its Table 1 on both metrics; the combination still falls short of the per-task better component, which the authors attribute to suboptimal policy invocation, the policy executing the fixed task instruction rather than the agent's subgoal, and grasps left unmaintained when control returns.

The premise it rests on. The load-bearing premise is that the agent's success on the 510 test episodes measures transfer from a disjoint training demonstration, but the paper acknowledges that for 31 of those episodes the demonstration records the test scene itself (Frame's demonstration records four of its 20 test scenes), and its own ablation shows the demonstration moves Frame from 1/20 to 13/20 successes; the companion premise that direct control and the harness ran under identical step budgets is also acknowledged as violated for Bottle and Shop (3,000 versus 5,000 physics steps).

If the paper is right (Pith's inferences, not the authors')

Hybrid control is the working recipe: an MLLM planner that delegates execution to a post-trained policy surpasses both of its components alone, reaching 56.0% SR and a 0.744 score with the stronger policy, the highest of the agent configurations and eight compared post-trained policies.

Delegation changes the economics of agentic robot control: token cost per success falls from 8.1M to 3.5-4.0M tokens (from USD 16.82 to about USD 5 per success), with the savings concentrated on the long-horizon tasks where direct control is costliest.

The frontier model's distinctive capabilities, visual grounding under distractors, stability across generalization shifts, in-episode recovery, and zero-shot composition of skills, are not reproduced by the post-trained policies, so scaling robot demonstration data alone is unlikely to yield them.

The remaining harness gap is an engineering target: the 3.1 to 4.8 point shortfall to the better component is traced to three named coordination failures, each with a proposed fix (passing subgoals to the policy, training the policy as a tool on mid-episode states, and maintaining grasps at control handover).

High-precision contact-rich manipulation is a shared unsolved ceiling: the best per-task choice among the agent and the two policies reaches only 30.0% SR on those tasks, below the strongest policy alone.

The paper's own per-task numbers imply a simple competence-aware delegation rule, invoke the policy only where the policy alone beats direct control, would recover most of the reported gap to the better component; the reported correlation between harness gain and policy-alone advantage (r ≈ 0.77 for the weaker policy) is the evidence for where the loose coordination lives.

Because token cost tracks the number of tool calls (r = 0.88 under direct control), since the agent resends its full context each call, a harness that compresses or summarizes past observations before re-prompting would compound the delegation savings without changing the control policy at all.

The agent's stated reasons on every call form a labeled corpus, 302 of 510 episodes name a slip, drop, or displacement together with the revised action that follows, and that corpus is a ready-made distillation set for training a small, fast corrective policy, which the authors propose as future work but do not test.

The protocol blemishes the authors disclose, 31 of 510 episodes whose demonstration records the test scene itself (Frame's demonstration records four of its 20 test scenes) and the 3,000-versus-5,000-step asymmetry on Bottle and Shop, mean the exact headline deltas would shift slightly under a strictly disjoint-demonstration, equal-budget rerun, and their own Frame ablation (1/20 to 13/20 with the

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated. In this report, we evaluate GPT-6-Astra on EBench as the core of an agent that controls a robot through an agent harness under the same visual and proprioceptive observation protocol as post-trained policies. We study two agent configurations: direct control, in which the agent generates waypoint commands directly, and harness with policy, in which it can also invoke a post-trained VLA or WAM policy for action execution. Under direct control, the agent achieves competitive overall performance relative to post-trained policies. It leads on short-horizon tasks and exhibits the least performance variation across generalization dimensions, whereas it underperforms post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks. In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost. However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control. Overall, our study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design. Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-27

The authors' main claim, as the classifier picked it out 0.76

“Overall, our study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design.”

How the main claim is stated: asserted 0.53

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.96 · human participants 0.03 · controlled comparison 0.85 · randomized assignment 0.02 · states quantitative result 0.10 · compares with prior results 0.96 · mentions code or data availability 0.43

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated. gap 0.92 0.05 0.03 asserted 0.72
In this report, we evaluate GPT-6-Astra on EBench as the core of an agent that controls a robot through an agent harness under the same visual and proprioceptive observation protocol as post-trained policies. contribution 0.69 0.94 0.04 asserted 0.95
We study two agent configurations: direct control, in which the agent generates waypoint commands directly, and harness with policy, in which it can also invoke a post-trained VLA or WAM policy for action execution. method 0.92 0.73 0.04 asserted 0.93
Under direct control, the agent achieves competitive overall performance relative to post-trained policies. result 1.00 0.94 0.14 asserted 1.00
It leads on short-horizon tasks and exhibits the least performance variation across generalization dimensions, whereas it underperforms post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks. result 1.00 0.94 0.12 asserted 1.00
In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost. result 0.99 0.94 0.36 asserted 1.00
However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control. limitation 0.60 0.92 0.12 asserted 1.00
Overall, our study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design. future work 0.53 0.70 0.60 asserted 0.53
Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/ contribution 0.33 0.06 0.02 unclear 0.98

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 7,128 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (9)
Figure 1 of the paper
Figure 1. Overview of the GPT-6-Astra control loop on EBench and executor timelines on a Microwave test scene. The input prompt combines the task instruction with a detailed task description annotated from a single training episode. The timeline bars compare active executors across an episode under direct control and under the harness with π0.5, with blue segments executed by the agent and purple segments by π0.5, and with keyframes showing right-wrist camera views from that episode at the marked steps. 26 tasks. Each episode is bounded by a task-specific physics-step limit ranging from 1,500…
Figure 2 of the paper
Figure 2. SR of the direct-control agent and of OpenWAM-α, InternW0-∆, Qwen-RobotManip, and π0.5 (a) across mobility, horizon, and precision task groups and (b) across the four generalization dimensions of the EBench test set. The radial axis of each panel ends at its largest SR, rounded up to a multiple of 10%. The gap between score and SR shows that the direct-control agent frequently earns stage credit in episodes that it does not finish. SR counts only complete successes, whereas the score also credits the stages completed before an unsuccessful episode ends. The SR of the direct-control…
Figure 3 of the paper
Figure 3. The direct-control agent across the 26 EBench test tasks by task group. (a) SR of the direct-control agent, range across the eight post-trained policies, and best post-trained policy, with its rank among the agent and the eight post-trained policies in the column on the right of the panel. (b) Shares of its episodes ending in complete success, partial credit, or a score of zero, averaged equally across tasks within each group and rounded to whole percent so that each bar sums to 100. only OpenWAM-α leads it. It drops to seventh on long-horizon and tabletop tasks and to eighth on…
Figure 4 of the paper
Figure 4. Adaptation in two episodes of the direct-control agent, with the agent’s stated reasons below the frames. Orange boxes show enlarged views of the contact and manipulation details. (a) Apple to bowl: after a slip, the agent takes a wider view and carries the apple with the arm alone, reaching a 1.00 score. (b) Coffee beans: after a sweep leaves the beans in place, the agent lowers the edge and tilts the fingertips, reaching a 0.50 score. The two episodes thus separate the agent’s diagnosis and adaptation from the precision of its physical execution. In both, the agent states a likely…
Figure 5 of the paper
Figure 5. Rollouts of the direct-control agent evaluated zero-shot, π0.5, and OpenWAM-α on a composite instruction outside the standard EBench tasks: placing a bookmark and then a pen onto the same open book. Keyframes are shown in chronological order with video time. 5.6 CASE STUDIES: IN-CONTEXT LEARNING, ADAPTATION, AND PRECISION In-context learning. On Frame and Gear, the annotated demonstration conveys task-level struc￾ture to the direct-control agent but not the precise action of a final contact step. Appendix D.3 compares runs with and without the demonstration on all 20 test episodes…
Figure 6 of the paper
Figure 6. SR of the GPT-6-Astra-based agent under direct control, of each policy alone, and of the harness with that policy, for π0.5 in (a, b) and InternW0-∆ in (c, d), across mobility, horizon, and precision task groups in (a, c) and across the four generalization dimensions of the EBench test set in (b, d). Panels in the same column share a radial axis that ends at the largest SR of the column, rounded up to a multiple of 10%. 6.3 CAPABILITY PROFILE ACROSS TASK GROUPS AND GENERALIZATION DIMENSIONS Task groups. The harness with policy improves task success mainly on long-horizon and…
Figure 7 of the paper
Figure 7. Per-task SR in percent across the 26 test tasks for the GPT-6-Astra-based agent under direct control, for each post-trained policy alone, and for the harness with π0.5 and with InternW0-∆, together with the share in percent of physics steps that each post-trained policy executes in its harness configuration. Tasks are grouped by precision and ordered within each group by the mean SR of the two post-trained policies alone. One blue-to-red scale shades SR and step share alike on an absolute 0–100% scale, values are rounded to whole percent, and bold marks the highest nonzero SR of…
Figure 8 of the paper
Figure 8. Policy invocation in the two harness configurations. (a) Share of the physics steps executed by the policy in each tenth of an episode, averaged over the successful and over the unsuccessful episodes. (b) Per task, the change in SR from direct control to the harness with policy against the SR difference between the policy alone and direct control, with the identity line. (c) Per task, the share of physics steps executed by the policy against the SR of the policy alone. Each marker in (b) and (c) is one of the 26 tasks, and r is the Pearson correlation over tasks. configurations lose…
Figure 9 of the paper
Figure 9. Token use of the GPT-6-Astra-based agent under direct control and in the harness with each policy on the 510 EBench test episodes. (a) Mean tokens per episode in successful and unsuccessful episodes. (b) Tokens per success on short-horizon and long-horizon tasks, dividing all tokens of each group, including those of its unsuccessful episodes, by its number of successful episodes. Within these shared failures, however, where an episode breaks along its chain of grasping, transfer, and contact depends on the policy. The two runs of an episode end with the same category of decisive…

Figures are the authors', reproduced from the paper.

References (41)
  1. [1] Gao, Ning and Zheng, Jinliang and Gao, Xing and Ma, Haoxiang and Wang, Hanqing and Wang, Yukai and Chen, Jiantong and Chen, Zanxin and Zhang, Shujie and Jia, Mingda and others , title =. arXiv preprint arXiv:2606.18239 , year = on PithPith's reviewarXiv
  2. [2] Gao, Ning and Chen, Yilun and Yang, Shuai and Chen, Xinyi and Tian, Yang and Li, Hao and Huang, Haifeng and Wang, Hanqing and Wang, Tai and Pang, Jiangmiao , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =. 2025 , doi... on PithPith's reviewarXiv
  3. [3] Black, Kevin and Brown, Noah and Darpinian, James and Dhabalia, Karan and Driess, Danny and Esmail, Adnan and Equi, Michael Robert and Finn, Chelsea and Fusai, Niccolo and Galliker, Manuel Y. and others , title =. Proceedings of The 9th Conference on Robot Learning , series = on Pith
  4. [4] Jia, Mengzhao and Lin, Yang and Zhang, Xixin and Zhang, Zhihan and Liu, Xiaobai and Jiang, Meng , title =. arXiv preprint arXiv:2609.12541 , year = on PithPith's reviewarXiv
  5. [5] Chen, Yanzhe and Bai, Zechen and Cao, Zhijun and Zeng, Wenzheng and Lin, Kevin Qinghong and Lin, Yiqi and Liang, Guoqiang and Ma, Kevin Yuchen and Huang, Qiming and Shou, Mike Zheng , title =. arXiv preprint arXiv:2609.10522 , year = on PithPith's reviewarXiv
  6. [6] Liu, Fangcheng and Shen, Yeqing and Cheng, Anda and Mi, Weishi and Tang, Chao and Liu, Chenyuan and Xiang, Yushun and Li, Tingguang and Li, Yong-Lu and Tang, Yehui , title =. arXiv preprint arXiv:2609.34261 , year = on PithPith's reviewarXiv
  7. [7] Wenbo Zhang and Kaixuan Wang and Yutao Ouyang and Xiaoyu Huang and Liyang Li and Kailun Su and Weiyang Jin and Wenhao Chai and Haotian Liang and Zhiyang Dou and others , title =. arXiv preprint arXiv:2609.24170 , year = on PithPith's reviewarXiv
  8. [8] Zhang, Yixian and Zhang, Huanming and Gao, Feng and Li, Xiao and Liu, Zhihao and Nie, Yi and Zhu, Chunyang and Qiu, Jiaxing and Yan, Yuchen and Liu, Jiyuan and others , title =. arXiv preprint arXiv:2607.08448 , year = on PithPith's reviewarXiv
  9. [9] Elmaaroufi, Karim and Svegliato, Justin and Kalade, Sarunas and Schelle, Graham and Seshia, Sanjit A. and Zaharia, Matei , title =. arXiv preprint arXiv:2606.16458 , year = on PitharXiv
  10. [10] Lu, Runyu and Wu, Yubo and Kou, Ethan and Fu, Letian and Xiao, Wenli and Mandlekar, Ajay and Xu, Yinzhen and Shi, Guanya and Goldberg, Ken and Chen, Ang and others , title =. arXiv preprint arXiv:2607.00272 , year = on PithPith's reviewarXiv
  11. [11] Xin Ding and Liang Mi and Mingzhe Huang and Zixuan Wang and Chao Zhang and Zixu Hao and Fu Chen and Xiangyu Li and Yikai Zheng and Yaoyu Guo and others , title =. arXiv preprint arXiv:2608.16590 , year = on PithPith's reviewarXiv
  12. [12] Zexi Li and Yehang Zhang and Haojian Huang and Bohan Zhou and Wenqian Li and Chenxu Wang and Yifan Chang and Yangkai Wei and Tianyi Zhang and Ying-Cong Chen and others , title =. arXiv preprint arXiv:2609.29394 , year = on PithPith's reviewarXiv
  13. [13] Junyi Tang and Jie Peng and Zezhen Ding and Yuan Shen and Tianlong Chen , title =. arXiv preprint arXiv:2609.29204 , year = on PithPith's reviewarXiv
  14. [14] Zeyu Lou and Yanhong Zeng and Yong Wang and Chenyang Si , title =. arXiv preprint arXiv:2609.28530 , year = on PithPith's reviewarXiv
  15. [15] Ye, Seonghyeon and Ge, Yunhao and Zheng, Kaiyuan and Gao, Shenyuan and Yu, Sihyun and Kurian, George and Indupuru, Suneel and Tan, You Liang and Zhu, Chuning and Xiang, Jiannan and others , title =. arXiv preprint arXiv:2602.15922 , year = on PithPith's reviewarXiv
  16. [16] Liang, Jacky and Huang, Wenlong and Xia, Fei and Xu, Peng and Hausman, Karol and Ichter, Brian and Florence, Pete and Zeng, Andy , title =. IEEE International Conference on Robotics and Automation (ICRA) , pages =. 2023 , doi = on Pith
  17. [17] Xu, Bingxin and Shang, Yuzhang and Ferrara, Emilio , title =. arXiv preprint arXiv:2608.16889 , year = on PithPith's reviewarXiv
  18. [18] ^. arXiv preprint arXiv:2511.14759 , year = on PithPith's reviewarXiv
  19. [19] Xin Wang and Wenhao Wu and Menghao Zhang and Zhi Wang and Kun Shao and Jian Luan and Yang Li and Qing Li and Shangding Gu and Huichi Zhou and others , title =. arXiv preprint arXiv:2609.29166 , year = on PithPith's reviewarXiv
  20. [20] You Don't Need To Stay in The Loop: An Agentic Robotics Loop for Robot-Policy Improvement , author =. arXiv preprint arXiv:2608.07555 , year = on PithPith's reviewarXiv
show 21 more
  1. [21] _0 : A Vision-Language-Action Flow Model for General Robot Control , author =. Proceedings of Robotics: Science and Systems , year = on Pith
  2. [22] Kim, Moo Jin and Pertsch, Karl and Karamcheti, Siddharth and Xiao, Ted and Balakrishna, Ashwin and Nair, Suraj and Rafailov, Rafael and Foster, Ethan P and Sanketi, Pannag R and Vuong, Quan and others , title =. Proceedings of the 8th Conference on Robot Learning , series =. 2... on Pith
  3. [23] Haoxiang Ma and Junhao Cai and Xiaoxu Xu and Hao Li and Yuyin Yang and Yang Tian and Jiafei Cao and Hongrui Zhu and Zherui Qiu and Zhaxizhuoma and others , title =. arXiv preprint arXiv:2607.04988 , year = on PithPith's reviewarXiv
  4. [24] Moo Jin Kim and Yihuai Gao and Tsung-Yi Lin and Yen-Chen Lin and Yunhao Ge and Grace Lam and Percy Liang and Shuran Song and Ming-Yu Liu and Chelsea Finn and others , title =. arXiv preprint arXiv:2601.16163 , year = on PithPith's reviewarXiv
  5. [25] Lin Li and Qihang Zhang and Yiming Luo and Shuai Yang and Ruilin Wang and Fei Han and Mingrui Yu and Zelin Gao and Nan Xue and Xing Zhu and others , title =. arXiv preprint arXiv:2601.21998 , year = on PithPith's reviewarXiv
  6. [26] Tianyuan Yuan and Zibin Dong and Yicheng Liu and Hang Zhao , title =. arXiv preprint arXiv:2603.16666 , year = on PithPith's reviewarXiv
  7. [27] Yifan Li and Jiaxu Wang and Dongming Wu and Yicheng Jiang and Ryan Ji and Xiangyu Yue and Yanwei Fu , title =. arXiv preprint arXiv:2610.05230 , year = on PithPith's reviewarXiv
  8. [28] arXiv preprint arXiv:2606.02800 , year = on PithPith's reviewarXiv
  9. [29] Jinliang Zheng and Jianxiong Li and Zhihao Wang and Dongxiu Liu and Xirui Kang and Yuchun Feng and Yinan Zheng and Jiayin Zou and Yilun Chen and Jia Zeng and others , title =. arXiv preprint arXiv:2510.10274 , year = on PithPith's reviewarXiv
  10. [30] Yude Zou and Yifei Yao and Guanbo Jiao and Jianming Ma and Guanjie Zheng and Xing Gao , title =. arXiv preprint arXiv:2610.26813 , year = on PithPith's reviewarXiv
  11. [31] Yuran Wang and Siqiao Huang and Mingleyang Li and Chenhao Zhang and Jiaqi Liang and Weiyang Jin and Yue Chen and Xuemin Chi and Donghao Zhou and Qize Yu and others , title =. arXiv preprint arXiv:2609.07398 , year = on PithPith's reviewarXiv
  12. [32] Jisong Cai and Yao Mu and Ganlin Yang and Zhe Cao and Zhangzheng Tu and Xing Gao and Kailin Li and Xinyu Zhan and Lixin Yang and Yangkun Zhu and others , title =. arXiv preprint arXiv:2609.27656 , year = on PithPith's reviewarXiv
  13. [33] Foundation Models in Robotics: Applications, Challenges, and the Future , author =. The International Journal of Robotics Research , volume =. 2025 , doi = on Pith
  14. [34] Yueen Ma and Zixing Song and Yuzheng Zhuang and Jianye Hao and Irwin King , journal =. A Survey on Vision-Language-Action Models for Embodied. 2026 , doi = on Pith
  15. [35] Qiuhong Shen and Shihua Zhang and Yue Liao and Qi Li and Zhenxiong Tan and Shizun Wang and Shuicheng Yan and Xinchao Wang , title =. arXiv preprint arXiv:2606.20781 , year = on PithPith's reviewarXiv
  16. [36] Haoqi Yuan and Zhixuan Liang and Anzhe Chen and Ye Wang and Haoyang Li and Pei Lin and Yiyang Huang and Zixing Lei and Tong Zhang and Jiazhao Zhang and others , title =. arXiv preprint arXiv:2606.17846 , year = on PithPith's reviewarXiv
  17. [37] arXiv preprint arXiv:2608.15875 , year = on PithPith's reviewarXiv
  18. [38] Xingyu Miao and Zizun Li and Baole Fang and Kaiwen Song and Tenghui Wang and Hanxue Zhang and Yating Wang and Xudong Li and Yuping He and Xueyuan Wei and others , title =. arXiv preprint arXiv:2609.31394 , year = on PithPith's reviewarXiv
  19. [39] Rao, Kanishka , title =. 2026 , month = on Pith
  20. [40] Cai, Zhongang and Cao, Ziang and Chen, Tailai and Chua, Han Chong and Dai, Linjun and Deng, Hanming and Deng, Hokin and Diao, Haiwen and Ding, Chenjing and Fan, Weichen and others , title =. arXiv preprint arXiv:2610.22471 , year = on PithPith's reviewarXiv
  21. [41] Qin, Chonghan and Wu, Dongming and Feng, Kaituo and Wang, Jiaxu and Yu, Xinlei and Sun, Peiwen and Fan, Yijia and He, Xilin and Chen, Bingao and Lian, Weitong and others , title =. arXiv preprint arXiv:2610.20442 , year = on PithPith's reviewarXiv
Cited by 4 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment