{"id":"550e52bd-454f-4814-a9fa-c51fb44d5f20","arxiv_id":"2610.18867","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A coding agent that distills physical trial-and-error into reusable code skills and diagnostic lessons achieves leaderboard-leading RoboDojo-Sim scores with frozen model weights.","lead":"EmbodiedEvo lets a coding agent practice one robot scene per task, then freezes the written skills and lessons it discovered so fresh agents on new scenes can reuse them without retraining. On 42 RoboDojo simulation tasks it reports a 37.82% average success rate, above the published Astra baseline and all leaderboard systems listed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The full-set claim of surpassing all leaderboard systems rests on unverifiable protocol equivalence: the 42 Playground scene IDs are undisclosed, no standard-vs-random layout split is reported, and the 5.6–6.4 pp margins over VPP2 and PhysicalRSI are within plausible task-cluster noise.","rationale":"The paper's central claim is that a coding agent can exceed learned-action policies on RoboDojo-Sim by distilling one Playground scene of experience per task into frozen Skills and Lessons. For that headline to hold, the full-set run must be protocol-equivalent to the official leaderboard rows, and the gains must come from cross-layout reuse rather than from the single scene the authors were free to choose. I checked internal consistency before writing this critique: category and overall averages reconstruct exactly (44.48 and 37.82 are the unweighted means of the five category Scores and SRs), Table 4 matches Table 9's averages, Table 11 reconstructs Figure 10's transfer deltas, and Appendix F gives a complete, self-consistent trace. The paired dev results (8.50 to 30.17 percent SR without wiki) and the transfer table support a real mechanism, and the paper acknowledges its residual limits (Precision bottleneck, non-identification of causal effects in traces, cost estimates). The load-bearing soft spot is the full-set comparison itself: the margins that justify the “surpasses VLA/WAM” framing (5.56 pp over VPP2, 6.44 pp over PhysicalRSI) are within about one task-cluster standard error, so the categorical claim needs intervals before it is established; and the 42 Playground scenes, the only free design choice, are undisclosed with no standard-versus-random layout analysis, so the “unseen layout” premise cannot be audited. These are specific, addressable gaps rather than evidence of error. The reader's CONDITIONAL verdict already requests the Playground list, artifacts, and logs; I would add the standard/random split and cluster-bootstrap CIs on Table 3 as explicit conditions. The verdict therefore stands unchanged.","tokens_in":30332,"tokens_out":20336,"duration_ms":140833,"concrete_test":"Once the authors release the 42 Playground scene IDs and per-layout Test logs aligned to the public per-layout records [53], re-analyze the full set in two ways: (i) split SR by the 25 standard versus 25 random layouts per task and test whether the gains over the leaderboard rows persist on random layouts; (ii) recompute the Table 3 margins with task-cluster bootstrap confidence intervals (categories as strata, 10,000 draws, same seed convention as Appendix E). The concern lands if gains concentrate on standard scenes or if the EmbodiedEvo−VPP2 and EmbodiedEvo−PhysicalRSI 95% intervals include zero; it is refuted if gains hold on random layouts and the margins are significant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that a frozen coding agent with one Playground scene of experience per task beats every listed VLA, WAM, and agent system on RoboDojo-Sim — requires the 2,100-episode full-set run to be protocol-equivalent to the official leaderboard rows and the gains to reflect transfer across genuinely unseen layouts. Neither condition is auditable from the manuscript, and the margins over the closest non-agent systems are statistically fragile. Three specific gaps. (1) §4.1 says each harness is acquired on “one Playground scene layout distinct from the Test layouts,” but the 42 Playground scene IDs are never given, even though layout IDs exist for case studies (Table 12). That one scene fully determines the executable Skills distilled per task; a scene chosen for typical geometry could fit the frozen harness to the official layout pool without genuine cross-layout transfer. (2) The Test pool is 25 standard plus 25 random layouts per task, yet no result is split by layout type; per-task dev and full-set means (Tables 9–10) pool them. If gains concentrate on standard layouts correlated with the Playground scene and vanish on random layouts, “unseen scene” transfer is overstated. (3) Table 3 reports point estimates only, although Appendix E shows the authors know how to bootstrap task-level differences, and the full-set run uses one seed. Full-set task SRs range 0–100 within a category, so the between-system standard error on the category-balanced average is roughly 8 pp; the 5.56 pp margin over VPP2 and 6.44 pp over PhysicalRSI are within about one SE, so “surpassing VLA/WAM systems” is not statistically established even under perfect protocol equivalence. The 15.34 pp margin over GPT-6-Astra and the paired dev results (8.50 to 30.17 percent SR) support a real mechanism, so this is missing evidence, not evidence of a broken claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"EmbodiedEvo equips a closed-loop coding agent (Astra/xhigh) with a Playground phase in which, on one scene per RoboDojo-Sim task, the agent interacts through a fixed programmatic interface and distills its experience into parameterized Python Skills and textual Lessons; these artifacts are frozen and then supplied to fresh agents on unseen Test scenes, with no model-weight updates. The paper claims: on the full 2,100-episode set, 44.48 Average Score and 37.82% SR, above every listed VLA/WAM/agent system in an October 9, 2026 leaderboard snapshot (including the published GPT-6-Astra result of 28.97/22.48); paired dev-set gains of 8.50 to 30.17% SR with task descriptions alone; cross-model transfer (Astra experience raising Sol and Luna SR by 8.86 and 5.50 percentage points); comparable performance to raw interaction history at 53.0% lower execution cost; and gains concentrated on Memory and Open tasks while Precision and Long-Horizon remain bottlenecks. Supporting material includes per-task dev and full-set tables, layout-identified case studies, and a complete successful execution trace with verbatim reuse of a frozen grasp function.","tokens_in":30756,"tokens_out":12265,"duration_ms":91148,"significance":"If the results hold, the paper's core contribution is a clean demonstration that a frozen coding agent can inherit physical experience across scenes without parameter updates: the paired matched-scene comparisons in Table 4 show large, consistent gains, and the strongest asset is the concrete evidence of actual reuse — Appendix F reproduces the frozen pick_from_above function and shows it prepended verbatim to Test execution 2, and Appendix D.1 provides layout-identified execution records, making the reuse claims auditable. The cross-model transfer experiment in Table 6 is a well-designed separation of knowledge production from knowledge consumption, and the unfinished-Playground case in Table 7 is a non-obvious, falsifiable finding. The authors also state their limitations plainly: fine contact control remains unsolved, causal isolation of Skills versus Lessons is explicitly not claimed, and the raw-history comparison is honestly labeled as not a formal equivalence result.","major_comments":[{"comment":"The headline result (44.48 Score / 37.82% SR, [b175]) requires that EmbodiedEvo's evaluation protocol be comparable to the official leaderboard rows, but the protocol contains an unverifiable free parameter: §4.1 states that each harness is acquired on 'one Playground scene layout distinct from the Test layouts,' yet the 42 Playground layout IDs are never disclosed, even though layout IDs are reported for the three case studies in Table 12 and the Test scene identities are pinned to public records [53]. Because the executable content of each task's Skills is distilled from that single scene, an undisclosed choice of typical or easy layouts could fit the frozen harness to the official layout pool and inflate the Table 3 comparison. Please list the 42 Playground layout IDs and confirm their disjointness from the Test layouts.","section":"§4.1, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The setup explicitly defines two layout types — 25 standard plus 25 random layouts per task on the full set, and 3 plus 2 on the dev set — but no result is ever split by layout type: every per-task row and category mean in Tables 9 and 10 pools them. The claim that experience transfers to genuinely unseen scenes requires showing that the gains are not concentrated on standard layouts that resemble the Playground scene; report standard-versus-random splits at least at category level for both scales.","section":"§4.1, Tables 9–10","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim of surpassing all listed systems rests on point estimates from a single-seed, 2,100-episode run against a leaderboard snapshot; the margins over VPP2 are 5.22 Score points and 5.56 SR percentage points, while Table 10 shows task-level SRs ranging from 0 to 100 within a category, so the category-balanced aggregate has a between-task standard error on the order of 6–8 percentage points. Since Appendix E already demonstrates a paired task-bootstrap procedure, report bootstrap intervals for the Table 3 comparison, state explicitly whether the leaderboard rows were computed with the same category-balanced aggregation formula, and either add a second seed or temper the 'surpassing all' formulation accordingly.","section":"Table 3, Appendix E","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The author name 'Y oujie Zheng' contains an errant space; it should read 'Youjie Zheng' to match the reference list entry [46].","section":"Author block","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states that Full Evaluation uses task description plus wiki, but it does not state whether the Playground acquisition chains behind the Table 3 run also received the wiki. Please specify the input setting for Playground in the full-set run.","section":"§4.1 Inputs and budgets","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The dev-set Astra baseline with wiki (20.67 Score / 13.36% SR) is well below the published full-set Astra result (28.97 / 22.48%); clarify whether the five dev scenes per task are a fixed subset of the 50 full-set scenes and whether this gap reflects scene difficulty or run-to-run variability.","section":"§4.2, Tables 3–4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The caption describes the rows as 'selected systems' while §4.2 says the comparison covers 'the VLA, WAM, and agent-based results in the October 9, 2026 leaderboard snapshot'; state explicitly whether Table 3 includes every verified row in that snapshot.","section":"Table 3 caption","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Worth your time, with one caveat. The genuinely new thing is the clean design: one Playground scene per task, distillation into executable Skills plus diagnostic Lessons, freeze, then fresh agents on unseen scenes. That mostly avoids the usual self-evolution circularity, because gains are measured against frozen baselines on matched scenes. The paired dev results (8.50% to 30.17% SR on task descriptions alone), the cross-model reuse numbers (Astra experience raises Sol and Luna SR by 8.86 and 5.50 pp), and the full 15-execution trace in Appendix F with verbatim code reuse are the best evidence that experience is actually inherited. That is more reproducible evidence than most papers in this area ship.\n\nThe soft spots are real but addressable. The headline 44.48 Score / 37.82% SR full-set run is one seed with no confidence interval, and the margins over VPP2 (5.56 SR pp) and PhysicalRSI (6.44 pp) are within plausible noise for a category-balanced average over tasks whose per-task SRs range 0 to 100. The authors know how to bootstrap—Appendix E has intervals for the raw-history comparison—so the absence of intervals on the headline comparison stands out. Two things are missing: the 42 Playground scene IDs, and any standard-vs-random split of the 25+25 Test layouts. The first matters because one scene fully determines the distilled Skills; scene selection could inflate transfer without genuine cross-layout generalization. The second matters because if gains concentrate on standard layouts resembling the Playground scene and vanish on random layouts, the “unseen scene” claim is overstated. The paper honestly notes that Playground success is evidence about one scene, but that does not fully cover the leaderboard claim.\n\nThe category breakdown also cuts against a blanket “coding agents beat VLA/WAM” reading: Precision SR is 13.25% and Long-Horizon 23.50%, well below the listed VLA and WAM systems, and the overall win is carried by Memory and Open. The leaderboard comparison uses an external snapshot the authors did not control, but they align input conditions with the published GPT-6-Astra evaluation and cite the public per-layout records. The proprietary model and simulator limit independent audit, though the trace appendix mitigates this.\n\nWho this is for: people working on embodied agent memory, skill libraries, and benchmark protocol design. It deserves a serious referee: the mechanism and trace evidence clear the bar, and the missing scene IDs, layout split, and intervals are all obtainable rather than fatal. I would bring it to our reading group and would cite it once the artifacts land.","headline":"The mechanism is real and well-evidenced in paired tests; the headline leaderboard claim is one-seed, undisclosed-scene, and needs artifacts before it should be treated as established.","tokens_in":31421,"tokens_out":3323,"duration_ms":24816},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A coding agent that distills its physical trial-and-error into reusable code Skills and diagnostic Lessons reaches a 37.82% success rate on the full 2,100-episode RoboDojo-Sim set, ahead of every listed VLA, world-action-model, and agent…","keywords":["self-evolving agents","embodied control","coding agents","robot manipulation","skill libraries","programmatic robot control","experience transfer across models","RoboDojo benchmark"],"falsifier":"Publish the 42 Playground scene identities and verify they are absent from both the 50 Test layouts and the official Astra evaluation records, then rerun the full 2,100-episode evaluation from a second, independently chosen Playground selection; if the 44.48 Score and 37.82% SR figures do not reproduce within sampling noise, or if a fresh Astra agent given only the wiki and public per-layout records with no Playground closes most of the gap, then the gain is scene selection or input leakage rather than distilled physical experience.","tokens_in":31807,"feed_emoji":"🤖","tokens_out":10106,"duration_ms":76370,"temperature":0.7,"pith_summary":"The paper claims that an embodied coding agent can learn from its own physical mistakes without changing any model weights, by writing its discoveries into two kinds of files: executable Skills, which are parameterized control procedures such as a grasp with a checked lift, and diagnostic Lessons, which record when an operation applies, what failure looks like, and how to recover. Experience is acquired on a single Playground scene per task, then frozen and handed to fresh agents in unseen scenes, making the harness rather than the network the evolving locus of knowledge. On RoboDojo-Sim this raises the agent's success rate from the published Astra baseline of 22.48% to 37.82%, with Average Score 44.48, ahead of the listed VLA, world-action-model, and agent+VLA systems; matched paired tests show the same executor jumping from 8.50% to 30.17% success when handed the experience with task descriptions alone. The authors take this as evidence that reusable control knowledge can be distilled, frozen, and transferred across scenes and even across weaker models, and that the remaining bottleneck is precise contact control, where success stays at 13.25%.","feed_headline":"Agent that saves its own skills tops every published RoboDojo baseline","feed_subtitle":"Freezing a coding agent's distilled skills lifts its RoboDojo score to 44.48, ahead of every listed baseline.","key_machinery":"The mechanism is a two-artifact evolving harness with a strict freeze boundary. A Skill is an executable, parameterized Python procedure composed from the fixed interface primitives (get_instruction, get_observation, step, and Playground-only reset), carrying its own preconditions, action budgets, tolerances, and execution checks; for example, pick_from_above descends, closes the gripper with a dwell, lifts slowly, and returns for visual retention verification rather than assuming the object is secured. A Lesson is text that records applicability conditions, observed failure modes, and recovery guidance while distinguishing verified findings from hypotheses. The freeze boundary does the causal work: Skills and Lessons are written only during Playground on one scene per task, then become read-only inputs to independent Test agents, so any Test improvement is attributable to the retained artifacts rather than to shared memory, updated weights, or within-Test learning. Raw interaction history is the same experience in undistilled form, and its comparison isolates what the distillation itself contributes: comparable observed performance at a 53.0% lower execution cost.","core_discovery":"On the paper's own terms, EmbodiedEvo establishes that a zero-shot coding agent is limited less by understanding the task than by establishing reliable operations and their preconditions, and that this missing experience can be manufactured in advance. A dedicated Playground phase lets the agent interact with one scene per task, revise its own control programs against physical feedback, and compress what it learns into parameterized Skills and textual Lessons; the harness is then frozen and carried into Test, where each unseen scene starts with a fresh agent, workspace, and simulator and no reset is allowed. The central result is the full-set evaluation: 44.48 Average Score and 37.82% success over 2,100 episodes, exceeding the published GPT-6-Astra result (28.97 Score, 22.48% SR) and the listed VPP2, Simate-beta, and PhysicalRSI systems. The gains concentrate in Memory tasks (71.90 Score, 69.67% SR) and Open tasks (55.52 Score, 53.00% SR), where reusable grasp-carry-place procedures and observation-grounded Lessons replace repeated within-episode rediscovery, while Precision (13.25% SR) and Long-Horizon (23.50% SR) stay below the strongest listed systems, which the paper attributes to unresolved contact transitions such as a receiver closing empty during handover. The same frozen artifacts improve weaker executors, with Astra's experience raising Sol's success rate by 8.86 percentage points and Luna's by 5.50, and they match raw interaction history in task performance at 53.0% lower test execution cost.","pith_inferences":["Because experience is stored as readable code and text rather than in weights, the harness makes robot behavior auditable in a way learned-action policies are not: a user could trace exactly which Lesson triggered a recovery and which Skill parameter decided a grasp; the paper's traces demonstrate this, though the authors do not elevate it into a claim about interpretability.","The mismatch-transfer result, where mismatched-task experience beats related-task experience by 4.47 Score points, suggests transfer value is carried by generic operations and applicability conditions rather than task-label similarity, so a natural extension the paper does not build is a retrieval or ranking mechanism that selects which frozen artifacts to load for each new scene.","The cost reduction concentrates on the input side, with 87.9% fewer uncached tokens, which implies that as inference pricing dominates agent economics, representation design could matter as much as model choice for embodied agents; one testable corollary is that well-indexed skill libraries will outcompete larger models carrying raw histories.","A caution worth testing rather than assuming: the paired 8.50% baseline is measured by the authors on five scenes per task without the wiki, while the published 22.48% figure is the official 50-scene wiki evaluation, so the largest single-number gain (30.17% vs 8.50%) and the headline comparison (37.82% vs 22.48%) rest on different scene sets and inputs; reproducing the paired analysis at full 50-"],"forward_implications":["If the full-set result holds, weight-free experience accumulation is viable: an embodied agent can improve across scenes by editing its own skill files, so improving at manipulation does not require gradient updates on robot data.","The transfer results imply that control knowledge is a portable asset: a stronger model's frozen Playground files raise Sol and Luna success rates by 8.86 and 5.50 percentage points, and even mismatched-task experience adds 18.65 Score points over no experience, suggesting manipulation libraries could be produced once and reused by many executors.","The representation matters economically: Skills and Lessons slightly outperform raw interaction history (38.64 vs 36.41 Score) while cutting test cost per episode from $12.20 to $5.73, an 87.9% reduction in uncached input, so distilling exploration logs is the cheaper way to carry experience forward.","The method's own boundary is contact: with Precision success at 13.25% and Long-Horizon success at 23.50%, the approach transfers operations and decision conditions but not the reliable contact transitions those operations still depend on."],"supporting_citations":[{"why":"The published GPT-6-Astra evaluation on RoboDojo supplies the official baseline (28.97 Score, 22.48% SR) and the protocol that EmbodiedEvo must beat and align with.","marker":"[6]"},{"why":"Agent as Policy establishes the coding-agent-as-closed-loop-policy paradigm and the observation that saved procedures speed repeated execution, which EmbodiedEvo extends into a frozen skill and lesson harness.","marker":"[20]"},{"why":"Voyager provides the executable skill-library idea that EmbodiedEvo adapts to physical, feedback-grounded manipulation.","marker":"[26]"},{"why":"RATs is the closest prior for acquiring and freezing robot skills through self-directed play, including transfer to other coding agents.","marker":"[27]"},{"why":"RoboProbe is the codebase from which the workspace protocol, wiki recipes, and per-layout evaluation records are drawn.","marker":"[51]"},{"why":"RoboDojo supplies the benchmark, the 42 tasks, the task-specific scoring rules, and the native action interface used in all reported evaluations.","marker":"[52]"},{"why":"The public per-layout evaluation records fix the Test scene identities so the full-set evaluation can be aligned with the official leaderboard snapshot.","marker":"[53]"},{"why":"The fixed task recipes (wiki) are the experiment input that makes the full evaluation comparable to the official GPT-6-Astra condition.","marker":"[54]"}],"fun_headline_variants":["Coding agent distills physical feedback into reusable skills, tops RoboDojo","Frozen skills lift self-evolving robot agent past all RoboDojo baselines","Robot agent saves its own skills to beat every published RoboDojo result","Self-distilled skills hit 44.48 RoboDojo score, beating all baselines","Self-evolving coder reuses distilled lessons to top all RoboDojo baselines"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The headline comparison assumes that the author-chosen Playground scene per task, whose layouts are never identified, is neutral, so that distilling experience there does not make Test episodes easier or leak task-specific scoring knowledge through the wiki recipes and public per-layout records beyond the conditions the published baselines faced.","fun_headline_variants_meta":{"raw":{"variants":["Coding agent distills physical feedback into reusable skills, tops RoboDojo","Frozen skills lift self-evolving robot agent past all RoboDojo baselines","Robot agent saves its own skills to beat every published RoboDojo result","Self-distilled skills hit 44.48 RoboDojo score, beating all baselines","Self-evolving coder reuses distilled lessons to top all RoboDojo baselines"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"cost":0.0001471288,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0001471288,"upstream_inference_prompt_cost":3.26508e-05,"upstream_inference_completions_cost":0.000114478},"total_tokens":1839,"prompt_tokens":1210,"completion_tokens":629,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1064,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":532}},"tokens_in":1210,"tokens_out":629,"cost_source":"openrouter_usage","duration_ms":7134,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":532,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-15T02:48:53.919041+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Publish the 42 Playground scene identities and verify they are absent from both the 50 Test layouts and the official Astra evaluation records, then rerun the full 2,100-episode evaluation from a second, independently chosen Playground selection; if the 44.48 Score and 37.82% SR figures do not reproduce within sampling noise, or if a fresh Astra agent given only the wiki and public per-layout records with no Playground closes most of the gap, then the gain is scene selection or input leakage rather than distilled physical experience.","supporting_citations":[{"cited_title":"EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics","cited_arxiv_id":"2609.27308","evidence_quote":"The published GPT-6-Astra evaluation on RoboDojo supplies the official baseline (28.97 Score, 22.48% SR) and the protocol that EmbodiedEvo must beat and align with."},{"cited_title":"Agent as Policy for Robotic Manipulation","cited_arxiv_id":"2609.12541","evidence_quote":"Agent as Policy establishes the coding-agent-as-closed-loop-policy paradigm and the observation that saved procedures speed repeated execution, which EmbodiedEvo extends into a frozen skill and lesson harness."},{"cited_title":"Query-as-context Pre-training for Dense Passage Retrieval","cited_arxiv_id":"2305.16291","evidence_quote":"Voyager provides the executable skill-library idea that EmbodiedEvo adapts to physical, feedback-grounded manipulation."},{"cited_title":"Playful Agentic Robot Learning","cited_arxiv_id":"2606.19419","evidence_quote":"RATs is the closest prior for acquiring and freezing robot skills through self-directed play, including transfer to other coding agents."},{"cited_title":"RoboProbe: An agentic robot manipulation codebase","cited_arxiv_id":null,"evidence_quote":"RoboProbe is the codebase from which the workspace protocol, wiki recipes, and per-layout evaluation records are drawn."},{"cited_title":"RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies","cited_arxiv_id":"2607.04434","evidence_quote":"RoboDojo supplies the benchmark, the 42 tasks, the task-specific scoring rules, and the native action interface used in all reported evaluations."},{"cited_title":"GPT-6 Astra per-layout RoboDojo evaluation records","cited_arxiv_id":null,"evidence_quote":"The public per-layout evaluation records fix the Test scene identities so the full-set evaluation can be aligned with the official leaderboard snapshot."},{"cited_title":"The arm reached its target, but load retention failed","cited_arxiv_id":null,"evidence_quote":"The fixed task recipes (wiki) are the experiment input that makes the full evaluation comparable to the official GPT-6-Astra condition."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":2}