{"id":"6d53132b-c6e5-4d73-b180-a036f01f0deb","arxiv_id":"2610.26848","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Six model systems improve at simulated robot tasks along five separable axes, and the capabilities that predict starting well (spatial reasoning) differ from those that predict recovery (perception).","lead":"What a model achieves on its first try at a robot task does not predict how quickly it learns from failures: this paper measures six AI systems with five separate improvement scores and shows the two qualities come apart. The result gives robotics teams a way to tell whether an agent is truly improving and what a model's public test reputation actually says about physical learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Capability-bottleneck attribution is unidentifiable at n=6: the Spatial-vs-Perception split (rho 0.94 vs 0.83, Section 4.1) is within sampling noise and no global-strength control is reported, so the Barrel Hypothesis rests on an untested premise.","rationale":"The reader's weakest_assumption identifies the same load-bearing point: the capability-association analysis presumes that the 17 public benchmark scores, grouped into four profiles, are valid and separable proxies for the capabilities used during physical refinement, and the six-system correlations cannot separate axis-specific capabilities from overall model strength. I agree, and I did not find a separate internal inconsistency in the axis definitions or in the workspace-intervention results. The five-axis profiles are clearly specified in Section 3.4, the Opus-vs-GPT-6.1 divergence in Table 1 is a direct measurement, and the ablation and frozen-workspace results are explicitly scoped to Astra and six tasks in Section 7. The capability-bottleneck attribution is the one claim that would be decisive for Q1, and it is not identifiable from six points after multiple-testing control. The paper's own honesty in Section 7 does not dissolve the problem: saying that capability bottlenecks remain hypotheses to test is correct, but Section 4.2 currently presents the barrel hypothesis as motivated by the correlation matrix, and Section 5.2's cross-model ranking reuses the same profiles, so the weak statistical base propagates into the scaling-law pillar. The proposed leave-one-out plus global-index check is cheap and can be run on the existing data; a seventh independently evaluated system, such as Gemini 4 on the full 18-task protocol, would be a stronger external check. The verdict stays CONDITIONAL: the measurement contribution is real, but Q1's capability attribution needs either the stability evidence or an explicit downgrade to pure description.","tokens_in":22110,"tokens_out":10735,"duration_ms":72952,"concrete_test":"Stability check for Section 4.1: construct G_i, the mean normalized rank over all 17 benchmarks for each system. (1) Leave-one-system-out: rebuild the four group profiles from the remaining five systems and recompute all axis-group Spearman rhos; record whether Spatial remains the top group for A1/A2/A4 and Perception for A3/A5 in all six deletions. (2) Compute rho(A_j, G) for each axis and report it alongside the named group rhos. If any deletion changes the top group for A1, A3, or A5, or if rho(A_j, G) equals or exceeds the named group rho for any axis, the axis-specific attribution is within noise of a general-strength effect and Section 4.2 should be presented as an untested conjecture rather than a motivated hypothesis. If the named groups win all six deletions and consistently beat G, the barrel attribution survives this six-system test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's Q1 claim is that different axes of physical RSI have different capability bottlenecks: Spatial reasoning is associated with A1/A2/A4 and Perception with A3/A5 (Section 4.1), motivating the Physical Barrel Hypothesis (Section 4.2). That attribution rests entirely on Spearman correlations over six systems between hand-grouped public-benchmark ranks and the five axes. At n=6 the gap between rho=0.94 (Spatial-A1) and the next-best rho=0.83 is small relative to the sampling resolution of a rank correlation, and Appendix D reports that the smallest Benjamini-Hochberg adjusted q from the exact permutation tests is 0.11, so no individual axis-group association is conventionally significant. The paper does not report the 4x4 cross-correlation matrix of the four group profiles, nor does it compare axis-group rhos with a single global-capability index such as the mean of all 17 benchmark ranks. Because the six systems are frontier models whose public profiles are likely correlated, the 'strongest association' labels may simply reflect overall model strength: Astra is best overall and also best in Spatial. The paper explicitly frames these bottlenecks as candidate hypotheses (Section 7, b265-b266), so this is not a soundness defect, but it is the load-bearing premise of Q1. The five-axis divergence itself (e.g., Opus A3=0.744 vs GPT-6.1 Sol A3=0.703 while A2=0.609 vs 0.822, Section 5.1) is a direct measurement and is not threatened by this concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper introduces PhysRSI-eval, a fixed-budget benchmark in which six frontier VLM systems (with a separate partial Gemini 4 run) refine persistent workspaces across 18 RoboDojo manipulation tasks, and proposes a five-axis profile — First-Attempt Performance (A1), Efficiency (A2), Success (A3), Budget (A4), and Improvement from failure (A5) — to quantify physical recursive self-improvement (RSI). It addresses three questions: (Q1) the capability origins of physical RSI, answered by rank-correlating four public-benchmark capability groups (Spatial, Harness, Perception, Coding) with the five axes and motivating the Physical Barrel Hypothesis; (Q2) the scaling of physical RSI, answered by cross-model prediction of best score and half-gain round (T50) from public capability profiles and by Weibull-template forecasting of a related model's trajectory from its first five rounds; and (Q3) whether gains reflect overfitting or generalization, answered by history-removal ablations, frozen-workspace tests on unseen layouts, and cross-task workspace transfer.","tokens_in":22504,"tokens_out":27846,"duration_ms":155519,"significance":"If the descriptive core holds, the paper makes a genuine methodological contribution: it argues credibly that endpoint success is insufficient for evaluating LLM-as-policy agents, that gain timing and physical cost are separable outcomes, and that public benchmark averages can misorder systems on these outcomes. The strengths are real: the metric definitions in Section 3.4 are precise about edge cases (undefined denominators, conditional task sets, no-interpolation tau90); task strata were fixed before evaluation; the ablation design controls individual information channels; the forecast is genuinely predictive (calibrated on K1-K5, evaluated on K6-K17) with paired-bootstrap bands; and Section 7 itemizes limitations candidly, including one chain per task, first-success stopping, and conditional A4/A5 sets. The weakest premise is the capability-bottleneck attribution: at n=6, with a smallest Benjamini-Hochberg adjusted q of 0.11 (Appendix D) and no global-strength control, the specific axis-capability labels are not statistically resolvable, so the Physical Barrel Hypothesis is presently an illustration rather than a supported finding.","major_comments":[{"comment":"The axis-specific capability attribution is not identified at n=6. The Spearman associations quoted (Spatial-A1 rho=0.94, Spatial-A2/A4 rho=0.83, Perception-A3/A5 rho=0.94) are computed over six systems whose four capability-group profiles are likely strongly rank-correlated with one another, and the smallest Benjamini-Hochberg adjusted q across the exact permutation tests (Appendix D) is 0.11, so no individual association is conventionally significant and the 0.94-versus-0.83 gap is within the sampling resolution of a rank correlation at this sample size. The paper does not report the 4x4 cross-correlation matrix of the group profiles, nor any control against a single global-capability index (e.g., the mean of all 17 benchmark ranks); because Astra, the strongest system overall, leads all five axes, the 'Spatial for A1, Perception for A3/A5' labels may reflect overall model strength rather than distinct bottlenecks. The Section 7 caveats and the 'candidate bottlenecks' framing in Section 4.2 are appropriate and are part of why this is not a soundness defect, but Section 4.1 itself should state that the axis-group differences are not statistically resolvable at this sample size, and should add the global-index control and the group-profile cross-correlation matrix. The five-axis divergence in Table 1 is direct measurement and is not affected by this concern.","section":"Section 4.1 and Appendix D","authors_acknowledge":"These associations suggest candi- date bottlenecks, not causal requirements.","authors_acknowledge_p":0.61},{"comment":"The definition of A2 (Efficiency) uses K0 as its reference, so the k=1 term (q1-q0)/(qH-q0) credits each system's full K0-to-K1 jump in its Efficiency score. Section 3.2 acknowledges that this jump 'mixes interface changes with improvement' and that the Harness rounds are the self-improvement segment; using q0 in Eq. (1) is in tension with that statement. Since q1 is not reported (Table 4 lists q0, q17, qmax, tau90, C1, C17, and recovery counts but not q1), the reader cannot assess how much of the headline divergence 'Opus has a higher Success score than GPT-6.1 Sol (0.744 versus 0.703), but lower Efficiency (0.609 versus 0.822)' (Section 5.1) is driven by differential one-shot interface benefits rather than refinement timing. Please either compute A2 from q1 forward, or report q1 (or an A2 variant with q1 as reference) so that the interface contribution is bounded.","section":"Section 3.4, Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The cross-model 'prediction' claim in Section 5.2 mixes an in-sample ranking with a leave-pair-out prediction. Ranking systems by the equal-weight average of the four capability profiles is computed from normalized ranks defined over all six systems, so reproducing the observed order of best score by K17 (q-bar-17) is a descriptive co-ranking, not a prediction for unseen systems; only the T50 analysis is actually leave-pair-out. The Figure 7 caption ('Each pair of systems is predicted from the other four') and the 'predicted/observed' labels on the best-by-K17 column conflate the two procedures. Please state the in-sample nature of the best-score ranking explicitly (or re-normalize profiles under leave-one-out for the ranking claim), and report whether the kernel-regression hyperparameters (gamma=0.0625, alpha=10, Appendix D) were fixed before or independently of the outcomes.","section":"Section 5.2 and Figure 7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Weibull transfer forecast rests on a single source-target pair. The template gA(t) = 1 - exp(-(t/tauA)^pA) fitted to Astra's K1-K17 and then calibrated on GPT-6.1 Sol's first five rounds achieves MAE 0.019 on the held-out rounds versus 0.112 for holding the last observed score, but no baseline isolates the contribution of the transferred template shape: a Weibull fitted to Sol's own first five rounds without the Astra prior, or a template taken from a less related system, would test the paper's transfer assumption ('A related model may share the shape of its improvement curve'). With one target system, the general claim that a reference trajectory can predict a related model's later performance is a demonstration, not an established result, and Section 5.3 should say so explicitly; adding at least one no-template or cross-template baseline would materially strengthen the analysis.","section":"Section 5.3 and Appendix D","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The caption says the matrix 'reports exploratory rank associations,' but the displayed numeric values (0.93, 0.78, 0.68, 0.44 for Astra, and so on) match the per-system capability profiles in Figure 7 rather than the Spearman rho values quoted in Section 4.1; please clarify what the matrix entries are, or plot the actual correlation matrix.","section":"Figure 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The rows for 'Reasoning budget per round' and 'Revision window' contain stray superscript markers that obscure which values are session-log estimates, and the row 'Selected K1-K17 entries' (91, 143, 255, 306, 175, 175) should state how a 'selected entry' is defined.","section":"Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sentence 'Nine GPT-5.6 Sol Harness entries summarize two observed runs' needs a specification of how two runs were collapsed into the nine entries used in the aggregates.","section":"Section 3.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The term 'rapid contraction region' is used as an explanatory device but is never defined; please define it or replace it with a direct description of the gap between the public-score and physical-profile orderings.","section":"Section 5.1 and Figure 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The matched-set check for the A4 comparison ('the gap persists on the ten tasks both systems solve') is good practice; please extend the same check to the A5 comparison, whose conditional task sets also differ across systems.","section":"Section 3.4 and Section 5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The phrase 'Observational Scaling Law' is stronger than the evidence base presented (six systems for the cross-model analysis, one source-target pair for the forecast); the body's 'exploratory' framing (Sections 5.2 and 7) is more accurate, and the title and abstract should be aligned with it.","section":"Title and abstract","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Short version: the five-axis PhysRSI-eval protocol is a genuine measurement contribution; the capability attribution and \"scaling law\" parts are exploratory correlations on six systems, which the authors mostly concede, so the benchmark deserves peer review but the law framing should be softened.\n\nWhat is actually new is the fixed-budget protocol: persistent workspace, judged physical attempts, K0 separated from K1–K17 refinement, and five axes with definitions precise enough to reimplement. That setup cleanly separates initial competence, gain timing, attained level, budget savings, and recovery from failure. The descriptive findings—Opus scores higher than GPT-6.1 Sol on Success but lower on Efficiency and Budget; systems with close public benchmark averages separate on the axes; removing history cuts Astra's late-round success from 41.7% to 8.3%; frozen workspaces succeed on unseen layouts—are direct measurements, and I see no reason to doubt them. The task strata were fixed before evaluation, the metric algebra is explicit in Section 3.4 and Appendix B, and Section 7 states the real scope limits: one chain per task, first-success stopping, conditional task sets, and a capability analysis without independent validation.\n\nThe soft spots are the interpretive layer. The capability associations are Spearman correlations on six systems, between hand-grouped public benchmark ranks and the axes. At n=6, the 0.94 versus 0.83 gap is well inside sampling noise, the smallest adjusted q is 0.11, and there is no control for overall model strength. The authors call these candidate bottlenecks, so this is not a case of hiding the weakness, but the Physical Barrel Hypothesis is an interpretation, not a supported result. The same goes for the two scaling analyses: leave-pair-out on six points with hand-fixed kernel hyperparameters is a worked example, and the Weibull forecast is one reference-target pair with no stated selection rule. The five axes are point estimates with no chain-level uncertainty, and the article itself releases no code, data, or prompts, which limits independent checking. The stress-test note is right about the attribution being unidentifiable at this n, though the direct five-axis divergence does not depend on it.\n\nBottom line: this is a solid measurement paper with a speculative overlay. It should go to a serious referee; the right response is to keep the benchmark, add uncertainty quantification and data release, and present the capability claims as hypotheses. People building LLM-as-policy evaluations will get value from the protocol; people looking for a scaling law should look elsewhere.","headline":"The five-axis PhysRSI-eval protocol is a real measurement contribution; the capability-bottleneck and scaling-law claims are exploratory correlations on six systems, which the authors mostly concede, so the benchmark deserves peer review but the 'law' framing should be softened.","tokens_in":23195,"tokens_out":3639,"duration_ms":25834},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Physical self-improvement in robot agents is five separate abilities, not one final score","keywords":["physical recursive self-improvement","LLM-as-policy","five-axis evaluation","Physical Barrel Hypothesis","observational scaling","robot manipulation","workspace transfer","capability bottleneck"],"falsifier":"Run the same fixed-budget loop on enough additional systems that the four capability profiles are not strongly rank-correlated with each other; if a system with low spatial benchmark ranks still shows high First-Attempt Performance, or a system with low perception ranks still shows high Success and Improvement from failure, the claimed bottleneck associations for A1 and A3/A5 collapse. A within-system version would ablate spatial or perceptual inputs and check whether A1 and A5 move independently as the hypothesis predicts.","tokens_in":23465,"feed_emoji":"🤖","tokens_out":9158,"duration_ms":58757,"temperature":0.7,"pith_summary":"This paper tries to establish that a robot agent's capacity to improve through physical trial and error is not a single ability. It builds a fixed-budget evaluation loop, PhysRSI-eval, in which six vision-language model systems act as robot policies by writing code and revising a persistent workspace across 18 dual-arm manipulation tasks, and it scores them on five axes: First-Attempt Performance, Efficiency, Success, Budget, and Improvement from failure. Its central claim, the Physical Barrel Hypothesis, is that different capabilities limit different parts of improvement: spatial reasoning shows the strongest association with starting well, while perception shows the strongest association with final score and recovery from failure. The paper further claims that public benchmark averages hide these differences, that public capability scores predict the relative order of physical improvement and its timing, and that retained workspace experience both helps and transfers to new layouts and tasks. If this is right, judging an LLM-as-policy agent by its final score alone is not just incomplete but can rank systems in a misleading order.","feed_headline":"Robot agents improve along five axes, not one score","feed_subtitle":"Spatial reasoning drives first attempts; perception drives recovery from failure, tests on 18 tasks show.","key_machinery":"The load-bearing mechanism is the measurement loop plus the five-axis profile it feeds. At K0 the agent gets one in-context first attempt with no workspace; from K1 to K17 each round executes one graded physical attempt, then the agent revises a persistent workspace of skills, tools, policy code and notes with robot actions disabled, and commits it for the next round. The five axes are defined on the resulting task-mean curves: A1 is the mean K0 score, A2 measures how early the terminal gain is realized, A3 is the mean score over the high window after reaching 90% of peak gain, A4 is the fraction of physical budget saved on solved tasks, and A5 is the fraction of K1 failures later solved. This design is what lets the paper separate starting well, improving early, finishing high, spending little, and overcoming failure, and then associate each axis with a different capability group.","core_discovery":"On its own terms, the paper's discovery is that physical recursive self-improvement is multidimensional. Across the 18-task suite, the systems' equal-weight five-axis mean spans 13.9–73.4 while their mean over 17 public benchmarks spans only 54.1–78.1, and a higher final score need not come with earlier or cheaper gains: Opus 5.5 reaches a higher Success score than GPT-6.1 Sol (0.744 versus 0.703) while having lower Efficiency (0.609 versus 0.822). The capability association analysis finds that spatial reasoning has the strongest observed association with First-Attempt Performance ($\\rho = 0.94$) and with Efficiency and Budget ($\\rho = 0.83$), while perception has the strongest association with Success and Improvement from failure ($\\rho = 0.94$). These associations are presented as candidate bottlenecks for the Physical Barrel Hypothesis, not as causal proof. On the generalization side, the paper reports that clearing history drops Astra's late-round success from 41.7% to 8.3%, frozen workspaces succeed in 25 of 60 unseen-layout trials, and three cross-task cases reach first success earlier when initialized from another task's workspace, so scene-specific adaptation and reusable experience appear to coexist.","pith_inferences":["Direct capability interventions would settle the barrel attribution: degrading spatial inputs should selectively lower first-attempt performance, and degrading perception should selectively lower recovery, if the hypothesis names the true bottleneck.","A practical screening rule follows from the forecasting result: a few early physical rounds on a reference system could be used to rank candidate models before committing a full interaction budget, turning the observational law into a cheap selection procedure.","Because the transfer evidence points to two carriers, executable tools and textual operating knowledge, future benchmarks should log workspace lineage and tool reuse as first-class measurements, since a system that reuses broadly may transfer better than one whose gains are concentrated in a few tools.","The five-axis separation suggests that error analysis for embodied agents should distinguish cannot-start failures from cannot-recover failures, since different capability fixes may apply to each."],"forward_implications":["Final-score rankings can mislead: a system that reaches a high score late and spends most of its physical budget can rank above a system that improves earlier and cheaper, so evaluation reports should include gain timing and budget use.","Public capability benchmarks carry predictive signal: ranking systems by averaged public capability ranks recovered the observed order of best score reached by K17, and all pairwise orderings of half-gain round T50 were recovered from the other systems in the set.","Early physical feedback can forecast later progress: using Astra's improvement shape and the first five rounds of GPT-6.1 Sol predicted its hidden rounds with mean absolute error 0.019, versus 0.112 from holding the last observed score.","Retained experience is doing real work: removing history cut Astra's late-round success from 41.7% to 8.3%, and the no-grader condition dropped it to 13.9%.","Some improvements are not overfitting to one scene: frozen workspace files solved 25 of 60 held-out-layout trials, and three cross-task transfers reached first success earlier."],"supporting_citations":[{"why":"Supplies the 18 RoboDojo manipulation tasks and the official per-task success rates used to fix the difficulty strata before evaluation.","marker":"[3]"},{"why":"Provides the Astra agent's reported physical-policy capability and the one-time-fix case that motivates tracking reliable improvement, not just success.","marker":"[9]"},{"why":"Establishes the code-as-policies mechanism by which the evaluated agents turn language-model reasoning into robot control programs.","marker":"[22]"},{"why":"Supplies the observational-scaling method of predicting held-out agent performance from benchmark-derived capability factors.","marker":"[34]"},{"why":"Provides the Roboflow Vision Evals scores used in the Spatial and Perception capability profiles.","marker":"[33]"},{"why":"Provides the ParseBench scores used for visual grounding and chart perception in the capability profiles.","marker":"[55]"},{"why":"Provides the LiveBench scores used for coding and instruction-following capability groups.","marker":"[44]"},{"why":"Provides the Terminal-Bench 4.0 scores used for harness and coding capability groups.","marker":"[28]"}],"fun_headline_variants":["Physical self-improvement splits into five axes, not one score","Spatial reasoning linked to first attempts, perception to failure recovery","Five-axis metric separates robot agents that public benchmarks cannot","Clearing history collapses late-round success for robot agents","Physical RSI is multidimensional and scene-specific experience persists"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that the 17 public benchmark scores, averaged into Spatial, Harness, Perception and Coding profiles on 15 October 2026, measure the same capabilities the agents actually use during physical trial-and-error revision; if those scores are not valid proxies, the barrel attributions and the rank predictions built on them lose their footing.","fun_headline_variants_meta":{"raw":{"variants":["Physical self-improvement splits into five axes, not one score","Spatial reasoning linked to first attempts, perception to failure recovery","Five-axis metric separates robot agents that public benchmarks cannot","Clearing history collapses late-round success for robot agents","Physical RSI is multidimensional and scene-specific experience persists"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"cost":0.0001425242,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0001425242,"upstream_inference_prompt_cost":3.24142e-05,"upstream_inference_completions_cost":0.00011011},"total_tokens":1690,"prompt_tokens":1085,"completion_tokens":605,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":911,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":537}},"tokens_in":1085,"tokens_out":605,"cost_source":"openrouter_usage","duration_ms":7430,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":537,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-20T05:08:27.123752+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the same fixed-budget loop on enough additional systems that the four capability profiles are not strongly rank-correlated with each other; if a system with low spatial benchmark ranks still shows high First-Attempt Performance, or a system with low perception ranks still shows high Success and Improvement from failure, the claimed bottleneck associations for A1 and A3/A5 collapse. A within-system version would ablate spatial or perceptual inputs and check whether A1 and A5 move independently as the hypothesis predicts.","supporting_citations":[{"cited_title":"RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies","cited_arxiv_id":"2607.04434","evidence_quote":"Supplies the 18 RoboDojo manipulation tasks and the official per-task success rates used to fix the difficulty strata before evaluation."},{"cited_title":"Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies","cited_arxiv_id":"2609.38537","evidence_quote":"Provides the Astra agent's reported physical-policy capability and the one-time-fix case that motivates tracking reliable improvement, not just success."},{"cited_title":"Code as Policies: Language Model Programs for Embodied Control","cited_arxiv_id":"2209.07753","evidence_quote":"Establishes the code-as-policies mechanism by which the evaluated agents turn language-model reasoning into robot control programs."},{"cited_title":"Observational Scaling Laws and the Predictability of Language Model Performance","cited_arxiv_id":"2405.10938","evidence_quote":"Supplies the observational-scaling method of predicting held-out agent performance from benchmark-derived capability factors."},{"cited_title":"Roboflow Vision Evals, 2026.https:// playground.roboflow.com/evals","cited_arxiv_id":null,"evidence_quote":"Provides the Roboflow Vision Evals scores used in the Spatial and Perception capability profiles."},{"cited_title":"ParseBench: A Document Parsing Benchmark for AI Agents","cited_arxiv_id":"2604.08538","evidence_quote":"Provides the ParseBench scores used for visual grounding and chart perception in the capability profiles."},{"cited_title":"LiveBench: A Challenging, Contamination-Limited LLM Benchmark","cited_arxiv_id":"2406.19314","evidence_quote":"Provides the LiveBench scores used for coding and instruction-following capability groups."},{"cited_title":"Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces","cited_arxiv_id":"2601.11868","evidence_quote":"Provides the Terminal-Bench 4.0 scores used for harness and coding capability groups."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}