Robotics · arXiv:2610.26848 v1 · submitted 19 Oct 2026 · 58 references
Where is Physical RSI capability from? An Observational Scaling Law of LLM-as-policy
The paper says
“In this paper, we systematically investigate the mechanisms, scaling behavior, and generalization of physical Recursive self-improvement. We introduce PhysRSI-eval to quantify physical RSI through a five-axis metric: First-Attempt Performance, Efficiency, Success, Budget and Improvement from failure.”
more of the abstract
Different VLMs are evaluated on 18 manipulation tasks under a fixed interaction budget. The analysis of experimental results motivates the Physical Barrel Hypothesis: different capabilities may constrain different parts of improvement. The five-axis metrics also distinguish systems with close public benchmark averages and show that high scores need not come with early or efficient gains. We study observational scaling from two perspectives: predicting the relative performance of VLM agents from their public capabilities, and forecasting later progress from early physical feedback. Together, these analyses ask how quickly and how far an agent can improve. Moreover, we conduct workspace interventions and then analyze what improvement leaves behind. Removing history reduces late-round success, frozen workspaces remain useful on unseen layouts, and three cross-task cases achieve success earlier. The results suggest that scene-specific adaptation and reusable experience can coexist.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 20 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. The five-axis PhysRSI-eval protocol is a real measurement contribution; the capability-bottleneck and scaling-law claims are exploratory correlations on six systems, which the authors mostly concede, so the benchmark deserves peer review but the 'law' framing should be softened. This line is wrong
4 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 58 references say.
The paper says, at Section 4.1 and Appendix D
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.61):
“These associations suggest candi- date bottlenecks, not causal requirements.”
Pith says: Objection 1 of 4 · major
The axis-specific capability attribution is not identified at n=6. The Spearman associations quoted (Spatial-A1 rho=0.94, Spatial-A2/A4 rho=0.83, Perception-A3/A5 rho=0.94) are computed over six systems whose four capability-group profiles are likely strongly rank-correlated with one another, and the smallest Benjamini-Hochberg adjusted q across the exact permutation tests (Appendix D) is 0.11, so no individual association is conventionally significant and the 0.94-versus-0.83 gap is within the sampling resolution of a rank correlation at this sample size.
more of objection 1
The paper does not report the 4x4 cross-correlation matrix of the group profiles, nor any control against a single global-capability index (e.g., the mean of all 17 benchmark ranks); because Astra, the strongest system overall, leads all five axes, the 'Spatial for A1, Perception for A3/A5' labels may reflect overall model strength rather than distinct bottlenecks. The Section 7 caveats and the 'candidate bottlenecks' framing in Section 4.2 are appropriate and are part of why this is not a soundness defect, but Section 4.1 itself should state that the axis-group differences are not statistically resolvable at this sample size, and should add the global-index control and the group-profile cross-correlation matrix. The five-axis divergence in Table 1 is direct measurement and is not affected by this concern.
The paper says, at Section 3.4, Eq. (1)
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
The definition of A2 (Efficiency) uses K0 as its reference, so the k=1 term (q1-q0)/(qH-q0) credits each system's full K0-to-K1 jump in its Efficiency score. Section 3.2 acknowledges that this jump 'mixes interface changes with improvement' and that the Harness rounds are the self-improvement segment; using q0 in Eq.
more of objection 2
(1) is in tension with that statement. Since q1 is not reported (Table 4 lists q0, q17, qmax, tau90, C1, C17, and recovery counts but not q1), the reader cannot assess how much of the headline divergence 'Opus has a higher Success score than GPT-6.1 Sol (0.744 versus 0.703), but lower Efficiency (0.609 versus 0.822)' (Section 5.1) is driven by differential one-shot interface benefits rather than refinement timing. Please either compute A2 from q1 forward, or report q1 (or an A2 variant with q1 as reference) so that the interface contribution is bounded.
The paper says, at Section 5.2 and Figure 7
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
The cross-model 'prediction' claim in Section 5.2 mixes an in-sample ranking with a leave-pair-out prediction. Ranking systems by the equal-weight average of the four capability profiles is computed from normalized ranks defined over all six systems, so reproducing the observed order of best score by K17 (q-bar-17) is a descriptive co-ranking, not a prediction for unseen systems; only the T50 analysis is actually leave-pair-out.
more of objection 3
The Figure 7 caption ('Each pair of systems is predicted from the other four') and the 'predicted/observed' labels on the best-by-K17 column conflate the two procedures. Please state the in-sample nature of the best-score ranking explicitly (or re-normalize profiles under leave-one-out for the ranking claim), and report whether the kernel-regression hyperparameters (gamma=0.0625, alpha=10, Appendix D) were fixed before or independently of the outcomes.
The paper says, at Section 5.3 and Appendix D
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
The Weibull transfer forecast rests on a single source-target pair. The template gA(t) = 1 - exp(-(t/tauA)^pA) fitted to Astra's K1-K17 and then calibrated on GPT-6.1 Sol's first five rounds achieves MAE 0.019 on the held-out rounds versus 0.112 for holding the last observed score, but no baseline isolates the contribution of the transferred template shape: a Weibull fitted to …
more of objection 4
… Sol's own first five rounds without the Astra prior, or a template taken from a less related system, would test the paper's transfer assumption ('A related model may share the shape of its improvement curve'). With one target system, the general claim that a reference trajectory can predict a related model's later performance is a demonstration, not an established result, and Section 5.3 should say so explicitly; adding at least one no-template or cross-template baseline would materially strengthen the analysis.
6 minor objections
The paper says, at Figure 4
Pith says: minor objection 1 of 6
The caption says the matrix 'reports exploratory rank associations,' but the displayed numeric values (0.93, 0.78, 0.68, 0.44 for Astra, and so on) match the per-system capability profiles in Figure 7 rather than the Spearman rho values quoted in Section 4.1; please clarify what the matrix entries are, or plot the actual correlation matrix.
The paper says, at Table 3
Pith says: minor objection 2 of 6
The rows for 'Reasoning budget per round' and 'Revision window' contain stray superscript markers that obscure which values are session-log estimates, and the row 'Selected K1-K17 entries' (91, 143, 255, 306, 175, 175) should state how a 'selected entry' is defined.
show 4 more
The paper says, at Section 3.2
Pith says: minor objection 3 of 6
The sentence 'Nine GPT-5.6 Sol Harness entries summarize two observed runs' needs a specification of how two runs were collapsed into the nine entries used in the aggregates.
The paper says, at Section 5.1 and Figure 6
Pith says: minor objection 4 of 6
The term 'rapid contraction region' is used as an explanatory device but is never defined; please define it or replace it with a direct description of the gap between the public-score and physical-profile orderings.
The paper says, at Section 3.4 and Section 5.1
Pith says: minor objection 5 of 6
The matched-set check for the A4 comparison ('the gap persists on the ten tasks both systems solve') is good practice; please extend the same check to the A5 comparison, whose conditional task sets also differ across systems.
The paper says, at Title and abstract
Pith says: minor objection 6 of 6
The phrase 'Observational Scaling Law' is stronger than the evidence base presented (six systems for the cross-model analysis, one source-target pair for the forecast); the body's 'exploratory' framing (Sections 5.2 and 7) is more accurate, and the title and abstract should be aligned with it.
What would settle it
Stability check for Section 4.1: construct G_i, the mean normalized rank over all 17 benchmarks for each system. (1) Leave-one-system-out: rebuild the four group profiles from the remaining five systems and recompute all axis-group Spearman rhos; record whether Spatial remains the top group for A1/A2/A4 and Perception for A3/A5 in all six deletions. (2) Compute rho(A_j, G) for each axis and report it alongside the named group rhos. If any deletion changes the top group for A1, A3, or A5, or if rho(A_j, G) equals or exceeds the named group rho for any axis, the axis-specific attribution is within noise of a general-strength effect and Section 4.2 should be presented as an untested conjecture rather than a motivated hypothesis. If the named groups win all six deletions and consistently beat G, the barrel attribution survives this six-system test.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 20 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Short version: the five-axis PhysRSI-eval protocol is a genuine measurement contribution; the capability attribution and "scaling law" parts are exploratory correlations on six systems, which the authors mostly concede, so the benchmark deserves peer review but the law framing should be softened.
What is actually new is the fixed-budget protocol: persistent workspace, judged physical attempts, K0 separated from K1–K17 refinement, and five axes with definitions precise enough to reimplement. That setup cleanly separates initial competence, gain timing, attained level, budget savings, and recovery from failure. The descriptive findings—Opus scores higher than GPT-6.1 Sol on Success but lower on Efficiency and Budget; systems with close public benchmark averages separate on the axes; removing history cuts Astra's late-round success from 41.7% to 8.3%; frozen workspaces succeed on unseen layouts—are direct measurements, and I see no reason to doubt them. The task strata were fixed before evaluation, the metric algebra is explicit in Section 3.4 and Appendix B, and Section 7 states the real scope limits: one chain per task, first-success stopping, conditional task sets, and a capability analysis without independent validation.
The soft spots are the interpretive layer. The capability associations are Spearman correlations on six systems, between hand-grouped public benchmark ranks and the axes. At n=6, the 0.94 versus 0.83 gap is well inside sampling noise, the smallest adjusted q is 0.11, and there is no control for overall model strength. The authors call these candidate bottlenecks, so this is not a case of hiding the weakness, but the Physical Barrel Hypothesis is an interpretation, not a supported result. The same goes for the two scaling analyses: leave-pair-out on six points with hand-fixed kernel hyperparameters is a worked example, and the Weibull forecast is one reference-target pair with no stated selection rule. The five axes are point estimates with no chain-level uncertainty, and the article itself releases no code, data, or prompts, which limits independent checking. The stress-test note is right about the attribution being unidentifiable at this n, though the direct five-axis divergence does not depend on it.
Bottom line: this is a solid measurement paper with a speculative overlay. It should go to a serious referee; the right response is to keep the benchmark, add uncertainty quantification and data release, and present the capability claims as hypotheses. People building LLM-as-policy evaluations will get value from the protocol; people looking for a scaling law should look elsewhere.
The strongest objection
The paper's Q1 claim is that different axes of physical RSI have different capability bottlenecks: Spatial reasoning is associated with A1/A2/A4 and Perception with A3/A5 (Section 4.1), motivating the Physical Barrel Hypothesis (Section 4.2). That attribution rests entirely on Spearman correlations over six systems between hand-grouped public-benchmark ranks and the five axes. At n=6 the gap between rho=0.94 (Spatial-A1) and the next-best rho=0.83 is small relative to the sampling resolution of a rank correlation, and Appendix D reports that the smallest Benjamini-Hochberg adjusted q from the exact permutation tests is 0.11, so no individual axis-group association is conventionally significant. The paper does not report the 4x4 cross-correlation matrix of the four group profiles, nor does it compare axis-group rhos with a single global-capability index such as the mean of all 17 benchmark ranks. Because the six systems are frontier models whose public profiles are likely correlated, the 'strongest association' labels may simply reflect overall model strength: Astra is best overall and also best in Spatial. The paper explicitly frames these bottlenecks as candidate hypotheses (Section 7, b265-b266), so this is not a soundness defect, but it is the load-bearing premise of Q1. The five-axis divergence itself (e.g., Opus A3=0.744 vs GPT-6.1 Sol A3=0.703 while A2=0.609 vs 0.822, Section 5.1) is a direct measurement and is not threatened by this concern.
The reader's weakest_assumption identifies the same load-bearing point: the capability-association analysis presumes that the 17 public benchmark scores, grouped into four profiles, are valid and separable proxies for the capabilities used during physical refinement, and the six-system correlations cannot separate axis-specific capabilities from overall model strength. I agree, and I did not find a separate internal inconsistency in the axis definitions or in the workspace-intervention results. The five-axis profiles are clearly specified in Section 3.4, the Opus-vs-GPT-6.1 divergence in Table 1 is a direct measurement, and the ablation and frozen-workspace results are explicitly scoped to Astra and six tasks in Section 7. The capability-bottleneck attribution is the one claim that would be decisive for Q1, and it is not identifiable from six points after multiple-testing control. The paper's own honesty in Section 7 does not dissolve the problem: saying that capability bottlenecks remain hypotheses to test is correct, but Section 4.2 currently presents the barrel hypothesis as motivated by the correlation matrix, and Section 5.2's cross-model ranking reuses the same profiles, so the weak statistical base propagates into the scaling-law pillar. The proposed leave-one-out plus global-index check is cheap and can be run on the existing data; a seventh independently evaluated system, such as Gemini 4 on the full 18-task protocol, would be a stronger external check. The verdict stays CONDITIONAL: the measurement contribution is real, but Q1's capability attribution needs either the stability evidence or an explicit downgrade to pure description.
The referee's report
If the descriptive core holds, the paper makes a genuine methodological contribution: it argues credibly that endpoint success is insufficient for evaluating LLM-as-policy agents, that gain timing and physical cost are separable outcomes, and that public benchmark averages can misorder systems on these outcomes. The strengths are real: the metric definitions in Section 3.4 are precise about edge cases (undefined denominators, conditional task sets, no-interpolation tau90); task strata were fixed before evaluation; the ablation design controls individual information channels; the forecast is genuinely predictive (calibrated on K1-K5, evaluated on K6-K17) with paired-bootstrap bands; and Section 7 itemizes limitations candidly, including one chain per task, first-success stopping, and conditional A4/A5 sets. The weakest premise is the capability-bottleneck attribution: at n=6, with a smallest Benjamini-Hochberg adjusted q of 0.11 (Appendix D) and no global-strength control, the specific axis-capability labels are not statistically resolvable, so the Physical Barrel Hypothesis is presently an illustration rather than a supported finding.
Pith's reading of the paper (a machine's summary, not the authors')
This paper tries to establish that a robot agent's capacity to improve through physical trial and error is not a single ability. It builds a fixed-budget evaluation loop, PhysRSI-eval, in which six vision-language model systems act as robot policies by writing code and revising a persistent workspace across 18 dual-arm manipulation tasks, and it scores them on five axes: First-Attempt Performance, Efficiency, Success, Budget, and Improvement from failure. Its central claim, the Physical Barrel Hypothesis, is that different capabilities limit different parts of improvement: spatial reasoning shows the strongest association with starting well, while perception shows the strongest association with final score and recovery from failure. The paper further claims that public benchmark averages hide these differences, that public capability scores predict the relative order of physical improvement and its timing, and that retained workspace experience both helps and transfers to new layouts and tasks. If this is right, judging an LLM-as-policy agent by its final score alone is not just incomplete but can rank systems in a misleading order.
What carries the argument. The load-bearing mechanism is the measurement loop plus the five-axis profile it feeds. At K0 the agent gets one in-context first attempt with no workspace; from K1 to K17 each round executes one graded physical attempt, then the agent revises a persistent workspace of skills, tools, policy code and notes with robot actions disabled, and commits it for the next round. The five axes are defined on the resulting task-mean curves: A1 is the mean K0 score, A2 measures how early the terminal gain is realized, A3 is the mean score over the high window after reaching 90% of peak gain, A4 is the fraction of physical budget saved on solved tasks, and A5 is the fraction of K1 failures later solved. This design is what lets the paper separate starting well, improving early, finishing high, spending little, and overcoming failure, and then associate each axis with a different capability group.
Core claim. On its own terms, the paper's discovery is that physical recursive self-improvement is multidimensional. Across the 18-task suite, the systems' equal-weight five-axis mean spans 13.9–73.4 while their mean over 17 public benchmarks spans only 54.1–78.1, and a higher final score need not come with earlier or cheaper gains: Opus 5.5 reaches a higher Success score than GPT-6.1 Sol (0.744 versus 0.703) while having lower Efficiency (0.609 versus 0.822). The capability association analysis finds that spatial reasoning has the strongest observed association with First-Attempt Performance ($\rho = 0.94$) and with Efficiency and Budget ($\rho = 0.83$), while perception has the strongest association with Success and Improvement from failure ($\rho = 0.94$). These associations are presented as candidate bottlenecks for the Physical Barrel Hypothesis, not as causal proof. On the generalization side, the paper reports that clearing history drops Astra's late-round success from 41.7% to 8.3%, frozen workspaces succeed in 25 of 60 unseen-layout trials, and three cross-task cases reach first success earlier when initialized from another task's workspace, so scene-specific adaptation and reusable experience appear to coexist.
The premise it rests on. The load-bearing premise is that the 17 public benchmark scores, averaged into Spatial, Harness, Perception and Coding profiles on 15 October 2026, measure the same capabilities the agents actually use during physical trial-and-error revision; if those scores are not valid proxies, the barrel attributions and the rank predictions built on them lose their footing.
If the paper is right (Pith's inferences, not the authors')
Final-score rankings can mislead: a system that reaches a high score late and spends most of its physical budget can rank above a system that improves earlier and cheaper, so evaluation reports should include gain timing and budget use.
Public capability benchmarks carry predictive signal: ranking systems by averaged public capability ranks recovered the observed order of best score reached by K17, and all pairwise orderings of half-gain round T50 were recovered from the other systems in the set.
Early physical feedback can forecast later progress: using Astra's improvement shape and the first five rounds of GPT-6.1 Sol predicted its hidden rounds with mean absolute error 0.019, versus 0.112 from holding the last observed score.
Retained experience is doing real work: removing history cut Astra's late-round success from 41.7% to 8.3%, and the no-grader condition dropped it to 13.9%.
Some improvements are not overfitting to one scene: frozen workspace files solved 25 of 60 held-out-layout trials, and three cross-task transfers reached first success earlier.
Direct capability interventions would settle the barrel attribution: degrading spatial inputs should selectively lower first-attempt performance, and degrading perception should selectively lower recovery, if the hypothesis names the true bottleneck.
A practical screening rule follows from the forecasting result: a few early physical rounds on a reference system could be used to rank candidate models before committing a full interaction budget, turning the observational law into a cheap selection procedure.
Because the transfer evidence points to two carriers, executable tools and textual operating knowledge, future benchmarks should log workspace lineage and tool reuse as first-class measurements, since a system that reuses broadly may transfer better than one whose gains are concentrated in a few tools.
The five-axis separation suggests that error analysis for embodied agents should distinguish cannot-start failures from cannot-recover failures, since different capability fixes may apply to each.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
In this paper, we systematically investigate the mechanisms, scaling behavior, and generalization of physical Recursive self-improvement. We introduce PhysRSI-eval to quantify physical RSI through a five-axis metric: First-Attempt Performance, Efficiency, Success, Budget and Improvement from failure. Different VLMs are evaluated on 18 manipulation tasks under a fixed interaction budget. The analysis of experimental results motivates the Physical Barrel Hypothesis: different capabilities may constrain different parts of improvement. The five-axis metrics also distinguish systems with close public benchmark averages and show that high scores need not come with early or efficient gains. We study observational scaling from two perspectives: predicting the relative performance of VLM agents from their public capabilities, and forecasting later progress from early physical feedback. Together, these analyses ask how quickly and how far an agent can improve. Moreover, we conduct workspace interventions and then analyze what improvement leaves behind. Removing history reduces late-round success, frozen workspaces remain useful on unseen layouts, and three cross-task cases achieve success earlier. The results suggest that scene-specific adaptation and reusable experience can coexist.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-20
The authors' main claim, as the classifier picked it out 0.77
“The results suggest that scene-specific adaptation and reusable experience can coexist.”
How the main claim is stated: hedged possibility 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.21 · human participants 0.04 · controlled comparison 0.42 · randomized assignment 0.02 · states quantitative result 0.56 · compares with prior results 0.51 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| In this paper, we systematically investigate the mechanisms, scaling behavior, and generalization of physical Recursive self-improvement. | contribution 0.97 | 0.76 | 0.05 | asserted 0.65 |
| We introduce PhysRSI-eval to quantify physical RSI through a five-axis metric: First-Attempt Performance, Efficiency, Success, Budget and Improvement from failure. | method 0.61 | 0.96 | 0.05 | asserted 1.00 |
| Different VLMs are evaluated on 18 manipulation tasks under a fixed interaction budget. | method 0.98 | 0.93 | 0.03 | asserted 0.98 |
| The analysis of experimental results motivates the Physical Barrel Hypothesis: different capabilities may constrain different parts of improvement. | interpretation 0.96 | 0.83 | 0.32 | hedged possibility 0.93 |
| The five-axis metrics also distinguish systems with close public benchmark averages and show that high scores need not come with early or efficient gains. | result 0.89 | 0.93 | 0.13 | asserted 1.00 |
| We study observational scaling from two perspectives: predicting the relative performance of VLM agents from their public capabilities, and forecasting later progress from early physical feedback. | contribution 0.81 | 0.89 | 0.06 | asserted 0.91 |
| Together, these analyses ask how quickly and how far an agent can improve. | interpretation 0.52 | 0.25 | 0.08 | unclear 0.63 |
| Moreover, we conduct workspace interventions and then analyze what improvement leaves behind. | method 0.51 | 0.94 | 0.05 | asserted 0.93 |
| Removing history reduces late-round success, frozen workspaces remain useful on unseen layouts, and three cross-task cases achieve success earlier. | result 1.00 | 0.94 | 0.11 | asserted 1.00 |
| The results suggest that scene-specific adaptation and reusable experience can coexist. | interpretation 1.00 | 0.89 | 0.32 | hedged possibility 1.00 |
What kind of paper the classifier reports this is: empirical study 0.98 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,357 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (10)
Showing 1 of the paper's 10 figures. Figures are the authors', reproduced from the paper.
References (58)
- [1] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patward- han, Lilian Weng, and Aleksander Mądry. MLE- bench: Evaluating machine learning agents on ma- chine learning engineering, 2024. https://ar... on PithPith's reviewarXiv
- [2] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endowing vision- language models with spatial reasoning capabilities, 2024.https://arxiv.org/abs/2401.12168 on PithPith's reviewarXiv
- [3] Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu... on PithPith's reviewarXiv
- [4] Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, and Qinhuai Na. AI4AI-Bench: Benchmarking LLM agents in algorithmic design for recursive self-improvement, 2026. https://arxiv. org/abs/2608.20318 on PithPith's reviewarXiv
- [5] Zijie Diao, Yitong Chen, Sicheng Xie, Tianyi Lu, Wujian Peng, Guojin Zhong, Houze Xu, Ziyi Ye, Zuxuan Wu, and Yu-Gang Jiang. LIBERO-Agent: Evaluating general-purpose agents for direct embod- ied manipulation, 2026. https://arxiv.org/abs/ 2609.39507 on PithPith's reviewarXiv
- [6] Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. EmbSpatial-Bench: Bench- marking spatial understanding for embodied tasks with large vision-language models, 2024. https: //arxiv.org/abs/2406.05756 on PithPith's reviewarXiv
- [7] Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Dantong Niu, Fei-Fei Li, Guanya Shi, Ji- ajun Wu, Shankar Sastry, Yuke Zhu, Ken Gold- berg, and Linxi “Jim” Fan. CaP-X: A framework for benchmarking and improving coding agents f... on Pith
- [8] XingyuFu, YushiHu, BangzhengLi, YuFeng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei- Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive, 2024. https://arxiv.org/abs/2404.12390 on PithPith's reviewarXiv
- [9] Galbot Team, Xuchuan Chen, Xiaoqian Cheng, Yu Deng, Lihe Ding, Shaocong Dong, Xiangjun Gao, Haozhe Jia, Zekai Li, Zhoujian Li, Yunrui Lian, Sikai Liang, Chenghuai Lin, Dairu Liu, Jia- hang Liu, Qingtao Liu, Yuxuan Ma, Zekun Qi, Jiayi Su, He Wang, Ruochen Xu, Tianyu Xu, Xudong ... on PithPith's reviewarXiv
- [10] Xingze Gao, Chuanrui Hu, Hongda Chen, Pengfei Yao, Zhao Wang, Yi Bai, Zhengwei Wu, Yunyun Han, Xiaofeng Cong, Jie Gui, Yafeng Deng, and Teng Li. EvoAgentBench: Benchmarking agent self-evolution via ability transfer, 2026.https://arxiv.org/abs/ 2607.05202 on PithPith's reviewarXiv
- [11] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, Steven Bohez, Konstantinos Bousmalis, An- thony Brohan, Thomas Buschmann, Arunkumar... on PithPith's reviewarXiv
- [12] Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Jo- hannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driess- che, Bogdan Damoc, Aurelia Guy, Si... on PithPith's reviewarXiv
- [13] Lisheng Huang, Chen Yang, Hao Zhou, Hua- tong Song, Zongchao Chen, Ran Le, Yang Song, Wayne Xin Zhao, and Tao Zhang. Evo-Bench: Can language models improve agent harness?, 2026. https://arxiv.org/abs/2608.09096 on PithPith's reviewarXiv
- [14] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied reasoning th... on PithPith's reviewarXiv
- [15] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Live- CodeBench: Holistic and contamination free eval- uation of large language models for code, 2024. https://arxiv.org/abs/2403.07974 on PithPith's reviewarXiv
- [16] Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. OmniSpatial: Towards comprehensive spatial rea- soning benchmark for vision language models, 2025. https://arxiv.org/abs/2506.03135 on PitharXiv
- [17] Sihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Tengfei Wang, Shisong Chen, Jing- hao Zhang, Tianjun Pan, Weijia Li, Jiaqing Liang, and Yanghua Xiao. SEA-Eval: A benchmark for eval- uating self-evolving agents beyond episodic assess- ment, 2026.https://arxiv.org/a... on PithPith's reviewarXiv
- [18] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?, 2023.https:// arxiv.org/abs/2310.06770 on PithPith's reviewarXiv
- [19] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scalinglawsforneurallanguagemodels, 2020. https: //arxiv.org/abs/2001.08361 on PithPith's reviewarXiv
- [20] Zixuan Ke, Vaidehi Patil, Haizhou Shi, Yang Li, Ye Liu, Sarath Shekkizhar, Anurag Koul, Jiayu Wang, Xuan Phi Nguyen, Semih Yavuz, Mohit Bansal, and Shafiq Joty. EvoHarnessBench: Can your agents keep pace with an evolving harness?, 2026.https://arxiv.org/abs/2609.04280 on PithPith's reviewarXiv
show 38 more
- [21] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking LLMs for embodied decision making, 2024.https... on PithPith's reviewarXiv
- [22] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model pro- grams for embodied control, 2022.https://arxiv. org/abs/2209.07753 on PithPith's reviewarXiv
- [23] Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, Dakuo Wang, Xiang Zhang, Benoit Dumoulin, Cihang Xie, Yuyin Zhou, Suhang Wang, and Hanqing Lu. Harness updat- ing is not harness benefit: Disentangling evolu- ti... on PithPith's reviewarXiv
- [24] Yiheng Lyu, Xueying Jiang, Wenhao Li, Shijian Lu, and Gongjie Zhang. CodeActionBench: Evaluating agentic code-as-policy for embodied manipulation, 2026.https://arxiv.org/abs/2609.33807 on PithPith's reviewarXiv
- [25] Haitong Ma, Chenxiao Gao, Rushi Qiang, Bo Dai, andNaLi. RLE-Bench: Aqualifyingexamforcoding agents as robot learning engineers, 2026. https: //arxiv.org/abs/2609.34210 on PithPith's reviewarXiv
- [26] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eu- reka: Human-level reward design via coding large language models, 2023. https://arxiv.org/abs/ 2310.12931 on PithPith's reviewarXiv
- [27] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Ma- jumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iter- ... on PithPith's reviewarXiv
- [28] Mike A. Merrill, Alexander G. Shaw, Nicholas Car- lini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhu- rina, Jenia Jitsev, Di Lu, Orfeas Menis Ma... on PithPith's reviewarXiv
- [29] Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Ameli... on Pith
- [30] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, T... on PithPith's reviewarXiv
- [31] Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, Jiazhao Zhang, Jiawei He, Jiayuan Gu, Xin Jin, Kaisheng Ma, Zhizheng Zhang, He Wang, and Li Yi. SoFar: Language-grounded ori- entationbridgesspatialreasoninga... on PitharXiv
- [32] Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko. PostTrainBench: Can LLM agents automate LLM post-training?, 2026. https://arxiv.org/abs/2603.08640 on PitharXiv
- [33] Roboflow. Roboflow Vision Evals, 2026.https:// playground.roboflow.com/evals. Accessed: 2026- 10-17 on Pith
- [34] Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the pre- dictability of language model performance, 2024. https://arxiv.org/abs/2405.10938 on PithPith's reviewarXiv
- [35] Zeyu Shen, Haoxiang You, Yilang Liu, Zhicheng Zheng, Lihan Zha, Kashu Yamazaki, Mingtong Zhang, Suning Huang, Jiankai Sun, Qianzhong Chen, Lucy He, Kaiyuan Liu, Haoran Chang, Katerina Fragkiadaki, Dhruv Shah, Mac Schwager, Peter Hen- derson, Ian Abraham, and Canwen Xu. Embod- ... on Pith
- [36] Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, Jiaheng Liu, Xinping Lei, Xinrong Zhang, Siyuan Fang, Zhewen Tan, Tianle Cai, Jun- hao Fang, Jiameng Huang, Yueyang Wang, Jinkai Liu, Yuxuan Zhang, Jian Yang, Zhoujun Li, Shen Yan, Wenhao Huang, and Ge Zhang. S3Gym... on PithPith's reviewarXiv
- [37] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal rein- forcement learning, 2023. https://arxiv.org/abs/ 2303.11366 on PithPith's reviewarXiv
- [38] Harman Singh, Anton Bakhtin, Rulin Shao, Gabriel Synnaeve, Ilia Kulikov, Rob Fergus, Sanjeev Arora, Kurt Keutzer, Jason Weston, Anuj Mahajan, and Anirudh Goyal. Agent plasticity: Measuring self- improvement through experience, 2026. https:// arxiv.org/abs/2610.08902 on PithPith's reviewarXiv
- [39] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating situated robot task plans using large language models, 2022. https://arxiv.org/abs/ 2209.11302 on PithPith's reviewarXiv
- [40] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Pat- wardhan. PaperBench: Evaluating AI’s ability to replicate AI research, 2025. https:... on PithPith's reviewarXiv
- [41] Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, and Zhibo Chen. VLA-JEPA: Enhancing vision- language-action model with latent world model, 2026. https://arxiv.org/abs/2602.10098 on PitharXiv
- [42] Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanx... on PithPith's reviewarXiv
- [43] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. https://arxiv.org/abs/2305.16291 on PithPith's reviewarXiv
- [44] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh- Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldbl... on PithPith's reviewarXiv
- [45] Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Holden Karnofsky, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, Will... on PithPith's reviewarXiv
- [46] Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, and Wenxuan Zhang. HarnessDev: Can LLMs create and e... on PithPith's reviewarXiv
- [47] Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Yuxuan Zhang, Xinping Lei, Junting Zhou, Zexuan Wang, Yuchen Wu, Huan Zhou, Duo Wang, Yinzhu Piao, Yongchang Peng, Yunfeng Shi, Jin Chen, Zuo Wang, Jinkai Liu, Jiaheng Liu, Wenxuan Zhang, Shen Yan, Wenhao Huang, and Ge Zhang. Aspire: Can m... on PithPith's reviewarXiv
- [48] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xi- aochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OS- World: Benchmarking multimodal agents fo... on PithPith's reviewarXiv
- [49] Shuhan Xue, Zixin Ding, Yichen Shen, Yinjie Wang, Zhenfei Yin, Yingcheng Wu, Yuxin Chen, Mengdi Wang, and Ling Yang. PAST-Bench: Benchmarking the foundations of recursive self-improvement in per- sonal agents, 2026. https://arxiv.org/abs/2608. 04003 on Pith
- [50] Jihan Yang, Shusheng Yang, Anjali W. Gupta, Ri- lyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How multimodal large language mod- els see, remember, and recall spaces, 2024.https: //arxiv.org/abs/2412.14171 on PithPith's reviewarXiv
- [51] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. Em- bodiedBench: Comprehensive benchmarking multi- modal large language models for vision-driven em... on Pith
- [52] Zhiqin Yang, Chenxin Li, Xiaomeng Hu, Yibin Liu, Weidong Huang, Jiankai Sun, Haitao Li, Zijian Wu, Yuzhi Huang, Fanding Huang, Hanwen Sun, Jiashun Liu, Jingqi Tong, Mingxin Huang, Shaoli Hu, Shi- jue Huang, Tianyi Bai, Xinyuan Wang, Yunlong Lin, Zhengyang Tang, Zhexin Zhang, Z... on PithPith's reviewarXiv
- [53] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024.https://arxiv.org/abs/2406.12045 on PithPith's reviewarXiv
- [54] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A m... on PithPith's reviewarXiv
- [55] Boyang Zhang, Sebastián G. Acosta, Preston Carl- son, Sacha Bron, Pierre-Loïc Doulcet, Daniel B. Os- pina, and Simon Suo. ParseBench: A document parsing benchmark for AI agents, 2026. https: //arxiv.org/abs/2604.08538 on PithPith's reviewarXiv
- [56] WenyaoZhang, HongsiLiu, ZekunQi, YunnanWang, Xinqiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, Fan Lu, He Wang, Zhizheng Zhang, Li Yi, Wenjun Zeng, and Xin Jin. DreamVLA: A vision- language-action model dreamed with comprehensive world knowledge, 2025. https://arxiv.org/abs/... on PithPith's reviewarXiv
- [57] Zhongbo Zhang, Jiayi Jin, Yifan Wang, Zaibin Zhang, Haiwen Diao, Lijun Wang, and Huchuan Lu. VABench: Measuring embodied spatial intelligence through visual demonstrations, active perception, and metric control, 2026.https://arxiv.org/abs/ 2609.19554 on PithPith's reviewarXiv
- [58] Congjie Zheng, Chuanyi Xue, Bin Liang, Jun Yang, and Changshui Zhang. SEAGym: An evaluation environment for self-evolving LLM agents, 2026. https://arxiv.org/abs/2606.17546. Appendix A Task and Protocol Details The 18 tasks keep their native grading and task- specific step lim... on PithPith's reviewarXiv
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Toward Recursive Self-Improvement in Physical AI cs.RO · Oct 2026
- ScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters cs.RO · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment