Pith.

Artificial Intelligence · arXiv:2608.14441 v1 · submitted 14 Aug 2026 · 54 references

PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

Yuhao Zhan, Bingxiang He, Zecong Tang, Chaojun Xiao

The paper says

“Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions and do not test recovery after those conditions change.”

more of the abstract

To address this gap, we introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark of 144 source-to-target adaptation pairs across six physics domains. Each pair links a source environment to a mutated target environment with the same goal and interface. A code-driven design that succeeds in the source fails in the target, where agents must iteratively adapt it into a working target design using diagnostic sandbox feedback within a limited attempt budget. We compare ten self-evolving methods from four paradigms. The benchmark remains far from saturated: Reflexion + Qwen3-14B succeeds on only 35.9% of full-benchmark pairs, while GPT-5.5 solves 66.7% of the Statics subset under the full budget. Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging. Even revealing exact physical changes does not raise the performance ceiling, pointing to mechanism redesign rather than parameter inference as the central bottleneck. Data and code are available at https://github.com/thunlp/PACE-Bench.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 27 Aug 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Solid, carefully-built benchmark for an underexplored capability; the difficulty-calibration loop weakens the headline saturation and 'know-how' claims but doesn't sink the resource. This line is wrong

3 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 54 references say.

The paper says, at §3.3, Appendix A.2–A.3

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The difficulty calibration loop in Phase 3 uses Qwen3-4B as the failure oracle: mutation parameters are escalated until Qwen3-4B fails two independent runs, and Appendix A.3 designs feedback.py diagnostics from Qwen3-4B failure logs. The full-benchmark evaluation then uses Qwen3-4B/8B/14B as the primary models (Table 2).

more of objection 1

This creates a selection-bias threat: tasks may be selected for being hard for the Qwen family specifically (e.g., due to code-generation style or prompt adherence), and the diagnostics may be overfit to Qwen failure modes, rather than for requiring intrinsic physics adaptation. The reported mitigations (from-scratch source 32.3% vs. mutated 6.4–13.4% in Table 16; GPT-5.5 66.7% on Statics in Table 3) show the tasks are not trivially easy, but they do not establish that Qwen3-4B's failure is a model-neutral difficulty measure. I recommend recalibrating a held-out subset with a non-Qwen oracle (e.g., GPT-5.5 or DeepSeek-V4-Pro) and reporting per-task difficulty correlations across model families, or explicitly framing the headline unsaturation result as conditional on Qwen-family calibration.

The paper says, at §4.2.3, Table 6

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The Change-Exposed (CE) versus Change-Hidden (CH) comparison is underpowered for the strong conclusion drawn. Table 6 is pooled over approximately 48 environment pairs, so a difference of ±5 percentage points corresponds to about 1–2 pairs; most of the 8B/14B deltas are within sampling noise, and Vanilla-14B's −7.2 point drop is about 3 pairs.

more of objection 2

Furthermore, the statement that the best CE result (Reflexion-14B, 14.6%) remains below the best CH result (Reflexion-8B, 17.9%) confounds the disclosure intervention with model scale. The conclusion that 'revealing exact physical changes does not raise the performance ceiling' needs per-pair paired analysis, confidence intervals, and ideally a larger sample; otherwise it should be reported as preliminary.

The paper says, at Appendix B, §4.2.2

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The trajectory-level error categories rest on thresholds that are presented without validation or sensitivity analysis: Design Fixation requires average code similarity >0.85 over the last 3 iterations, score trend <3, and best score <50; Late Convergence requires a >2 point second-half improvement; Stagnation and Exploration are defined by unique failure-type counts.

more of objection 3

Since the paper's central qualitative claims about memory anchoring, tree-search non-convergence, and the fixation/exploration split are all derived from this taxonomy (Table 5, Figure 4, Section 4.2.2), the thresholds should be checked for robustness (e.g., vary 0.85 and the trend/threshold values and report how category shares and method-level conclusions change). Without this, the RQ2 failure analysis is not yet a reliable quantitative contribution.

8 minor objections

The paper says, at Title, page 1

Pith says: minor objection 1 of 8

The title and running header contain missing spaces ('PhysicsAdaptation', 'BenchmarkingP hysicsA daptation'); please fix throughout the manuscript.

The paper says, at §3.1, Eq. (3)

Pith says: minor objection 2 of 8

Equation (3) uses 'task progress' without defining it; give the formula or refer to the evaluator module where it is computed.

show 6 more

The paper says, at §4.1.2, Table 4

Pith says: minor objection 3 of 8

Table 4 states that scores are fitted to the nearest 1/24 increment; the fitting procedure is not described in the main text and should be explained or moved to the appendix.

The paper says, at Table 6

Pith says: minor objection 4 of 8

Table 6 says 'approximately 48 environment pairs'; report the exact denominators for each cell, as done for Table 2 in Appendix D.3.

The paper says, at Figure 4

Pith says: minor objection 5 of 8

Figure 4 uses category labels S, K, D, F, C, E that are not defined in the caption; add a legend or caption text.

The paper says, at Appendix D.3

Pith says: minor objection 6 of 8

Appendix D.3 reports an average denominator of 139.20 across Table 2; including per-cell denominators in the main table would improve transparency.

The paper says, at References

Pith says: minor objection 7 of 8

The reference list contains typographical issues such as 'Anthropic. 2 025' and inconsistent spacing in model names (e.g., 'MiniMax-M2.7'); please proofread the references.

The paper says, at Limitations

Pith says: minor objection 8 of 8

The Limitations section acknowledges the 2D simulator and 20-attempt budget but does not mention the calibration-oracle overlap with the Qwen evaluator family; adding a sentence about this threat would be useful.

What would settle it

Re-run the Phase 3 calibration for all 144 pairs using a non-Qwen oracle (e.g., DeepSeek-V4-Pro, with the same two-failure threshold and solvability checks), then evaluate Vanilla and Reflexion with Qwen3-14B and GPT-5.5 on the re-calibrated pairs. If full-benchmark or Statics pass rates rise substantially relative to Tables 2 and 3 (e.g., Reflexion+Qwen3-14B above roughly 60%, or GPT-5.5 on Statics above roughly 90%), the reported saturation level is partly an artifact of oracle choice. If rates stay close to 35.9% and 66.7%, the concern does not land and the headline conclusions are robust.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 27 Aug 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 44 s · 38,828 tokens read, 4,348 written
  3. Editor wrote the verdict above 35 s · 39,589 tokens read, 3,721 written
  4. Referee major revision: 3 major objections, 8 minor 50 s · 39,003 tokens read, 5,832 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

PACE-Bench is a genuinely useful benchmark for a capability nobody else has benchmarked: adapting a working, code-level physical design after the environment silently changes. The construction pipeline is the most careful I've seen in this space—automated audits, human verification, from-scratch solvability checks, run-reliability statistics, and full task specs in the appendix. That alone justifies the paper's existence. The novelty claim in Table 1 holds up: no prior benchmark combines environment mutation, executable design, and self-evolving evaluation.

The main results are plausible but not bulletproof. The weakest link is the difficulty-calibration loop in §3.3. Tasks are mutated until Qwen3-4B fails two runs, and then the headline full-benchmark results come from Qwen3-4B/8B/14B. If Qwen3-4B tends to fail for reasons tied to its code-generation or physics-reasoning style, the calibration selects mutations that are specifically hard for the Qwen family, which inflates the 'far from saturated' reading and distorts cross-model comparisons. The paper's mitigations—from-scratch source 32.3% vs target 6.4–13.4%, and GPT-5.5 at 66.7% on Statics—show the tasks are not trivial for other models, but they don't prove the Qwen-based calibration is a neutral difficulty metric. The honest statement would be 'unsaturated for the Qwen family' plus a separate note that frontier models still fail a third of Statics.

The CE/CH experiment that drives the 'know-how vs know-what' conclusion is also smaller than it looks—about 48 pairs pooled across two domains, with only a few comparisons outside ±5 points. The claim that revealing exact physical changes doesn't raise the ceiling is interesting but underpowered; I'd want a larger sample or a regression to believe it. The VLM video-feedback section is explicitly exploratory and should stay that way.

None of this sinks the benchmark. The resource is valuable, the documentation is unusually transparent, and the error taxonomy, though subjective, is disclosed with priority ordering. I'd send this to serious peer review. A good reviewer should ask for a calibration check using a different oracle model family on a subset, hedging the saturation claim, and reframing the CE/CH and VLM results as suggestive rather than definitive. With those changes, PACE-Bench deserves to become a standard testbed.

The strongest objection

Section 3.3 Phase 3 calibrates each task by escalating mutation severity until Qwen3-4B fails two independent runs, and Appendix A.2 confirms this loop uses Qwen3-4B as the difficulty oracle. The full-benchmark evaluation then uses Qwen3-4B/8B/14B as the primary models (Table 2). If Qwen3-4B tends to fail for reasons correlated with its code generation, prompt adherence, or physics reasoning style, the calibration selects mutations that are specifically hard for the Qwen family; the headline "far from saturated" (35.9% Pass@2 for Reflexion+Qwen3-14B) and the cross-model ordering could then be inflated by oracle/evaluator overlap. The paper's mitigations (from-scratch source pass 32.3% vs mutated 6.4-13.4%; GPT-5.5 at 66.7% on Statics) show the tasks are not trivially easy for other models, but they do not establish that Qwen3-4B's failure is a neutral difficulty measure. The central "know-how vs know-what" bottleneck conclusion also depends on the same task set. This is a selection-bias threat to benchmark validity, not an internal inconsistency.

I read the paper as constructing a benchmark whose core value is measuring whether self-evolving agents can adapt executable designs after an environment change, with the headline evidence being that current methods (especially the Qwen family) are far from saturation. The construction pipeline is unusually careful: automated audits, graduate-level human verification, solvability and non-triviality conditions, run-reliability statistics, and public code all support the benchmark infrastructure. The weakest load-bearing assumption is the coupling between the difficulty oracle and the main evaluator: Section 3.3 and Appendix A.2 use Qwen3-4B to decide when a mutation is "difficult enough", and Section 4.1.1 uses Qwen3-4B/8B/14B as the primary full-benchmark models. If Qwen3-4B fails for family-specific reasons, the calibration target is not a neutral proxy for physics-adaptation difficulty. I agree with the reader's weakest_assumption; the concern is concrete and testable. I do not think it warrants rejection: the benchmark itself is released, the audits are strong, and the GPT-5.5 results on a subset suggest the tasks are nontrivial beyond Qwen. However, the saturation claim and the central bottleneck claim should be re-verified with a calibration oracle from a different model family before strong conclusions are drawn. Hence I would keep the reader's CONDITIONAL verdict rather than raising or lowering it.

The referee's report

If the benchmark's validity holds, it fills a genuine gap: existing self-evolution benchmarks either fix execution conditions or test transfer, whereas PACE-Bench tests adaptation of an executable design after a controlled environment shift. The construction pipeline is a strength: it includes automated cross-module audits, three human verification passes (86% of tasks with moderate-to-critical issues in pass 1, none in pass 3), explicit solvability and non-triviality checks (x0 in X0\Xk), and run-reliability statistics (86.6% pass/fail agreement between runs, Spearman rho up to 0.976 at 14B). The cross-model results are falsifiable and the public code/data release supports reproducibility. The benchmark's central claims, however, depend on the neutrality of its difficulty calibration, which is currently the main validity risk.

Pith's reading of the paper (a machine's summary, not the authors')

PACE-Bench asks a question existing self-evolving benchmarks skip: can an agent take a code-driven design that works in one simulated environment and revise it after physical parameters — friction, gravity, material strength, dynamics — are secretly mutated so the old design fails? The paper builds 144 source-to-target pairs across six physics domains and runs ten self-evolving methods within a 20-attempt budget. It reports that the benchmark is far from saturated: Reflexion with Qwen3-14B solves 35.9% of full-benchmark pairs, while GPT-5.5 solves 66.7% of the Statics subset. The paper's central finding is that simulator-grounded reflection helps, unverified self-revision hurts, memory can anchor agents to early designs, and telling the agent exactly what changed does not raise the performance ceiling — mechanism redesign, not parameter inference, is the bottleneck.

What carries the argument. The central object is the source-to-target pair $\tau_0 \to \tau_k$: the same task template, goal, and API, but a mutated environment $E_k=(\theta_k,\phi_k)$ in which the source solution $x_0$ fails while a reference solution passes. The orchestrating mechanism is the evaluator returning $(v(x), s(x), d(x))$ — fractional constraint satisfaction, a score that transitions from constraint penalty to task progress once $v(x)=1$, and a structured diagnostic report — which drives iterative revision under a 20-attempt budget. The design enforces information asymmetry: visible variables are disclosed, invisible physics such as gravity, friction, and damping is scrubbed from prompts, and a Uniform Suffix warns that conditions may differ without naming them. This forces agents to infer the change from feedback and redesign accordingly.

Core claim. The discovery is a measurable gap: after a controlled physical mutation that keeps goal and interface fixed, self-evolving agents mostly fail to adapt executable designs within 20 sandbox attempts. The strongest tested configuration reaches 35.9% Pass@2 over 144 pairs; even GPT-5.5, the best model on the Statics subset under the full budget, fails a third of its pairs. The paper attributes the gap to a know-how bottleneck: revealing the exact changed variables and values does not lift performance above the hidden-setting ceiling, so the limiting step is redesigning the mechanism rather than inferring what changed.

The premise it rests on. The benchmark's difficulty calibration treats Qwen3-4B failing twice as proof that a task is hard, and the same model family is then the main evaluator; if those failures reflect Qwen-specific blind spots rather than intrinsic physics-adaptation difficulty, the unsaturated benchmark result and the model comparisons are distorted.

If the paper is right (Pith's inferences, not the authors')

The benchmark is far from saturated: Reflexion + Qwen3-14B achieves 35.9% Pass@2 on the full 144-pair benchmark, and GPT-5.5 reaches only 66.7% on the Statics subset under the full budget.

Simulator-grounded reflection (Reflexion) is the most reliable context-based method, whereas Self-Refine's unverified inner-loop revisions never exceed 7.1% Pass@2 and compound errors.

Memory-augmented methods can anchor agents to early designs: ACE and ReasoningBank beat vanilla at 8B but underperform it at 14B, with code-similarity trends showing progressive fixation.

Broad inference-time search (ToT) is efficient but converges poorly, with near-zero design fixation and stagnation around 50%.

Revealing the exact changed variables and values does not beat the hidden-setting performance ceiling, supporting mechanism redesign ("know how") as the binding bottleneck rather than parameter inference ("know what").

Beyond the paper: the know-what/know-how split implies that benchmarks which disclose parameter changes in text may overstate agent competence, since knowing the numbers is easier than redesigning the mechanism.

Beyond the paper: a model-panel calibration check — running several 4B-class models through the difficulty-calibration step — would reveal whether the unsaturated benchmark reflects intrinsic difficulty or Qwen-family-specific blind spots.

Beyond the paper: with only a 20-attempt budget, budget exhaustion and convergence are conflated; a variable-budget condition could separate agents that plateau from agents that would converge with more attempts.

Beyond the paper: the VLM feedback result — helping memory methods but hurting context methods — suggests feedback modality and memory architecture interact; a controlled study varying feedback richness on a fixed solver would test the causal claim.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions and do not test recovery after those conditions change. To address this gap, we introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark of 144 source-to-target adaptation pairs across six physics domains. Each pair links a source environment to a mutated target environment with the same goal and interface. A code-driven design that succeeds in the source fails in the target, where agents must iteratively adapt it into a working target design using diagnostic sandbox feedback within a limited attempt budget. We compare ten self-evolving methods from four paradigms. The benchmark remains far from saturated: Reflexion + Qwen3-14B succeeds on only 35.9% of full-benchmark pairs, while GPT-5.5 solves 66.7% of the Statics subset under the full budget. Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging. Even revealing exact physical changes does not raise the performance ceiling, pointing to mechanism redesign rather than parameter inference as the central bottleneck. Data and code are available at https://github.com/thunlp/PACE-Bench.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.58

“Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.14 · states limitation 0.82 · human participants 0.02 · controlled comparison 0.37 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.89 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions and do not test recovery after those conditions change. gap 0.86 0.07 0.03 asserted 1.00
To address this gap, we introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark of 144 source-to-target adaptation pairs across six physics domains. contribution 1.00 0.91 0.05 asserted 0.98
Each pair links a source environment to a mutated target environment with the same goal and interface. method 0.94 0.78 0.02 asserted 0.99
A code-driven design that succeeds in the source fails in the target, where agents must iteratively adapt it into a working target design using diagnostic sandbox feedback within a limited attempt budget. method 0.95 0.60 0.05 asserted 0.97
We compare ten self-evolving methods from four paradigms. method 0.86 0.88 0.02 asserted 0.93
The benchmark remains far from saturated: Reflexion + Qwen3-14B succeeds on only 35.9\% of full-benchmark pairs, while GPT-5.5 solves 66.7\% of the Statics subset under the full budget. result 1.00 0.94 0.09 asserted 1.00
Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging. interpretation 0.99 0.92 0.55 asserted 1.00
Even revealing exact physical changes does not raise the performance ceiling, pointing to mechanism redesign rather than parameter inference as the central bottleneck. interpretation 0.89 0.92 0.36 asserted 0.96
Data and code are available at https://github.com/thunlp/PACE-Bench. contribution 0.85 0.41 0.02 asserted 0.94

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from simulation or computation 0.64

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,966 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. Existing self-evolving benchmarks test trans
Figure 2 of the paper
Figure 2. Overview of the three-phase dataset construction pipeline.
Figure 3 of the paper
Figure 3. Overview of PACE-BENCH. Left: the 36 tasks organized across six physics categories. Right: rendered snapshots of reference solutions succeeding on the source environments. 3.4 Dataset Overview
Figure 4 of the paper
Figure 4. Error type distribution by category (left) and method (right).
Figure 5 of the paper
Figure 5. Pass@2 and Score@2 by model on Statics, Vanilla method, five-attempt budget. 1 5 10 15 20 Iterations 0 10 20 30 Pass@2 Rate (%) 1 5 10 15 20 Iterations 0 10 Avg Score Qwen3-4B Qwen3-8B Qwen3-14B
Figure 6 of the paper
Figure 6. Pass@2@k and Score@2@k by model (all methods, all categories). T = 0.7, top-p = 0.95, and a maximum of 65,536 output tokens. The Box2D sandbox runs at 60 FPS (PPM = 40 pixels/meter) with up to 10,000 simulation steps per attempt. Each environment pair (τ0 → τk) is evaluated with two independent runs. Pass@2 records whether either run passes, while Score@2 averages the best score obtained within each run. History Management. Under the 20-attempt bud￾get, naive retention of the full interaction history (x i , si , di ) t−1 i=0 rapidly exceeds LLM context limits (especially for
Figure 7 of the paper
Figure 7. Model × category Pass@2 heatmap (all meth￾ods aggregated). ing. Agents explore diversely (radicality 0.311) but show 34.1% Exploration without convergence. Because “almost balanced” structures still collapse, iterative refinement lacks a gradient. Category 2 (Kinematics, K) tests linkage topology design. The smallest scaling gain (+1.9) and 44.5% Bud￾get Exhaustion suggest that combinatorial search, rather than reasoning depth, may be the main bot￾tleneck. Category 3 (Dynamics, D) tests coupled energy, momentum, and timing. All three trajectory failures co-occur (30.6% Fixation, 30.1

Figures are the authors', reproduced from the paper.

References (54)
  1. [1] Phy-Q as a Measure for Physical Reasoning Intelligence , author =. Nature Machine Intelligence , volume =. 2022 , doi = on Pith
  2. [2] Physbench: Benchmarking and enhancing vision-language models for physical world understanding , author=. International Conference on Learning Representations , volume= on Pith
  3. [3] Shenghe Zheng and Qianjia Cheng and Junchi Yao and Mengsong Wu and Ning Ding and Yu Cheng and Shuyue Hu and Lei Bai and Dongzhan Zhou and Ganqu Cui and Peng Ye , booktitle =. Scaling Physical Reasoning with the on Pith
  4. [4] Evolution gym: A large-scale benchmark for evolving soft robots , author=. Advances in Neural Information Processing Systems , volume= on Pith
  5. [5] Robogrammar: graph grammar for terrain-optimized robot design , author=. ACM Transactions on Graphics (TOG) , volume=. 2020 , publisher= on Pith
  6. [6] Causalworld: A robotic manipulation benchmark for causal structure and transfer learning , author=. arXiv preprint arXiv:2010.04296 , year= on PithPith's reviewarXiv
  7. [7] Minedojo: Building open-ended embodied agents with internet-scale knowledge , author=. Advances in Neural Information Processing Systems , volume= on Pith
  8. [8] Newtonbench: Benchmarking generalizable scientific law discovery in llm agents , author=. arXiv preprint arXiv:2510.07172 , year= on PitharXiv
  9. [9] Self-refine: Iterative refinement with self-feedback , author=. Advances in neural information processing systems , volume= on Pith
  10. [10] Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in neural information processing systems , volume= on Pith
  11. [11] Self-adapting language models , author=. Advances in Neural Information Processing Systems , number= on Pith
  12. [12] Physreason: A comprehensive benchmark towards physics-based reasoning , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  13. [13] Buildarena: A physics-aligned interactive benchmark of llms for engineering construction , author=. arXiv preprint arXiv:2510.16559 , year= on PithPith's reviewarXiv
  14. [14] Toward engineering AGI: Benchmarking the engineering design capabilities of LLMs , author=. Advances in Neural Information Processing Systems , number= on Pith
  15. [15] Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization , author=. arXiv preprint arXiv:2604.12290 , year= on PithPith's reviewarXiv
  16. [16] Scaling physical reasoning with the physics dataset , author=. Advances in Neural Information Processing Systems , number= on Pith
  17. [17] Learning to discover at test time , author=. arXiv preprint arXiv:2601.16175 , year= on PithPith's reviewarXiv
  18. [18] Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning , author=. arXiv preprint arXiv:2504.20073 , year= on PithPith's reviewarXiv
  19. [19] Nature-inspired population-based evolution of large language models , author=. arXiv preprint arXiv:2503.01155 , year= on PithPith's reviewarXiv
  20. [20] Absolute zero: Reinforced self-play reasoning with zero data , author=. Advances in Neural Information Processing Systems , number= on Pith
show 34 more
  1. [21] Thetaevolve: Test-time learning on open problems , author=. arXiv preprint arXiv:2511.23473 , year= on PithPith's reviewarXiv
  2. [22] Alphaevolve: A coding agent for scientific and algorithmic discovery , author=. arXiv preprint arXiv:2506.13131 , year= on PithPith's reviewarXiv
  3. [23] Codeevolve: An open source evolutionary coding agent for algorithm discovery and optimization , author=. arXiv preprint arXiv:2510.14150 , year= on PithPith's reviewarXiv
  4. [24] Expel: Llm agents are experiential learners , author=. Proceedings of the AAAI Conference on Artificial Intelligence , number= on Pith
  5. [25] Reasoningbank: Scaling agent self-evolving with reasoning memory , author=. arXiv preprint arXiv:2509.25140 , year= on PithPith's reviewarXiv
  6. [26] Voyager: An open-ended embodied agent with large language models , author=. arXiv preprint arXiv:2305.16291 , year= on PithPith's reviewarXiv
  7. [27] A survey of self-evolving agents: On path to artificial super intelligence , author=. arXiv preprint arXiv:2507.21046 , volume= on PithPith's reviewarXiv
  8. [28] Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume= on Pith
  9. [29] Agentic context engineering: Evolving contexts for self-improving language models , author=. arXiv preprint arXiv:2510.04618 , year= on PithPith's reviewarXiv
  10. [30] Qwen3 Technical Report , author=. 2025 , eprint= on Pith
  11. [31] DeepSeek-V4 Technical Report , author=. 2025 , url= on Pith
  12. [32] Introducing GPT-5.5 , author=. 2025 , url= on Pith
  13. [33] Gemini 3.1 Pro: Our most capable model for agents and coding , author=. 2025 , url= on Pith
  14. [34] Claude Opus 4.7 , author=. 2 025 , url= on Pith
  15. [35] A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems , author=. 2025 , eprint= on Pith
  16. [36] PhysMem: Scaling Test-time Physical Memory for Robot Manipulation , author=. arXiv preprint arXiv:2602.20323 , year= on PithPith's reviewarXiv
  17. [37] World Model Implanting for Test-time Adaptation of Embodied Agents , author=. 2025 , eprint= on Pith
  18. [38] PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement , author=. 2026 , eprint= on Pith
  19. [39] ProgramBench: Can Language Models Rebuild Programs From Scratch? , author=. 2026 , eprint= on Pith
  20. [40] GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models , author=. 2025 , eprint= on Pith
  21. [41] Kimi K2.6 , author=. 2026 , url= on Pith
  22. [42] MiniMax M2.7 , author=. 2026 , url= on Pith
  23. [43] OpenAI Gym , author=. 2016 , eprint= on Pith
  24. [44] CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning , author=. 2021 , eprint= on Pith
  25. [45] Gemma 4 Technical Report , author=. 2026 , eprint= on Pith
  26. [46] StreamBench: Towards Benchmarking Continuous Improvement of Language Agents , author=. 2024 , eprint= on Pith
  27. [47] LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners , author=. 2025 , eprint= on Pith
  28. [48] AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning , author=. 2025 , eprint= on Pith
  29. [49] SE-Bench: Benchmarking Self-Evolution with Knowledge Internalization , author=. 2026 , eprint= on Pith
  30. [50] SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment , author=. 2026 , eprint= on Pith
  31. [51] SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks , author=. 2026 , eprint= on Pith
  32. [52] SEAGym: An Evaluation Environment for Self-Evolving LLM Agents , author=. 2026 , eprint= on Pith
  33. [53] EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer , author=. 2026 , eprint= on Pith
  34. [54] Executable Code Actions Elicit Better LLM Agents , author=. 2024 , eprint= on Pith
Cited by 2 Pith papers
How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 21 sentences without a confident label

Cites as background 20 papers
  • Recent work spans four paradigms: context-based revision through iterative refinement and self-critique [ref] ; memory-augmented approaches that maintain structured experience across attempts [ref] ; inference-time search over solution candidates via evolutionary algorithms [ref] ; and parameter-based training through test-time RL [ref] , SFT [ref] , genetic search [ref] , or self-generated data [ref] .

    Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models · Related Work · 0.99

  • Recent work spans four paradigms: context-based revision through iterative refinement and self-critique [ref] ; memory-augmented approaches that maintain structured experience across attempts [ref] ; inference-time search over solution candidates via evolutionary algorithms [ref] ; and parameter-based training through test-time RL [ref] , SFT [ref] , genetic search [ref] , or self-generated data [ref] .

    Learning to Discover at Test Time · Related Work · 0.99

  • Recent work spans four paradigms: context-based revision through iterative refinement and self-critique [ref] ; memory-augmented approaches that maintain structured experience across attempts [ref] ; inference-time search over solution candidates via evolutionary algorithms [ref] ; and parameter-based training through test-time RL [ref] , SFT [ref] , genetic search [ref] , or self-generated data [ref] .

    Nature-Inspired Population-Based Evolution of Large Language Models · Related Work · 0.99

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment