Pith. sign in

REVIEW 4 major objections 7 minor 30 references

Verified scientific-computing runs can be turned into procedures that teach a weaker model durable coding skill—even when it cannot use those procedures at runtime.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 14:22 UTC pith:BZHW4YWP

load-bearing objection Real control and a clean capacity gap on SciCode; the transferable-consolidation claim rests on a one-problem held-out delta and should be dialed back. the 4 major comments →

arxiv 2607.24459 v2 pith:BZHW4YWP submitted 2026-07-27 cs.AI

From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis

Liwei Dong , Jiahao Zhao , Nan Xu This is my paper
classification cs.AI
keywords scientific computingexperience consolidationprocedural knowledgeabstraction-execution gapsupervised fine-tuningSciCodeself-improving agentscode generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language models that write scientific code get rich feedback when their programs run, but that feedback usually dies with the single problem. This paper argues that the missing step is experience consolidation: contrast verified successes and failures, abstract cross-task computational procedures, validate them on held-out development data, and expand training with new answer-free queries built from failure priors. A weaker model often cannot follow those abstract procedures directly—an abstraction–execution gap the authors measure by showing that the same procedures help a 27B model at inference time but barely help a 9B model. The fix is temporary: a stronger model uses the procedures while writing concrete code, then the 9B student is trained with ordinary supervised fine-tuning and evaluated with no procedures in the prompt. On SciCode, that student beats both its base model and a matched no-procedure teacher control under procedure-free deployment, establishing a practical path from runtime traces to lasting scientific-coding capability.

Core claim

On SciCode, the same validated procedures raise Qwen3.6-27B by about +3.85 sub-step and +6.26 main-problem points at runtime, yet give almost no aggregate main-problem gain to Qwen3.5-9B—operational evidence of an abstraction–execution gap. When a stronger model concretizes those procedures into code and the 9B model is trained on that supervision, the student, evaluated without any runtime procedure, gains +3.89/+6.25 over a matched no-procedure SFT control and +5.62/+11.25 over the original 9B model.

What carries the argument

SciConsolidate: contrastive induction of family-level procedures from verified successes and failures, a development-validation gate that only keeps procedures with transfer utility, failure-informed answer-free query synthesis, and procedure-guided concretization by a stronger teacher into procedure-free SFT data (with a matched no-procedure teacher branch as control).

Load-bearing premise

That the measured gains—especially the modest lift on a small frozen held-out split, after filtering that leaves slightly unequal training sets—really show transferable capability rather than mostly improvement on the development distribution plus leftover teacher effects.

What would settle it

Train the same guided and no-procedure 9B students on perfectly size-matched data and evaluate only on a larger frozen held-out SciCode-style split (or a second scientific-computing benchmark never used in procedure or mixture selection); if the guided-over-control main-problem gap disappears, the consolidation claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Useful scientific-computing experience need not stay as a permanent runtime prompt or memory entry; it can be compiled into ordinary model weights via procedure-guided code supervision.
  • Procedure utility must be judged by the consuming model: artifacts that help a stronger model can still be worthless at inference for a weaker one.
  • Answer-free, failure-prior query synthesis plus executable filtering can enlarge consolidation data without needing pre-existing gold solutions.
  • External procedures are best treated as data-construction conditions for smaller deployed models, not as permanent inference dependencies.
  • Residual weak buckets (broader numerical methods, specification/interface transfer) mark where the next consolidation loop should collect targeted experience.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the abstraction–execution gap is mainly capacity, intermediate-size models or staged internalization (procedures → scaffolds → free policy) may shrink the need for a permanent stronger teacher.
  • The same success/failure contrast recipe could apply to other verifiable scientific loops—unit-checked derivations, simulation configs, lab-protocol scripts—wherever execution exposes recurring operational failures.
  • Closing the loop by re-running consolidation on the updated student is the natural next measurement; one-pass gains do not yet prove stable multi-round self-improvement.
  • Yield differences after executable filtering hint that procedural guidance may also raise the fraction of teacher solutions that pass tests, a second channel beyond pure behavioral transfer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper studies whether verified execution experience on scientific-coding tasks can be converted into durable model capability. The proposed SciConsolidate pipeline (i) collects target-model rollouts on SciCode and contrasts successes/failures to induce cross-task "failure families" (F1–F4), (ii) synthesizes family-level procedural artifacts (SOPs) gated by a development-validation split, (iii) expands the query pool via failure-informed, answer-free synthetic task generation, and (iv) has a stronger 27B model concretize the procedures into code supervision for standard procedure-free SFT of a 9B student, with a pipeline-matched no-procedure teacher branch as control. The two empirical claims are: (1) an abstraction–execution gap — runtime procedure injection helps Qwen3.6-27B (+3.85/+6.26 sub-step/main points) but not Qwen3.5-9B (−0.30/+0.00); (2) procedure-guided concretization yields a 9B student that, under procedure-free inference, beats the no-procedure SFT control by +3.89/+6.25 on the 80-problem aggregate and +1.46/+3.33 on the frozen 30-problem held-out split.

Significance. If the results hold, the paper contributes three things of real value. First, it operationalizes an "abstraction–execution gap" — the same validated procedures help a 27B model (+3.85/+6.26) but not a 9B model — which is a concrete, falsifiable observation relevant to the growing skill/memory literature, most of which implicitly assumes consumers can use what they retrieve. Second, the experimental design is unusually disciplined for this area: matched query budget and pipeline for the procedure-guided vs no-procedure teacher branches, a development-validation gate for procedure promotion, an explicit frozen held-out split, and procedure-free student evaluation. This is the right control structure and sets a standard for consolidation claims. Third, the answer-free query synthesis plus executable filtering pipeline is practical and the authors promise public code. The authors also deserve credit for an honest limitations section (they flag the dev-valid dual role, the yield asymmetry, and the modest held-out effect themselves). Significance is currently bounded by scale — one benchmark, 80 problems, 30 frozen, one seed — so the contribution is a well-controlled proof of concept of"

major comments (4)
  1. [§4.3.1, Table 3] §4.3.1, Table 3 (held-out column): the only untouched evidence for transferable consolidation is +1.46 sub-step / +3.33 main-problem points, i.e. roughly 2 of 140 sub-steps and exactly 1 of 30 main problems. Appendix B.2 states all training runs use a single fixed seed (42) with no variance estimate. At N=30 main problems, one flipped problem — well within ordinary seed-to-seed SFT noise — would zero the held-out main-problem effect. This is load-bearing for the paper's framing claim ('durable capability', 'transfer extends beyond the experience-source split'). The authors do hedge ('positive but modest'), but the Conclusion and Abstract still assert consolidation without any uncertainty quantification. Required: either (a) multiple training seeds with a mean±sd or a sign test on the held-out split, or (b) at minimum bootstrap confidence intervals on the paired per-problem differences, p
  2. [§4.3.1, Table 3] Table 3, held-out sub-step row: the procedure-guided student (27.86) is *below* the original 9B base (29.29), and the no-procedure control is lower still (26.40). So on the frozen split, both SFT variants regress on the finer-grained metric relative to base, and the entire positive held-out signal is carried by main-problem chaining. This is compatible with the data teaching split-specific chaining patterns rather than general sub-step competence. The manuscript's interpretation (§4.3.1, §4.5) does not engage with this regression at all — it reports only the guided-over-control delta. Please analyze why held-out sub-step accuracy degrades under both SFT conditions and discuss what this implies for the 'capability consolidation' interpretation versus catastrophic-forgetting-style side effects of 1-epoch full-parameter SFT on 2k examples.
  3. [§4.3.1 / App. B.1] §4.3.1 and Appendix B.1 (Table 6): the guided and control pools differ by 110 examples (2,095 vs 1,985) after executable filtering. The authors acknowledge this, but the comparison is the paper's central causal contrast and the asymmetry is not neutral: a higher filter yield for the guided branch is itself consistent with procedure-guided solutions being systematically easier to verify, so the treatment (procedure guidance) and the retained-data composition are entangled in construction. A ~5.5% data-size advantage plus a selection-on-verifiability channel could plausibly account for a sizable fraction of the +3.89/+6.25 aggregate gap. Required: a size-matched ablation (e.g., subsample the guided pool to 1,985 examples, or match per-family retained counts) reported alongside the main result, so the reader can separate 'procedure guidance changed the supervision' from 'procedure guidance
  4. [§4.2, Table 1] §4.2, Table 1: the abstraction–execution gap claim rests on the aggregate contrast (27B: +3.85/+6.26; 9B: −0.30/+0.00), but the split-level pattern is noisier than the narrative suggests: the 9B model loses 13.33 main-problem points on development-valid under procedure injection, and the 27B model loses 6.52 sub-step points on the same split. With 46 sub-steps and 15 main problems in dev-valid, per-split swings of this size are expected under noise, which cuts both ways: it weakens confidence in the *magnitude* of the 27B gain (also single-run, presumably greedy/single-sample — the inference sampling configuration is never stated) while supporting the qualitative direction. Please (a) state the decoding/evaluation protocol and number of inference samples per cell in Table 1, and (b) temper the language around specific point values or add repeated inference runs.
minor comments (7)
  1. [Table 1] Table 1, Qwen3.6-27B 'No' held-out row reports 3.30 main accuracy while every other cell in the table is rounded to two decimals ending in .33/.67; 3.30 vs 3.33 suggests 1/30 was recorded differently. Verify the arithmetic.
  2. [§4.4] §4.4/Figure 2/Table 7: the interpretation that reintroducing procedures post-SFT is 'disruptive because the behavior is compiled in' is plausible but equally consistent with a plain input-distribution shift. The text already hedges ('does not uniquely prove'); consider softening 'consistent with' further or adding a simple control (e.g., irrelevant-procedure injection).
  3. [§3.3, Eq. (7)] Eq. (7): G is defined only verbally ('such as paired improvement'). Since procedure promotion is a load-bearing selection step, state the exact gate statistic used in the experiments (metric, split, threshold, ties).
  4. [Tables 5 and 8] Table 5 vs Table 8 use different step counts for the same families (F1: 29 vs 33; F2: 14 vs 15; F4: 59 vs 62). A footnote explaining the different inclusion criteria would prevent confusion.
  5. [Throughout] Terminology: 'Qwen3.6-27B' is called both 'concretizer' (Table 1) and 'teacher' (Table 7, App. B.5); Figure 2 says 'compiler'. Pick one term.
  6. [§3.1] The descriptor vector z(x) (§3.1) is introduced but its exact fields and how difficulty is estimated are never specified; a short table in Appendix A would help reproducibility.
  7. [Abstract] Abstract and §1 claim '+3.89/+6.25 over the no-procedure SFT control' without noting this is the development-mixed aggregate; adding '(aggregate; +1.46/+3.33 on the frozen split)' would align the abstract with the paper's own split-discipline argument.

Circularity Check

0 steps flagged

No significant circularity: SciCode results are empirical accuracy deltas with a matched no-procedure teacher control, not quantities forced by definition, fit, or self-citation.

full rationale

SciConsolidate is a data-centric systems paper whose load-bearing claims are benchmark comparisons (runtime procedure injection on 9B vs 27B; procedure-guided vs no-procedure SFT on SciCode), not a closed-form derivation. Procedural artifacts are induced from verified train rollouts, promoted only by a development-validation utility gate (Eq. 7), and then used only as teacher-side conditioning during data construction; the student is trained and evaluated with procedure-free inputs (Eqs. 3–5, §3.5). The matched no-procedure branch (Eq. 12) isolates procedural guidance from generic 27B distillation. Reported numbers are executable sub-step/main-problem accuracies on an external benchmark with a frozen held-out split never used for iterative decisions (§3.1, §4.1). There is no self-definitional identity (X defined as Y then “predicted”), no fitted parameter renamed as a prediction, no uniqueness theorem or ansatz imported from overlapping-author prior work as a forcing premise, and no renaming of a known law as a new first-principles result. Development-valid participation in selection and failure-family-shaped synthetic data are ordinary generalization/selection risks, not circular reductions of the central claims to their inputs. Weak held-out effect sizes and single-seed training affect evidence strength, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

This is an empirical ML-systems paper. The claim rests on standard supervised fine-tuning assumptions, SciCode’s deterministic tests as ground truth for ‘capability,’ and design choices about splits, teacher/student roles, and what counts as a transferable procedure. No physical constants or fitted scientific laws; free parameters are pipeline hyperparameters and selection gates.

free parameters (4)
  • SFT learning rate and schedule = 2e-6, 1 epoch, seed 42
    Full-parameter SFT uses lr 2e-6, cosine schedule, 0.03 warmup, one epoch, seed 42; student quality depends on these choices.
  • Development-validation procedure promotion gate G
    Candidate SOPs are kept only if they improve paired utility on D_val; gate definition and revision count shape the retained skill bank.
  • Synthetic query balance and retained pool sizes = 2095 vs 1985 retained examples
    Balanced generation plus executable filtering yields 2,095 guided vs 1,985 control examples; mixture weights and filter yield affect measured gains.
  • Train/val/test split sizes on SciCode = 35 train / 15 val / 30 held-out main problems
    35/15/30 main-problem split is a design choice that defines what counts as source, selection, and generalization evidence.
axioms (5)
  • domain assumption Executable SciCode pass/fail (sub-step and main-problem) is a valid proxy for scientific-computing capability.
    All utility U and consolidation claims are defined via m(·,x) on SciCode (§2, §4.1).
  • domain assumption Standard next-token SFT on final code answers can internalize teacher behavior without runtime procedures.
    Capability objective in Eqs. 3–5 assumes procedure-free SFT transfers concretized solutions into endogenous student policy.
  • ad hoc to paper Cross-task failure families induced from contrasts capture reusable computational mechanisms rather than only source-specific fixes when validation-gated.
    Core methodological premise of §§3.2–3.3; necessary for calling artifacts ‘procedural knowledge.’
  • domain assumption A stronger teacher (27B) plus external labeler can stand in for roles a fully self-improving target cannot yet perform.
    Stated pathway uses MT concretization and GLM-5.1 labeling; limitations section notes autonomy is incomplete.
  • standard math Background mathematical and coding facts needed by SciCode tasks are available to the models or recoverable from context.
    Implicit in evaluating numerical methods, simulations, and scientific calculations without supplying new scientific laws.
invented entities (3)
  • SciConsolidate experience-to-capability pathway independent evidence
    purpose: Name the end-to-end process from verified rollouts to validation-gated procedures to procedure-guided SFT data to procedure-free student.
    Framework label for the pipeline in Eq. 5 and Figure 1; evaluated as one pass, not a new physical entity.
  • Abstraction–execution gap U(M_T,s) > U(M_θ,s) independent evidence
    purpose: Operational definition of consumer-dependent inability to instantiate abstract procedures.
    Defined in Eq. 2 and evidenced by Table 1; useful construct, not an unfalsifiable object.
  • Failure families F1–F4 and family-level SOPs no independent evidence
    purpose: Cluster recurring operational failure axes and attach reusable decision-rule skills.
    Induced via LLM labeling/distillation prompts in Appendix A; existence is internal to the pipeline though effects are tested on SciCode.

pith-pipeline@v1.2.0-grok45-kimik3 · 28330 in / 3582 out tokens · 67847 ms · 2026-07-31T14:22:52.734421+00:00 · methodology

0 comments
read the original abstract

Large language models increasingly solve scientific-computing tasks, but executable feedback from one problem rarely becomes durable capability on subsequent problems. We study scientific-computing experience consolidation: converting verified runtime experience into transferable procedural knowledge and persistent model improvement. This setting presents two challenges: trajectory-derived artifacts may encode source-specific repairs rather than cross-task computational mechanisms; and a weaker target model may be unable to operationalize an otherwise valid abstract procedure - an abstraction-execution gap. We introduce SciConsolidate, which contrasts verified successes and failures to induce cross-task procedures, selects them through a development-validation gate, and uses failure-informed, answer-free query synthesis to expand the consolidation data without requiring pre-existing reference answers. Because the target model may not directly execute these abstractions, a stronger model concretizes them into executable code supervision for standard, procedure-free SFT; a matched no-procedure teacher branch isolates the value of procedural guidance. On SciCode, runtime procedure injection improves Qwen3.6-27B by +3.85/+6.26 sub-step/main-problem points, but yields almost no aggregate main-problem gain for Qwen3.5-9B, providing operational evidence of the abstraction-execution gap. After procedure-guided concretization, the 9B student improves under procedure-free deployment by +3.89/+6.25 points over the no-procedure SFT control and by +5.62/+11.25 over the original 9B model. These results establish an experience-to-capability pathway for scientific computing and provide a practical starting point for scaling self-improving scientific assistance.

Figures

Figures reproduced from arXiv: 2607.24459 by Jiahao Zhao, Liwei Dong, Nan Xu.

Figure 1
Figure 1. Figure 1: Overview of scientific-computing experience consolidation via procedural knowledge [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Runtime-procedure effects under the background-conditioned aggregate evaluation. The [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Synthetic-data composition after stratified balancing. From left to right, the panels report [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 27 linked inside Pith

  1. [1]

    SciGym: Benchmarking scientific reasoning agents in real-world laboratories.arXiv preprint arXiv:2505.24853,

    Yubo Ma et al. SciGym: Benchmarking scientific reasoning agents in real-world laboratories.arXiv preprint arXiv:2505.24853,

    Pith/arXiv arXiv

  2. [3]

    Towards an AI co-scientist.arXiv preprint arXiv:2502.18864,

    Juraj Gottweis et al. Towards an AI co-scientist.arXiv preprint arXiv:2502.18864,

    Pith/arXiv arXiv

  3. [4]

    The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066,

    Kyle Swanson et al. The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066,

    Pith/arXiv arXiv

  4. [5]

    Shubham Gandhi, Dhruv Shah, Manasi Patwardhan, Lovekesh Vig, and Gautam Shroff

    doi: 10.1101/ 2025.05.30.656746. Shubham Gandhi, Dhruv Shah, Manasi Patwardhan, Lovekesh Vig, and Gautam Shroff. Research- CodeAgent: An LLM multi-agent system for automated codification of research methodologies. arXiv preprint arXiv:2504.20117,

    Pith/arXiv arXiv 2025

  5. [6]

    Truong, Weixin Liang, Fan-Yun Sun, and Nick Haber

    Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T. Truong, Weixin Liang, Fan-Yun Sun, and Nick Haber. ResearchCodeBench: Benchmarking LLMs on implementing novel machine learning research code.arXiv preprint arXiv:2506.02314,

    Pith/arXiv arXiv

  6. [7]

    CFD-LLMBench: A benchmark suite for evalu- ating large language models in computational fluid dynamics.arXiv preprint arXiv:2509.20374,

    Nithin Somasekharan, Ling Yue, Yadi Cao, Weichao Li, Patrick Emami, Pochinapeddi Sai Bhargav, Anurag Acharya, Xingyu Xie, and Shaowu Pan. CFD-LLMBench: A benchmark suite for evalu- ating large language models in computational fluid dynamics.arXiv preprint arXiv:2509.20374,

    Pith/arXiv arXiv

  7. [8]

    FEM-Bench: A struc- tured scientific reasoning benchmark for evaluating code-generating LLMs.arXiv preprint arXiv:2512.20732,

    Saeed Mohammadzadeh, Erfan Hamdi, Joel Shor, and Emma Lejeune. FEM-Bench: A struc- tured scientific reasoning benchmark for evaluating code-generating LLMs.arXiv preprint arXiv:2512.20732,

    Pith/arXiv arXiv

  8. [9]

    Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026a

    Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, Chujun Tao, Binxu Li, Xuandong Zhao, Hejia Geng, Xiaojun Wu, Junwei Zhou, Xiaokun Chen, Hanwen Xing, Yubo Li, Qunhong Zeng, Di Wang, Yuanli Wang, Roey Ben Chaim, Penghao Jiang, Haotian Shen, Luyang Kong, Xinyi Liu, Runhui Wang,...

    Pith/arXiv arXiv

  9. [11]

    Skillopt: Executive strategy for self-evolving agent skills.arXiv preprint arXiv:2605.23904,

    Ke Yang, Yuchen Zhuang, Ji Qi, Yunlong Mao, Weizhi Wu, Yichi Zhang, Yu Liu, Ziyuan Jia, Geng Chen, Xing Xie, Da Yin, and Bryan Hooi. Skillopt: Executive strategy for self-evolving agent skills.arXiv preprint arXiv:2605.23904,

    Pith/arXiv arXiv

  10. [12]

    Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister

    Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. Reasoningbank: Scaling agent self-evolving with reasoning memory.arXiv preprint arXiv:2509.25140,

    Pith/arXiv arXiv

  11. [13]

    Memskill: Learning and evolving memory skills for self-evolving agents.arXiv preprint arXiv:2602.02474, 2026a

    13 Haozhen Zhang, Quanyu Long, Jianzhu Bao, Tao Feng, Weizhi Zhang, Haodong Yue, and Wenya Wang. Memskill: Learning and evolving memory skills for self-evolving agents.arXiv preprint arXiv:2602.02474, 2026a. Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, and Botian Shi. Evolver...

    Pith/arXiv arXiv

  12. [14]

    Memevolve: Meta-evolution of agent memory systems.arXiv preprint arXiv:2512.18746,

    Guibin Zhang, Haotian Ren, Chong Zhan, Zhenhong Zhou, Junhao Wang, He Zhu, Wangchunshu Zhou, and Shuicheng Yan. Memevolve: Meta-evolution of agent memory systems.arXiv preprint arXiv:2512.18746,

    Pith/arXiv arXiv

  13. [15]

    SKILL0: In-context agentic reinforcement learning for skill internalization.arXiv preprint arXiv:2604.02268,

    Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. SKILL0: In-context agentic reinforcement learning for skill internalization.arXiv preprint arXiv:2604.02268,

    Pith/arXiv arXiv

  14. [16]

    Skill0.5: Joint skill internalization and utilization for out-of-distribution generalization in agentic reinforcement learning.arXiv preprint arXiv:2605.28424, 2026a

    Jiapeng Zhu, Jianxiang Yu, Yibo Zhao, Chengcheng Han, Qi Gu, Xunliang Cai, Xiang Li, and Weining Qian. Skill0.5: Joint skill internalization and utilization for out-of-distribution generalization in agentic reinforcement learning.arXiv preprint arXiv:2605.28424, 2026a. Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi Gu, Xunliang Cai, Xiang...

    Pith/arXiv arXiv

  15. [17]

    Skillrl: Evolving agents via recursive skill-augmented reinforcement learning.arXiv preprint arXiv:2602.08234,

    Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. Skillrl: Evolving agents via recursive skill-augmented reinforcement learning.arXiv preprint arXiv:2602.08234,

    Pith/arXiv arXiv

  16. [18]

    Skill-to-lora: From using skills to learning behaviors for token- efficient llm agents.arXiv preprint arXiv:2606.16769,

    Tianyi Zhang and Zhonghao Qi. Skill-to-lora: From using skills to learning behaviors for token- efficient llm agents.arXiv preprint arXiv:2606.16769,

    arXiv

  17. [19]

    Scicode: A research coding benchmark curated by scientists.arXiv preprint arXiv:2407.13168,

    Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huert...

    Pith/arXiv arXiv

  18. [20]

    Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou

    Accessed: 2026-07-07. Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory.arXiv preprint arXiv:2504.07952,

    Pith/arXiv arXiv 2026

  19. [21]

    Memp: Exploring agent procedural memory.arXiv preprint arXiv:2508.06433,

    Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Memp: Exploring agent procedural memory.arXiv preprint arXiv:2508.06433,

    Pith/arXiv arXiv

  20. [22]

    Memrl: Self-evolving agents via runtime reinforcement learning on memory.arXiv preprint arXiv:2601.03192, 2026b

    Shengtao Zhang, Jiaqian Wang, Ruiwen Zhou, Junwei Liao, Yuchen Feng, Zhuo Li, Yujie Zheng, Weinan Zhang, Ying Wen, Zhiyu Li, Feiyu Xiong, Yutao Qi, Bo Tang, and Muning Wen. Memrl: Self-evolving agents via runtime reinforcement learning on memory.arXiv preprint arXiv:2601.03192, 2026b. Libin Qiu, Zhirong Gao, Junfu Chen, Yuhang Ye, Weizhi Huang, Xiaobo Xue...

    Pith/arXiv arXiv

  21. [23]

    Recursive self-evolving agents via held-out selection.arXiv preprint arXiv:2606.28374,

    Michael Nguyen, Quoc Nguyen, and Paul Vuong. Recursive self-evolving agents via held-out selection.arXiv preprint arXiv:2606.28374,

    Pith/arXiv arXiv

  22. [24]

    Organizing, orchestrating, and benchmarking agent skills at ecosystem scale.arXiv preprint arXiv:2603.02176, 2026b

    Hao Li, Chunjiang Mu, Jianhao Chen, Siyue Ren, Zhiyao Cui, Yiqun Zhang, Lei Bai, and Shuyue Hu. Organizing, orchestrating, and benchmarking agent skills at ecosystem scale.arXiv preprint arXiv:2603.02176, 2026b. 14 Wenlong Huang, Weilin Cai, Wanyu Wang, Weizhi Wu, Yuchen Zhuang, Hao Qin, Da Yin, and Bryan Hooi. From raw experience to skill consumption: A ...

    arXiv

  23. [25]

    Jingzhi Gong, Ruizhen Gu, Zhiwei Fei, Yazhuo Cao, Lukas Twist, Alina Geiger, Shuo Han, Dominik Sobania, Federica Sarro, and Jie M. Zhang. Skillmoo: Multi-objective optimization of agent skills for software engineering.arXiv preprint arXiv:2604.09297,

    Pith/arXiv arXiv

  24. [26]

    Skillevolver: Skill learning as a meta-skill.arXiv preprint arXiv:2605.10500, 2026c

    Li Zhang, Zihao Ma, Jiabang He, Hangyu Hu, Xiaobo Hu, Yixin Yang, Changjiang Fan, Fanyi Xiong, Yulong Lu, Linxi Liu, Rui Wang, Haotong Qin, Huan Su, Junbo Zhu, and Shiyang Wen. Skillevolver: Skill learning as a meta-skill.arXiv preprint arXiv:2605.10500, 2026c. Hongyi Liu, Haoyan Yang, Tao Jiang, Bo Tang, Feiyu Xiong, Yuyu Luo, and Zhiyu Li. Skillsvote: L...

    Pith/arXiv arXiv

  25. [27]

    Skillwiki: A living knowledge infrastruc- ture for agent skills.arXiv preprint arXiv:2606.16523, 2026b

    Dingcheng Huang, Yuda Ding, Bingshuo Liu, Qingbin Liu, Xi Chen, Jiang Bian, Hongliang Sun, Zhiying Tu, Dianhui Chu, Xiaoyan Yu, and Dianbo Sui. Skillwiki: A living knowledge infrastruc- ture for agent skills.arXiv preprint arXiv:2606.16523, 2026b. Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, and Yutao Yue. Skill- coach: Self-evolvin...

    arXiv

  26. [28]

    Agenttrap: Measuring runtime trust failures in third-party agent skills.arXiv preprint arXiv:2605.13940,

    Haomin Zhuang, Hanwen Xing, Yujun Zhou, Yuchen Ma, Yue Huang, Yili Shen, Yufei Han, and Xiangliang Zhang. Agenttrap: Measuring runtime trust failures in third-party agent skills.arXiv preprint arXiv:2605.13940,

    Pith/arXiv arXiv

  27. [29]

    Skills are not islands: Measuring dependency and risk in agent skill supply chains.arXiv preprint arXiv:2607.01136,

    Changguo Jia, Tianqi Zhao, Runzhi He, and Minghui Zhou. Skills are not islands: Measuring dependency and risk in agent skill supply chains.arXiv preprint arXiv:2607.01136,

    Pith/arXiv arXiv

  28. [30]

    Opd-evolver: Cultivating holistic agent evolver via on-policy distillation.arXiv preprint arXiv:2606.17628, 2026d

    Guibin Zhang, Xun Xu, Yanwei Yue, Zikun Su, Wangchunshu Zhou, Xiaobin Hu, and Shuicheng Yan. Opd-evolver: Cultivating holistic agent evolver via on-policy distillation.arXiv preprint arXiv:2606.17628, 2026d. Zhiyuan Fan, Wenwei Jin, Feng Zhang, Bin Li, Yihong Dong, Yao Hu, and Jiawei Li. Evolving-rl: End-to-end optimization of experience-driven self-evolv...

    Pith/arXiv arXiv

  29. [2025]

    Training a scientific reasoning model for chemistry.arXiv preprint arXiv:2504.16084,

    Taicheng Guo et al. Training a scientific reasoning model for chemistry.arXiv preprint arXiv:2504.16084,

    Pith/arXiv arXiv

  30. [2026]

    Inducing programmatic skills for agentic tasks.arXiv preprint arXiv:2504.06821,

    Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, and Daniel Fried. Inducing programmatic skills for agentic tasks.arXiv preprint arXiv:2504.06821,

    Pith/arXiv arXiv