Pith.

Artificial Intelligence · arXiv:2611.18533 v1 · submitted 13 Nov 2026 · 25 references

ModHarness-RL: Recasting Harnesses as Learnable and Evolving Programs for Agentic RL

Zelin He, Siyu Zhu, Jacob Helwig, Yujian Liu, Li An, Rohit Jain, Shiyu Chang, Alborz Geramifard

The paper says

“Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout.”

more of the abstract

However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains. We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Nov 2026 · read the full text (114,619 characters) · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A principled method for learning and evolving the harness (Compression and Steering programs over typed interaction blocks) alongside RL training, with thorough CodeSearch results and honest limitations. This line is wrong

8 minor · 1 open question

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

The paper, first character to last

The reviewer was given the whole text, 114,619 characters. Not checked: figures, code, what the 25 references say.

8 minor objections

The paper says, at §1

Pith says: minor objection 1 of 8

The paper uses many custom macros (e.g., CsOursLo, BenchCsSteer) that expand to numbers. While this is common in LaTeX, it makes the text hard to read without the macro definitions. Consider using explicit numbers or a table for clarity.

The paper says, at §4.1, Table 1

Pith says: minor objection 2 of 8

The table caption says 'F1 averaged over the three granularities' but the table shows 'Overall' which is the average. It would be helpful to explicitly state that the F1 is the mean of file, module, and entity F1 scores.

show 6 more

The paper says, at §4.2, Figure 3

Pith says: minor objection 3 of 8

The caption says 'Prefill tokens are the prompt tokens not served from the prefix cache, summed over an episode and measured by replaying the recorded episodes'. This is a good definition, but it would be helpful to also report the raw prompt token counts for completeness.

The paper says, at §5

Pith says: minor objection 4 of 8

The Limitations section acknowledges the proprietary creator and the scope of comparison. However, it does not discuss the potential for the LLM creator (GPT-5-mini) to introduce biases or the reproducibility concerns of using a proprietary model. The paper should mention that the results may not be fully reproducible if the creator model is not publicly available.

The paper says, at §A.2, Table 5

Pith says: minor objection 5 of 8

The ablation table shows that 'w/o generation' and 'w/o judging' both reduce performance. However, the paper does not explain what 'w/o generation' means in detail. From context, it means the creator proposes no new triggers during training, but the paper should state this explicitly.

The paper says, at §A.1

Pith says: minor objection 6 of 8

The paper states that 'every method that learns from rollouts uses GPT-5-mini as its creator'. This is good for fairness, but it also means that the baselines are not necessarily the original implementations. The paper should clarify whether the baselines are re-implemented or adapted, and whether any hyperparameter tuning was done for them.

The paper says, at §3.2

Pith says: minor objection 7 of 8

The algorithm (Algorithm 1) is presented in a wrapfigure environment, which may cause formatting issues. The pseudocode is clear, but the use of exttt{Call} functions without definitions makes it hard to follow. Consider providing more detailed pseudocode or a clearer description of each subroutine.

The paper says, at §4.2, Figure 4

Pith says: minor objection 8 of 8

The figure shows 'F1 against inference time' but the inference time is measured in seconds per episode. It would be helpful to also report the number of turns per episode, as inference time depends on episode length.

Open questions (1)

Pith raised these and could not confirm them against the text of the paper it read. They are questions for the authors, not findings.

  1. §3.2, §4.3 Pith raised a concern here that it could not check against the text of the paper it was given, so it is not published as an objection.

What would settle it

If one trained a model with ModHarness-RL on a structured environment like CodeSearch and found that the learned Compression and Steering programs, when applied to a held-out set, consistently perform worse than a simple append-only baseline with the same context budget, the central claim would be falsified. The paper already reports such comparisons and finds improvements, so a falsifier would be a reproduction where the improvements fail to replicate, or where the programs learned are consistently inert (do not change the context) or unfaithful (remove task-relevant information).

from Pith's summary of the paper

How the review went

5 passes by deepseek-v4-flash · 16 Nov 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic concern withdrawn: the paper already says this 7 s · 39,933 tokens read, 749 written
  3. Referee 8 minor 11 s · 40,219 tokens read, 1,731 written
  4. Check against the paper kept 0 of 5 objections 24 s · 81,851 tokens read, 4,759 written
  5. Editor wrote the verdict above 6 s · 40,638 tokens read, 498 written

Not run on this paper: the simulated authors' defence.

Pith's passes raised 5 objections. A check against the paper's text kept 0, withdrew 4 (4 the paper already says) and left 1 as a question.

The review in full

The review in brief, machine-written

This paper introduces ModHarness-RL, which treats the harness of an agentic RL system as programs over typed interaction blocks and learns/evolves these programs during training. The core idea is clean: represent Compression (how history is assembled into context) and Steering (when to supply guidance) as deterministic programs over a structured State, propose candidates from training trajectories, verify by execution, and gate on held-out tasks. The method is evaluated on CodeSearch (in depth), ALFWorld, and BIRD SQL, with models from 1.5B to 8B under two context budgets.

The paper does several things well. The representation of harness components as deterministic programs is genuinely novel and addresses a real bottleneck: harness design is currently manual and task-specific. The replay mechanism that checks whether a failure persists before installing guidance is sensible. The CodeSearch results are strong: ModHarness-RL improves over the fixed-harness baseline by 14-24% and outperforms memory, skill, prompt, and harness evolution baselines. The ablation study and the analysis of the judge's decisions are informative. The paper is clearly written and the limitations section is unusually honest.

The soft spots are real but acknowledged. The full method (Compression + Steering) is only evaluated on CodeSearch; ALFWorld and BIRD SQL test individual components. The programs are written by a proprietary model (GPT-5-mini), which is a reproducibility concern—though the paper notes this and uses the same creator for all baselines. The cost comparison with GPT compaction uses estimated figures. The gate's held-out set is small, which the authors flag. These are not fatal flaws, but they mean the central claim is not fully supported across all claimed settings.

This paper is for researchers working on LLM agent training, particularly those frustrated with manual harness design. It deserves a serious referee. The method is well-motivated, the experiments are thorough for the primary environment, and the limitations are stated rather than hidden. A referee should ask for the full method on at least one more environment and for the generated programs to be released, but the core contribution is solid.

The stress test's concern

Withdrawn: the paper already says this.

The paper says

“The programs are written by a proprietary model, GPT-5-mini. To enable a fair comparison, every learning baseline uses the same creator. GPT-5-mini writes the \Compression\ and \Steering\ programs of \ours, the memories, skills, prompts and harness edits of the baselines, and the summaries of the compaction baseline.”

The referee's report

Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.

Pith's reading of the paper (a machine's summary, not the authors')

The paper introduces ModHarness-RL, a method that recasts the harness (the scaffold around an LLM agent that decides what it sees and what guidance it receives) as deterministic programs over a structured State of typed interaction blocks. Two core components are learned and evolved: a Compression program that assembles the accumulated history into the model's context under a budget, and a Steering program that supplies guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. The paper reports that on CodeSearch, ModHarness-RL improves over the fixed-harness baseline by 14 to 24% in every setting and scores highest against memory, skill, prompt and harness evolution baselines. On ALFWorld and BIRD SQL, each program alone improves over the base harness in most settings.

What carries the argument. The central object is the harness represented as deterministic programs over a structured State (s_t), which records each turn as typed blocks (reasoning, tool call, tool result). Two programs are learned: Compression, which reads the blocks and returns a new view that fits the context budget while keeping key information, and Steering, which is a set of triggers (small programs over s_t) that fire on detectable failure modes and return guidance. The key mechanism is the Replay process: because each trigger is a program, it can be executed over all stored trajectories from past training steps to determine whether a failure is persisting or being learned away, and a judge decides whether to install the guidance. The programs are proposed by an LLM creator (GPT-5-mini) from training trajectories, verified by execution, and gated on held-out training tasks.

Core claim. The central claim is that the harness for agentic RL can be represented as deterministic programs over typed interaction blocks (State), and that these programs can be learned from data and evolved with the model during RL training. The paper proposes two such programs: Compression, which re-renders earlier turns to fit a context budget while keeping key information, and Steering, which detects specific failure patterns and appends guidance only when they fire. The key insight is that the harness should guide only what the model cannot learn on its own, so Steering programs are replayed over past training trajectories and installed only for failures that persist. The paper demonstrates that this approach improves held-out performance over a fixed hand-written harness in most settings across three environments (CodeSearch, ALFWorld, BIRD SQL) with models from 1.5B to 8B parameters.

The premise it rests on. The harness can be effectively represented as deterministic programs over typed interaction blocks, and the patterns worth steering (e.g., repeated tool calls) can be detected by simple rules written by an LLM from training trajectories.

If the paper is right (Pith's inferences, not the authors')

If the harness can be learned from data and evolved with the model, then agentic RL systems can automatically adapt their scaffolding to the task, data, model, and context budget without manual tuning.

The Compression program's ability to reduce prefill tokens while maintaining or improving performance suggests that learned context management can be more efficient than append-only history, even under sufficient budgets.

The Steering program's replay mechanism provides a principled way to decide when to intervene: only for failures that the policy does not learn to fix on its own, avoiding unnecessary guidance that could interfere with learning.

The approach outperforms methods that learn memory, skills, prompts, or harness edits from the same creator model, suggesting that representing harness components as deterministic programs over typed blocks is a more effective representation for learnable scaffolding.

The method beats zero-shot GPT-5-mini (the creator of its harness) in every setting, indicating that the gain is not simply distillation from a stronger model.

The reliance on a proprietary model (GPT-5-mini) to write the programs is a practical limitation, but the framework itself is model-agnostic: any sufficiently capable LLM could serve as the creator, and open-source models may eventually match this capability.

The deterministic program representation limits the harness to patterns detectable by explicit rules over typed interaction blocks. For open-ended domains like conversation where relevant signals are semantic, a learned program representation may need to incorporate learned embeddings or neural detectors, which would sacrifice the computational efficiency and replayability that the current approac

The paper's ablation showing that evolving Compression (rather than fitting it once) reduces performance suggests a trade-off: context stability during policy training may be more important than adapting the compression to the changing policy. This could be tested by varying the frequency of Compression updates.

The replay mechanism for Steering programs could be extended to automatically propose new triggers by mining trajectories for recurring patterns, rather than relying on an LLM to hypothesize failures from mixed-outcome groups.

Check this review

What was read

The passes were given all 114,619 characters of the paper's LaTeX source.

Fingerprint of that text (SHA-256): 2cdda69d7f1101c345a040c88435edad0153b0916efe6632f0fb584463394805

Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:

PYTHONPATH=. python scripts/review_input_check.py 2611.18533

The passes, in the order they ran

  1. Reader deepseek-v4-flash · medium effort · $0.0043 · instructions of the Reader, fingerprint f9ead7300382…
  2. Skeptic deepseek-v4-flash · medium effort · $0.0038 · instructions of the Skeptic, fingerprint 8f88c0f61488…
  3. Referee deepseek-v4-flash · medium effort · $0.0040 · instructions of the Referee, fingerprint a852d99aea0d…
  4. Check against the paper deepseek-v4-flash · medium effort · $0.0054 · instructions of the Check against the paper, fingerprint 6c7a6883d551…
  5. Editor deepseek-v4-flash · medium effort · $0.0038 · instructions of the Editor, fingerprint e7211f6a3d27…
  6. Circularity check deepseek-v4-flash · medium effort · $0.0037 · instructions of the Circularity check, fingerprint e9946e39d67b…
  7. Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0041 · instructions of the Summary for the page, fingerprint e4f56bf568e7…

The passes cost $0.029 in model calls together.

Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.d60f54226f00

Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.

curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum

prints the fingerprint of the reader's instructions. The same works for every pass.

The receipt

The review's fingerprint was taken on 16 Nov 2026. The review you are reading still matches it.

Fingerprint of the review (SHA-256): 6224e5a71e52f50dad09c8fae905ac32d63338ccf6e5724f9402bf61ba6c36ec

curl -s 'https://pith.science/api/v1/verdicts/2611.18533?form=canonical' | sha256sum

prints the fingerprint of the review as it stands now. The rule, in four steps.

A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout. However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains. We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-16

The authors' main claim, as the classifier picked it out 1.00

“These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model.”

How the main claim is stated: hedged possibility 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.14 · human participants 0.03 · controlled comparison 0.96 · randomized assignment 0.01 · states quantitative result 0.19 · compares with prior results 0.96 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. background 1.00 0.05 0.02 asserted 0.99
Most existing approaches train under a hand-written harness that stays fixed throughout. background 0.85 0.05 0.02 asserted 1.00
However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains. gap 0.62 0.08 0.06 asserted 1.00
We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. contribution 0.99 0.90 0.15 asserted 1.00
ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates. method 0.97 0.82 0.04 asserted 0.99
During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. method 1.00 0.89 0.07 asserted 0.99
We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. method 0.70 0.92 0.02 asserted 0.90
The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. result 1.00 0.95 0.23 asserted 1.00
These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model. interpretation 1.00 0.77 0.90 hedged possibility 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.94

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-16: 1 request, 6,922 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (14)
Figure 1 of the paper
Figure 1. Left: Base Harness + RL (top) and MODHARNESS-RL (bottom). MODHARNESS-RL exploits the modularity of an agent’s interaction to learn the harness from its trajectories and evolve it with the model. Right: Performance improvement of MODHARNESS-RL on CodeSearch over SWE-Bench Lite, Verified and Pro, with a context budget that runs out mid-episode (limited) or is rarely reached (sufficient). ∗ Equal contribution. † Work done during an internship at LinkedIn. 1 The Pennsylvania State University 2 LinkedIn 3 University of California, Santa Barbara 1 arXiv:2611.18533v1 [cs.AI] 13 Nov 2026
Figure 2 of the paper
Figure 2. The harness as programs over STATE. Compression and Steering are programs that read the blocks of STATE. Compression returns a new view of STATE that re-renders earlier turns, keeping their key information and offloading what is no longer needed. Steering checks the trace for specific conditions and, when one holds, returns a message that is appended to steer the model’s behavior. reruns each candidate STEERING program over trajectories from past training steps and installs it only for the failures that persist as the model trains. We train models from 1.5B to 8B with GRPO in three…
Figure 3 of the paper
Figure 3. Learning and evolving the harness with the policy. compress is optimized against the base policy, and policy updates under the latest harness are then alternated with steer updates. Both optimizations propose programs from the reservoir, verify them by execution, and install a winner only through a held-out gate. renders this record directly as one growing transcript, appending every turn, c flat t = flatten(st) = q ⊕ u0 ⊕ · · · ⊕ ut−1, (1) where ⊕ denotes concatenation. This append-only context leaves two problems, and we address each with a program over st . compress As the…
Figure 4 of the paper
Figure 4. COMPRESSION and STEERING each applied to the base harness on three benchmarks. Models are Qwen3 on CodeSearch, Qwen2.5 on ALFWorld and Qwen3.5 on BIRD SQL, and the tick gives the scale. Whiskers are ±1 SD over repeated evaluations of each trained checkpoint. †SFT warm start. holding steady or rising, and the curves of all triggers together give the replay evidence E. ADMIT. A judge reads each trigger’s replay evidence together with its guidance and the installed set, and returns one of three verdicts: (1) intervene when the failure holds steady or rises, installing the trigger and…
Figure 5 of the paper
Figure 5. F1 against inference time, Qwen3-8B on CodeSearch. Each panel is one context budget. F1 is averaged over the three granularities and over all validation tasks, as in
Figure 7 of the paper
Figure 7. (a) A learned COMPRESSION and STEERING program on CodeSearch, as high-level pseudocode. (b) The STEERING judge’s decision in different cases on CodeSearch. Dashed lines show the run the judge did not choose. et al., 2026; Kim et al., 2026) or by training the policy with RL to manage its own context (Shao et al., 2026). These approaches rely on a model to rewrite the harness or to operate on the context directly. MODHARNESS-RL instead writes harness components as programs over typed blocks that capture rich interaction, which run without a model call, keep the KV cache largely…
Figure 8 of the paper
Figure 8. Prefix-cache reuse under COMPRESSION. Each benchmark multiplies the prompt tokens an episode sends, summed over its model calls, by the share of them the cache misses. Results are collected from Qwen2.5-7B on ALFWorld and Qwen3-8B on CodeSearch. Whiskers are ±1 SD over repeated evaluations of each trained checkpoint. 1.5B† 7B 1.5B† 7B Context limited Context sufficient 7 8 9 10 Score ↑ higher is better Seen 1.5B† 7B 1.5B† 7B Context limited Context sufficient 7 8 9 10 Unseen 1.5B† 7B 1.5B† 7B Context limited Context sufficient 7 8 9 10 Overall 1.5B† 7B 1.5B† 7B Context limited…
Figure 9 of the paper
Figure 9. ALFWorld validation, base harness against COMPRESSION. Every model is Qwen2.5 and the tick gives its scale. Whiskers are ±1 SD over repeated evaluations of each trained checkpoint. †SFT warm start (Appendix A). tokens an episode sends times the share the cache misses. COMPRESSION serves 58 to 72% of prompt tokens from cache against 64 to 95% for the append-only base, so the rate falls without collapsing, and because it is taken on far fewer tokens the prefill tokens are lower in 3 of 4 cells, by up to 35%. Measured instead as the prompt at an episode’s last model call, served…
Figure 10 of the paper
Figure 10. CodeSearch validation, base harness against COMPRESSION. Every model is Qwen3 and the tick gives its scale, and Overall is the instance weighted mean. This is the per-split view of the aggregate in
Figure 11 of the paper
Figure 11. Learned STEERING on top of each context policy, CodeSearch overall validation score. COMPRESSION + STEERING is MODHARNESS-RL. Whiskers are ±1 SD over repeated evaluations of each trained checkpoint. 0 10 20 30 observed projected Failure rate the judge sees (%) judge: install installed if not installed 0 5 10 Failure rate after (%) ↓ 0 50 100 150 200 250 training step 0 10 20 30 observed projected judge: do not install not installed if installed 100 125 150 175 200 training step 0 5 10 Case 1 Steering needed Case 2 Steering not needed
Figure 12 of the paper
Figure 12. The judge’s decision and the failure rate after it. A failure is an episode that ends without the model submitting an answer. The judge sees only the training history before its decision. The runs with and without the guidance continue from the same checkpoint, with two runs each in Case 1 and one each in Case 2. 22
Figure 13 of the paper
Figure 13. A compress program applied to a stored training episode. Top: pseudocode of the generated program. Bottom: turns 1–4 of the episode as recorded (left) and as shown to the model (right); numbered rules match the program. Grey lines are dropped, and teal lines are written by the program. 23
Figure 14 of the paper
Figure 14. A steer trigger serving guidance in a live training episode. Top: pseudocode of the generated check and its guidance. Bottom: the episode as recorded, with the check evaluated after every turn (right); the numbered rule matches the program. The check fires once, after turn 4, and the guidance is appended as a message before the model’s next turn. 24

Showing 13 of the paper's 14 figures. Figures are the authors', reproduced from the paper.

References (25)
  1. [3] URL https://arxiv.org/ abs/2610.09832. 11 Prannay Hebbar, Yogendra Manawat, Samuel Verboomen, Alesia Ivanova, Selvam Palanimalai, Kunal Bhatia, and Vignesh Baskaran. SIA: Self improving AI with harness & weight updates, on PithPith's reviewarXiv
  2. [4] URL https://arxiv.org/abs/2605.27276. Kelly Hong, Anton Troynikov, and Jeff Huber. Context rot: How increasing input tokens impacts LLM performance. Technical report, Chroma, July on PithPith's reviewarXiv
  3. [5] URLhttps://arxiv.org/abs/2609.00196. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Sympo... on PithPith's reviewarXiv
  4. [7] URLhttps://arxiv.org/abs/2609.37105. 12 Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. ToolRL: Reward is all tool learning needs. InAdvances in Neural Information Processing Systems, volume 38, pages 105523–105553, on PithPith's reviewarXiv
  5. [8] URLhttps://arxiv.org/abs/2609.37725. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, on PithPith's reviewarXiv
  6. [10] URL https://arxiv.org/abs/ 2607.21419. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, on PithPith's reviewarXiv
  7. [11] URL https: //arxiv.org/abs/2603.17829. Lev S. Vygotsky.Mind in Society: Development of Higher Psychological Processes. Harvard University Press, Cambridge, MA, on PitharXiv
  8. [12] doi: 10.18653/v1/2026.acl-long.69. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Fei-Fei Li, Lijuan Wang, Yejin Choi, and Manling Li. RAGEN:... on Pithdoi:10.18653/v1/2026.acl-long.69
  9. [13] URLhttps://arxiv.org/abs/2504.20073. Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. WebAgent-R1: Training web agents via end-to- end multi-turn reinforcement learning. InProceedings of... on PithPith's reviewarXiv
  10. [14] doi: 10.18653/v1/2025.emnlp-main.401. David Wood, Jerome S. Bruner, and Gail Ross. The role of tutoring in problem solving.Journal of Child Psychology and Psychiatry, 17(2):89–100, on Pithdoi:10.18653/v1/2025.emnlp-main.401
  11. [16] URLhttps://arxiv.org/abs/2509.13313. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying LLM-based software engineering agents.Proceedings of the ACM on Software Engineering, 2(FSE):801–824, on PitharXiv
  12. [17] URLhttps://arxiv.org/abs/2602.08234. Can Xie, Yuyi Zhou, Wen Yang, Ziyi Zhang, Siyao Song, Yingzhuo Deng, Shuo Ren, and Jiajun Zhang. EDGE: Experience-distillation for guided exploration in agentic reinforcement learning, on PithPith's reviewarXiv
  13. [18] URL https://arxiv.org/abs/2608.21946. Zekun Xu, Siyu Xia, Chuhuai Yue, Jiajun Chai, Mingxue Tian, Xiaohan Wang, Wei Lin, Haoxuan Li, and Guojun Yin. MTIR-SQL: Multi-turn tool-integrated reasoning reinforcement learning for text-to-SQL, on PithPith's reviewarXiv
  14. [19] URL https://arxiv. org/abs/2512.24601. Han Zhou, Xingchen Wan, Ivan Vuli´ c, and Anna Korhonen. Agentic policy optimization via instruction- policy co-evolution, on PithPith's reviewarXiv
  15. [20] URLhttps://arxiv.org/abs/2512.01945. Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Kongcheng Zhang, Jiale Zhao, Jingwen Yang, Yihe Zhou, Jianwei Lv, Tongya Zheng, Hengtong Lu, Chen Wei, Xie Yan, and Mingli Song. Breaking the exploration bottleneck: Rubric-scaffolded reinforce... on PitharXiv
  16. [21] in verl (Sheng et al., 2025), and vLLM (Kwon et al., on Pith
  17. [22] samples a group of G trajectories {τi}Gi=1 from the current policy and normalizes their rewardsR i =r(τ i)within the group to form advantages, ˆAi = Ri −mean({R j}Gj=1)std({R j}Gj=1) . (3) The policy is updated by the clipped surrogate LGRPO(θ) = 1G G∑i=1 1|τi| |τi|∑t=1 min ρi... on Pith
  18. [23] and 88 from SWE-Bench Pro (Deng et al., 2026). We use Qwen3-4B and Qwen3-8B. Each step samples 8 tasks and 8 episodes per task, and makes one policy update on the whole batch. We use a constant learning rate of 10−6 with no warmup, a low-variance KL loss with coefficient 0.001... on Pith
  19. [40] and every 20 steps (K= 10). Table 4 shows that an interval of 10 scores the highest with Qwen3-4B and ties with an interval of 20 with Qwen3-8B. Update interval (steps) 5 10 20 Qwen3-4B .404.424.404 Qwen3-8B .429.460 .461 Table 4:Overall score of MODHARNESS-RL on CodeSearch un... on Pith
  20. [128] No program is chosen on an evaluation set, and Dholdtrain is disjoint from every evaluation set we report. A.4 History Budget and Overflow We follow the design of verl-agent (Feng et al., 2025), which drops earlier turns from the history and tells the policy that they existed.... on Pith
show 5 more
  1. [1976] doi: 10.1111/j.1469-7610.1976.tb00381.x. Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, and Botian Shi. From interactions to principles: Experience-driven self-distillation for evolving LLM agents. ... on Pithdoi:10.1111/j.1469-7610.1976.tb00381.x
  2. [2023] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024a. doi: 10.1162/tacl_a_00638. X... on Pithdoi:10.1162/tacl_a_00638
  3. [2024] URLhttps://arxiv.org/abs/2402.03300. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. InProceedings of the Twentieth European Conference on Computer Systems, ... on PithPith's reviewarXiv
  4. [2025] URL https: //arxiv.org/abs/2506.11425. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R. Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, ... on PithPith's reviewarXiv
  5. [2026] URL https://trychroma.com/research/ context-1. Mingju Chen, Can Lv, Guibin Zhang, Heng Chang, and Shiji Zhou. HarnessForge: Joint harness and policy evolution for adaptive agent systems, 2026a. URLhttps://arxiv.org/abs/2606.01779. Zhengyu Chen, Teng Xiao, Huaisheng Zhu, Yige Y... on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment