{"id":"8944b2ed-82ea-4eeb-afc7-b33095e1296c","arxiv_id":"2610.18216","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"TextualRL uses group-based comparisons of multiple task rollouts to propose and refine textual instructions for frozen LLM agents, achieving consistent accuracy gains over strong baselines.","lead":"This paper introduces TextualRL, a method that improves how AI agents learn from their own task attempts by rewriting their instructions instead of retraining the underlying model. It groups multiple tries at the same task to spot useful behaviors and then validates changes against a separate set of tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Single-run results for four of five targets leave the narrow 0.9–3.2 pp advantage over SkillOpt unquantified for variability.","rationale":"The reader's weakest_assumption highlighted the generalization gap from validation to test, which is an acknowledged limitation. I instead focus on the statistical robustness of the narrow advantage over SkillOpt across all target models. The paper's own stability analysis for one target is commendable, but it does not cover the other four targets where only single runs are reported. Since the central claim is that TextualRL achieves the highest six-benchmark average for every target, and the margins are small, verifying that the advantage is not due to random seed variation is critical. The concrete test would settle this. Despite this concern, the paper is well-executed and the evidence is strong; my recommendation is to leave the verdict unchanged pending optional follow-up.","tokens_in":23240,"tokens_out":2418,"duration_ms":27124,"concrete_test":"For each of the four remaining target models (GPT-5.5, GPT-5.4, Qwen3.5-4B, Qwen3.6-35B-A3B), repeat the full TextualRL and SkillOpt optimization with at least three random seeds (varying batch composition and initial seed). If TextualRL’s six-benchmark average exceeds SkillOpt’s by at least 0.5 percentage points in every target (or if the one-sided confidence intervals exclude zero), the claim that TextualRL achieves the highest average for every target is supported. If any target shows overlapping intervals or a reversal, the central claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main results (Table 1) report a single run per target–benchmark pair. The central claim rests on a small advantage over the strongest baseline SkillOpt (0.9–3.2 percentage points). For one target (Qwen3.8-27B), a three-run stability study (Table 3) shows the advantage is consistent, with an overall mean advantage of 3.25 pp and non-overlapping standard deviations. However, for the other four target models only a single optimization is reported. Without repeated runs, it is unknown whether the observed advantage for those targets is robust or could be eliminated by seed variation. Given that the gains are narrow, this is a load-bearing concern for the generality of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper introduces TextualRL, a gradient-free reinforcement learning framework that optimizes the shared textual context of a frozen LLM agent by using groups of same-task rollouts as the unit of critique. Groupwise Policy Critique contrasts successful and failed rollouts within mixed-outcome tasks, and uses cross-task evidence for homogeneous-outcome tasks. Cross-Group Policy Update reviews consolidated proposals against pooled evidence before constructing a bounded textual patch, which is accepted or rejected based on held-out validation rewards. The method is evaluated on five target models (GPT-5.5, GPT-5.4, Qwen3.5-4B, Qwen3.6-35B-A3B, Qwen3.8-27B) across six benchmarks (SearchQA, Spreadsheet, OfficeQA, DocVQA, LiveMath, ALFWorld). TextualRL achieves the highest six-benchmark average for every target, with gains of 11.4–24.4 percentage points over the base model and 0.9–3.2 points over the strongest baseline SkillOpt.","tokens_in":23544,"tokens_out":2638,"duration_ms":34856,"significance":"If the results hold, TextualRL offers a practical and principled approach to adapting LLM agents without parameter updates, by organizing trajectory feedback into group-wise comparisons and evidence-based proposal review. The method is model-agnostic, can be applied to any frozen agent, and the ablation studies confirm that both the outcome-aware critique and cross-group review contribute meaningfully. The validation-gated selection provides a safeguard against overfitting. The paper includes thorough ablations, run stability analysis (for one target), optimizer sensitivity, skill transfer, and an evidence review audit, making the empirical contribution strong. The approach is clearly motivated by the limitations of existing trajectory-level methods and provides a coherent framework that could influence future work on context optimization.","major_comments":[{"comment":"The central claim of a consistent 0.9–3.2 pp advantage over SkillOpt relies on single optimization runs for four of the five target models. Table 3 provides three-run stability only for Qwen3.8-27B, where the advantage is robust (3.18–3.36 pp). For the other four targets (GPT-5.5, GPT-5.4, Qwen3.5-4B, Qwen3.6-35B-A3B), no repeated runs are reported, leaving the variability of the narrow advantage unquantified. The authors should either provide repeated-run statistics for at least one additional target model, or explicitly discuss how seed and run variability might affect the reported advantages and hedge the generality claim accordingly.","section":"Table 1 and §4.4 (Run stability)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The default optimizer is GPT-5.5, and Table 7 shows that using Qwen3.8-27B as the optimizer reduces the overall score by 1.3 percentage points relative to GPT-5.5 (71.0% vs. 69.7%). While the paper notes this, it does not discuss whether the method's effectiveness is inherently tied to a strong optimizer model. Given that the optimizer performs critical inference (critique, consolidation, review), the sensitivity to optimizer strength should be characterized more explicitly. What happens with a significantly weaker optimizer (e.g., Qwen3.5-4B)? Without such analysis, the practical applicability of TextualRL when a strong proprietary optimizer is unavailable remains unclear.","section":"Table 7 and §4.4 (Optimizer choice)","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The subscripts on TextualRL scores show percentage-point gains over the Base Model. For readability, consider also including the percentage-point gain over SkillOpt in the table or a separate column, given that SkillOpt is the strongest baseline and the main comparison.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sentence 'Further optimization does not consistently improve unseen performance' is supported by the OfficeQA and LiveMath curves. It would be helpful to explicitly report the maximum test score reached and whether the validation-based checkpoint selection could miss a better test-performing context due to the validation gap.","section":"§4.4 (Training dynamics, paragraph 2)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Line 4 references critique calls 'in parallel'. Clarify whether this parallelism is implemented within a single process (e.g., batched LLM API calls) or across multiple API endpoints, as this affects runtime and cost estimates.","section":"Algorithm 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The five-benchmark average excludes ALFWorld. For consistency with Table 1, consider reporting results on the same set of benchmarks or justifying the exclusion.","section":"§4.4 (Agent harnesses, Table 5)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The note that ALFWorld uses 18 validation environments instead of the 140 described in the SkillOpt paper is important for reproducibility. The potential impact of this smaller validation set on checkpoint selection should be discussed briefly.","section":"Appendix B.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"'Consolidate duplicate trajectories while preserving their outcome labels, original rollout references, and multiplicities.' It is not entirely clear whether deduplication happens within the same task or across tasks. The Appendix B.2 clarifies it is within the same task, but consider moving that clarification to the main text.","section":"§3.2 (line below eq. 3)","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"TextualRL is a clean engineering contribution to the prompt-optimization subfield. The core idea—treating a group of same-task rollouts as the unit of critique, then cross-referencing evidence across groups before applying edits—is genuinely new and well-motivated. The authors back it with thorough ablations, run-stability data for one target (Qwen3.8-27B), and a transparent audit of the evidence-review step. They also ship code and acknowledge the generalization gap from validation overfitting, which I respect.\n\nThat said, the empirical case is softer than the main table suggests. The headline advantage over SkillOpt is 0.9–3.2 percentage points across targets, and for four out of five target models only a single optimization run is reported. The three-run stability study on Qwen3.8-27B shows the advantage holds there (mean 3.25 pp with non-overlapping SDs), but we have no idea whether the same is true for GPT-5.5, GPT-5.4, or the other Qwen variants. Given that the gains are narrow, this is a real weak spot. The authors do not hide it—they say the Table 1 scores are from individual runs—but the paper’s central claim (“highest six-benchmark average for every target”) rests on numbers that could shift with seed variation.\n\nA secondary concern is that the method depends on GPT-5.5 as the optimizer. The ablations show Qwen3.8-27B as optimizer still beats the base model by 14.1 pp on average, but the advantage over SkillOpt likely shrinks further with a weaker optimizer. This is not a flaw but it limits the headline generality.\n\nWho benefits from this paper? Researchers working on prompt/context adaptation will find the grouping and cross-group review ideas worth trying. The ablations give good signal about which components matter. The paper is clearly written and the experimental design is otherwise thorough (five targets, six benchmarks, sensible baselines).\n\nRecommendation: send to peer review. The core algorithm is novel, the experiments are honest about limitations, and the work is reproducible. A referee should ask for repeated runs on the other four targets to quantify variance, and for discussion of what the method buys over just running SkillOpt with more rollouts. But the paper deserves a chance to revise on that point.","headline":"Novel grouping-based RL for context optimization, but the tiny gains over SkillOpt (0.9–3.2 pp) and single-run results for 4 of 5 targets make the central claim thinner than the packaging suggests.","tokens_in":23970,"tokens_out":1529,"duration_ms":15509},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"TextualRL optimizes the textual context of frozen LLM agents using grouped rollout comparisons and validation-gated updates, outperforming prior methods across six benchmarks.","keywords":["reinforcement learning","context management","LLM agents","prompt optimization","gradient-free optimization","textual policy","skill learning","groupwise critique"],"falsifier":"If TextualRL is run with a weak or random optimizer (e.g., a small model fine-tuned on unrelated tasks) and its averaged test accuracy across the six benchmarks fails to exceed the base model by at least 5 percentage points (or falls below SkillOpt), the central claim that group-based critique with LLM optimization drives improvement would be undermined. Additionally, if a post‑hoc analysis of validation‑score trajectories shows that many accepted updates actually decrease test performance (contradicting the assumption that validation selection is a reliable proxy), the method’s validation gate would be shown to be insufficient.","tokens_in":24812,"feed_emoji":"🤖","tokens_out":2610,"duration_ms":42216,"temperature":0.7,"pith_summary":"The paper introduces TextualRL, a gradient-free reinforcement learning method that optimizes the shared textual context (instructions and procedures) of a frozen LLM agent to maximize task rewards. Instead of relying on isolated trajectory feedback, TextualRL groups multiple rollouts from the same task under the current context and uses their success–failure patterns to propose targeted edits. A groupwise critique module contrasts successful and failed rollouts within a task, or draws on cross-task evidence when outcomes are uniform, to generate edit proposals and trajectory evidence cards. A cross-group update module then reviews all proposals against the entire pool of evidence—including observations that did not trigger an edit—to refine content and applicability before assembling a bounded textual patch. The candidate context is accepted only if its validation reward exceeds a stored reference score. On five target agents and six benchmarks, TextualRL achieves the highest six-benchmark average for every target, with gains of 11.4–24.4 percentage points over the base model and 0.9–3.2 points over the strongest baseline SkillOpt. The method is significant because it enables parameter-free policy optimization through context updates, and the group-based critique addresses the challenge of turning sparse execution outcomes into reusable guidance.","feed_headline":"Grouped rollouts boost frozen LLM agents by 11–24 points","feed_subtitle":"TextualRL uses success-failure contrasts within task rollouts to generate reusable context patches, outperforming prior skill learning…","key_machinery":"The central mechanism is the pairing of Groupwise Policy Critique and Cross-Group Policy Update. Groupwise Policy Critique organizes same-task rollout groups by their reward vector (mixed, all-success, all-failure) and produces at most one atomic edit per group plus evidence cards describing observable behavior, even when no edit is proposed. Cross-Group Policy Update then reviews all proposals against the pooled evidence cards—including those from other groups—retaining, narrowing, or revising instructions based on support, limits, or contradictions. This evidence-based review is what allows the method to avoid overfitting to specific task patterns and to produce bounded textual patches that are then validated on a held-out set.","core_discovery":"TextualRL establishes that a group of same-task rollouts under the same context, rather than an isolated trajectory, can serve as the effective unit for policy critique in textual context optimization. By routing groups into mixed-outcome, all-success, or all-failure analysis, and by retaining trajectory evidence independently of edit proposals, the method can generate context updates that generalize better than those produced by prior methods that either analyze trajectories individually or merge proposals without cross-evidence review. Across 30 target–benchmark settings, TextualRL outperforms all baselines in 29 and ties in one, demonstrating consistent improvements.","pith_inferences":["The explicit separation of evidence cards from edit proposals is a design insight that could generalize beyond context optimization: any modular policy learning system might benefit from decoupling behavioral observations from the actions they suggest, allowing cross-source review.","The generalization gap observed (e.g., OfficeQA validation 62.5% vs test 47.1%) implies that the success of TextualRL depends critically on the quality and representativeness of the validation set; practitioners using this method should invest in diverse validation tasks to maximize transfer to unseen tasks.","Because the optimizer LLM (GPT-5.5) itself provides the critiques and reviews, the method inherits any biases or limitations of that model; future work could explore whether weaker or more specialized optimizer models yield similar improvements."],"forward_implications":["TextualRL provides a practical recipe for improving frozen LLM agents without fine-tuning, making it applicable when model parameters are inaccessible or too costly to update.","The group-based critique approach reveals that comparative evidence from multiple rollouts is more informative than isolated trajectory feedback for identifying which instructions to preserve or revise.","The method demonstrates that context optimization can substantially boost performance even on small, weak target models (e.g., Qwen3.5-4B gains 51.5 points on ALFWorld), suggesting a low-resource path to agent improvement.","The validation-gated update rule introduces a principled defense against harmful edits: only context patches that improve held-out validation accuracy are adopted, preventing reward-hacking or overfitting to training tasks."],"supporting_citations":[{"why":"Supplies the strongest baseline (SkillOpt) that TextualRL compares against and often outperforms; SkillOpt also uses bounded skill updates with validation selection.","marker":"[8]"},{"why":"Trace2Skill is a prior method for distilling trajectory-local lessons into agent skills; TextualRL contrasts its own group-based approach against this individual‑trajectory method.","marker":"[7]"},{"why":"Reflexion established the paradigm of incorporating verbal feedback into subsequent attempts; TextualRL builds on this idea but moves from single‑attempt reflection to grouped comparisons.","marker":"[2]"},{"why":"GRPO uses relative rewards within groups of same‑question responses for parameter updates; TextualRL adapts this group‑relative concept to gradient‑free context updates.","marker":"[9]"},{"why":"ExpeL extracts insights from same‑task success–failure pairs; TextualRL extends this comparative learning to cross‑task evidence and a formal proposal‑review‑validation loop.","marker":"[10]"},{"why":"ProTeGi uses natural‑language feedback to guide prompt updates; TextualRL differs by using grouped rollouts and evidence review rather than gradient‑style feedback.","marker":"[4]"},{"why":"EvoSkill proposes skills from failed trajectories and validates skill repositories; TextualRL includes this as a baseline and adds within‑task contrast and cross‑group review.","marker":"[17]"}],"fun_headline_variants":["Same-task rollouts teach frozen LLMs to self-correct","TextualRL: groupwise contrast sharpens context for LLM agents","Contrasting failed and successful runs yields reusable context edits","No fine-tuning needed: group rollouts update LLM agent context","Gradient-free RL uses rollout groups to patch agent context"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The optimization process assumes that an LLM optimizer (GPT-5.5 by default) can produce meaningful critiques and edit proposals from grouped trajectory evidence, and that a held-out validation set reliably indicates which context updates will generalize to unseen tasks; the paper acknowledges that validation scores can improve far more than test scores, revealing a generalization gap that is only partially addressed.","fun_headline_variants_meta":{"raw":{"variants":["Same-task rollouts teach frozen LLMs to self-correct","TextualRL: groupwise contrast sharpens context for LLM agents","Contrasting failed and successful runs yields reusable context edits","No fine-tuning needed: group rollouts update LLM agent context","Gradient-free RL uses rollout groups to patch agent context"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":6.4e-05,"raw_usage":{"cost":6.421926e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":6.421926e-05,"upstream_inference_prompt_cost":4.085046e-05,"upstream_inference_completions_cost":2.33688e-05},"total_tokens":1169,"prompt_tokens":909,"completion_tokens":260,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":172}},"tokens_in":909,"tokens_out":260,"cost_source":"openrouter_usage","duration_ms":4137,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":172,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-14T13:13:29.658145+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If TextualRL is run with a weak or random optimizer (e.g., a small model fine-tuned on unrelated tasks) and its averaged test accuracy across the six benchmarks fails to exceed the base model by at least 5 percentage points (or falls below SkillOpt), the central claim that group-based critique with LLM optimization drives improvement would be undermined. Additionally, if a post‑hoc analysis of validation‑score trajectories shows that many accepted updates actually decrease test performance (contradicting the assumption that validation selection is a reliable proxy), the method’s validation gate would be shown to be insufficient.","supporting_citations":[{"cited_title":"SkillOpt: Executive Strategy for Self-Evolving Agent Skills","cited_arxiv_id":"2605.23904","evidence_quote":"Supplies the strongest baseline (SkillOpt) that TextualRL compares against and often outperforms; SkillOpt also uses bounded skill updates with validation selection."},{"cited_title":"Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills","cited_arxiv_id":"2603.25158","evidence_quote":"Trace2Skill is a prior method for distilling trajectory-local lessons into agent skills; TextualRL contrasts its own group-based approach against this individual‑trajectory method."},{"cited_title":"Gopinath, Karthik Narasimhan, and Shunyu Yao","cited_arxiv_id":null,"evidence_quote":"Reflexion established the paradigm of incorporating verbal feedback into subsequent attempts; TextualRL builds on this idea but moves from single‑attempt reflection to grouped comparisons."},{"cited_title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","cited_arxiv_id":"2402.03300","evidence_quote":"GRPO uses relative rewards within groups of same‑question responses for parameter updates; TextualRL adapts this group‑relative concept to gradient‑free context updates."},{"cited_title":"ExpeL: LLM agents are experiential learners","cited_arxiv_id":null,"evidence_quote":"ExpeL extracts insights from same‑task success–failure pairs; TextualRL extends this comparative learning to cross‑task evidence and a formal proposal‑review‑validation loop."},{"cited_title":"Automatic Prompt Optimization with “Gradient Descent” and Beam Search","cited_arxiv_id":null,"evidence_quote":"ProTeGi uses natural‑language feedback to guide prompt updates; TextualRL differs by using grouped rollouts and evidence review rather than gradient‑style feedback."},{"cited_title":"EvoSkill: Automated Skill Discovery for Multi-Agent Systems","cited_arxiv_id":"2603.02766","evidence_quote":"EvoSkill proposes skills from failed trajectories and validates skill repositories; TextualRL includes this as a baseline and adds within‑task contrast and cross‑group review."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}