{"id":"bcfbb42d-ec00-4259-913d-f3d9682d732d","arxiv_id":"2501.04393","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SEO automatically optimizes natural-language rule sets that, when added to prompts, improve LLM accuracy on several tasks and transfer to out-of-distribution data.","lead":"This paper introduces SEO, an automatic method that writes and improves a small set of natural-language rules, called experiences, to guide a language model on a task. The method tests candidate rule sets on random samples of training data and keeps the one that scores best, improving GPT-3.5 and Llama-2 on question answering, translation, and classification without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported improvements are not statistically supported because validation scores are noisy, many candidates are selected over, and no error bars or repeated runs are provided; the acknowledged En-Zh exception is a concrete symptom.","rationale":"The reader's weakest assumption is that the 50-example stochastic validation set reliably predicts test improvement; my concern is the same assumption, sharpened by the multiple-comparison structure and the absence of variance reporting. The ablation in Section 5.5 does show that removing validation hurts (test EM 37.6 vs 45.2), which supports the method's internal logic, and the OOD transfer in Table 7 is a genuine positive signal. But the same ablation shows only a small gap between fixed and stochastic validation (44.4 vs 45.2), and no repeated runs are reported. The paper explicitly says code and optimized experiences will be released only upon acceptance, which limits independent verification. Given these factors, a CONDITIONAL verdict is appropriate: the method is plausible and internally coherent, but the central empirical claim requires additional evidence, particularly multi-seed results and significance testing. I therefore recommend no change to the reader's verdict.","tokens_in":18178,"tokens_out":11527,"duration_ms":115381,"concrete_test":"Run SEO on HotpotQA with GPT-3.5 under at least 5 independent seeds, varying only the random draws of the stochastic validation sets Dt and the Mopt sampling, and report the mean and 95% confidence interval of test EM for the dev-selected experience in each seed. If the confidence interval overlaps Direct or the selected experience differs substantially across seeds, the claim of consistent improvement is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SEO reliably finds experiences that improve LLM performance, with the stochastic validation step (Section 2.3, Eq. 3-4) as the mechanism that is supposed to guarantee the update direction. However, the reported experiments do not establish that this guarantee holds. With k=3 candidates per example, batch sizes of 1 and 3, and up to 200 training steps (Section 3.2), the pipeline evaluates hundreds of candidate experiences on m=50-example validation sets. For EM and accuracy, a 50-example binary score has a standard error of up to about 7 points; for COMET, differences of 0.1-0.5 points are likely within metric noise. Selecting the maximum validation score over many candidates, and then choosing the final experience from two runs using a 200-example dev set, is a multiple-comparison procedure that can select experiences that were lucky on the validation or dev samples rather than genuinely better. The paper reports no error bars, no significance tests, and no repeated runs. The one acknowledged exception (En-Zh Llama-2-13b: SEO 73.23 vs Direct 73.41, Table 2) is exactly the kind of outcome expected if part of the reported gains are selection noise. Additionally, candidate experiences are generated using the ground-truth answer Y of a single training example (Section 2.2); the noisy 50-example filter is the only defense against single-example overfitting, and its reliability is not independently tested. The headline claim of 'consistently improved performance' is therefore not yet supported at the level claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Stochastic Experience Optimization (SEO), an inference-time procedure that iteratively refines a natural-language 'experience' (a list of rules) for a frozen generator LLM. At each training step, an optimizer LLM samples k candidate updates conditioned on a single training example, its gold answer, the generator's trials with and without the current experience, and the current experience's effect; the candidates are then evaluated on a stochastically resampled, 50-example validation set, and the best candidate that beats the current experience on that set is kept as the next experience. Experiments on HotpotQA, WMT23 machine translation (four directions), and GLUE classification (SST-2, CoLA) with GPT-3.5 and Llama-2-7b/13b report performance improvements over direct prompting and over applying the initial experience, along with ablations, an error analysis, and an out-of-distribution transfer test to 2WikiMultiHopQA.","tokens_in":18475,"tokens_out":5620,"duration_ms":57857,"significance":"If the central claim were statistically established, SEO would be a practical, model-agnostic and task-agnostic inference-time optimization tool: it improves LLM performance without parameter updates, uses only a frozen optimizer LLM, and the described algorithm is clear enough to reimplement. The paper has genuine strengths: Algorithm 1 is explicit, the ablation in Table 8 isolates the validation step as a major contributor, the error analysis in Section 5.2 distinguishes format gains from correctness gains, and the OOD transfer result in Table 7 is a falsifiable prediction that goes beyond same-distribution tuning. However, the empirical evidence is currently under-powered: there are no error bars, no significance tests, no repeated runs, and the reported numbers are selected maxima over many candidate evaluations and over two optimization runs. The one acknowledged exception (En-Zh Llama-2-13b) is precisely the kind of outcome expected if part of the reported gains are selection noise. The method is plausible and worth further investigation, but the headline claim of 'consistently improved performance' is not yet supported by the experiments as presented.","major_comments":[{"comment":"The central claim of consistent improvement is not statistically established. The paper reports no repeated runs, no error bars, and no significance tests. For the MT results in Table 2, many reported gains are 0.1-0.5 COMET points (e.g., GPT-3.5 En-De: SEO 84.23 vs Direct 83.27; De-En: 85.47 vs 85.36), which are likely within metric noise for a 200-example test set. The one acknowledged exception, En-Zh Llama-2-13b (SEO 73.23 vs Direct 73.41), is exactly the kind of result expected under selection noise, and it undermines the abstract's unqualified phrase 'consistently improved performance.'","section":"Section 4, Tables 1-3"},{"comment":"The stochastic validation step does not, by itself, guarantee a reliable update direction. With n=3, k=3, and up to 200 training steps, the algorithm compares up to 1,800 candidate scores against the current experience score on 50-example validation sets. For EM and accuracy, a 50-example binary metric has a standard error of roughly 7 points; under the null hypothesis that candidates are equivalent to Et, the maximum of many candidate scores will frequently exceed St by chance. No multiple-comparison correction, calibration, or stability analysis is provided. The statement in Section 2.3 that validation 'make[s] sure that an updated experience actually improves the performance of Mgen on Dt' is only true for the particular sampled Dt, not for the test distribution, and the paper should be revised to replace 'ensure' with a claim that is actually supported.","section":"Section 2.3, Eq. (4)"},{"comment":"Candidate experiences are generated using the ground-truth answer Y of a single training example, and the optimizer prompt only softly instructs the model to avoid sentence-specific content. The 50-example stochastic validation filter is therefore the only objective defense against single-example overfitting, but its reliability is not independently tested. The paper should report, for example, the correlation between validation-set scores and held-out dev scores for accepted candidates, or the fraction of accepted candidates that fail to generalize to a fresh validation sample. Without such an analysis, it is unclear whether the validation step is selecting for genuine general rules or for experiences that happen to score well on the small sampled set.","section":"Section 2.2, Figure 2"},{"comment":"The final reported performance is the result of an additional selection stage: the best experience is chosen from all valid training steps using a 200-example held-out dev set, and the best of two runs (batch size 1 and batch size 3) is selected. The tables therefore report the maximum over many candidate trajectories, not the expected performance of a single SEO run. This inflates the apparent improvement and further justifies the need for repeated runs or a nested evaluation protocol. At minimum, the paper should report the number of valid steps, the dev scores of the selected runs, and the performance variance across runs.","section":"Section 2.3 and Section 3.2"}],"minor_comments":[{"comment":"The abstract states that experiences optimized by SEO 'can achieve consistently improved performance,' but Section 4 acknowledges 'except for one case' and Table 2 shows SEO below Direct for En-Zh Llama-2-13b. Please align the wording (e.g., 'generally' or 'in all but one case').","section":"Abstract and Section 4"},{"comment":"The difference between the stochastic-validation setting (45.2 test, 51.0 dev) and the fixed-validation setting (44.4 test, 47.0 dev) is small, and the text says fixed validation 'may lead to less effective experience.' Without repeated runs, this conclusion is not supported; please soften the claim or provide variance estimates.","section":"Table 8"},{"comment":"The matrix in Figure 3 reports differences in COMET scores when an experience optimized for one direction is applied to another direction, but the row and column labels are somewhat confusing as printed. Please make explicit whether rows correspond to the experience's training direction and columns to the tested target direction, and clarify the color scale.","section":"Figure 3"},{"comment":"The error-type analysis reports differences between SEO and Direct (e.g., Delta Correct = 10, 14, 6) without giving the base counts of each error type in the Direct condition. Please include the total number of dev examples and the per-type counts for each method, so the reader can assess the magnitude of these differences.","section":"Section 5.2, Table 5"},{"comment":"The paper says code and optimized experiences will be released 'upon acceptance.' Given that the experiments depend on specific API model versions (gpt-3.5-turbo-0613, gpt-4-0613), releasing the optimized experiences and, where possible, the exact prompts and sampling seeds as supplementary material would substantially improve reproducibility.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The main concern is statistical: the reported improvements are plausible but not demonstrated, because the evaluation protocol involves many candidate selections and no variance estimates. The one acknowledged En-Zh exception is a concrete symptom of this. If the authors can supply repeated runs with error bars or confidence intervals, a significance test for the main comparisons (at least on HotpotQA and the larger MT differences), and a more measured statement of the consistency claim, the paper could be publishable. I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimately new piece of prompt/experience optimization, and the key new mechanism—the stochastic validation filter—is supported by the ablation. But the empirical claims are presented without any uncertainty quantification, and the single acknowledged exception looks like what you'd expect from selection noise. It deserves a serious referee, and it needs revision.\n\nThe new thing is the validation step. Previous work like CLIN and ExpeL refine experience from trial comparisons but accept updates without a performance check. SEO generates k candidates per example, scores them on a freshly sampled 50-example set, and only updates if a candidate beats the current experience. The ablation is the most convincing part of the paper: removing the validation step drops HotpotQA EM from 45.2 to 37.6, essentially back to Direct's 37.2. That tells you the filter is doing the work, not just the extra prompt engineering. Feeding the per-example effect delta into the optimizer is a small but sensible addition. I also give the authors credit for acknowledging the En-Zh Llama-2-13b exception, for testing OOD transfer, and for a reasonable reimplementation of the baselines. The design avoids test-label contamination: the dev set is held out, and the search never optimizes against test labels. For a purely empirical paper, that's a clean setup.\n\nThe soft spots are exactly where the stress-test note points. There are no error bars, no repeated runs, no significance tests. With m=50, a binary metric has a standard error up to roughly 7 points; many of the reported gains, especially in COMET (0.1 to 0.3 points), are plausibly within metric noise. The final experience is selected from two optimizer runs (batch size 1 and 3) plus up to 200 steps of candidate sampling, all on a 200-example dev set. That's a lot of comparisons, and it invites selection of a lucky experience. The one failing cell—En-Zh Llama-2-13b at 73.23 vs Direct 73.41—is exactly the kind of result you'd expect if some fraction of the 'consistent improvement' is selection noise. The paper's own language in the abstract ('consistently improved') is stronger than what the data show; the body properly says 'almost all tasks.' I also wish the code and optimized experiences were available at submission; for a method whose value is empirical, that's a real gap.\n\nThe central idea is not in doubt. The validation step as a filter is a sound engineering idea, and the paper is worth refereeing. But the quantitative claims need hardening: repeated runs or bootstrap intervals, a proper multiple-comparison correction, and a caveat on the 50-example proxy. My recommendation: send it out, with a request for major revisions focusing on uncertainty quantification and code release.","headline":"Useful new twist on prompt optimization with an honest ablation, but the headline claim outruns the statistics.","tokens_in":19044,"tokens_out":3326,"would_cite":false,"duration_ms":32437,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM can search over natural-language rule lists, keep only edits that pass a fresh 50-example check, and thereby improve its own task performance without any parameter update.","keywords":["stochastic experience optimization","LLM inference-time adaptation","natural-language rule optimization","prompt optimization","stochastic validation","model-specific experience","multi-hop question answering","machine translation"],"falsifier":"Run the same SEO loop on one task and record, at every accepted update, whether the held-out dev-set score also improves; if accepted updates regularly lower the dev score or perform no better than randomly selected candidate experiences on the held-out test set, the 50-example stochastic validation is not identifying a reliable update direction.","tokens_in":17963,"feed_emoji":"🧠","tokens_out":11240,"duration_ms":89484,"temperature":0.7,"pith_summary":"SEO is an iterative procedure for finding a short list of natural-language rules—an 'experience'—that makes a given large language model perform better at a given task, without changing the model's weights. The paper claims that the resulting experiences reliably beat both the no-experience baseline and the initial experience across three tasks, three generator LLMs, and seven datasets. The authors argue that the method works because every candidate rewrite is screened by a stochastic validation set before it is kept, giving each optimization step a concrete direction rather than relying on the optimizer's guess about what should help. If correct, this makes inference-time prompting a tunable asset, so frozen or API-only models can be adapted to a task by growing a better instruction list.","feed_headline":"No retraining: rule-list experiences lift LLM scores on three tasks","feed_subtitle":"An optimizer LLM edits rule lists, keeping only changes that pass a fresh 50-example check, beating handcrafted prompts.","key_machinery":"The load-bearing object is the 'experience': a compact natural-language rule list inserted into the generator model's prompt. The mechanism that carries the argument is the stochastic validation step, which simulates a descent direction for an optimization problem where no true gradient exists: a candidate experience is accepted only if its score on a freshly resampled 50-example validation set exceeds the current experience's score on that same set, and the training loop otherwise keeps the old experience. This acceptance rule is what distinguishes SEO from earlier reflection-based experience learners, which update experiences without checking whether the update actually helps.","core_discovery":"The paper's central claim is that an 'experience'—a list of roughly ten general rules written in natural language—can be treated as the analogue of model parameters and optimized for a specific LLM on a specific task. At each step, the generator LLM answers the same training question with and without the current experience, and the optimizer LLM sees the question, the correct answer, both trial outputs, and the score difference $\\delta$ between them, then samples $k$ candidate rewrites. Every candidate is scored on a freshly sampled 50-example validation subset of the training data, and the experience moves forward only if the best candidate beats the current experience's score on that subset; otherwise the old experience is kept. The authors report that this loop improves exact-match accuracy on HotpotQA, COMET scores across four WMT23 translation directions, and accuracy on SST-2 and CoLA for GPT-3.5, Llama-2-13b, and Llama-2-7b, and that the optimized experience transfers to an out-of-distribution multi-hop QA dataset and across translation directions.","pith_inferences":["If the 50-example validation proxy is as reliable as the paper's results suggest, the same loop could be applied to other prompt attachments—few-shot example selection, chain-of-thought scaffolding, output-format instructions, or safety constraints—treating any text in the context window as a tunable parameter.","A testable consequence of the design is that SEO's advantage should shrink as the generator model approaches its own performance ceiling, a pattern already visible in the small MT gains for GPT-3.5; one could measure this directly by running the loop on increasingly strong generators.","The cross-lingual MT results imply that some learned rules are language-general; an extension the paper does not run is to test whether a HotpotQA-optimized experience transfers to other reasoning benchmarks, not just 2WikiMultiHopQA, and whether transfer decays with task distance.","Because the optimizer LLM is GPT-4 while all generators are weaker, an open question the paper leaves implicit is how much of SEO's success depends on the optimizer being stronger than the generator; a direct test would pair each generator with an optimizer of matched strength."],"forward_implications":["On HotpotQA, the SEO-optimized experience raises exact-match accuracy for GPT-3.5 from 37.2 to 45.2, for Llama-2-13b from 19.4 to 29.8, and for Llama-2-7b from 19.0 to 22.6, and it beats both the CLIN and ExpeL baselines.","On MT, SEO improves COMET over the direct, no-experience baseline in essentially all model/language-direction combinations; for the weaker Llama-2-7b model it also reduces failure-to-translate cases, for example from 143 to 8 in the En-Zh direction.","The optimized experience generalizes out of distribution: experience optimized on HotpotQA lifts exact-match on 2WikiMultiHopQA from 23.0 to 35.4 for GPT-3.5 and from 20.2 to 27.0 for Llama-2-13b.","When validation is removed, HotpotQA test accuracy drops from 45.2 to 37.6, and a fixed validation set performs worse than the stochastic one, which the paper reads as evidence that the random resampling and acceptance rule are doing the work.","Experience optimized for one translation direction typically improves other directions for Llama-2-7b, and in some cases beats the experience optimized for the target direction itself."],"supporting_citations":[{"why":"Closest prior experience-learning baseline (ExpeL) that updates experiences without a validation step; the paper compares against it and also uses its human-crafted HotpotQA insights as an alternative initialization.","marker":"Zhao et al., 2023a"},{"why":"Baseline (CLIN) that iteratively learns experiences from trials without validation; shows what happens when updates are not screened.","marker":"Majumder et al., 2023"},{"why":"Prompt-optimization method whose candidate-sampling approach SEO follows for generating k candidate experiences.","marker":"Zhou et al., 2023"},{"why":"Textual 'gradient descent' prompt optimizer that motivates the descent analogy and the use of sampled validation data for selection.","marker":"Pryzant et al., 2023"},{"why":"LLM-as-optimizer framework (OPRO) that provides the optimizer-prompt formulation with scores, which SEO adapts for experience updates.","marker":"Yang et al., 2024"},{"why":"Source for the difficulty of computing exact gradients in natural-language optimization and for the practice of validating on 50-100 examples.","marker":"Tang et al., 2024"},{"why":"Supplies the HotpotQA dev/test splits and few-shot examples used for training, evaluation, and one initialization condition.","marker":"Trivedi et al., 2023"},{"why":"Source of the HotpotQA dataset used for multi-hop QA experiments and the official dev set from which training data is drawn.","marker":"Yang et al., 2018"},{"why":"2WikiMultiHopQA is the out-of-distribution dataset used to test whether HotpotQA-optimized experience transfers.","marker":"Ho et al., 2020"}],"fun_headline_variants":["LLMs improve via stochastically validated rule-list tweaks","No parameter updates: iterative rule edits lift LLM accuracy","Stochastic experience optimization: fresh 50-example checks per step","Rule lists as learnable experiences: SEO iterates to better prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's steering depends on a randomly sampled 50-example validation set from the training data being a trustworthy proxy for what will help on the held-out test distribution.","fun_headline_variants_meta":{"raw":{"variants":["LLMs improve via stochastically validated rule-list tweaks","No parameter updates: iterative rule edits lift LLM accuracy","Stochastic experience optimization: fresh 50-example checks per step","Rule lists as learnable experiences: SEO iterates to better prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1853,"prompt_tokens":914,"completion_tokens":939,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":867}},"tokens_in":530,"tokens_out":939,"duration_ms":8717,"temperature":1.0,"reasoning_tokens":867,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:33:29.289271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SEO loop on one task and record, at every accepted update, whether the held-out dev-set score also improves; if accepted updates regularly lower the dev score or perform no better than randomly selected candidate experiences on the held-out test set, the 50-example stochastic validation is not identifying a reliable update direction.","supporting_citations":[{"cited_title":"Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers","cited_arxiv_id":"2402.17564","evidence_quote":"Source for the difficulty of computing exact gradients in natural-language optimization and for the practice of validating on 50-100 examples."}],"review_version":1}