{"id":"aa3527e9-7e73-432d-b227-08b495d3b00d","arxiv_id":"2507.10326","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A grammar-guided evolutionary search that composes prompt edits outperformed PromptWizard, OPRO, and RL-Prompt on small LLMs across four domain-specific tasks.","lead":"This paper introduces a way to automatically improve prompts for small language models by evolving them with grammar-guided genetic programming and a local search refinement step. It reports that the method beats three existing prompt-optimization systems on four technical tasks, while the baselines often made small models perform worse.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Budget parity for baselines is asserted but never quantified; the paper's own Section 5.2 concedes RL-Prompt ran with fewer steps than recommended, so the claimed outperformance may reflect compute disparity rather than search quality.","rationale":"The reader's weakest assumption identifies exactly the same issue: the comparison assumes the search budgets were matched, but Section 4.2 provides no step counts, total LLM calls, or runtime, and Section 5.2 explicitly concedes that RL-Prompt was run with fewer steps than recommended. I agree with this assessment, and I think it is the most load-bearing weakness because it directly threatens the headline claim of outperforming three state-of-the-art methods. The proposed test is feasible: the authors already control the evaluation harness, so logging calls and rerunning RL-Prompt at its recommended settings requires no new methodology. If the budget logs confirm parity and the reruns reproduce the reported numbers, the empirical claim stands; if not, the correct conclusion is that G3P+LS is a strong search method but not demonstrated to outperform baselines at equal cost. The rest of the paper -- the grammar design, the surrogate-assisted local search, the failure-mode analysis, and the held-out test evaluation -- remains a solid contribution regardless, so the verdict should remain conditional pending this audit. I would not move to accept or reject because the missing information is recoverable and the method itself is plausible and described in unusual detail.","tokens_in":20495,"tokens_out":6769,"duration_ms":80632,"concrete_test":"Instrument every method to log (i) total task-model forward passes on training and validation data, (ii) total LLM calls used for prompt editing or generation, and (iii) wall-clock time for each of the 12 task-model combinations; rerun RL-Prompt at its authors' recommended number of steps, and rerun OPRO and RL-Prompt with total budgets (LLM edit calls plus training forward passes) matched to G3P+Local Search, over at least five seeds. If G3P+LS is not best or second-best on all 12 combinations under equalized budgets, the \"outperforms\" claim should be withdrawn or weakened to \"competitive at higher compute.\"","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim -- that G3P+Local Search outperforms PromptWizard, OPRO, and RL-Prompt -- rests on an unverified budget-matching assumption. Section 4.2 says the authors \"limit the LLM-based evaluations on training data to match that of our approach\" by using 10 training rows per step, but it never reports the number of steps, the number of candidate prompts per step (except RL-Prompt's 10), the total number of task-model forward passes, the number of LLM calls used for prompt editing, or wall-clock time for any method. This is not a minor reporting gap: G3P's own budget includes 20 generations with a population of 50 and an offspring population of 50, 20 training rows per evaluated individual, one validation evaluation per generation, surrogate training on the resulting data, and a local-search phase that really evaluates 50 neighbours on both validation and training data; in addition, every phenotype containing paraphrase or summarise operations consumes LLM calls before fitness is measured. None of these costs is counted against the baselines.\n\nThe paper itself flags the problem in Section 5.2: for RL-Prompt, \"the reduced number of steps for which RL-Prompt was allowed to run was possibly too few to effectively apply the approach,\" and the authors \"were unable to use as many steps as the authors recommended.\" A baseline that is explicitly run below its recommended budget is not a fair comparator. Consequently, the Table 4 mean relative gains of +56% versus +5%, -12%, and -42% may measure relative compute, not relative search effectiveness. Since the abstract's central claim is \"outperforms,\" this unquantified budget disparity is the most load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes G3P DPO, a two-phase discrete prompt optimisation method aimed at smaller general-purpose LLMs on long, domain-specific prompts. Phase 1 uses grammar-guided genetic programming to evolve sequences of list-manipulation, dictionary-based, and LLM-based edit operations over six prompt sections; phase 2 applies surrogate-screened local search to the best evolved prompt. The method is evaluated on PubMedQA, ETHOS, TAT-QA, and ConvFinQA with Llama3.2 3B, Llama3 8B, and Gemma2 9B, and compared against PromptWizard, OPRO, and RL-Prompt. The authors report that G3P plus local search is the best or second-best optimisation method in all 12 task-model combinations, with a mean relative gain over the baseline prompt of +56%, versus +5%, -12%, and -42% for the three baselines.","tokens_in":20833,"tokens_out":9625,"duration_ms":102563,"significance":"If the empirical claims are supported, the paper makes a useful contribution. It addresses a genuine gap: most discrete prompt optimisation work assumes very large LLMs and short prompt templates, while this work targets smaller models on long, structured, domain-specific prompts. The use of grammar-constrained search, external task metrics as fitness signals, and a surrogate model to prune local-search candidates are sensible design choices, and the appendices provide useful detail on the grammar, hyperparameters, and baseline prompts. The qualitative failure-mode analysis of the baselines in Section 5.2 is also informative. However, the central contribution is a comparative claim, and the current evidence is weakened by two load-bearing issues: the baseline compute budgets are asserted but never quantified, and all results are single test-set numbers from a stochastic procedure with no repeated seeds or confidence intervals. These issues should be addressed before the comparative claims can be accepted.","major_comments":[{"comment":"The claim that OPRO and RL-Prompt were run at matched compute is not substantiated. Section 4.2 says these baselines were limited to 10 training rows per step 'to match that of our approach', and that RL-Prompt used 10 candidate prompts per step, but no step counts, total numbers of LLM forward passes, total numbers of LLM edit calls, or runtimes are reported for any method. G3P's own budget is substantial: 20 generations with population and offspring sizes of 50, 20 training rows per evaluated individual, one validation evaluation per generation, surrogate-model training, and a local-search phase that really evaluates 50 neighbours on both validation and training data. In addition, every phenotype containing paraphrase or summarise operations consumes LLM calls before fitness is measured. Section 5.2 explicitly concedes that RL-Prompt was run with fewer steps than its authors recommend. Because the headline result in Table 4 is a comparison of search methods, the evaluation budgets must be quantified and actually matched; as written, the reported superiority could be an artefact of compute disparity rather than search design. For PromptWizard, no budget information is reported at all beyond 'default settings'.","section":"Sec. 4.2 and Table 4"},{"comment":"All test results are single numbers from a single run, although the search procedure is stochastic in several places: training rows are resampled every generation, the initial population is generated with PTC2, local-search neighbours are constructed from randomly selected index perturbations, and the surrogate ensemble uses bootstrap sampling. Figures 3 and Appendix G plot the mean and standard deviation across individuals within one run, not across repeated runs. Several headline differences are small: for ETHOS with Gemma2 9B, G3P plus local search gives 83.7 versus 83.5 for RL-Prompt and 83.0 for the baseline prompt; for TAT-QA with Gemma2 9B, G3P plus local search gives 40.6 versus 40.5 for the baseline prompt. Without repeated seeds, confidence intervals, or a paired significance test, the statements that G3P plus local search is 'best or second-best for all combinations' and that the approach 'outperforms' three baselines are not quantitatively supported. This is a load-bearing issue because the paper's central claim is comparative and the observed margins include cases within plausible run-to-run noise.","section":"Table 4, Sec. 3.3, Sec. 3.4"}],"minor_comments":[{"comment":"The abstract's 'outperforms three state-of-the-art prompt optimisation approaches' is stronger than Table 4, where PromptWizard achieves the highest score in 3 of 12 task-model combinations (PubMedQA with Llama3 8B, ETHOS with Llama3.2 3B, and TAT-QA with Llama3 8B); please qualify the claim as an average or overall comparison.","section":"Abstract and Sec. 5.1"},{"comment":"For ConvFinQA with Gemma2 9B, G3P plus local search (54.5) remains 12% below the baseline prompt (61.9); the abstract's 'only incurring minimal degradation' and the 'best or second-best' framing should state explicitly that the ranking is among optimisation methods and should discuss this case.","section":"Table 4 and Sec. 5.1"},{"comment":"The phrase 'sorrugated ensemble' appears to be a typo for 'surrogate ensemble'.","section":"Sec. 3.4"},{"comment":"The production for ⟨cot_expr⟩ has mismatched angle brackets (given as ⟨cot_expr>⟩), and several terminal strings in the grammar are sentiment-classification specific despite the grammar being presented as general; please clarify whether these are illustrative placeholders.","section":"Appendix C"},{"comment":"The x-axis in the training-fitness plots ends at generation 15, while Appendix D specifies 20 generations; if the plots are truncated or generations are indexed from 0 to 19 with a display quirk, this should be explained.","section":"Appendix G and Fig. 3"},{"comment":"The phrase 'syntactic comparison to ground-truth labels' is potentially misleading for TAT-QA and ConvFinQA, where evaluation requires executing or parsing formulas; please clarify how the metrics are computed for these tasks.","section":"Sec. 4.1"},{"comment":"The paper does not state code or data availability and does not list the random seeds used; given the many hyperparameters in Appendices D and E, releasing code and seed configurations would materially improve reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a CL or benchmarking venue, and the small-model/long-prompt setting is a reasonable distinct contribution. The main risk is that the comparative claims rest on unverified budget parity and on single runs of a stochastic algorithm; both are fixable with additional reporting and repeated runs. I would not reject the paper on novelty grounds, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for anyone in discrete prompt optimization, especially the small-model end. The paper's real contribution is the combination: grammar-guided genetic programming over prompt-edit programs, with dynamic chunking levels and a surrogate-assisted local search phase. That specific mix is not in the cited PLUM, SAMMO, or GRIPS lines, and the authors are honest that the ingredients are known. They also target a genuine gap—long, domain-specific prompts on 3B–9B models—and evaluate on held-out test sets with ground-truth labels, avoiding LLM-as-judge circularity.\n\nThe failure-mode analysis in Section 5.2 is the best part. Watching PromptWizard inject a training question into the prompt, or OPRO turn 'hateful/not hateful' into 'Positive/Negative', is instructive and fairly reported. The authors do not hide that the baselines can collapse.\n\nNow the soft spots, in proportion. The budget-parity claim is load-bearing and it is not actually established. Section 4.2 says evaluations were limited to match, but no step counts, total LLM calls, or runtimes are reported anywhere. Section 5.2 then admits RL-Prompt ran with fewer steps than recommended. That undercuts the headline 'outperforms' claim—you cannot tell whether the +56% mean relative gain reflects search quality or compute disparity. This is not a minor omission; it is the central comparison.\n\nSecond, there are no error bars. Every number is a single test-set run despite stochastic training resampling and population initialization. Some gains are tiny (ETHOS Gemma2: 83.7 vs 83.0), so it is hard to tell which results are real. Third, no code or data released, which would help calibrate these concerns.\n\nThe method itself is plausible. The search is described in unusual detail, the grammar is in the appendix, and the fitness signal is external accuracy, not self-evaluation. The authors also state their own limitations clearly. I would not call the paper circular or incoherent—the thinking is solid.\n\nWho gets value: researchers or practitioners working on prompt optimization for smaller deployed models, and people building on evolutionary DPO. The paper deserves a serious referee, but the revision must supply actual budget logs, repeated seeds with variance, and ideally code and prompts. Without those, the empirical contribution is a strong existence proof, not a validated comparison.","headline":"A promising but under-verified empirical recipe for prompt optimization on small LLMs: the method is genuinely novel in combination, but the superiority claim rests on unquantified budget matching and single test-set numbers.","tokens_in":21435,"tokens_out":2407,"would_cite":true,"duration_ms":28802,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grammar-guided evolutionary search reliably optimises long, detailed prompts for small language models, outperforming three leading prompt optimisers that often degrade the same task-model combinations.","keywords":["discrete prompt optimisation","grammar-guided genetic programming","evolutionary search","local search","surrogate model","small language models","prompt sensitivity"],"falsifier":"Rerun PromptWizard, OPRO, and RL-Prompt on the same 12 task-model combinations with the total number of LLM evaluation calls equalised to the G3P + Local Search budget (20 generations x 50 individuals evaluated on 20 training rows, plus 50 local-search evaluations), reporting step counts and wall-clock time; if any baseline reaches or exceeds the +56% mean relative gain under matched calls, the claimed superiority would be shown to reflect budget allocation rather than search design.","tokens_in":20320,"feed_emoji":"🧬","tokens_out":9461,"duration_ms":96722,"temperature":0.7,"pith_summary":"Large language models become more sensitive to prompt wording as they get smaller, and the automated prompt optimisers that work on big models often collapse on small models and on long, detail-heavy prompts. This paper argues that the fix is to stop asking the model to edit its own prompt and instead run a grammar-guided evolutionary search over small, discrete edits to a manually written prompt, evolving the edit program rather than the prompt text. Across four domain-specific tasks and three small models (Llama3.2 3B, Llama3 8B, Gemma2 9B), the method improves on the baseline prompt in 10 of 12 task-model pairs, and a follow-up local search phase makes it the best or second-best method in all 12. If this holds, practitioners can deploy small, cheap models on specialised tasks without fine-tuning and without the performance cliff that current prompt-automation tools impose on them.","feed_headline":"Grammar-guided evolution beats three prompt optimizers","feed_subtitle":"On small LLMs, the search lifts scores a mean 56 percent while rivals often degrade them.","key_machinery":"The central machinery is a Backus–Naur-form grammar whose productions compose prompt-editing functions into executable programs, a setup the paper calls G3P DPO. Each individual in the population is an integer genotype that maps through the grammar to a phenotype: a set of edit sequences, one per prompt section, each a tree of list-manipulation and semantically aligned operations (LLM-based paraphrase and summarise, dictionary-based stop-word removal and synonym substitution) at word, sentence, or phrase chunking levels. Evolution runs 20 generations of 50 individuals with tournament selection, subtree crossover, and subtree mutation, evaluating fitness on 20 training rows per generation; a post-hoc local search then perturbs each index parameter of the best program, screens the resulting neighbours with a 10-model neural ensemble surrogate trained on Sentence-BERT embeddings of prompts, and real-evaluates the 25 highest-mean and 25 highest-variance neighbours.","core_discovery":"The paper claims that discrete prompt optimisation for small LLMs should be treated as a program-synthesis problem, not a text-editing problem. Its G3P approach evolves sequences of edit operations — swap, remove, re-add, duplicate, paraphrase, summarise, stop-word removal, and synonym substitution — applied at word, sentence, or phrase granularity to six modular sections of a base prompt (persona, task, output format, in-context demonstrations, context, chain-of-thought). A formal grammar restricts the space so every candidate is a syntactically valid prompt-producing program, and fitness is the target model's accuracy on a sample of training rows. The authors report a mean relative gain of +56% over the base prompts (against +5% for PromptWizard, −12% for OPRO, and −42% for RL-Prompt), best-or-second-best results in all 12 task-model combinations, and a narrowing of the performance gap between the weakest and strongest models from 35.6 points to 10.5, because the weakest model gains the most. They also document specific failure modes of the baselines — PromptWizard injecting a single training question into the instruction, OPRO rewriting class labels into invalid values, RL-Prompt producing near-zero performance — and argue that evolutionary selection and elitism naturally filter out such destructive edits.","pith_inferences":["The grammar's modular section structure could double as an automatic ablation tool: which sections the evolved program keeps, edits, or drops gives a per-task diagnosis of what a small model actually needs in its prompt.","A natural extension is to seed the population with the baselines' own best outputs, or to add the information-injecting operations the paper lists as missing (instruction induction, self-reflection, phrase injection), combining content discovery with G3P's selection pressure.","Because the surrogate is trained only on prompts from one G3P run, its cost-saving benefit should be tested by scaling generations and checking whether the ensemble's screening accuracy keeps pace with the growing neighbourhood.","The claim that selection filters destructive edits could be probed directly by counting how many early-generation low-fitness candidates exhibit the same failure modes documented for the baselines and confirming they are eliminated by selection rather than never generated."],"forward_implications":["Practitioners can optimise long, detail-heavy prompts for small open-source models without fine-tuning, with the largest relative gains landing on the weakest model, such as Llama3 8B rising from 19.6% to 33.5% on ConvFinQA.","The inter-model performance spread across the four tasks drops from 35.6 points on the base prompts to 14.4 with G3P and 10.5 after local search, so the choice among small models matters less once prompts are optimised this way.","Because the grammar makes persona, output-format, and chain-of-thought sections optional, the search itself decides which sections a small model actually needs; 7 of the 12 optimised prompts left at least one section untouched.","The surrogate-assisted local search adds a further improvement in 11 of 12 task-model combinations beyond the evolved individual, meaning the local refinement phase is complementary rather than redundant to the evolutionary search.","The baselines' documented failures — unparseable outputs, invalid labels, and training-set artefacts copied into instructions — show that LLM-self-editing optimisers do not transfer safely to the small-model, long-prompt regime."],"supporting_citations":[{"why":"The PromptWizard baseline the method must beat, and the source of the report that it degrades on Llama3 8B.","marker":"[1]"},{"why":"The OPRO baseline; the paper raises its 500-character prompt limit and caps its training evaluations to match budget.","marker":"[43]"},{"why":"The RL-Prompt baseline, providing the token-selection comparison against edit-based search.","marker":"[8]"},{"why":"Whigham's grammar-guided genetic programming, the framework the search algorithm is built on.","marker":"[41]"},{"why":"SAMMO, the prior source of combined structural, syntactic, and LLM-based edit operations the grammar draws on.","marker":"[36]"},{"why":"PLUM, the metaheuristic prompt search whose list-manipulation operations (swap, remove, duplicate) are reused.","marker":"[30]"},{"why":"The PubMedQA dataset, one of the four evaluation tasks used to compare all methods.","marker":"[16]"},{"why":"The ConvFinQA dataset, one of the four evaluation tasks used to compare all methods.","marker":"[6]"},{"why":"The TAT-QA dataset, one of the four evaluation tasks used to compare all methods.","marker":"[47]"}],"fun_headline_variants":["Grammar-guided evolution finds better prompts for small LLMs","Evolving prompt programs beats text editing by 56% on small models","Prompt optimisation as program synthesis outperforms three baselines","Small LLMs benefit most from grammar-guided prompt search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes the baselines ran on a search budget comparable to G3P's 20 generations of 50 individuals, but the paper caps OPRO and RL-Prompt at 10 training rows per step without reporting step counts, total LLM calls, or runtime, and it concedes in Section 5.2 that RL-Prompt's reduced steps were possibly too few.","fun_headline_variants_meta":{"raw":{"variants":["Grammar-guided evolution finds better prompts for small LLMs","Evolving prompt programs beats text editing by 56% on small models","Prompt optimisation as program synthesis outperforms three baselines","Small LLMs benefit most from grammar-guided prompt search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1651,"prompt_tokens":1066,"completion_tokens":585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":682,"tokens_out":585,"duration_ms":6979,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:33:55.624466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun PromptWizard, OPRO, and RL-Prompt on the same 12 task-model combinations with the total number of LLM evaluation calls equalised to the G3P + Local Search budget (20 generations x 50 individuals evaluated on 20 training rows, plus 50 local-search evaluations), reporting step counts and wall-clock time; if any baseline reaches or exceeds the +56% mean relative gain under matched calls, the claimed superiority would be shown to reflect budget allocation rather than search design.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The OPRO baseline; the paper raises its 500-character prompt limit and caps its training evaluations to match budget."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Whigham's grammar-guided genetic programming, the framework the search algorithm is built on."},{"cited_title":"Schnabel and J","cited_arxiv_id":null,"evidence_quote":"SAMMO, the prior source of combined structural, syntactic, and LLM-based edit operations the grammar draws on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PLUM, the metaheuristic prompt search whose list-manipulation operations (swap, remove, duplicate) are reused."}],"review_version":1}