{"id":"9469a7b1-674e-46c3-aa28-b7034987319c","arxiv_id":"2412.09722","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A gradient-based discrete prompt optimizer that uses reasoning chains to let small LMs self-optimize prompts, outperforming text-feedback baselines on reasoning benchmarks.","lead":"GReaTer is a prompt optimization method that lets smaller language models improve their own prompts using loss gradients computed over their generated reasoning, without needing a large proprietary model. On GSM8K, BBH, and FOLIO, prompts tuned this way beat several state-of-the-art text-based prompt optimizers and often transfer well to other models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4's top-k intersection can be empty; Algorithm 1 has no fallback, so the candidate-selection step is undefined (or silently no-ops) whenever the q sampled inputs disagree.","rationale":"The central claim is that gradient-guided token selection over reasoning lets small models optimize prompts reliably. For that to be true, Eq. 4 must yield a non-empty candidate set at each optimized position; otherwise the replacement rule in Eq. 7 has no candidates to score. The paper's hyperparameters (k=10, q=5) are not justified against this requirement, and no fallback is described in Algorithm 1, Section 4.2, or the appendix. The reader's CONDITIONAL verdict already captures this and the lack of significance testing. I agree that this is the primary structural weakness. The evaluation limitations (no error bars, unreported seed for the 'random' BBH-5 subset) are real but secondary: they affect how strongly the empirical claim is supported, whereas the empty-intersection issue affects whether the algorithm is even well-defined as written. A code-level check can settle it. If the implementation already handles empty intersections (e.g., union or skip), the concern reduces to a documentation gap; if not, the results need qualification.","tokens_in":22687,"tokens_out":11648,"duration_ms":123360,"concrete_test":"Run the released GReaTer code with k=10, q=5 on all reported tasks (21 BBH, GSM8K, FOLIO) and instrument the candidate-proposal stage to log |candidates_i| for every position and every iteration. Count how often the intersection is empty and what the code path does in that case. Then repeat with k=5 and q=10; if empty intersections occur with no explicit fallback, or if their frequency is substantial, the algorithm as specified is not well-defined and the reported results need a documented repair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 and Eq. 4 define the candidate set at each prompt position as the intersection of per-sample top-k token sets: candidates_i = ∩_{x_j∈D_q} cand_{i,j}. With k=10 and q=5, this intersection can be empty whenever the five sampled inputs disagree on the top-10 continuations of the current prefix. The paper gives no fallback: Algorithm 1 line 8 builds a one-hot indicator for candidates_i, and line 14 takes top-μ of −∂L/∂ε_i; an empty candidate set makes these operations undefined (or, if the implementation silently includes the current token, the position becomes a no-op that cannot be improved). The claim of consistent improvement therefore depends on the unstated assumption that this intersection is non-empty at every optimized position for every task, an assumption that is not checked or reported. Since the method optimizes many positions over T=105 steps, even a modest empty-intersection rate can stall optimization and make the reported gains non-reproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GReaTer, a prompt optimization method for small language models that uses gradient information over generated reasoning to select replacement tokens at each prompt position. Token candidates are proposed by taking the intersection of the top-k token sets over a small sample of inputs; the model generates a reasoning chain and answer logits, and the loss gradient with respect to a one-hot token indicator guides the choice among candidates. Experiments on GSM8K, BBH, and FOLIO with Llama-3-8B-Instruct and Gemma-2-9B-it report consistent improvements over text-feedback baselines, better transferability across model sizes, and in some cases parity with prompts optimized by larger proprietary models. The paper also includes ablations, initial prompt sensitivity analysis, and a comparison with very small models.","tokens_in":22885,"tokens_out":7362,"duration_ms":73037,"significance":"If the results are reliable, the method demonstrates that small, open-source models can self-optimize prompts without relying on large proprietary LLMs, which is a practically important contribution. The idea of combining discrete prompt search with a reasoning-aware loss is novel, and the public code release is a strength. The main tables cover a wide range of tasks and two backbone models, and the transferability experiments give additional evidence for the method's usefulness. However, the core algorithm has an undefined case (empty candidate intersection) that needs to be addressed, and the absence of error bars or significance tests makes it difficult to assess the strength of the claimed consistent gains.","major_comments":[{"comment":"The candidate set at each prompt position is defined as the intersection of per-sample top-k token sets: candidates_i = ∩_{x_j∈D_q} cand_{i,j}. This intersection can be empty when the q sampled inputs disagree on the top-k continuations, and Algorithm 1 provides no fallback: line 8 builds a one-hot indicator over an empty set, and lines 14-15 perform gradient selection over no candidates, leaving the update undefined. The paper neither proves non-emptiness nor reports the frequency of empty intersections, yet the algorithm is claimed to run for T=105 steps at every position. This is a load-bearing gap in the method description; please add a fallback (e.g., current token or union) and report how often it is triggered.","section":"Section 4.2, Eq. (4), Algorithm 1 line 7"},{"comment":"The central claim of consistent gains rests on single-run accuracy values without error bars, standard deviations, or significance tests. In Table 1, the gains on FOLIO/Gemma-2-9B (69.1 vs. 67.5) and GSM8K/Gemma-2-9B (89.4 vs. 88.6) are small; without variance estimates, these differences may not be reliable. Please report scores over multiple seeds (at least three) or a significance test for the main tables.","section":"Tables 1 and 6, Figure 6"},{"comment":"The five BBH tasks are described as \"randomly selected,\" but no random seed or deterministic selection protocol is provided. Although the task names are listed, the selection process is not reproducible, and the comparison to prompts optimized by large proprietary models is based on this subset. Provide the seed or a justification for the chosen subset.","section":"Section 5.3, Tables 2 and 3"}],"minor_comments":[{"comment":"The description of the one-hot indicator is unclear: it states \"with a value of one only for pi and zeros for all other candidates,\" which conflicts with the need for gradients over all candidate tokens. Clarify that ϵ_i is a vector over the candidate set, with a one at the index of the current token.","section":"Section 4.2"},{"comment":"The set D used in lines 10, 12, and 15 is not defined in the algorithm input; it should be either D_task or D_q consistently.","section":"Algorithm 1"},{"comment":"When the prompt length changes dynamically (as described in Appendix B), the modulo operation should use the current prompt length rather than length(p_init).","section":"Algorithm 1, line 19"},{"comment":"The notation \"arg max_{ϵ_i}\" is not well-defined for a vector; use a set of candidate indices or a similar clear formulation.","section":"Section 4.4, Eq. (7)"},{"comment":"The ablation study of \"Gradient Over Reasoning\" reports only two BBH tasks; adding more tasks would strengthen the claim that the reasoning component is essential.","section":"Figure 3"},{"comment":"The paper does not specify whether the reported scores in Table 1 are from a single run or averaged; please clarify the number of runs in the experimental setup.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically relevant problem, and the empirical claims are plausible. The empty-intersection issue in the core algorithm is a clear correctness gap but is fixable with a small modification or an explicit empirical study. The lack of error bars and the non-reproducible random subset selection are standard fixable concerns. I recommend major revision; the paper does not require a new conceptual framework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Have you seen GReaTer? The core idea is genuinely new and it mostly works. Gradient-based discrete prompt search is not new—AutoPrompt, PEZ, GCG—but those methods target classification or jailbreak settings with fixed templates. GReaTer extends the idea to reasoning tasks by generating a chain of thought, extracting the final answer logits, and backpropagating through the reasoning to score candidate token replacements. That is a real extension, and the ablation in Figure 3 supports the claim that the reasoning part is what makes the gradient useful.\n\nThe paper does several things well. The comparison against text-feedback baselines (APE, APO, PE2, TextGrad) is fair in the sense that all methods use the same small optimizer model, and GReaTer consistently wins or ties. The transfer experiments—optimized prompts moving across models, and from small to larger—are a useful addition, and the claim that small-model-optimized prompts can beat GPT-4-optimized prompts is interesting even if the margins on GSM8K are thin.\n\nNow the soft spots. First, there are no error bars or significance tests anywhere. Some of the headline gains are small: Gemma-2-9B goes from 88.6 to 89.4 on GSM8K; that is a 0.8-point difference and could easily be noise. Second, the 'five randomly chosen BBH tasks' in Tables 2 and 3 are not reproducible—no seed, no selection procedure. That is a research-completeness issue, not a fatal one. Third, and more substantive: Eq. 4 defines the candidate set as the intersection of per-sample top-k sets. With k=10 and q=5, that intersection will sometimes be empty, and Algorithm 1 has no fallback. If the implementation silently falls back to the current token, that position becomes a no-op; if it produces an empty gradient, the optimization stalls. The paper does not report how often this happens or what the code does. Given the method optimizes every position over 105 steps, this needs to be addressed before I'd fully trust the numbers.\n\nThe comparison to AutoPrompt and PEZ is limited to two tasks (Appendix C), so the 'reasoning extension' novelty is plausible but not thoroughly explored. I'd have liked a broader comparison.\n\nOverall: the central argument holds up. The method is clever, the ablation is convincing, and the results are probably real. But the missing fallback and lack of variance estimates make it a conditional accept rather than a clear pass. If you work on prompt optimization or use small LMs in production, this is worth reading and citing. I'd send it to a serious referee—ideally one who will check the code for the empty-intersection case.","headline":"Gradient-over-reasoning prompt optimization is a genuine advance, and the reported gains are plausible, but the paper needs an explicit fallback for empty candidate sets and some statistical rigor.","tokens_in":23426,"tokens_out":2639,"would_cite":true,"duration_ms":27283,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GReaTer claims that small open-source language models can optimize their own prompts by computing loss gradients through generated reasoning chains, beating text-feedback prompt optimizers that rely on GPT-4 on GSM8K, BBH, and FOLIO.","keywords":["prompt optimization","gradient over reasoning","small language models","discrete token search","chain-of-thought","reasoning tasks","transferability"],"falsifier":"Run GReaTer with a configuration where the top-k candidate intersection across the q sampled inputs is empty at some position (for example, k=1 or a large q); if the procedure has no fallback, it is ill-defined at that step. Alternatively, replace the gradient-ranked token choice of Equation 7 with a random choice from the same candidate set: if task accuracy does not drop, the loss gradient is not what drives the reported gains.","tokens_in":22488,"feed_emoji":"⚙️","tokens_out":4767,"duration_ms":50682,"temperature":0.7,"pith_summary":"The paper claims that prompt optimization does not have to depend on large, closed-source language models for textual feedback. GReaTer lets a small open-source model improve its own prompts by turning promising token suggestions into discrete candidates and ranking them with true numerical gradients, computed by backpropagating the task loss through the model's generated reasoning chain. Across GSM8K, BBH, and FOLIO reasoning benchmarks, the authors report that prompts found this way consistently outperform those found by state-of-the-art text-feedback methods, including baselines that use GPT-4 as the optimizer. If correct, this would let smaller models self-optimize for reasoning tasks more cheaply and without any reliance on expensive proprietary judges.","feed_headline":"Gradients over reasoning let small LMs optimize their own prompts","feed_subtitle":"Token-level gradient search beats text-feedback baselines on reasoning tasks without any closed-source optimizer.","key_machinery":"The load-bearing mechanism is the 'gradient over reasoning' computation: the loss is taken from final answer logits after the model has generated a reasoning chain, so the backward pass flows through that reasoning and guides token selection. This is coupled with a candidate proposal stage that projects the vocabulary down to the intersection of top-k tokens over q sampled inputs, encoded as a one-hot token indicator so the discrete prompt becomes locally differentiable.","core_discovery":"GReaTer's central claim is that a smaller language model can serve as its own prompt optimizer if the optimization signal is a real loss gradient rather than natural language critique. The paper proposes token candidates at each prompt position using the model's own top-k probabilities over a handful of task inputs, then generates a reasoning chain for each sample, extracts final answer logits through a formatted extractor prompt, and computes a cross-entropy loss with a perplexity regularizer. The gradient of that loss with respect to a one-hot token indicator at the position is used to rank the candidates, with the top few candidates verified by a forward pass. The authors report that this 'gradient over reasoning' approach outperforms APE, APO, PE2, and TextGrad on GSM8K, BBH, and FOLIO when optimizing Llama-3-8B-Instruct and Gemma-2-9B-it, and that the resulting prompts transfer across small models and to larger models.","pith_inferences":["The same candidate-proposal-plus-gradient machinery could be reused for other discrete text optimization problems, such as instruction selection in agents or few-shot example mining, whenever an end-to-end loss over model output is available.","A clean causal test of the 'reasoning' component would be to freeze the candidate set and replace gradient ranking with random selection: if the accuracy gain vanishes, the reasoning-aware gradient is what carries the result, not the candidate proposal stage alone.","Because the paper notes that some optimized prompts read awkwardly, layering a fluency constraint or dynamic top-k on top of the gradient selection is a natural extension that could make the method more deployable.","The reported success on a 1B-parameter model suggests the approach could eventually enable on-device prompt adaptation where no external optimizer is reachable."],"forward_implications":["Small open-weight models can improve their own reasoning prompts without calling closed-source LLMs for feedback, removing a major cost and dependency.","Because each iteration costs one forward and one backward pass per sample, prompt optimization becomes cheaper than text-feedback methods that generate long critiques on chained error samples.","Optimized prompts transfer across small models and to larger models, so one optimization run on a small model can serve multiple deployment targets.","Gradient-guided discrete token search, previously practical only for fixed-input-output classification settings, is extended to free-form reasoning tasks with generated reasoning chains.","The performance gains reported are consistent and often exceed those from prompts optimized with GPT-4, PaLM-2-L, and GPT-3.5 on the same tasks."],"supporting_citations":[{"why":"Introduces the 'textual gradient' concept and APO, a primary baseline that GReaTer outperforms on most tasks.","marker":"Pryzant et al., 2023"},{"why":"TextGrad is a state-of-the-art text-feedback baseline that GReaTer compares against and whose library the authors use for evaluation.","marker":"Yuksekgonul et al., 2024"},{"why":"APE is the original LLM-as-prompt-optimizer baseline used as a comparison point.","marker":"Zhou et al., 2022"},{"why":"PE2 is a baseline, and the paper uses their implementations for APO, APE, and PE2, plus GPT-4-optimized prompts for comparison.","marker":"Ye et al., 2023"},{"why":"Provides the gradient-based discrete token search technique on one-hot token indicators that GReaTer adapts for prompt optimization.","marker":"Zou et al., 2023"},{"why":"AutoPrompt is the earlier gradient-guided hard prompt search that GReaTer extends to complex reasoning tasks.","marker":"Shin et al., 2020"},{"why":"Chain-of-thought prompting is the basis for generating the reasoning chain that GReaTer differentiates through in its loss.","marker":"Wei et al., 2022"}],"fun_headline_variants":["Small LMs self-optimize prompts via reasoning gradients","Reasoning gradients let small models master prompt tuning","GReaTer: gradient-guided prompt self-optimization for small LMs","Small language models sharpen prompts with reasoning gradients","Gradients over reasoning: small LMs ditch closed-source optimizers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that at every prompt position being optimized, the common top-k token candidates proposed across the sampled inputs form a non-empty set, leaving at least one token available to score.","fun_headline_variants_meta":{"raw":{"variants":["Small LMs self-optimize prompts via reasoning gradients","Reasoning gradients let small models master prompt tuning","GReaTer: gradient-guided prompt self-optimization for small LMs","Small language models sharpen prompts with reasoning gradients","Gradients over reasoning: small LMs ditch closed-source optimizers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1403,"prompt_tokens":1029,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":290}},"tokens_in":645,"tokens_out":374,"duration_ms":4465,"temperature":1.0,"reasoning_tokens":290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:47:32.223879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GReaTer with a configuration where the top-k candidate intersection across the q sampled inputs is empty at some position (for example, k=1 or a large q); if the procedure has no fallback, it is ill-defined at that step. Alternatively, replace the gradient-ranked token choice of Equation 7 with a random choice from the same candidate set: if task accuracy does not drop, the loss gradient is not what drives the reported gains.","supporting_citations":[],"review_version":1}