{"id":"8a88bd14-bfb8-43b3-801f-b505eba99d0e","arxiv_id":"2504.20355","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Restricting prompt edits to flagged tokens yields small accuracy gains over global rewriting in three prompt optimization methods, but the faster-convergence claim is not consistently supported.","lead":"Local Prompt Optimization asks the optimizing model to flag a few prompt words with <edit> tags and then change only those words, instead of rewriting the whole prompt. The reported benchmark gains are modest, and the paper's evidence is single-run, partly contradictory, and missing error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LPO's gains are attributed to a restricted edit scope, but the paper never verifies that the proposal LLM edits only tagged tokens, and it changes the meta-prompt (adding a token-identification step) at the same time, so the claimed search-space reduction may not be the cause.","rationale":"The reader's weakest_assumption and this stress-test pass align. The central claim—that restricting edits to flagged tokens shrinks the search space and improves prompt optimization—requires two conditions. First, the proposal LLM must actually restrict its edits to the marked tokens. Second, the experimental comparison must isolate the edit-scope restriction from the newly added token-identification instruction. Neither condition is checked in the paper. Section 2.2 only gives the instruction; Sections 3 and 4 report aggregate scores without any token-level edit statistics. The abstract and Section 4 assert faster convergence, but Table 2's APE row shows local optimization taking more steps than global, and the 27-subtask convergence plot in Fig. 3a lacks error bars and is based on a single run with a search budget of 3 steps. Since the mechanism is unverified and the comparison is confounded, the headline conclusions are unsupported as stated. This does not mean LPO is wrong; a revised version with the proposed ablation, edit-locality measurements, multiple seeds, and corrected convergence claims could be acceptable. Therefore the reader's REJECT verdict stands.","tokens_in":9825,"tokens_out":4408,"duration_ms":48919,"concrete_test":"Run a three-arm ablation on GSM8k and a BBH subset. Arm A: global baseline (PE2 or APO as reported). Arm B: LPO as in §2.2. Arm C: same meta-prompt as Arm B including the Fig. 2 token-identification step, but with the final instruction changed so the identified tags are only suggestions and the optimizer may edit any token. If Arm C matches Arm B's accuracy and convergence, the gains come from the added identification step, not local scope. Additionally, token-diff the saved Arm B proposals at every step and report the fraction of changed tokens outside the <edit> spans; if this fraction is material, or if tagged spans cover most of the prompt, the 'local' premise fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 defines LPO by asking Mproposal to mark candidate edit tokens with <edit> tags and then to 'Reply with the new instruction without the <edit>, </edit> tags.' The paper reports accuracy and convergence improvements (Tables 2 and Fig. 3) but never measures whether the proposed prompt differs from the previous one only inside the tagged spans. If the optimizer rewrites untagged regions or tags most of the prompt, LPO is effectively global optimization with a modified meta-prompt, and the claimed reduction in search space is not realized. The comparison is also confounded: local runs add the token-identification instruction of Fig. 2, which itself may improve proposals via an extra chain-of-thought step, so even perfect tag compliance would not show that locality—rather than the added reasoning step—drives the gains. The convergence claim is additionally undercut by Table 2's APE row, where local optimization takes 4 steps versus 2.5 for global. Without a token-level edit-locality audit and an ablation that keeps the identification instruction while allowing edits anywhere, the central mechanism is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Local Prompt Optimization (LPO), a wrapper around existing automatic prompt engineering methods such as APE, APO, and PE2. LPO adds a preliminary step in which the proposal LLM marks candidate edit tokens in the current prompt with <edit> tags, and then asks the proposal LLM to return a revised prompt without the tags. The intended effect is to restrict each optimization step to a small, explicitly identified subset of prompt tokens, thereby shrinking the search space and improving both final accuracy and convergence speed. The authors evaluate LPO on GSM8K, MultiArith, BIG-bench Hard, and an internal production prompt, reporting average accuracy gains of 1.5% on math reasoning and 2.3% on BIG-bench Hard, together with faster convergence.","tokens_in":10068,"tokens_out":3284,"duration_ms":35791,"significance":"If the mechanism were verified, LPO would be a useful, low-cost addition to existing prompt optimization pipelines, particularly for long production prompts where controlled editing is desirable. The paper correctly identifies a real limitation of global prompt mutation, and its integration with three existing methods follows a clean experimental template. The authors also deserve credit for using standard train/dev/test splits from PE2 and for stating the limitations of their study candidly. However, the current manuscript does not verify that the proposed mechanism actually operates as claimed, and several load-bearing empirical claims are either contradicted by the reported numbers or rest on unreadable graphical evidence. The contribution is incremental but potentially valuable; the missing verification is the main obstacle.","major_comments":[{"comment":"The central mechanism is unverified. The paper instructs Mproposal to 'Reply with the new instruction without the <edit>, </edit> tags,' but it never measures whether the proposed prompt actually differs from the previous prompt only inside the tagged spans. If the proposal LLM rewrites untagged regions, or tags most of the prompt, LPO degenerates into global optimization with a modified meta-prompt. The authors should provide a token-level edit-locality audit, e.g., the fraction of changed tokens lying outside the tagged spans across optimization steps, for each method and task.","section":"Section 2.2 and Figure 2"},{"comment":"The comparison is confounded by the additional token-identification step. The LPO pipeline changes two variables at once: it adds a chain-of-thought-style identification stage and restricts edit scope. The reported gains could therefore come from the extra reasoning step rather than from locality. An ablation that retains the token-identification instruction but allows the proposal LLM to edit anywhere is needed to attribute the improvements to local optimization specifically.","section":"Section 2.2 and Section 4"},{"comment":"The faster-convergence claim is directly contradicted by the APE row of Table 2: local optimization takes 4 steps versus 2.5 for global optimization on the math reasoning benchmarks. Since the abstract and Section 4 claim that LPO converges faster, this discrepancy must be explained, or the claim must be restricted to the methods and tasks where the evidence supports it.","section":"Table 2"},{"comment":"All reported numbers appear to come from a single run of a stochastic pipeline. LLM-based prompt optimization is highly sensitive to sampling and evaluation variance, and the claimed gains are small (e.g., +0.3% on GSM8K for APE, +0.5% on MultiArith for PE2). The authors should report multiple independent runs with means and standard deviations, or at least the number of seeds, so the reader can judge whether the improvements exceed noise.","section":"Section 3.3 and Table 2"},{"comment":"The BIG-bench Hard and production-prompt claims are supported only by figures, and in the manuscript the figures are rendered as unreadable glyph sequences rather than actual plots. No numerical table or task-level breakdown is provided for the 2.3% average improvement or the 6% production gain. These results must be reported in a machine-readable numerical form, ideally in a table or appendix, before they can be assessed.","section":"Section 4 and Figure 3"},{"comment":"The production-prompt experiment is not reproducible: the prompt is internal, and the paper does not describe the evaluation data, the metric, the number of queries, or the procedure for selecting which sections of the 8k-token prompt LPO is allowed to edit. At minimum, the authors should describe the task distribution, report performance before and after optimization with error bars, and state exactly which prompt sections were targeted.","section":"Section 3.1 and Section 4"}],"minor_comments":[{"comment":"There are several grammatical and typographical errors, e.g., 'the use of prompts ... have increased dramatically' and 'where in all the prompt tokens have to be optimized.' A careful copyedit is needed.","section":"Abstract and Section 1"},{"comment":"The '# steps' column is not defined. The paper should state explicitly what is measured (e.g., the first step at which the best prompt was found, with ties resolved how?) and confirm that lower is better.","section":"Table 2"},{"comment":"The formatting of the instruction 'Reply with the new instruction without the <edit>, </edit> tags.' contains stray spacing and inconsistent quotation marks; this should be cleaned up.","section":"Section 2.2"},{"comment":"The sentence 'We hypothesize that since Local Optimization reduces the optimization tokens ... and introduces a Chain-of-Thought approach' acknowledges the confound but does not test it; this hypothesis should be stated earlier and explicitly treated as a claim requiring ablation.","section":"Section 4"},{"comment":"The limitations section notes that local optimization sometimes drives dev scores close to 99%, suggesting possible overfitting; the paper should discuss whether the reported test improvements are stable under this overfitting and whether early stopping was applied.","section":"Section 6"},{"comment":"The reference list contains formatting issues inherited from the Llama 3 authors list, such as 'Vishal V ogeti' and 'Guangyi, Zhang'; these should be normalized.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I chose major_revision rather than reject because the missing evidence is well-defined and obtainable: a token-level locality audit, an ablation separating the identification step from the edit restriction, multi-seed runs, and numerical reporting for the BBH and production results. If the authors cannot provide these, the central claim that local search-space restriction drives the gains would fail. The current single-run, unverified-mechanism presentation is not publishable as-is, but the idea is worth one more round."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing. First, the core idea is genuinely new in a narrow sense: no prior APE/APO/PE2 method asks the proposal LLM to mark <edit> spans and then limits rewriting to those spans. Second, the paper never checks that the optimizer actually obeys the tags, and its own Table 2 undercuts the faster-convergence headline (APE goes from 2.5 to 4 steps). So the central mechanism is asserted, not demonstrated.\n\nWhat the paper does well: it integrates the local-scope step cleanly into three existing methods, reuses PE2's splits, and reports held-out accuracy. The math gains are internally consistent: the GSM8k/MultiArith rows average to about the claimed 1.5%. The production-prompt result is a nice practical example, and the limitations section honestly flags overfitting and closed-source reproducibility.\n\nThe soft spots are load-bearing. The stress-test concern holds: Section 2.2's instruction is the only thing restricting edits to marked tokens, and there is no token-level audit. The comparison is confounded because local runs add the identification instruction, which itself is an extra chain-of-thought step; the authors even acknowledge this in Section 4 when they credit both 'reduced tokens' and 'CoT' for the gains. The convergence claim is contradicted by their own APE row, and the BBH figure is unreadable in the PDF I have, with no tabular numbers to check the 2.3% average. Single-run evaluations, no error bars, and an internal production prompt round it out.\n\nThe math, data, and citation pattern look solid enough — no circular tuning, no invented baselines. The problem is verification, not integrity.\n\nWho this is for: prompt-engineering practitioners and anyone working on APE variants. As a reviewer, I would not desk reject it; the idea is concrete and the gap is fixable. But as stated, I would ask for a substantial revision: report edit-locality rates, add an ablation that keeps the identification step but allows global edits, run multiple seeds, and fix or soften the convergence claim. If those come in, this could be a useful paper. The reader's REJECT verdict is about right for the current version.","headline":"A small, plausible prompt-optimization trick whose claimed mechanism and speedup are not actually verified in the paper as written.","tokens_in":10615,"tokens_out":3906,"would_cite":false,"duration_ms":33383,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Editing only flagged tokens improves prompt optimization accuracy by 2.3%.","keywords":["local prompt optimization","prompt engineering","textual gradients","token-level editing","BIG-bench Hard","GSM8K","MultiArith","convergence speed"],"falsifier":"Measure, across optimization steps, how many output tokens fall outside the tagged spans from the previous round. If substantial edits occur outside the tags, or if the model tags most of the prompt, LPO has not actually localized the search and the reported gains would not be attributable to the mechanism. A second check is to run LPO with randomly placed tags: if random tags perform as well as model-chosen ones, the gain comes from the extra meta-instruction rather than from selecting the right tokens.","tokens_in":9626,"feed_emoji":"🎯","tokens_out":6389,"duration_ms":58331,"temperature":0.7,"pith_summary":"The paper proposes Local Prompt Optimization (LPO), a plug-in step for any automatic prompt optimizer. Instead of letting the optimizer rewrite the whole prompt, LPO first asks the proposal model to wrap the tokens that need changing in <edit> tags, then tells the model to revise only those marked spans. The authors report that this constraint improves accuracy by an average of 1.5% on math reasoning tasks and 2.3% across BIG-bench Hard subtasks, while also reaching the best prompt in fewer optimization steps. The practical point is that prompt optimization becomes more controllable: long production prompts can be edited section by section without risking regressions elsewhere.","feed_headline":"Local prompt edits beat global rewriting by 2.3%","feed_subtitle":"Marking edit tokens with <edit> tags makes prompt optimizers converge faster on math and reasoning tasks.","key_machinery":"The <edit> tag mechanism. LPO's pipeline is: (1) a meta-prompt tells the proposal LLM to locate the tokens responsible for incorrect predictions and wrap each span of at most five words in <edit>...</edit>; (2) the same LLM then produces the new prompt from those marks, instructed to reply without the tags. This turns 'where to edit' into an explicit step, shrinking the effective search space from all tokens and the full vocabulary to a handful of marked spans, and it is the component that can be inserted into any existing optimizer.","core_discovery":"The central claim is that the bottleneck in prompt optimization is not the optimizer's vocabulary but the size of the span being edited: global mutation treats every token as movable, so the proposal LLM spends its effort on large rewrites and drifts away from the task. LPO removes that by making token selection explicit. A meta-prompt first identifies the problematic spans, each capped at five words, and marks them with <edit> tags; a second instruction asks for the revised instruction with the tags stripped. The paper shows that this two-stage, chain-of-thought style editing lifts accuracy on GSM8K, MultiArith, and BIG-bench Hard across three optimizers, and on an 8,000-token production prompt it yields a 6% gain while leaving untouched sections intact.","pith_inferences":["The reported gains presume the optimizer obeys the tags; if that holds, the same mechanism could be inverted into a user-controlled feature where practitioners mark the regions they allow to change, which LPO currently leaves to the model's judgment.","Faster convergence should translate into measurable dollar savings per optimization run; a cost-per-accuracy-point comparison, which the paper does not report, would likely show larger relative gains than the accuracy deltas.","The token-selection step resembles sparse-gradient selection in deep learning, suggesting a testable extension: replace LLM-chosen tags with uncertainty-based token scoring and check whether the accuracy gains persist or improve."],"forward_implications":["On math reasoning, adding LPO raises accuracy by about 1.5% on average (GSM8K and MultiArith) across three optimizers.","On BIG-bench Hard's 27 subtasks, LPO beats global optimization by an average of 2.3% across the same methods.","LPO reaches its best prompt in fewer steps, so each optimization run consumes fewer LLM proposals and less compute.","Because only tagged spans change, optimization can be confined to a production prompt's relevant section, which the paper ties to a 6% accuracy gain and no regression in other intents."],"supporting_citations":[{"why":"Supplies the PE2 baseline, the train-dev-test splits, and the hyperparameter setup used across all experiments.","marker":"Ye et al., 2024"},{"why":"Provides the APO baseline and the textual-gradient feedback that LPO augments with edit-token identification.","marker":"Pryzant et al., 2023"},{"why":"Provides the Iterative APE baseline that LPO wraps with its local editing step.","marker":"Zhou et al., 2023b"},{"why":"Defines the BIG-bench Hard benchmark whose 27 subtasks measure the average 2.3% gain.","marker":"Suzgun et al., 2023"},{"why":"Supplies the GSM8K math word-problem dataset used in the math reasoning evaluation.","marker":"Cobbe et al., 2021"},{"why":"Supplies the MultiArith dataset used alongside GSM8K for math reasoning evaluation.","marker":"Roy and Roth, 2015"},{"why":"Provide the 'Let's think step by step' initialization and the chain-of-thought rationale motivating token-level edit direction.","marker":"Kojima et al., 2022; Wei et al., 2022"}],"fun_headline_variants":["Local token edits outperform global prompt rewrites","Spot-fix prompts: 2.3% gain over full rewrites","Local edits converge faster, up to 6% on 8k prompts","Mark <edit> tags to speed prompt optimization","Edit only problem spans for faster prompt tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proposal model actually edits only the tokens wrapped in the <edit> tags, so the search space genuinely shrinks instead of the tags being decorative.","fun_headline_variants_meta":{"raw":{"variants":["Local token edits outperform global prompt rewrites","Spot-fix prompts: 2.3% gain over full rewrites","Local edits converge faster, up to 6% on 8k prompts","Mark <edit> tags to speed prompt optimization","Edit only problem spans for faster prompt tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000542,"raw_usage":{"total_tokens":2555,"prompt_tokens":859,"completion_tokens":1696,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":1615}},"tokens_in":475,"tokens_out":1696,"duration_ms":17023,"temperature":1.0,"reasoning_tokens":1615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:32:00.540257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, across optimization steps, how many output tokens fall outside the tagged spans from the previous round. If substantial edits occur outside the tags, or if the model tags most of the prompt, LPO has not actually localized the search and the reported gains would not be attributable to the mechanism. A second check is to run LPO with randomly placed tags: if random tags perform as well as model-chosen ones, the gain comes from the extra meta-instruction rather than from selecting the right tokens.","supporting_citations":[],"review_version":1}