{"id":"5acd1387-6bdc-4e2d-8a97-686b70074731","arxiv_id":"2511.19829","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An execution-free evaluator that predicts prompt-quality metrics guides per-query prompt rewriting, but the reported consistent gains are not supported by the paper's own tables.","lead":"This paper trains a small model to predict, without running the large language model, whether a prompt will succeed, then uses those predictions to rewrite prompts per query. The authors report gains over baselines, but the comparisons are weakened by unfair query-editing, missing error bars, and results that sometimes contradict their own claims.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Query-side augmentation in §3.4 changes the test input for 'Ours' only; Table 3 also shows baselines winning multiple cells, so 'consistently surpass' is unsupported.","rationale":"The reader's REJECT verdict is appropriate, and my analysis reinforces it. I focus on the query-side augmentation in Section 3.4 because it is the most direct invalidation of the central comparison: the method is not solving the same task as the baselines when it can add clarifications and constraints to the test query. This is a protocol-level flaw, not a tuning issue. The internal Table 3 inconsistencies strengthen the conclusion: even under the reported (unfair) protocol, Ours is not consistently the best, and on some cells it is worse than the LLM-only baseline. A single ablation—disabling query-side augmentation—would settle whether the claimed gains survive a fair comparison. I partially agree with the reader because they also identify the query-overlap issue in the evaluator's 83.7% accuracy, which is a separate and important concern, but the query-augmentation flaw alone is sufficient to reject the headline claim as currently supported. The paper may have a salvageable core idea, but the submitted evidence does not establish it.","tokens_in":12851,"tokens_out":8268,"duration_ms":82454,"concrete_test":"Disable the query-entropy branch in Section 3.4 so that the optimizer cannot augment the query side, and rerun the full Table 3 comparison (8 datasets × 3 backbones) using the same evaluator and iteration budget. Report per-cell accuracy and the win rate against the best baseline. If Ours no longer beats the best baseline on a majority of cells, or if the gain over LLM-only drops below the gains of Self-Refine/ProRefine/TextGrad/APE, then the claimed consistent superiority is an artifact of test-input modification rather than prompt optimization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core comparison is invalidated by Section 3.4: for the query-entropy dimension, the optimizer's fixes are 'applied by augmenting the query side with minimal clarifications and constraints.' This means the proposed method is allowed to modify the test question itself, while all baselines (Self-Refine, ProRefine, APE, TextGrad) can only modify the prompt or template. Any improvement on LegalBench, MedQA, or GPQA could therefore come from injecting extra task-relevant information or constraints into the query, not from better prompt optimization. The paper's 'consistent superiority' claim cannot be interpreted without an ablation that disables this query-side editing.\n\nIndependently, the paper's own Table 3 contradicts 'consistently surpass': on bbh web of lies, Ours scores 0.69 vs TextGrad 0.74 (llama3) and 0.68 vs APE 0.71 / TextGrad 0.73 (llama3.1), and even falls below the LLM-only baseline of 0.69 on llama3.1; on bbh disambiguation qa gpt-4o, TextGrad 0.71 > Ours 0.69; on GPQA Diamond llama3.1, ProRefine 0.29 > Ours 0.27. No error bars or repeated seeds are reported, so 'stable' and 'consistent' are unsupported even setting aside the unfair comparison. The reader's additional concern about train/validation query overlap in the 11,530-prompt pool is real but secondary; even with a clean split, the query-augmentation issue makes the headline result uninterpretable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified evaluation-instructed prompt optimization framework. It constructs a diverse prompt pool (11,530 prompt–query pairs) via static templates, LLM-generated styles, and evolutionary recombination; selects four prompt-quality metrics (NLL, stability, MI, query entropy) using XGBoost importance; fine-tunes a LLaMA-8B LoRA evaluator that predicts these metrics and an overall success probability from text without execution; and uses gradient-based attributions from the evaluator to guide query-dependent, metric-aware prompt rewriting. The authors claim the evaluator reaches 83.7% validation accuracy and that the resulting optimization pipeline 'consistently surpasses' static-template and query-dependent baselines across eight datasets and three backbone models.","tokens_in":13254,"tokens_out":3484,"duration_ms":37642,"significance":"If the claims were substantiated, this would be a meaningful step toward efficient, interpretable prompt optimization: an execution-free evaluator that produces dimension-specific signals could be plugged into multi-agent systems at low cost. The paper does include a systematic metric selection, a multi-component ablation, and cross-backbone experiments, which are strengths. However, the headline claims are currently undermined by (i) an unfair comparison setup in which the proposed method may modify the test query itself, (ii) multiple cells in Table 3 where the proposed method does not outperform baselines, and (iii) an unspecified train/validation split for the evaluator. The contribution is potentially valuable, but the empirical evidence as presented does not support the core claim of consistent superiority.","major_comments":[{"comment":"The optimizer is allowed to 'augment the query side with minimal clarifications and constraints' when addressing the query-entropy dimension. This changes the test input for 'Ours' only; all baselines (Self-Refine, ProRefine, APE, TextGrad) can only modify the prompt/template. Therefore, improvements on tasks like LegalBench, MedQA, GPQA, and ambiguous BBH subsets could stem from injecting task-relevant information into the query rather than from better prompt optimization. An ablation disabling query-side augmentation—or equivalently giving baselines the same ability—is necessary to make the comparison interpretable. This is load-bearing because the central claim of consistent superiority rests on Table 3.","section":"§3.4, Query Entropy"},{"comment":"The abstract and conclusion state that the approach 'consistently surpasses' baselines, but Table 3 contains multiple counterexamples. On bbh web of lies (llama3.1), Ours scores 0.68, below both the LLM-only baseline (0.69) and TextGrad (0.73); on GPQA Diamond (llama3.1), Ours scores 0.27 versus ProRefine's 0.29; on bbh disambiguation qa (gpt-4o), Ours scores 0.69 versus TextGrad's 0.71. Several other rows are ties or near-ties. No error bars, confidence intervals, or repeated-seed results are reported, so 'stable' and 'consistent' are unsupported even setting aside the fairness issue. At minimum, the claims need to be moderated to match the actual results, and statistical uncertainty needs to be quantified.","section":"Table 3"},{"comment":"The evaluator is trained on 80% of the 11,530-prompt pool and validated on 20%, but the paper does not state whether the split is query-disjoint. Because the pool contains multiple prompt candidates per sampled question, overlapping queries between train and validation would let the evaluator memorize query-level difficulty rather than judge prompt quality, inflating the 83.7% accuracy. The paper should clarify the split construction and, if necessary, rerun with a query-disjoint split. This is load-bearing for the evaluator's headline accuracy, which is the foundation of the optimization.","section":"§4.3, Evaluator validation split"},{"comment":"The claim of consistent wins 'across eight datasets and on three backbone models' is not reflected in the table: MATH500 results are reported only for GPT-4o. For LLaMA-3 and LLaMA-3.1, the table shows dashes for all optimization methods, meaning no optimization comparison exists on that dataset for two backbones. This overstates the empirical coverage and should be corrected by either reporting results or explicitly restricting the claim to the seven datasets where all backbones were evaluated.","section":"§4.4 and Table 3, MATH500"}],"minor_comments":[{"comment":"The four metrics are selected on the same 11,530-prompt pool used to train and evaluate the evaluator, with a gain threshold of 10%. The sensitivity of the optimization results to this selection is not examined; a brief robustness check (e.g., using all eight metrics or an alternative threshold) would strengthen the claim that the selected metrics are the right ones.","section":"§3.2"},{"comment":"The appendix cites 'Kojima et al., 2022', 'Zhou et al., 2022a', 'Hulbert, 2023', and 'Liang et al., 2023' but these are not in the reference list. Reference [24] and [25] are the same paper (Prompt-OIRL) with different numbering; this should be de-duplicated.","section":"References"},{"comment":"Minor typos and inconsistent terminology: 'ect.' in §3.4, 'ProteGi' vs 'ProTeGi', and inconsistent capitalization of 'LLaMA-8B' vs 'LLaMA-3-8B'. The paper would also benefit from stating the number of optimization iterations used for the main results (it says max 3) and whether the same iteration budget was used for all baselines.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper tackles an important problem and the general architecture is plausible, but the experimental protocol has a fundamental fairness flaw and the reported results do not support the 'consistent superiority' claim. I would like to see a revised version that (1) removes or symmetrizes the query-side augmentation, (2) provides a query-disjoint evaluator split, (3) reports variance and error bars, and (4) tempers the claims to match the actual data. If these issues cannot be resolved, the paper may be better suited to a workshop or as a position paper rather than a full archival publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nTwo things to know. The closed-loop design — an execution-free evaluator that predicts several prompt-quality metrics and uses classification-loss gradients to guide per-query rewriting — is genuinely new and worth thinking about. But the evaluation does not support the headline claim of consistent superiority. The main problem is that the optimizer is allowed to edit the test query itself for the query-entropy fixes, while no baseline gets that option.\n\nWhat is actually good. The evaluator is a LLaMA-8B with LoRA, jointly trained for binary quality prediction and regression of NLL, stability, MI, and query entropy, with metric-aware fusion. The 83.7% validation accuracy over an XGBoost baseline (69%) is impressive, assuming the split is query-disjoint. The ablation in Table 1 gives useful evidence that the components matter. Using gradient magnitudes w.r.t. predicted metrics to decide which diagnostic module to invoke is a nice, interpretable way to connect evaluation and optimization.\n\nWhere it falls down. Section 3.4 says that for the query-entropy dimension, fixes are applied by augmenting the query side with minimal clarifications and constraints. That changes the test input. Baselines only rewrite the prompt. Any gain on LegalBench, MedQA, or GPQA could come from injecting extra task-relevant information, not from better prompt optimization. There is no ablation isolating this effect. That alone invalidates the comparison.\n\nSecond, Table 3 contradicts 'consistently surpass.' On bbh web of lies llama3.1, the method scores 0.68 vs LLM-only 0.69, APE 0.71, and TextGrad 0.73. On disambiguation qa gpt-4o, TextGrad wins 0.71 vs 0.69. On GPQA Diamond llama3.1, ProRefine wins 0.29 vs 0.27. No error bars or repeated seeds appear anywhere, so 'stable' and 'consistent' are unsupported. MATH500 is omitted for the two LLaMA backbones; the limitation section gives a plausible reason (arithmetic capability), which is honest, but the omission still narrows the empirical claim.\n\nThird, the 80/20 split on the 11,530-prompt pool is not stated to be query-disjoint. If the same question appears in both train and validation, the 83.7% accuracy is inflated by memorizing query difficulty. The paper needs to clarify this.\n\nThe core idea has merit, and the evaluator could be a useful building block if the protocol is cleaned up. As submitted, the evidence does not back the abstract and conclusion. A revision that disables query-side editing, reports seeds and error bars, and clarifies the split would make a credible contribution.\n\nPrompt-optimization researchers will want to see this, but no one should trust the optimization results as they stand. It deserves peer review — the reviewers should push on the comparison protocol. I would not cite it in its current form.\n\nBest,\n[Your name]","headline":"Novel closed-loop evaluator–optimizer, but the evaluation is spoiled by query-side editing and selective reporting; the central claims are unsupported as submitted.","tokens_in":13772,"tokens_out":5050,"would_cite":false,"duration_ms":46093,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt quality can be decomposed into four text-predictable metrics, and an evaluator trained on those metrics can guide query-by-query prompt rewrites that beat both static templates and existing query-dependent optimizers on eight benchma","keywords":["prompt optimization","execution-free evaluation","query-dependent prompting","prompt quality metrics","interpretable optimization","multi-agent systems","LLM evaluation","gradient attribution"],"falsifier":"Check the 11,530-prompt pool for query overlap between the training and validation splits; if any query appears in both, recompute evaluator accuracy on a query-disjoint split. If accuracy falls substantially (toward the ~69% of the embedding-based baseline), the central claim of execution-free evaluation is unsupported. Separately, rerun the optimization on a held-out set while forbidding any edits to the query side; if the gains over baselines disappear, the reported improvements are largely attributable to query augmentation rather than diagnostic prompt rewriting.","tokens_in":12712,"feed_emoji":"🧠","tokens_out":7800,"duration_ms":74272,"temperature":0.7,"pith_summary":"Most prompt-optimization methods either tune one fixed template for a whole task or rely on executing the model repeatedly to get feedback. This paper tries to close that loop differently: it defines prompt quality as a small set of performance-relevant dimensions—confidence in the correct answer, output stability, prompt–response mutual information, and intrinsic query difficulty—and trains a lightweight evaluator that predicts those dimensions and the probability of success directly from the prompt text, without running the model. The evaluator's per-dimension attributions then tell an optimizer exactly which facet to fix for each query. The paper reports that the evaluator predicts prompt performance with 83.7% accuracy, and that the resulting query-dependent optimization beats static-template and existing query-dependent baselines across eight benchmarks and three different backbone models, including a medical domain the system never trained on. If true, this would make prompt optimization cheap enough to run inside multi-agent systems and interpretable enough to trust.","feed_headline":"Prompt quality predicted without running the LLM, 83.7% accurate","feed_subtitle":"A four-metric evaluator guides query-level rewrites that beat static and adaptive baselines on 8 benchmarks.","key_machinery":"The load-bearing object is the evaluation-instructed optimizer loop, built from three pieces: (1) a diverse training pool of 11,530 prompts assembled from five static templates, six LLM-directed prompt styles, and genetic-style recombination, so the evaluator sees prompts with varied structures and quality; (2) an execution-free evaluator that predicts the four metric scores and the binary quality label, using a bi-level training objective that lets metric regression serve classification only when useful, a gradient-informed weighting scheme that up-weights metrics with stronger influence on classification, and a metric-aware feature fusion; and (3) gradient-based attribution, where the part","core_discovery":"The paper's central claim is that a good prompt is one that scores well on four complementary, performance-reflective metrics: negative log-likelihood of the correct answer (how confidently the prompt routes the model to the right output), semantic stability (how consistent the outputs are across runs), mutual information (how much the prompt shapes the output beyond the query alone), and query entropy (how ambiguous the question itself is). It then fine-tunes an 8-billion-parameter instruction-tuned language model, with lightweight adapters, to predict these four scores and a binary quality label (whether average execution accuracy would exceed 50%) from the combined text of the query and p","pith_inferences":["One testable extension: train a much smaller distilled evaluator (under a billion parameters) on the same four-metric scoring task and check whether optimization gains survive; the paper uses an 8B model, so the cost-savings claim for multi-agent settings would become stronger with a lighter scorer.","The dominance of query entropy suggests a division of labour worth testing: for high-entropy queries, optimizing the question formulation may return more value than optimizing the reasoning template; a suite could route queries accordingly.","Because the optimizer is permitted to augment the query with clarifications while the baselines only rewrite prompts, a cleaner test would compare all methods under identical action spaces; if the gains persist without query-side edits, the improvement is attributable to prompt rewriting rather than query augmentation.","The binary quality threshold of 50% accuracy is coarse; a graded label would let the evaluator discriminate between 'just above chance' and 'clearly correct' prompts, and might change which failure attributions the optimizer trusts."],"forward_implications":["Prompt optimization no longer requires repeated model execution at inference time: the evaluator scores a prompt in one forward pass, then the optimized loop can run inside a multi-agent system without multiplying token costs.","The same evaluator, trained on prompts from a single backbone, transfers its optimization signals to other backbones, including a stronger commercially hosted model, so the approach is model-agnostic rather than tied to the training model.","Because the evaluator's learned weights rank query entropy highest (32.7% of the classification weight), the method implies that the largest prompt-side gains come from clarifying the question itself, not from inventing fancier reasoning templates.","The framework generalizes to a held-out medical dataset with consistent 5–6% gains, indicating that the four metrics capture performance-reflective dimensions that are not domain-specific.","Reducing a noisy set of eight candidate metrics to four performance-reflective ones improves evaluator accuracy from 79.6% to 83.7%, evidence that focusing the evaluation on a few complementary dimensions is better for learning."],"fun_headline_variants":["Prompt quality scores without running the LLM: 83.7% accuracy","Four-metric evaluator predicts prompt quality, guides rewrites","Execution-free prompt evaluator hits 83.7% on performance prediction","Query-level prompt optimization driven by evaluation signals","Predict prompt performance from text alone: 83.7% accurate"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the evaluator learns to judge prompt quality rather than recall query difficulty—so the 83.7% accuracy assumes the train/validation split is query-disjoint—and that the reported gains come from prompt rewriting, not from the optimizer's extra freedom to edit the query itself.","fun_headline_variants_meta":{"raw":{"variants":["Prompt quality scores without running the LLM: 83.7% accuracy","Four-metric evaluator predicts prompt quality, guides rewrites","Execution-free prompt evaluator hits 83.7% on performance prediction","Query-level prompt optimization driven by evaluation signals","Predict prompt performance from text alone: 83.7% accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000591,"raw_usage":{"total_tokens":2603,"prompt_tokens":734,"completion_tokens":1869,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1795}},"tokens_in":478,"tokens_out":1869,"duration_ms":12459,"temperature":1.0,"reasoning_tokens":1795,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:23:32.590747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the 11,530-prompt pool for query overlap between the training and validation splits; if any query appears in both, recompute evaluator accuracy on a query-disjoint split. If accuracy falls substantially (toward the ~69% of the embedding-based baseline), the central claim of execution-free evaluation is unsupported. Separately, rerun the optimization on a held-out set while forbidding any edits to the query side; if the gains over baselines disappear, the reported improvements are largely attributable to query augmentation rather than diagnostic prompt rewriting.","supporting_citations":[],"review_version":1}