{"id":"39dc175d-8292-4a72-8336-43c89250ce70","arxiv_id":"2505.10736","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An evaluation-set selection method that adds real-time model feedback to semantic sampling improves the accuracy and stability of three prompt optimization methods on two datasets.","lead":"This paper proposes IPOMP, a method for choosing which examples are used to evaluate prompts while optimizing a language model's instructions. It reports that IPOMP improves prompt optimizer accuracy by about two percent and makes results much more stable, at tiny extra cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-2 gain is not tied to the logit-correlation redundancy signal: the ablation only compares with/without Stage 2, so the core mechanism remains unvalidated.","rationale":"The reader's weakest_assumption points to the same component: the validity of the logit-correlation redundancy signal in Algorithm 2. I agree that this is the most load-bearing assumption, but I sharpen it: even within the reported experiments, the signal's causal role is not established because no control for generic replacement is run. The reader framed the risk as cross-domain transferability ('if this signal fails on a new task or model'); my concern is more immediate, namely that the existing ablation cannot distinguish the proposed mechanism from simple resampling. That is why agreement is 'partial' rather than 'agree'. The concrete problems the reader also lists are real and should be fixed: the abstract/body inconsistency (1.6-3.1% vs 1.6-5.3% and 50% vs 57%), the APE overhead in Table 3 (120.23 vs 109.84 seconds, about 9.5%, contradicting 'below 1%'), the Algorithm 1 loop bound using alpha*N instead of (1-alpha)*N, and the absence of significance tests with only five runs. However, these are presentation, reporting, and statistical-evidence issues; they do not get at whether the central mechanism works. If the proposed replacement-strategy ablation shows that random replacement performs as well as logit-correlation-guided replacement, then the method may still improve over baselines, but its claimed novelty and the universal-plugin conclusion would be substantially weakened. Conversely, if the ablation shows a clear advantage for the guided replacement, the central claim becomes much more credible. Given the positive full-method results and the missing control, the appropriate verdict remains conditional rather than rejection or acceptance.","tokens_in":17677,"tokens_out":6704,"duration_ms":69548,"concrete_test":"Run a replacement-strategy ablation as in Algorithm 2 but vary only the replacement source and target: (a) logit-correlation-selected redundant samples replaced by least-similar samples (IPOMP); (b) logit-correlation-selected redundant samples replaced by random samples; (c) a random set of beta*|S| samples replaced by least-similar samples; and (d) a random set replaced by random samples. Use the same optimizers, datasets, 20-sample evaluation budget, and, ideally, 30 seeds; compare final held-out accuracy and standard deviation. If (b) or (d) matches (a) within noise, the logit-correlation redundancy signal and the dissimilar-replacement rule are not responsible for Stage 2's benefit, and the 'model-performance-guided' claim should be weakened accordingly.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central novelty of IPOMP is Algorithm 2's real-time model-performance-guided refinement (Section 3.2): samples whose logits are highly correlated across candidate prompts (CT=0.9) are deemed redundant and replaced by semantically least-similar training samples. The claim that this mechanism drives the reported effectiveness and stability gains rests on the ablation in Table 2, which removes Stage 2 in whole. That ablation does not isolate the redundancy signal from the act of resampling: a control that replaces the same beta fraction of samples with random (or merely new) samples could plausibly produce similar gains, especially since Stage 1 selects only 20 samples from datasets of 392 to 10,240 examples, so any replacement increases coverage. Appendix A.7 (Table 8) only shows that 'Dissimilar' replacement lowers pairwise logit correlations relative to 'Similar' and 'Random' replacement; it never measures final prompt accuracy on the held-out test set. Therefore the paper does not establish that sample-level logit correlation is a valid redundancy signal for prompt evaluation, nor that the specific selection-and-replacement logic, rather than generic resampling or diversity, causes the improvement. Because the same Stage 2 is also claimed to be a universal plugin (Section 5.2), this unvalidated mechanism is load-bearing for both the method's novelty and the stability claims (e.g., SD reductions in Tables 1 and 2).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IPOMP, a two-stage method for selecting evaluation data subsets to be used during automated prompt optimization. Stage 1 selects representative and diverse samples by combining semantic clustering with boundary-case selection. Stage 2 then iteratively replaces samples that are deemed redundant based on high logit correlations across candidate prompts, substituting the most semantically dissimilar training samples. The method is evaluated on BIG-bench and LIAR using GPT-3.5 and GPT-4o-mini across three prompt optimizers (APE, APO, EVOPROMPT), with accuracy and standard deviation reported. The authors claim effectiveness gains of 1.6% to 3.1% over the best baseline, stability improvements of at least 50%, and computational overhead below 1%, and they further claim that Stage 2 can be used as a universal enhancement to other data-selection baselines.","tokens_in":17834,"tokens_out":6936,"duration_ms":61004,"significance":"If the results hold, IPOMP addresses a practical gap in prompt optimization: evaluation subset selection. The paper's empirical scope is reasonable for a first study, with two datasets, two LLMs, three prompt optimizers, multiple baselines, an ablation study, sensitivity analysis, and cost/overhead measurements. The idea of using real-time model performance to refine evaluation samples is interesting and potentially useful, especially because it does not require pre-collected performance data. However, several central numerical claims are not backed by the reported tables, and the mechanism that distinguishes Stage 2 from generic resampling is not directly validated. These issues materially affect the paper's conclusions but are fixable with additional analyses and corrections.","major_comments":[{"comment":"The numeric claims are internally inconsistent and not fully supported by Table 1. The abstract states '1.6% to 5.3%' and 'at least 57%' improvement, while the full-text abstract and introduction state '1.6% to 3.1%' and '50% to 55.5%'. More importantly, row-level differences against Anchor-Point in Table 1 are as small as 0.2% (EVOPROMPT on GPT-3.5 BIG-bench: 0.776 vs 0.774; EVOPROMPT on GPT-4o-mini BIG-bench: 0.758 vs 0.756), and the GPT-3.5 BIG-bench average difference is 1.2% (0.757 vs 0.745). The claim of 'at least 1.6% to 3.1%' needs to be reconciled with these numbers or the statistic must be redefined.","section":"Abstract, Section 5.1"},{"comment":"The paper claims computational overhead of less than 1%, but Table 3 shows APE with IPOMP taking 120.23 seconds versus 109.84 seconds for random selection, which is a 9.5% overhead. The average overhead across APO, APE, and EVOPROMPT is 1.34% (401.61 vs 396.31 seconds). The claim should be corrected or the aggregation clarified.","section":"Section 5.4, Table 3"},{"comment":"The ablation removes Stage 2 as a whole, so it does not test whether the logit-correlation redundancy signal causes the observed gains. Since Stage 1 selects only 20 samples from training sets of 392 to 10,240 examples, any resampling, including random replacement, expands coverage. A control that replaces the same beta fraction of samples with randomly selected or minimally similar samples and reports final held-out accuracy is needed to support the mechanism claim and the 'universally applied' claim. Table 8 only reports pairwise logit correlations of replacement strategies, not the resulting prompt accuracy or stability.","section":"Section 5.2, Table 2, Appendix A.7"},{"comment":"The sensitivity analysis states that accuracy 'consistently improves' as alpha increases from 0.1 to 0.9, yet the default configuration uses alpha=0.5. No justification is given for choosing alpha=0.5, and the main results may therefore not reflect the better-performing configuration. This is important because alpha and other hyperparameters are selected on the same datasets used for evaluation.","section":"Section 4.4, Section A.5, Figure 4"},{"comment":"The stability claims are based on standard deviations computed over five runs, but the paper does not report confidence intervals or significance tests for either accuracy or standard-deviation differences. Given that several differences are small (e.g., 0.2% in Table 1), the reader cannot assess whether the reported stability improvements are robust rather than noise.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"There is a typo: 'sage 2' should be 'stage 2'.","section":"Section 5.2"},{"comment":"The conclusion says evaluation was on the BIG-bench dataset, but the experiments also use LIAR; the conclusion should mention both.","section":"Conclusion"},{"comment":"The phrase 'at least 50% to 55.5%' is ambiguous; consider using 'from 50% to 55.5%' or a similar explicit range.","section":"Abstract"},{"comment":"The definition of the runtime performance matrix MP_runtime is not fully specified for open-ended or multi-label outputs; please clarify how the logit-based representation is constructed in those cases.","section":"Section 3.2, Algorithm 2"},{"comment":"The caption of Figure 1 does not state which dataset, model, and prompt optimizer are shown; the text refers to APE but the caption should be self-contained.","section":"Section 5.1, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to have been revised, and the metadata abstract and full-text abstract give different headline numbers; the conclusion also omits LIAR. Please ask the authors to harmonize all numerical claims across the abstract, introduction, and results. If the random-replacement control for Stage 2 shows that generic resampling produces similar gains, the paper's central novelty framing will need significant rework, so the control should be added before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This one is worth a serious referee, but it is not ready as-is. The central claim — that a cheap, model-agnostic evaluation-subset selection layer improves prompt optimization accuracy by roughly 2 to 3 percent and cuts run-to-run variance — is plausible and supported by a consistent pattern across three optimizers, two datasets, and two models. But the paper undercuts itself with an internal numbers mismatch, a false overhead claim, and a missing control for its main mechanism.\n\nWhat's genuinely new: it is the first data-selection method designed explicitly for prompt optimization, rather than repurposed from LLM benchmarking. The two-stage idea — semantic clustering plus boundary sampling, then iterative replacement using runtime logit correlations — is a sensible assembly of known components. The plugin experiment (baselines + stage 2) is a nice addition, and the related work is honestly surveyed. The limitations section is candid about the limited model/optimizer coverage.\n\nThe soft spots are concrete. First, the preprint abstract reports 1.6% to 5.3% accuracy gains and 57% stability gains; the full abstract and Section 5.1 report 1.6% to 3.1% and 50% to 55.5%. Those are different numbers. Second, the 'less than 1%' overhead claim is contradicted by the paper's own Table 3: for APE, IPOMP takes 120.23s versus 109.84s for random sampling, about 9.5% overhead, and the average is about 1.3%. Third, Algorithm 1's loop condition `n ≤ αN` selects αN boundary samples rather than the intended (1-α)N, so the subset size is right only when α=0.5; it may be a typo, but the pseudocode needs correcting.\n\nFourth and most important: the stage-2 ablation removes the whole stage, so it does not isolate the redundancy signal. A control that replaced the same fraction of samples randomly, or with any new samples, would tell you whether the logit-correlation logic actually helps. The appendix only shows that dissimilar replacements lower pairwise correlations; it never shows that this translates into better final prompts. Since the plugin claim depends on the same mechanism, this is load-bearing.\n\nAlso worth noting: five runs per cell with no significance tests, and some standard deviations overlap, so the magnitude of the gains is not nailed down. That is a robustness issue rather than a fatal one. The baseline adaptations, especially prescreening 200 examples for Anchor-Point, may slightly disadvantage the comparison, but the direction of the main effect is consistent.\n\nBottom line: this paper deserves a serious referee. It fills a real gap, the experimental pattern is consistent, and the problems are fixable rather than fundamental. I would not take the headline numbers at face value yet, but I would ask for a revision that aligns the claims, fixes the pseudocode, adds a random-replacement control, and either re-measures or softens the overhead statement.","headline":"A plausible and novel evaluation-selection method for prompt optimization, undermined by internal inconsistencies and an unvalidated core mechanism; worth a serious referee and a careful revision.","tokens_in":18497,"tokens_out":3441,"would_cite":true,"duration_ms":31613,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that IPOMP, a two-stage evaluation-data selection method that combines semantic clustering, boundary sampling, and real-time model-performance-guided replacement of redundant samples, makes automated prompt optimization…","keywords":["prompt optimization","evaluation data selection","coreset selection","large language models","semantic clustering","boundary sampling","model performance guidance","iterative refinement"],"falsifier":"On a held-out task, compare the correlation between evaluation-set accuracy and full-test accuracy across candidate prompts for the stage-1 set before and after stage-2 refinement; if refinement does not raise that correlation, the redundancy signal is not doing the claimed work. A second check: swap in randomly chosen replacements instead of least-similar ones; if final prompt accuracy matches IPOMP, the dissimilarity rule is not the active ingredient.","tokens_in":17330,"feed_emoji":"🎯","tokens_out":8637,"duration_ms":77215,"temperature":0.7,"pith_summary":"The paper claims that the evaluation subsets used during automated prompt optimization should not be chosen at random or by semantics alone. It introduces IPOMP, a two-stage approach that first selects a diverse seed set by semantic clustering plus boundary samples, then iteratively swaps out redundant samples identified by correlated logits across candidate prompts. On BIG-bench and LIAR with GPT-3.5 and GPT-4o-mini, IPOMP reports accuracy gains of 1.6% to 3.1% over the best baseline and at least 50% lower standard deviation, at under 1% extra runtime. If correct, it supplies a cheap, model-agnostic layer that makes any iterative prompt optimizer more reliable.","feed_headline":"Swap redundant eval samples to make prompt tuning 50% steadier","feed_subtitle":"Automated prompt tuning with IPOMP beats random sampling and prior coreset baselines by 1.6–3.1% at under 1% extra cost.","key_machinery":"The central object is the per-sample performance vector across candidate prompts, a matrix of logits of size |S| by (|output| times |candP|). Two samples are redundant when their logit rows are highly correlated across candidate prompts; hierarchical clustering with threshold 0.9 groups them, and a fraction beta = 0.5 is replaced by the most semantically dissimilar training examples via HNSW search. That mechanism converts a static semantic selection into a dynamic one, and the ablation shows it is responsible for most of the stability gain, with standard deviation dropping by a factor of 2.83 when stage 2 is removed.","core_discovery":"The central claim is that evaluation data for prompt optimization can be selected and continuously refined by combining semantic diversity with real-time model behavior, and that this beats both random sampling and existing coreset methods designed for model benchmarking. Stage 1 builds a small 20-sample set from K-means semantic clusters and from farthest-apart boundary pairs. Stage 2 uses logits produced by candidate prompts during optimization: samples whose performance vectors are highly correlated (threshold 0.9) are treated as redundant, and a fraction beta = 0.5 are replaced each iteration by the least semantically similar training samples. The paper reports that this raises accuracy by 1.6% to 3.1% over Anchor-Point, the strongest baseline, cuts standard deviation by at least 50%, adds less than 1% overhead, and that the stage-2 refinement alone improves the Random, Boundary, Clustering, and Prediction-based baselines when attached to them.","pith_inferences":["A testable extension not explored in the paper: the same redundancy logic could be applied to any loop that scores many candidate configurations on a fixed sample set, such as few-shot example selection or hyperparameter search, as long as per-sample confidence scores are available.","The paper fixes the replacement target as the least semantically similar sample; an alternative that follows directly from its own logit-correlation logic would be to swap in samples whose confidence vectors are least correlated with the retained set, which could outperform semantic dissimilarity on tasks where semantics and model behavior diverge.","The 0.9 correlation threshold is static, yet the paper's case study shows redundancy drops from 19% to 10% after one round; adapting the threshold as the prompt set evolves is a natural way to squeeze out further stability gains."],"forward_implications":["Any iterative prompt optimizer can carry the cost of IPOMP: the added runtime is under 1% of the optimizer's execution time, so the method can be layered onto APE, APO, or EVOPROMPT without a separate warm-up stage.","Small evaluation sets become viable: IPOMP outperforms all baselines even with only 5 samples, and the paper's sweep suggests 20 samples is the sweet spot before returns flatten.","Existing coreset selectors inherit the stability gain: attaching stage-2 refinement to Random, Boundary, Clustering, Anchor-Point, and Prediction-based improves their accuracy by 0.3% to 2.3% and reduces their standard deviation by up to 60% in the reported configurations.","The method avoids the cost of prior performance data: unlike Anchor-Point, which needs a preliminary stage evaluating the whole training set on about 10 prompts, IPOMP collects its performance signal during optimization itself."],"supporting_citations":[{"why":"Defines the Anchor-Point coreset method that IPOMP is compared against and that requires a costly preliminary confidence-collection stage.","marker":"Vivek et al., 2023"},{"why":"Supplies the Prediction-based baseline that trains an assessor on existing LLM performance data, the main contrast for IPOMP's real-time performance signal.","marker":"Pacchiardi et al., 2024"},{"why":"Defines APE, one of the prompt optimizers used as the testbed for evaluation-data selection.","marker":"Zhou et al., 2022b"},{"why":"Defines APO, a prompt optimizer whose minibatch feedback makes evaluation-data quality especially important.","marker":"Pryzant et al., 2023"},{"why":"Defines EVOPROMPT, an evolutionary prompt optimizer used as another testbed.","marker":"Guo et al., 2023"},{"why":"Provides the Sentence-BERT embeddings used for semantic clustering and dissimilarity search in both stages.","marker":"Reimers, 2019"},{"why":"Provides the HNSW approximate nearest-neighbor search used to find least-similar replacement samples efficiently.","marker":"Malkov and Yashunin, 2018"},{"why":"Supplies the boundary-case selection idea of choosing least-similar pairs in semantic space.","marker":"Huang et al., 2024"},{"why":"Provides the BIG-bench dataset used for the main experiments.","marker":"bench authors, 2023"}],"fun_headline_variants":["Model-guided eval data selection makes prompt tuning steadier and smarter","IPOMP uses real-time model performance to pick better eval samples","Prompt tuning gains 1.6–5.3% via iterative model-performance eval selection","Performance-driven eval selection beats random and coreset baselines","Iterative model feedback trims redundant data for sharper prompt tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that when two evaluation examples show nearly identical confidence scores across candidate prompts, one of them is redundant for judging prompts, and swapping it for a very different example makes the small evaluation set a better predictor of which prompt will win on the full test set.","fun_headline_variants_meta":{"raw":{"variants":["Model-guided eval data selection makes prompt tuning steadier and smarter","IPOMP uses real-time model performance to pick better eval samples","Prompt tuning gains 1.6–5.3% via iterative model-performance eval selection","Performance-driven eval selection beats random and coreset baselines","Iterative model feedback trims redundant data for sharper prompt tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000844,"raw_usage":{"total_tokens":3680,"prompt_tokens":955,"completion_tokens":2725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2648}},"tokens_in":571,"tokens_out":2725,"duration_ms":18604,"temperature":1.0,"reasoning_tokens":2648,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:04:13.523272+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out task, compare the correlation between evaluation-set accuracy and full-test accuracy across candidate prompts for the stage-1 set before and after stage-2 refinement; if refinement does not raise that correlation, the redundancy signal is not doing the claimed work. A second check: swap in randomly chosen replacements instead of least-similar ones; if final prompt accuracy matches IPOMP, the dissimilarity rule is not the active ingredient.","supporting_citations":[],"review_version":1}