{"id":"2d01d69b-aa06-44bd-879c-af065b0ef83b","arxiv_id":"2607.20497","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Error-driven prompt optimization (ERGO) beats demonstration and search methods on boundary-learnable tasks (TREC 90.0, CLINC150 94.4), but no paradigm dominates overall.","lead":"The paper compares three prompt-optimization strategies for LLM text classification and finds they win on different task types, then proposes ERGO, an error-driven method that turns misclassifications into decision rules. For practitioners, it gives a practical rule of thumb: start with demonstrations, use error-driven optimization when boundaries between specific labels are confusing, and use search-based optimization for many-class tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Boundary-learnable taxonomy is post hoc; paradigm-selection framework lacks prospective validation.","rationale":"The reader's verdict (CONDITIONAL) is appropriate: the empirical observations are honestly reported and the limitations section is unusually candid. However, the single most load-bearing issue is not just the batch-coverage limitation (which ERGO's own results on MASSIVE already concede), but the fact that the paradigm-selection framework—a headline contribution—depends on a 'boundary-learnable' label that is assigned post hoc. The paper defines boundary-learnable tasks as those where ERGO wins, then advises applying ERGO to boundary-learnable tasks, which is circular unless a prospective measure exists. The reader's weakest_assumption mentions this separately ('the boundary-learnable label is assigned after observing where ERGO wins'), so my concern partially overlaps; I sharpen it by identifying it as the primary threat to the central claim. A concrete pre-registered test on new datasets can settle whether the taxonomy has predictive power. If the test fails, the paper's contribution reduces to a set of benchmark-specific observations, which is still valuable but does not support the general selection guidance. I therefore keep the reader's CONDITIONAL verdict but emphasize that the condition should include prospective validation of the framework, not merely code release or additional baselines. I give credit to the paper for its honest limitations, the interpretability of ERGO's rules, and the transfer experiments, which are independent evidence that the method works on the reported tasks. But the central 'complementarity framework' remains unproven without a non-circular operationalization.","tokens_in":14645,"tokens_out":8920,"duration_ms":98002,"concrete_test":"Pre-register 10–15 held-out classification datasets not used in the paper. For each, run a cheap baseline (e.g., ICL-Diversity with 20 demonstrations) on a validation set, compute the concentration of errors into the top-k confused label pairs (e.g., the fraction of all errors involving the most frequent off-diagonal entry, or 1 − normalized entropy of the confusion matrix). Rank datasets by this concentration score. Then run ERGO and the strongest baselines (ICL-Diversity, DSPy, GEPA) with fixed budgets. If ERGO's accuracy advantage over the best alternative is not significantly positively correlated with the pre-registered concentration score (e.g., Spearman ρ > 0.5 with a pre-specified threshold), and ERGO does not win on the top-ranked boundary-concentrated datasets, the framework's predictive claim is falsified. The test must be fully specified before observing ERGO's results on the","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central practical contribution is the paradigm-selection framework (Section 7.3, Table 4): practitioners should \"apply ERGO if classification errors concentrate in specific confused label pairs.\" But the term \"boundary-learnable\" is assigned after observing where ERGO wins (TREC, CLINC150, RTE, Ethos), and no independent, pre-hoc operationalization is provided. To use the framework, a practitioner must already know whether errors concentrate in confused pairs—which typically requires running a classifier and inspecting its confusion matrix. The recommendation is therefore circular: ERGO's own error-diagnosis loop is the only proposed way to detect the condition that justifies using ERGO. The paper provides qualitative evidence that ERGO finds such boundaries (Section 6), but it never shows a baseline confusion-based concentration measure that predicts ERGO's relative gain on held-out tasks. Without that, the claim that ERGO is \"a useful tool for a specific, identifiable class of tasks\" is not established: the class is identifiable only retrospectively. The reader's weaker-assumption about 20-item batch coverage is secondary because ERGO explicitly loses on many-class tasks like MASSIVE; the load-bearing defect is the unvalidated, post hoc taxonomy underpinning the selection guidance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an empirical comparison of three prompt-optimization paradigms for few-shot text classification—demonstration selection (ICL-Uniform/ICL-Diversity), exploration-based search (APE, DSPy, GEPA), and the authors' newly proposed error-driven diagnosis method ERGO. ERGO iterates over non-overlapping 20-example training batches, diagnoses confused label pairs from classification errors, and rewrites the full prompt (instruction, demonstrations, guidelines) via a diagnose-prescribe-rewrite meta-prompt, selecting the best prompt on a validation set. Across 8 benchmarks (2–150 classes), no method dominates: ICL wins on coverage-dependent tasks, DSPy/GEPA on many-class tasks, and ERGO on tasks the authors call 'boundary-learnable' (TREC, CLINC150, RTE, Ethos), with best accuracy on TREC (90.0%) and CLINC150 (94.4%), convergence in 3–5 iterations, interpretable rules, and transfer across model families. The paper also proposes a paradigm-selection framework based on task characteristics.","tokens_in":14919,"tokens_out":9054,"duration_ms":92862,"significance":"If the results hold, the paper makes a useful empirical contribution: a nuanced map of when iterative prompt optimization pays off, concrete evidence that error-driven rule refinement can discover non-obvious labeling conventions, and a practical recommendation to start from zero-cost ICL. Strengths include the honest statement that ERGO is not universally better, the inclusion of ablations, cross-model and cross-model-transfer experiments, seed variation, and detailed case studies with sample-level support, plus the appendix-level algorithm and meta-prompt for reproducibility. The 'no single method dominates' claim is well evidenced by Table 1's small average differences. However, the significance of the central practical contribution—the paradigm-selection framework—is currently limited because the key condition for choosing ERGO ('boundary-learnable') is defined post hoc and not measured prospectively; the paper also contains an internal inconsistency in the cross-model sample count and reports individual-win p-values without correction. These are fixable but affect the strength of the headline guidance.","major_comments":[{"comment":"The proposed decision rule — 'if classification errors concentrate in specific confused label pairs, apply ERGO' — cannot be applied without an independent, pre-hoc measure of error concentration. The term 'boundary-learnable' is introduced after observing that ERGO wins on TREC, CLINC150, RTE, and Ethos, and the only operational test offered is to run a classifier and inspect its confusion matrix. This makes the paradigm-selection framework circular in practice: a practitioner must already have a working classifier and know its error pattern before deciding to use ERGO, and no evidence is given that a cheap baseline's error-concentration statistic predicts ERGO's relative gain on held-out tasks. Please (a) define a concrete, computable measure (e.g., top-1 confused-pair share or error entropy from ICL-Diversity predictions on validation data); (b) report its values for the eight benchma","section":"§7.3, Table 4"},{"comment":"The significance infrastructure is not calibrated for the headline comparisons. The paper says 'paired t-test at 5% level across all 40 configurations' but then reports individual p-values for dataset-level wins (e.g., CLINC150 p=0.043, Yahoo p=0.046, 20Newsgroups p=0.013) without any multiple-comparison correction. With 8 datasets and multiple method pairs, these marginal p-values would not survive a Bonferroni or FDR control; the CLINC150 win in particular is very weak evidence as reported. Similarly, 'all pairwise p>0.6' for non-dominance is presented as if it established equivalence, but this ignores the multiple-comparison problem and the fact that absence of significant difference is not evidence of equality. Please report corrected p-values, confidence intervals, or explicitly demote the dataset-level win claims to descriptive observations. The qualitative complementarity story ca","section":"§4.1–4.2, Table 1"},{"comment":"There is a quantitative inconsistency in the homogeneous-transfer experiment. The text says 'five models from four families' and 'Pooled across all 200 (model, dataset, seed) triples,' but Table 2 and Table 10 list only four models (Claude Haiku 4.5, Claude Sonnet 4.5, Gemma 3 27B, Mistral Large 3). Four models × 8 datasets × 5 seeds = 160 triples, not 200. This discrepancy directly affects the reported pooled p-value (p=1.5×10−9) and the cross-model generalization claim. Please reconcile: either add the missing model and its table rows, or correct the text and recompute the pooled statistics. As written, the cross-model claim rests on an internally inconsistent N.","section":"§5.2, Tables 2 and 10"}],"minor_comments":[{"comment":"Algorithm 1 sets T = ceil(|D_train|/k), while §5.3 states the default is T=10. For a 150-example training set with k=20 this would give T=8, not 10. Align the pseudocode with the actual implementation or explain why T is fixed at 10 in the experiments.","section":"Algorithm 1 vs §5.3"},{"comment":"CLINC150 is listed with 150 training examples and 150 classes, implying exactly one training example per class. This makes the batch-coverage concern in the Limitations section even more severe and should be stated in the table caption.","section":"Table 5"},{"comment":"The p-value annotation uses stars only. Since the table reports one non-significant model (Gemma 3 27B) and several significance levels, include the exact p-values in the table or a footnote.","section":"Table 2"},{"comment":"Figure 1 is referenced in the overview but does not appear in the submitted text. Ensure the figure is included in the camera-ready version.","section":"§3.2 / Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The empirical core is solid and the paper is unusually honest about non-dominance, but the central 'paradigm selection' contribution needs a prospective operationalization of 'boundary-learnable,' and the N=200/5-model discrepancy in Section 5.2 must be checked. If those are fixed, this could become a strong empirical contribution; I recommend major revision rather than rejection because the path is clear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ERGO is a genuinely useful empirical study of prompt-optimization paradigms, with an honest \"no method dominates\" message. The main new contribution is the joint diagnose→prescribe→rewrite loop that updates instruction, demonstrations, and decision rules together from non-overlapping error batches, and the case studies show it actually learns transferable rules (TREC's \"companies count as human beings\" is a nice example). The paper also does the field a service by showing that ICL-Diversity, the cheapest baseline, is already optimal on most tasks, and by quantifying where iterative search pays off. That's a real finding.\n\nThe soft spots are real but not fatal. The \"boundary-learnable\" category is introduced after the authors saw where ERGO wins; Section 7.3 tells practitioners to apply ERGO when errors concentrate in confused pairs, but there is no independent measure of that condition that predicts ERGO's gain on held-out tasks. So the selection framework is more a useful organizing story than a validated decision procedure. I'd like to see a confusion-matrix concentration score computed from baselines, tested across new tasks before trusting that guidance. That's the load-bearing weakness, and the stress-test note is right about it.\n\nSecond, the significance testing is not corrected for the 40 configurations; the p=0.043 and p=0.046 wins would not survive a Bonferroni correction. The overall \"no dominance\" claim is robust, but the specific pairwise claims should be softer.\n\nThird, ProTeGi is cited as the closest instruction-only optimiser but not benchmarked, which weakens the claim that joint rewriting is the reason for ERGO's gains. And there is no code or data release, which limits reproducibility.\n\nThe batch-coverage limitation on many-class tasks is acknowledged in the limitations section, and ERGO loses on MASSIVE as expected, so that concern is handled honestly.\n\nWho is this for? Practitioners choosing among prompt optimization methods, and researchers working on when optimization helps. It deserves a serious referee; the core empirical map is valuable even if the framework needs prospective validation. I'd push for code release and a pre-registered or at least pre-hoc boundary-learnability measure before accepting.","headline":"Solid empirical map of prompt-optimization paradigms; the 'boundary-learnable' label is post hoc, but the core finding that no method dominates holds.","tokens_in":15373,"tokens_out":2366,"would_cite":true,"duration_ms":25493,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt optimization methods are complementary, and an error-driven method called ERGO wins specifically on classification tasks where errors concentrate in a few confused label pairs, by turning mistakes into explicit decision rules.","keywords":["prompt optimization","text classification","error-driven learning","in-context learning","decision rules","large language models","few-shot classification","paradigm comparison"],"falsifier":"Run ERGO on a many-class task where errors are spread thinly across many label pairs rather than concentrated: if it fails to beat a zero-cost demonstration baseline, then the boundary-learnable condition is doing the work, not the error-driven mechanism. More directly, take a task with known confusable pairs, hold those pairs out of the training batches, and check whether the learned rules still fix test errors; if they do not, the single-error-to-rule generalization fails.","tokens_in":14524,"feed_emoji":"🎯","tokens_out":5056,"duration_ms":52604,"temperature":0.7,"pith_summary":"This paper argues that the three main approaches to prompt optimization for text classification—selecting demonstrations, searching over prompt variants, and diagnosing errors—are complementary, not competing: no single method outperforms the others across eight benchmarks. It introduces ERGO, which iterates over the training set in non-overlapping batches, asks the language model to diagnose why examples are misclassified, and rewrites the instruction, demonstrations, and decision rules in one pass. On tasks where errors cluster in specific confused label pairs ('boundary-learnable' tasks), ERGO reaches 90.0% on TREC and 94.4% on CLINC150, converging in 3-5 iterations and producing human-readable rules. The practical payoff is a selection framework: start with free demonstration-based in-context learning, use ERGO when boundaries are learnable from error patterns, and use exploration-based search for many-class tasks.","feed_headline":"Mistakes become rules: ERGO hits 94.4% on 150-class intent","feed_subtitle":"Diagnosing a few errors beats searching prompts: 90.0% on TREC, 94.4% on CLINC150, in 3-5 iterations.","key_machinery":"ERGO's diagnose→prescribe→rewrite meta-prompt: a loop that classifies a non-overlapping 20-example batch, partitions the outputs into mistakes and correct predictions, asks the LLM to identify confused label pairs and explain why, then jointly rewrites the instruction, demonstrations, and decision guidelines. The work it does: single errors become generalizable rules (e.g., one misclassified company query yields 'organizations → human beings,' fixing many test queries), and the best-on-validation prompt is selected across iterations.","core_discovery":"On the paper's own terms, the central discovery is that error-driven refinement—diagnosing a small batch of mistakes, prescribing rules, and rewriting the full prompt—captures task conventions that neither demonstration selection nor accuracy-based search can, and that this makes ERGO the best method on boundary-learnable tasks while remaining statistically tied overall. The paper does not claim universal dominance; it claims a complementary niche. Evidence includes case studies where a single misclassified query triggers a rule that fixes dozens of test examples, and transfer experiments where prompts refined by a stronger model serve a weaker model with no measurable loss.","pith_inferences":["The 'boundary-learnable' property could be measured before optimization, e.g., by the concentration of a confusion matrix on a seed prompt, turning the retrospective label into a testable predictor of when ERGO will win.","Combining ERGO's explicit rules with demonstration-based in-context learning might capture both precise boundaries and broad coverage; the paper's ablation hints that demonstrations carry much of the signal, so a hybrid is a natural next step.","The same diagnose-prescribe-rewrite loop may apply beyond classification to structured outputs such as slot filling or relation extraction, where label pairs become semantic confusion classes.","Because ERGO's rules are plain text, they could be distilled into a small model or used to synthesize training data for fine-tuning, extending the transfer result beyond prompt reuse."],"forward_implications":["Practitioners can choose the cheapest adequate method: a zero-cost demonstration baseline first, ERGO for tasks with concentrated confusions, and exploration-based search for many-class tasks with overlapping categories.","Rules learned by a strong model transfer to a weaker model with no measurable accuracy loss, so one optimization run can be reused for cheaper inference.","ERGO's interpretable rules double as annotation guidelines or audit trails, unlike black-box searched prompts.","On boundary-learnable tasks, prompt optimization is not a coin flip: error-driven feedback converges in 3-5 iterations and is more stable across seeds than search methods."],"fun_headline_variants":["Error-driven ERGO tops boundary tasks: 94.4% on CLINC150","Mistakes to rules: ERGO wins on confused label pairs","ERGO's diagnose-rewrite loop beats search on tricky classes","From errors to rules: ERGO excels where boundaries blur","ERGO: error diagnosis yields rules for hard intent distinctions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"ERGO assumes that the mistakes in a random 20-example batch reflect the task's true confusions, and that the rules written from those few errors generalize to the whole test distribution; on many-class tasks a 20-item batch covers only 10-15% of classes per iteration, so this is exactly where ERGO is weakest.","fun_headline_variants_meta":{"raw":{"variants":["Error-driven ERGO tops boundary tasks: 94.4% on CLINC150","Mistakes to rules: ERGO wins on confused label pairs","ERGO's diagnose-rewrite loop beats search on tricky classes","From errors to rules: ERGO excels where boundaries blur","ERGO: error diagnosis yields rules for hard intent distinctions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001026,"raw_usage":{"total_tokens":4167,"prompt_tokens":756,"completion_tokens":3411,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":3320}},"tokens_in":500,"tokens_out":3411,"duration_ms":25697,"temperature":1.0,"reasoning_tokens":3320,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T11:03:09.832404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ERGO on a many-class task where errors are spread thinly across many label pairs rather than concentrated: if it fails to beat a zero-cost demonstration baseline, then the boundary-learnable condition is doing the work, not the error-driven mechanism. More directly, take a task with known confusable pairs, hold those pairs out of the training batches, and check whether the learned rules still fix test errors; if they do not, the single-error-to-rule generalization fails.","supporting_citations":[],"review_version":1}