{"id":"31d7b236-0ebc-4267-9384-30312a034ad2","arxiv_id":"2506.17700","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A review that categorizes 45 prompt optimization strategies into 11 classes and surveys their use across NLP tasks, models, and datasets, but with inconsistent counts and overlapping categories.","lead":"This paper surveys 45 published methods for optimizing prompts given to large language models, sorting them into 11 categories based on how they work. It is a reference map for researchers who want to know which prompt optimization technique to use for a given NLP task, though its usefulness is limited by internal errors and an incomplete taxonomy.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 11-class taxonomy is not a partition: BPO appears in two classes, AutoPrompt's class is ambiguous, and the '3 soft / 42 hard' count contradicts the methods listed, so the paper's central map is not well-defined.","rationale":"The reader's verdict of REJECT is supported by the paper's internal inconsistencies. The central claim is that the paper provides 'unique and comprehensive insights' by categorizing 45 prompt optimization strategies into 11 distinct classes. This requires the classification to be a partition, but the paper's own text contradicts that requirement: BPO is placed in two classes, AutoPrompt's category is ambiguous, and the soft/hard counts are irreconcilable with the methods listed. These are not matters of disagreement with an external consensus; they are internal contradictions in the manuscript's core contribution. A review paper whose main output is a taxonomy cannot serve as a reliable reference if the taxonomy's membership rules are not well-defined. The concrete test proposed would settle whether the taxonomy can be repaired: if independent re-reading yields a consistent one-class-per-method assignment, the contradictions might be typographical; if not, the central contribution is invalid as stated. Since the reader already identified this same load-bearing concern and I find it confirmed, no adjustment to the verdict is needed.","tokens_in":49109,"tokens_out":3740,"duration_ms":37136,"concrete_test":"Independently re-read each method description in Section 4 alongside the 11 class definitions and the soft/hard distinction from Section 1, and assign each of the 45 methods to exactly one class. Verify three things: (1) every method receives exactly one class label, so no row of the 45-by-11 assignment matrix has more than one entry; (2) BPO appears in exactly one of Human LLM Collaboration and Bayesian Optimization; (3) the soft-prompt count is recomputed by listing every method whose Section 4 description says 'learnable continuous vectors' or 'soft prompts', including at minimum Prompt-Tuning, P-Tuning, BBT, FedBPT, DEPT, LoPT, Prefix-Tuning, P-Tuning v2, and BBTv2. If the resulting count differs from 3, or if any method is assigned to two classes, then the partition assumption fails and the taxonomy needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is a structured map of 45 prompt optimization strategies into 11 distinct classes; for this to be meaningful, the classes must be a partition: every method belongs to exactly one class, and counts must reconcile. The paper's own text violates this. Section 4.10 says BPO is the 'sole approach' of Human LLM Collaboration, while Section 4.11 lists BPO under Bayesian Optimization; under standard reading it is in two classes, so the categories are not mutually exclusive. AutoPrompt is described in Section 4.1 as gradient-based, yet it is also a discrete hard-prompt method; the paper never specifies whether classification is by optimization signal or by prompt representation, making its assignment ambiguous. The soft/hard counts also fail: Section 1 states '3 soft prompts and 42 hard prompt', but Section 4 describes at least Prompt-Tuning, P-Tuning, BBT, FedBPT, DEPT, LoPT, Prefix-Tuning, P-Tuning v2, and BBTv2 as learnable continuous prompts, before even counting InstructZero's soft-prompt phase or 'Soft Prompt' itself. Additionally, Section 7's model-count percentages (21% + 34% + 34% + 7% + 8% + 18%) sum to about 122%, indicating non-exclusive or erroneous categories. Since the taxonomy is the 'unique and comprehensive' contribution, its failure to be a partition means the central claim lacks a well-defined object.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a systematic review of prompt optimization strategies for large language models in NLP. The authors describe a two-stage literature selection process (379 candidate articles reduced to 45), propose an 11-class taxonomy of prompt optimization methods organized by 'working paradigm' (gradient-based, single-layer, multi-layer, interpretable, reinforcement-learning, enumeration, evolutionary, in-context learning, LLM-based, human-LLM collaboration, Bayesian optimization), and survey the NLP tasks, pretrained language models, and benchmark datasets on which these methods have been evaluated, culminating in ten performance tables and a call for standardized benchmarking. The paper's central claim is that it provides 'unique and comprehensive insights' and 'a robust foundation for future comparative studies.'","tokens_in":49398,"tokens_out":14703,"duration_ms":117513,"significance":"The survey addresses a timely and genuinely useful need: a structured map of prompt optimization methods together with the tasks, models, and datasets used to evaluate them. Its strengths are the breadth of the compilation (45 methods, ten detailed tables), an explicit and reproducible-looking selection methodology that follows prior review protocols, and a fair critical discussion of non-standardized evaluation practices, particularly the inconsistencies in train/test splits and metrics across studies. The paper also makes falsifiable quantitative claims—method counts, soft/hard prompt counts, and model-usage percentages—which is a credit to its transparency, though several of these claims fail when checked against the paper's own content. If the internal inconsistencies identified below are corrected, the survey would be a serviceable reference; as it stands, the central organizing claims are not reliable.","major_comments":[{"comment":"The soft/hard prompt count stated in Section 1 ('3 soft prompts and 42 hard prompt based optimization strategies') is contradicted by the paper's own classifications. Section 4.2–4.3 and Figure 6 identify at least ten continuous-prompt methods—Prompt-Tuning, P-Tuning, Prefix-Tuning, P-Tuning v2, Soft Prompt, BBT, BBTv2, FedBPT, DEPT, and LoPT—and Section 4.11 adds InstructZero's soft-prompt refinement stage. Because the soft/hard dichotomy is the paper's own stated organizing frame for the 45 methods, the enumeration that anchors the entire survey is inaccurate and must be recomputed and reconciled with Figure 6.","section":"Section 1; Section 4; Figure 6"},{"comment":"The distribution of methods by number of pretrained language models used does not sum to 100%: the text reports 21% (one PLM) + 34% (two) + 34% (four) + 7% (three) + 8% (five) + 18% (six to seventeen) = 122%. These percentages cannot all be correct for a partition of the 45 methods, so the quantitative claims about model-usage diversity in Section 7 are not supported and need to be recomputed from the underlying data.","section":"Section 7; Figure 16"},{"comment":"The task taxonomy is not internally consistent. Section 5 opens by promising 'nine key NLP tasks' and includes Information Retrieval (5.7), but Section 6—which opens by also saying '9 different classes of NLP Tasks'—contains ten subsections, replaces Information Retrieval with Semantic Parsing (6.5) and Knowledge and Contextual Understanding (6.9), and never returns to IR. The claimed comprehensive task overview must be reconciled so that the tasks enumerated in Section 5 are exactly those analyzed in Section 6.","section":"Section 5; Section 6"},{"comment":"The central taxonomy is not a well-defined partition. BPO is introduced in Section 4.10 as the 'sole approach' of Human LLM Collaboration, yet by the criteria given in Section 4.11 (methods that employ Bayesian optimization over candidate prompts with LLM-provided feedback) BPO belongs equally to the Bayesian Optimization class; InstructZero likewise spans soft-prompt tuning and Bayesian optimization; and AutoPrompt is simultaneously gradient-based (Section 4.1) and a hard-prompt method. The paper never states whether the 11 classes are mutually exclusive or orthogonal facets, nor which axis is primary for assignment, so a reader cannot assign a new method to exactly one class and the claim of '11 distinct classes' is undefined.","section":"Section 4.10; Section 4.11; Figure 6"}],"minor_comments":[{"comment":"The paragraph on PROMPTBREEDER contains a leftover drafting artifact—'Here's a comprehensive rephrasing of your statement, followed by five candidate sentences:'—which must be deleted.","section":"Section 4.7"},{"comment":"The screening arithmetic is off: 232 articles pass title/abstract screening, but 185 discarded plus 45 kept equals 230, leaving two articles unaccounted for.","section":"Section 2.2"},{"comment":"Section 7 refers to 'Table 16' where the referenced object is Figure 16; the decoder-based model list contains the fragment '-tuning'; and Figure 16 labels 'OPRP' where the text consistently uses OPRO.","section":"Section 7; Figure 16"},{"comment":"Method names are used inconsistently throughout (MOP vs. MoP, PROMPTBREEDER vs. Prompt BREEDER, STablePRompt in Table 9, Doscrete-v in Figure 6), and several methods appearing in the tables are never cited at first mention or at all (e.g., Soft Prompt, Active Examples, AEO, ABO before [60], LongPrompt, and PROPANE in Figure 16).","section":"Section 4; Tables 1-10"},{"comment":"The disambiguation-qa rows for MoP and AEO are merged into a single entry with one accuracy value and one model column, so it is unclear which method the reported accuracy of 68 refers to.","section":"Table 1"},{"comment":"Numerous typos and grammatical slips remain, including 'multilable' (5.1), 'doamin' (Section 2 sample query), 'aaccuracy' (Section 6.1), 'Diaglouge' (Table 5), and 'To best of our knowledge' in the Abstract and Section 1.","section":"Section 5.1; Section 2.1; Section 6.1"},{"comment":"The search engine list in Figure 3 includes both 'Elsevier' and 'ScienceDirect' as separate items although ScienceDirect is Elsevier's platform, and the text names 'the archive' without specifying arXiv.","section":"Figure 3"},{"comment":"The order of classes in Figure 6 (Single Layer first) differs from the order in Section 4 (Gradient based first), and the figure's method-to-column assignments for FluentPrompt and the reinforcement-learning group are not clearly consistent with Sections 4.1 and 4.5; the figure and the text should be carefully aligned and checked against the numbered categories.","section":"Section 4; Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript requires a substantial editorial overhaul and a recomputation of the quantitative claims before it can be considered further. If the authors cannot reconcile the soft/hard counts and the Section 7 percentages, the central claims of the survey would remain unsupported; the deeper issue is the undefined partition rule for the 11 classes, which should be resolved by explicitly stating the primary classification axis. The self-citations [7,8] are not load-bearing and could be replaced with more standard NLP-task references."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe honest headline: this is a broad, earnest survey of 45 prompt optimization methods with a genuinely useful compilation of task/benchmark/model tables. The authors also land a real point: evaluation in this field is so inconsistent—different splits, metrics, sample sizes—that direct comparisons are nearly impossible. That observation is worth something.\n\nWhat is new: grouping the methods into 11 working-paradigm classes, with a timeline and per-task performance tables, is a framing I have not seen at this scale. If the taxonomy were clean, it would give researchers a convenient map of the area.\n\nIt is not clean. The central contribution is load-bearing and it fails on the paper's own evidence. BPO is called the sole approach of Human LLM Collaboration (Section 4.10) and is also listed under Bayesian Optimization (Section 4.11). AutoPrompt is classified as gradient-based (Section 4.1) but is a discrete hard-prompt method; the paper never states whether classification follows optimization signal or prompt representation. The Introduction says “3 soft prompts and 42 hard prompt” strategies, but Section 4 alone describes at least Prompt-Tuning, P-Tuning, BBT, FedBPT, DEPT, LoPT, Prefix-Tuning, P-Tuning v2, BBTv2, InstructZero's soft phase, and “Soft Prompt” itself—all learnable continuous prompts. The count does not add up. Section 7's model-count percentages (21+34+34+7+8+18) sum to about 122%, so the categories overlap or the numbers are misreported; either way the analysis is not reliable.\n\nSecondary issues run in the same direction: the article-selection protocol (379 → 232 → 45) is not reproducible—no search dates, no excluded-article list, no stated criteria beyond “domain experts”; several methods appear in tables (AEO, PROMST, PROPANE, Soft Prompt) without being defined in the taxonomy; Section 5 lists information retrieval as a covered task but no IR evaluation appears elsewhere; and the text contains leftover drafting language (“Here's a comprehensive rephrasing...”). The self-citations to the authors' own work are not load-bearing, so I see no circularity problem there.\n\nCredit where due: the benchmark-standardization concern is argued in detail, the tables are extensive, and the survey format is legitimate even without new results.\n\nWho is this for: someone entering prompt optimization who wants a quick census of methods and a catalogue of evaluation inconsistencies. But the map is not trustworthy until the taxonomy is fixed. I would not cite it in the current form, and I would send it back for major revision rather than desk-reject it: the raw material is there, but the internal contradictions must be resolved before the classification can function as a reference.","headline":"A broad compilation of prompt optimization methods with a useful benchmark critique, but the central 11-class taxonomy is internally inconsistent and needs major correction before it can be used.","tokens_in":49909,"tokens_out":4659,"would_cite":false,"duration_ms":43124,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This review claims that 45 prompt-optimization strategies can be grouped into 11 distinct working-paradigm classes, and it catalogs the tasks, models, and datasets used to evaluate them.","keywords":["prompt optimization","large language models","taxonomy","soft prompts","hard prompts","reinforcement learning","evolutionary algorithms","Bayesian optimization"],"falsifier":"Counting whether any method appears in more than one class of the paper's Figure 6, or comparing the stated 3-soft/42-hard split against the paper's own list of continuous prompt methods, would settle whether the 11 classes partition the 45 strategies.","tokens_in":1349,"feed_emoji":"🗺️","tokens_out":1526,"duration_ms":46021,"temperature":0.7,"pith_summary":"This paper attempts to fill a gap in the NLP literature: many surveys cover prompt engineering, but few analyze prompt optimization as a distinct field. It claims to provide comprehensive insights by studying 45 prompt-optimization strategies and grouping them into 11 classes based on their underlying working paradigms, such as gradient-based, reinforcement learning, evolutionary, and Bayesian approaches. It also compiles the NLP tasks, pretrained models, and benchmark datasets used to evaluate these strategies. If the classification holds, researchers gain a structured map of the field and a foundation for standardized comparisons of prompt optimization methods.","feed_headline":"One map sorts 45 prompt-optimization methods into 11 classes","feed_subtitle":"A review of the field maps where each strategy works and why fair comparisons are still missing.","key_machinery":"The organizing device is the working-paradigm taxonomy of Figure 6, which assigns each of the 45 methods to one of 11 classes based on the optimization principle used to refine prompts. The classes distinguish whether a method optimizes continuous embeddings or discrete text, and whether it relies on gradients, single-layer or multi-layer insertion, reinforcement learning, enumeration, evolutionary search, in-context learning, an LLM as optimizer, human collaboration, or Bayesian optimization. This taxonomy carries the argument by converting a scattered set of methods into a structured map that can be used for comparison and method selection.","core_discovery":"On the paper's own terms, the central discovery is that the existing 45 prompt-optimization strategies can be understood through a taxonomy of 11 working paradigms, ranging from soft prompts optimized at single or multiple layers to hard prompts refined by gradients, reinforcement learning, evolutionary search, in-context learning, human-LLM collaboration, and Bayesian optimization. The paper further claims that mapping each method to its paradigm, its evaluation tasks, and its pretrained model reveals where the field is concentrated and where evaluation practices are inconsistent. It presents this taxonomy and the accompanying performance tables as a foundation for future comparative studies of prompt optimization and LLM-based predictive pipelines.","pith_inferences":["The paper's contribution is stronger as a catalog than as a rigorous partition: several of its own placements, such as assigning BPO to both human-LLM collaboration and Bayesian optimization, and AutoPrompt to both gradient-based and hard-prompt categories, suggest the 11 classes overlap and should be treated as a menu of paradigms rather than a mutually exclusive grouping.","A natural extension of the review would be to convert the taxonomy into a decision procedure that selects a class by prompt type (discrete versus continuous) and model access (white-box versus black-box), which the paper describes but does not formalize.","The inconsistency the paper documents could be turned into a testable recommendation: evaluate a common suite of methods on one shared set of datasets, metrics, and splits to see whether the reported performance gaps persist or shrink.","The paper's observation that decoder-based and GPT-family models dominate evaluation implies that prompt-optimization results may be architecture-specific, and testing the same 45 strategies on encoder-only and encoder-decoder models would likely change the comparative picture."],"forward_implications":["Researchers can locate a new prompt-optimization method within an existing paradigm and compare it against direct relatives rather than against unrelated strategies.","The review makes it possible to see which NLP tasks, benchmark datasets, and pretrained models dominate evaluation, exposing gaps such as limited multilingual and domain-specific testing.","The documented inconsistencies in dataset splits, evaluation metrics, and sample sizes imply that reported accuracy numbers cannot yet be read as reliable cross-method rankings.","The taxonomy highlights emerging directions, especially hybrid approaches that combine human expertise with automated search, as candidates for further investment.","Because most methods are evaluated on only a few models, the field still lacks evidence on how well prompt-optimization strategies generalize across different model architectures and scales."],"supporting_citations":[{"why":"AutoPrompt anchors the gradient-based hard-prompt class as the first method in that category.","marker":"[19]"},{"why":"Prompt-Tuning anchors the single-layer soft-prompt class and is cited as its foundation.","marker":"[21]"},{"why":"Prefix-Tuning anchors the multi-layer class by introducing trainable prefix vectors.","marker":"[27]"},{"why":"Black-Box Tuning grounds the black-box and evolutionary line of single-layer optimization.","marker":"[23]"},{"why":"APE anchors the LLM-based class by formulating prompt optimization as a search task performed by an LLM.","marker":"[59]"},{"why":"OPRO anchors the LLM-as-optimizer paradigm and is evaluated across numerous tasks in the survey.","marker":"[53]"},{"why":"PromptBreeder anchors the evolutionary class through genetic-algorithm prompt evolution.","marker":"[41]"},{"why":"InstructZero anchors the Bayesian optimization class with its two-stage black-box instruction search.","marker":"[58]"},{"why":"BPO anchors the human-LLM collaboration class by combining human-crafted prompts with Bayesian optimization.","marker":"[56]"}],"fun_headline_variants":["45 prompt-optimization methods, 11 paradigms: a new map","New taxonomy sorts 45 prompt-optimization strategies into 11 groups","Prompt optimization: 45 methods, 11 paradigms, one roadmap","11 paradigms order 45 prompt-optimization methods","A field guide to 45 prompt-optimization techniques, sorted into 11 families"],"cache_read_input_tokens":52096,"weakest_assumption_plain":"The load-bearing premise is that each of the 45 methods belongs to exactly one of the 11 classes, so the taxonomy is a clean partition; the paper's own double assignments, such as BPO appearing under both human-LLM collaboration and Bayesian optimization, would undermine that premise.","fun_headline_variants_meta":{"raw":{"variants":["45 prompt-optimization methods, 11 paradigms: a new map","New taxonomy sorts 45 prompt-optimization strategies into 11 groups","Prompt optimization: 45 methods, 11 paradigms, one roadmap","11 paradigms order 45 prompt-optimization methods","A field guide to 45 prompt-optimization techniques, sorted into 11 families"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000722,"raw_usage":{"total_tokens":3216,"prompt_tokens":901,"completion_tokens":2315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2221}},"tokens_in":517,"tokens_out":2315,"duration_ms":16288,"temperature":1.0,"reasoning_tokens":2221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:02:30.140885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Counting whether any method appears in more than one class of the paper's Figure 6, or comparing the stated 3-soft/42-hard split against the paper's own list of continuous prompt methods, would settle whether the 11 classes partition the 45 strategies.","supporting_citations":[],"review_version":1}