{"id":"e193f615-3ab5-4e3b-989c-1c3ef77b793a","arxiv_id":"2501.06689","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"TAPO is a prompt optimization framework that selects task-specific evaluation metrics and evolves prompts, reporting small and partly conflicting gains over four baselines on six reasoning datasets.","lead":"This paper introduces TAPO, a prompt optimization system that lets a large language model pick task-specific scoring criteria and then evolves prompts through repeated selection and mutation. A generalist could read it to see how automated prompt tuning is becoming task-aware, though the reported gains over baselines are small and uneven.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 'similarity scores' are also the optimization objective, so TAPO's gains may reflect overfitting to cosine similarity rather than real task improvement; no data split is described.","rationale":"The reader's weakest_assumption correctly identifies that the metric used for evaluation is also the optimization objective and that no data split is described. My stress-test pass converges on the same point: the empirical superiority claim in Section III-B rests on Table I's similarity scores, which are computed with the same MiniLM cosine similarity that appears in the scoring function in Eq. (1). This creates a direct selection-bias risk: the evolutionary loop can inflate the reported number without improving the underlying task, and the absence of a held-out split prevents even a partial check. I agree with CONDITIONAL: the concern is concrete and addressable, not a refutation. The method description is clear enough that the missing experimental details could be supplied in a revision, and the public code means the check I propose is feasible. I do not see a separate, more load-bearing concern: the lack of error bars and the inconsistency in Table I are symptoms of the same underlying issue rather than an independent failure, because a proper out-of-sample evaluation with task-native metrics would resolve both. Therefore no verdict change beyond the reader's conditional assessment is warranted.","tokens_in":7903,"tokens_out":1643,"duration_ms":18272,"concrete_test":"Run TAPO and all baselines with a strict train/validation/test split, selecting prompts only on the validation fold and reporting once on the held-out test fold. Report task-native metrics (exact-match accuracy for GSM8K, AddSub, MultiArith, SingleEQ, SVAMP; per-task and aggregate accuracy for BBH) alongside the MiniLM cosine similarity score. If TAPO's advantage over CoT, APE, PE2, and PromptBreeder disappears or reverses under task-native metrics, the central claim of consistent task-level improvement fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in Section III-B is that TAPO 'consistently outperforms baseline methods by dynamically selecting and weighting task-specific metrics.' But the reported outcome in Table I is the 'similarity score' computed with all-MiniLM-L6-v2 cosine similarity (Section III-A), and the same cosine similarity is a component of the objective S(P) in Eq. (1) used for prompt selection and evolution (Sections II-C and II-D). The paper describes no train/validation/test split, so it is possible that prompts are selected directly on the same examples used to report final scores. Two concrete failure modes follow. First, evolutionary search over a prompt space can exploit the specific embedding model: a prompt that makes outputs lexically or semantically close to reference strings will raise the reported metric even if it does not improve correct reasoning, exact answer match, or other task-native quality. Second, without a held-out test split, any selection procedure—even one optimizing a benign metric—can overfit the evaluation set, and the reported numbers then overstate generalization. This concern is load-bearing because the entire comparison rests on Table I and Figure 2, whose numbers are computed from the very metric being optimized. The paper itself concedes TAPO does not always win (e.g., GSM8K GPT-3.5-turbo: CoT 83.70 vs TAPO 83.40; AddSub GPT-4o: CoT 100.00 vs TAPO 96.32), so the claim of consistent superiority already requires careful statistical support that the reported aggregate similarity scores do not provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents TAPO, a framework for automated prompt optimization in which an LLM classifies the task, selects evaluation metrics (similarity, diversity, perplexity, complexity) with weights, and then uses an evolutionary loop of mutation and tournament selection to refine prompts under the weighted scoring function S(P)=sum_i w_i*M_i(P). The authors compare TAPO against Zero-Shot CoT, APE, PE2, and PromptBreeder on BBH, GSM8K, AddSub, MultiArith, SingleEQ, and SVAMP using GPT-3.5-turbo, GPT-4o, and Llama3-8B-Instruct, and they report ablations that remove prompt optimization and multi-metric scoring. The main claim is that TAPO consistently outperforms baseline methods by dynamically selecting and weighting task-specific metrics.","tokens_in":8331,"tokens_out":4064,"duration_ms":38451,"significance":"If the empirical claim were supported, TAPO would be a useful contribution: it targets multi-task adaptation in prompt optimization, ships public code, includes ablations, and covers both closed and open models. However, the headline results are similarity scores computed with the same MiniLM cosine-similarity model that appears inside the optimization objective, and the paper gives no data split or uncertainty quantification. The contribution is therefore conditional on a re-evaluation with an objective-independent, held-out metric.","major_comments":[{"comment":"The evaluation metric in Table I and Figure 2 is described as cosine similarity between generated and reference texts using all-MiniLM-L6-v2, while Eq. (1) makes the same embedding-based similarity a component of S(P), the score used for tournament selection and mutation in §II-D. The reported numbers therefore measure the objective being optimized, so they cannot by themselves show that TAPO improves task-solving quality (exact answers, correct reasoning, etc.) rather than similarity to reference strings. Please report task-native metrics (e.g., exact-match accuracy) and/or a metric excluded from the optimization objective.","section":"§II-C, §III-A, Table I, Fig. 2"},{"comment":"The experiment settings do not describe any train/validation/test split. Without a held-out test set, prompt selection during evolution may be performed on the same examples used to compute the final reported scores, which would make the comparison reflect selection on the test set. Please specify how many examples are used for metric selection and prompt evolution, and evaluate the final prompts on a disjoint held-out set.","section":"§III-A"},{"comment":"The claim that TAPO consistently outperforms baseline methods is not supported by the point estimates in Table I: on GSM8K with GPT-4o, PromptBreeder scores 88.61 while TAPO scores 88.40, and on SVAMP with GPT-3.5-turbo, CoT scores 94.38 while TAPO scores 92.72. In addition, the asterisks for p<0.05 are not accompanied by variance, number of independent runs, or a description of the significance test. Please quantify uncertainty and either provide a precise definition of consistent or soften the claim.","section":"§III-B, Table I"}],"minor_comments":[{"comment":"The table header contains the typo 'SigleEQ'; it should be 'SingleEQ'.","section":"Table III"},{"comment":"The string 'V oke-k' appears to be a rendering artifact, and the correspondence to reference [4] is unclear; please correct the citation and the model name.","section":"Section I"},{"comment":"Table II includes a translation error detection example, but no translation dataset is listed in the datasets paragraph; please specify which of the six datasets or BBH tasks this example comes from.","section":"Section III-A, Table II"},{"comment":"The sentence 'Dynamically adapts to various tasks by selecting appropriate evaluation metrics...' lacks a grammatical subject; consider beginning 'TAPO dynamically adapts...'.","section":"Section II-A"},{"comment":"The figure caption and surrounding text do not state exactly which quantity is plotted (similarity score versus accuracy) or which datasets are aggregated in the reported average improvements; please clarify the axes and the aggregation procedure.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is serious because the paper's headline metric is the optimization objective, and the absence of a data split undermines the generalization claim. However, these defects are fixable with additional experiments using task-native accuracy metrics and a proper held-out evaluation, and the public code repository makes such a re-evaluation feasible; I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful nugget here is the dynamic metric selection module: TAPO asks the LLM to pick and weight evaluation metrics per task rather than using a fixed scoring function. That is not in APE, PE2, or PromptBreeder, and it is a sensible thing to try. The paper is also clearly written, and the code is public, which is real credit.\n\nThe problems are in the evaluation, and they are load-bearing. Table 1 reports \"similarity scores\" computed with the same all-MiniLM-L6-v2 cosine similarity that appears in the optimization objective in Eq. (1). So the numbers you are comparing are exactly what the optimizer is maximizing. That makes the gains hard to interpret even before you ask about overfitting. The paper describes no train/validation/test split, so there is nothing ruling out selection directly on the test set. On top of that, the prose in Section III-B says TAPO \"consistently outperforms\" baselines, but the table itself shows counterexamples: GSM8K with GPT-4o (PB 88.61 vs TAPO 88.40) and SVAMP with GPT-3.5 (CoT 94.38 vs TAPO 92.72). The paper even concedes this in the next paragraph, which is honest but still contradicts the earlier claim. There are no error bars, no number of runs, and the asterisks for p<0.05 are not backed by any description of a statistical test.\n\nThese are addressable issues, not fatal ones. The idea is reasonable, and the architecture is coherent. But as it stands, the central empirical assertion is not supported. If the authors re-run on a proper held-out split, report task-native metrics (exact match, accuracy) alongside similarity, and give variance or at least multiple seeds, the paper could be worth publishing. Without that, the numbers do not do the work they are asked to do.\n\nMy take: this deserves a serious referee, because the metric-selection idea is worth engaging with and the flaws are fixable in revision. But the referee should push for a complete redo of the experimental section. I would not cite it in its current form.","headline":"TAPO has a genuinely new metric-selection idea, but its own table contradicts the 'consistently outperforms' claim and the reported metric is the same cosine similarity used in optimization, so the empirical case needs major revision.","tokens_in":8788,"tokens_out":1509,"would_cite":false,"duration_ms":15552,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TAPO reports that letting each task choose its own evaluation metrics beats CoT, APE, PE2, and PromptBreeder on six datasets.","keywords":["automated prompt optimization","task-aware metric selection","evolutionary prompt search","multi-metric evaluation","large language models","cosine similarity scoring","chain-of-thought prompting","math word problem reasoning"],"falsifier":"Run TAPO with a strict split: optimize prompts on a training fold, validate on a validation fold, then evaluate on a never-seen test fold using exact-match or human-judged correctness instead of cosine similarity; if higher similarity scores do not accompany higher correctness on the held-out test fold, the central claim fails.","tokens_in":7687,"feed_emoji":"🤖","tokens_out":8552,"duration_ms":71988,"temperature":0.7,"pith_summary":"TAPO is an automated prompt optimization framework built around one idea: the right evaluation metric for a prompt depends on the task, so the optimizing LLM should choose and weight the metrics itself. For each dataset, the framework first classifies the task type, then selects and weights metrics from a pool of similarity, diversity, perplexity, and complexity, producing a task-specific scoring function. Candidate prompts are scored with that function, and the best ones are repeatedly mutated and re-selected through a tournament mechanism. On the paper's reported numbers, this scheme outperforms or matches CoT, APE, PE2, and PromptBreeder across BBH, GSM8K, AddSub, MultiArith, SingleEQ, and SVAMP on GPT-3.5-turbo and GPT-4o, and it improves Llama3-8B-Instruct's math reasoning scores by about 10.2% over CoT. The authors take the ablation results as evidence that both the multi-metric scoring and the prompt-optimization loop contribute to the gains.","feed_headline":"Task-picked metrics beat fixed prompts on six datasets","feed_subtitle":"Per-task weighted metrics edge out CoT, APE, PE2, and PromptBreeder on three LLMs and six datasets.","key_machinery":"The load-bearing mechanism is the pairing of Dynamic Metric Selection with the weighted scoring function $S(P)=\\sum_{i=1}^{n} w_i M_i(P)$, where the $w_i$ are assigned by the LLM according to task priority and the $M_i$ are similarity, diversity, perplexity, and complexity scores. That function turns 'adapt to the task' into a concrete objective that drives the third module, Evolution-Based Prompt Optimization, which applies small strategy mutations such as 'break the task into steps' and uses tournament selection to keep the highest-scoring prompts. The named machinery is the metric-selection-then-evolve loop, and the claim is that it transfers across math, reasoning, and language tasks.","core_discovery":"On its own terms, the paper's central discovery is that prompt optimization benefits from making the fitness function task-dependent instead of fixed. TAPO operationalizes this with a three-module loop: an LLM-driven module selects task-relevant metrics and assigns weights, a scoring module combines them into $S(P)=\\sum_{i=1}^{n} w_i M_i(P)$, and an evolution module mutates and tournament-selects the best prompts across iterations. The empirical core is Table I, where TAPO is either the top or second-best method in nearly every dataset-model pairing, with particularly large margins on arithmetic tasks such as AddSub and MultiArith under GPT-3.5-turbo. The paper interprets this as showing that dynamic metric selection plus evolutionary refinement yields prompts that are better adapted to each task's demands.","pith_inferences":["Because MiniLM cosine similarity is both the optimization objective and the reported evaluation metric, part of the gain could be improved embedding resemblance rather than improved reasoning; a held-out exact-match accuracy check would separate the two.","The same weighted-metric loop could be applied to tasks without reference answers by replacing cosine similarity with self-consistency or entailment-based proxies, but TAPO as presented does not test those settings.","Since no train/validation/test split is described, the reported numbers may be optimistic; rerunning the pipeline with separate optimization and evaluation partitions is a direct way to measure that optimism.","Letting the LLM choose its own evaluation criteria introduces a possible gaming channel: the model may select metrics its own outputs happen to satisfy. A control condition with fixed or randomly chosen metrics would isolate how much of the gain comes from adaptation rather than from the evolutionary search itself."],"forward_implications":["The reported gains are largest on arithmetic reasoning: with GPT-3.5-turbo, TAPO reaches 88.15% on AddSub and 89.26% on MultiArith, versus 58.61% and 69.00% for zero-shot CoT.","The improvement is not universal: on GPT-4o GSM8K, TAPO scores 88.40% versus PromptBreeder's 88.61%, and on GPT-3.5-turbo SVAMP it scores 92.72% versus CoT's 94.38%, so the paper's claim is consistency near the top rather than outright dominance.","Ablations indicate both components matter: removing multi-metric scoring drops SingleEQ from 89.06% to 75.91%, and removing the prompt-optimization loop drops MultiArith from 89.26% to 83.04%.","On Llama3-8B-Instruct, TAPO improves average math-reasoning similarity scores by 10.2% over CoT and 6.2% over PE2, suggesting the method also helps weaker open models keep output format during multi-step reasoning."],"supporting_citations":[{"why":"Introduces APE, the automated prompt-engineering baseline that TAPO compares against and aims to surpass.","marker":"[2]"},{"why":"Introduces PromptBreeder, the evolutionary prompt optimization baseline that supplies the closest comparison for TAPO's mutation-and-selection loop.","marker":"[6]"},{"why":"Introduces zero-shot chain-of-thought prompting, the generic 'Let's think step by step' baseline that TAPO outperforms on most datasets.","marker":"[15]"},{"why":"Introduces PE2, the iterative prompt refinement baseline used in both the GPT-model and Llama3 comparisons.","marker":"[16]"},{"why":"Specifies GPT-3.5-turbo, one of the three LLMs TAPO uses for optimization and evaluation.","marker":"[17]"},{"why":"Specifies Llama3-8B-Instruct, the open-source model on which TAPO reports consistent gains.","marker":"[19]"},{"why":"Supplies the all-MiniLM-L6-v2 embeddings used to compute the cosine similarity metric in the scoring function and in the reported table.","marker":"[20]"},{"why":"Supplies GPT-2 perplexity, the fluency metric used in the multi-metric scoring function.","marker":"[21]"},{"why":"Provides GSM8K, the multi-step math word problem dataset used for evaluation.","marker":"[13]"},{"why":"Provides BIG-Bench Hard, the 23-task benchmark used to test TAPO's generalization across diverse reasoning tasks.","marker":"[14]"}],"fun_headline_variants":["Task-aware metric selection sharpens prompt evolution","Per-task metric weights improve prompt optimization across six datasets","TAPO: Adaptive metrics for prompt optimization","Dynamic metric weighting boosts prompt optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM-chosen metrics, especially MiniLM cosine similarity, and their weights in the scoring function faithfully measure task quality, so that optimizing prompts against them transfers to genuinely better task solving.","fun_headline_variants_meta":{"raw":{"variants":["Task-aware metric selection sharpens prompt evolution","Per-task metric weights improve prompt optimization across six datasets","TAPO: Adaptive metrics for prompt optimization","Dynamic metric weighting boosts prompt optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000886,"raw_usage":{"total_tokens":3780,"prompt_tokens":855,"completion_tokens":2925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":2869}},"tokens_in":471,"tokens_out":2925,"duration_ms":21788,"temperature":1.0,"reasoning_tokens":2869,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:54:26.963219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TAPO with a strict split: optimize prompts on a training fold, validate on a validation fold, then evaluate on a never-seen test fold using exact-match or human-judged correctness instead of cosine similarity; if higher similarity scores do not accompany higher correctness on the held-out test fold, the central claim fails.","supporting_citations":[{"cited_title":"Gpt-3.5 turbo model documentation,","cited_arxiv_id":null,"evidence_quote":"Specifies GPT-3.5-turbo, one of the three LLMs TAPO uses for optimization and evaluation."}],"review_version":1}