{"id":"abce94c7-48db-4564-b9ca-3dc7bc9f168c","arxiv_id":"2608.02919","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FLARE, an error-driven prompt optimizer that combines reflective rewriting with a small few-shot reference set, reports higher scores than GEPA on all tested GPT-5 task-model pairs.","lead":"FLARE is a method that improves AI prompts by having a large language model diagnose its own mistakes on labeled examples and rewrite the instructions accordingly. The paper reports that this error-driven approach beats a strong recent prompt optimizer, GEPA, on retrieval, tool calling, and emotion classification benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains are selected from a test-based validation-size sweep and the final prompt memorizes validation examples, so the central data-efficiency claim is not yet established.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the reported gains may reflect memorization of validation examples and test-based selection rather than generalizable instruction improvements. This is exactly where the paper's central claim is least secure. The appendix provides direct evidence of memorization, and the validation-size sweep is explicitly evaluated on the test split, so the headline number is a selected maximum rather than an unbiased estimate. The data-efficiency narrative—reaching peak performance with 100 validation examples—depends entirely on that test-selected sweep, so the concern is not peripheral. The RAG and tool-calling results are less affected, but the GoEmotions result is the flagship and is used to argue that FLARE is 'strikingly data-efficient.' If the corrected protocol shows the gap persists, the claim is supported; if not, the paper's central argument weakens substantially. The reader's CONDITIONAL verdict is appropriate: the flaws are addressable, but the current evidence is insufficient to accept the claims as stated.","tokens_in":15200,"tokens_out":4898,"duration_ms":44102,"concrete_test":"Re-run the GoEmotions GPT-5.1 experiment with a corrected protocol: (1) choose validation-set size and iteration count using only a validation/development split (or nested CV), never test scores; (2) quantify and exclude test examples that are exact or near-duplicate matches of validation examples; (3) run a variant where the meta-optimizer is forbidden from emitting verbatim example overrides (or strip section H). If the FLARE-vs-GEPA gap on a truly held-out test set drops below significance, or if the 100-example peak disappears, the central data-efficiency claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that FLARE's error-aware optimization yields generalizable, data-efficient improvements over GEPA. Two protocol details undermine this. First, although Algorithm 1 states the test set is 'tracked for reporting only' (line 8), Table 1 reports FLARE's 'best configuration across the validation-set-size sweep,' and the sweep in Table 2 is evaluated on the full held-out test split. Selecting the validation-set size (and GEPA's light/heavy budget) from test scores means the headline +15.3 is a maximum over many comparisons, not an unbiased estimate. Second, Appendix A's Iteration 20 prompt contains a section titled 'SPECIAL OVERRIDE CASES (MEMORIZE)' with verbatim validation examples such as 'BLOCK BLOCK BLOCK him!' and 'downvoted heavily but i understand your experience. salute, soldier.' Hard-coding validation texts into the returned prompt is direct evidence that the optimizer is memorizing validation instances rather than inducing general rules. Even if the test set is disjoint, this raises the possibility that the gain reflects overfitting to the validation distribution, and the test-based selection guarantees the reported number is optimistic. The load-bearing assumption is that the reported gain would survive a protocol that uses only validation data for all decisions and excludes memorized examples.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FLARE, an iterative prompt-optimization framework in which a GPT-5-series meta-optimizer rewrites a task prompt based on per-instance error signals computed over a validation set, together with a small set of few-shot reference examples. The method is compared with GEPA, Promptomatix, and OpenAI's Prompt Optimizer on multi-label emotion classification (GoEmotions), a tool-calling benchmark (tau2-bench), and three retrieval-augmented reasoning tasks (HotPotQA, MedQA, 2WikiMultiHopQA). The central claims are that FLARE consistently outperforms GEPA on every task-model pair, reaches its GoEmotions peak with as few as 100 validation examples, and is more stable across seeds than GEPA. Appendix A documents the iterative prompt-optimization process, including the final prompt that was selected as validation-best.","tokens_in":15421,"tokens_out":3216,"duration_ms":31060,"significance":"If the reported results are reliable, FLARE would be a meaningful contribution to prompt optimization: it challenges GEPA's claim that instruction-only reflective evolution outperforms few-shot-grounded optimization, and its validation-set-size sweep provides a useful data-efficiency analysis. The paper's strengths include running configurations over three seeds with reported standard deviations, comparing multiple baselines, including an appendix with actual iterative prompts, and releasing code. However, two protocol issues bear directly on the central generalizability and data-efficiency claims: the headline GoEmotions configuration is selected using test-set performance, and the returned prompt contains verbatim validation examples as memorized override cases. These issues make the reported gains, as they stand, an optimistic upper bound rather than a reliable estimate of FLARE's generalizable performance.","major_comments":[{"comment":"The headline GoEmotions number is selected from a test-based validation-size sweep. Algorithm 1 states that the test set is 'tracked for reporting only,' but Table 1 reports FLARE's best configuration across the validation-set-size sweep, and Table 2 evaluates all configurations on the full held-out test split. Selecting the validation-set size (and GEPA's light/heavy budget) from test scores means the reported +15.3 gain in Table 1 is a maximum over eight validation sizes and multiple seeds, not an unbiased estimate of FLARE's expected performance. The authors should re-select the configuration using validation-only scores and report the resulting test performance; the current protocol cannot support the claim that FLARE 'reaches its peak performance using as few as 100 validation examples.'","section":"Section 5.2 / Table 2 / Algorithm 1"},{"comment":"The final prompt selected as validation-best contains a section titled 'SPECIAL OVERRIDE CASES (MEMORIZE)' that hard-codes exact validation texts such as 'BLOCK BLOCK BLOCK him!' and 'downvoted heavily but i understand your experience. salute, soldier.' with forced labels. This is direct evidence that the optimizer is memorizing validation instances rather than learning generalizable instruction rules. Even if the test set is disjoint, this overfitting to the validation distribution can inflate test scores when the test distribution is similar, and it directly undermines the paper's data-efficiency interpretation. The authors should quantify performance when these memorized override cases are removed, or adopt a protocol that explicitly prohibits verbatim validation examples from appearing in the returned prompt.","section":"Appendix A, Iteration 20"},{"comment":"The claim that FLARE is 'markedly more stable across seeds than GEPA' is based on comparing standard deviations of the selected best configurations, but the selection process itself uses test scores. Since GEPA's light/heavy budgets and the validation size are selected after seeing test results, the reported stability comparison is also affected by selection effects. A fairer comparison would hold the selection protocol fixed for both methods and report the stability of the selected configuration under that protocol.","section":"Section 5.2 / Figure 1"}],"minor_comments":[{"comment":"There is a typo in the heading 'CURIOUSITY [7] vs NEUTRAL [27]'; it should be 'CURIOSITY.'","section":"Appendix A, Iteration 1"},{"comment":"The RAG composite metric (0.3 x retrieval + 0.7 x generation quality) is not fully defined; please specify the exact retrieval and generation submetrics and how they are normalized before combination.","section":"Section 4.2"},{"comment":"Algorithm 1 passes the complete training set T to the meta-optimizer but does not specify how many reference exemplars are sampled or how they are selected; Section 4.1 mentions a fixed reference exemplar set of 50, but the relationship between T and the few-shot reference examples should be clarified in the algorithm pseudocode.","section":"Section 3, Algorithm 1"},{"comment":"The caption for Table 4 says 'Best optimized scores per dataset-LLM combination' but the RAG comparison is restricted to FLARE and GEPA; the caption should state this restriction explicitly, as is done in the body text.","section":"Table 4 caption"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's the honest read on FLARE. The core idea—using an LLM to diagnose per-instance errors and rewrite prompts, plus a fixed few-shot reference set—is a legitimate extension of GEPA/OPRO-style reflective optimization, and the empirical scope is genuinely solid: three task families, multiple models, seeds with error bars, and linked code. That part deserves credit.\n\nThe problem is the load-bearing claim of data-efficient, generalizable improvement. Two protocol details directly undercut it. First, Table 1 reports FLARE's \"best configuration across the validation-set-size sweep,\" and that sweep in Table 2 is scored on the held-out test split. Selecting the validation-set size from test scores means the headline +15.3 is a maximum over many comparisons, not an unbiased estimate. GEPA's light/heavy budget is similarly selected. Second, Appendix A's final prompt contains a section literally titled \"SPECIAL OVERRIDE CASES (MEMORIZE)\" with verbatim validation texts like \"BLOCK BLOCK BLOCK him!\" and \"downvoted heavily but i understand your experience. salute, soldier.\" Hard-coding validation examples into the returned prompt is direct evidence the optimizer is memorizing the validation distribution. Even if the test set is disjoint, that is not prompt induction of general rules; it is a lookup table. The authors are transparent about this—it's in the appendix—but transparency does not fix the validity of the central claim.\n\nThe stress-test holds up. I'd add one minor nuance: the RAG and tool-calling results show no memorization patterns, so those might be more trustworthy, but they share the same test-based selection of GEPA budgets, so the bias concern remains for the comparative numbers. There's also no significance testing beyond seed SDs; that's minor given the means.\n\nBottom line: a well-executed empirical paper with a serious methodological leak. The fix is straightforward—use only validation data for all selection decisions and remove or ablate the memorized override cases before re-reporting. If the advantage survives, it's a useful result. As written, the headline numbers cannot be taken at face value.\n\nFor peer review: I'd send it out. The method is clearly described, the flaw is specific and addressable, and the empirical backbone is otherwise careful. A good referee could turn this into a valid finding.","headline":"FLARE's error-aware prompt optimization is a real idea with careful experiments, but test-based configuration selection and memorized validation examples undermine the headline gains.","tokens_in":15961,"tokens_out":1845,"would_cite":false,"duration_ms":17937,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLARE beats the prior optimizer on every task-model pair it tests, with gains up to +15.3 points.","keywords":["prompt optimization","few-shot learning","reflective feedback","error-aware optimization","large language models","retrieval-augmented generation","tool calling","multi-label emotion classification"],"falsifier":"Take the final optimized prompt documented in the appendix and run it on a held-out set that excludes or paraphrases the exact strings the prompt memorizes, such as 'BLOCK BLOCK BLOCK him!' and 'downvoted heavily but i understand your experience. salute, soldier.'. If the GoEmotions gain collapses toward the baseline on those paraphrases, the advantage is memorization; if the gain persists, it is a generalizable rule.","tokens_in":14984,"feed_emoji":"🎯","tokens_out":6334,"duration_ms":52005,"temperature":0.7,"pith_summary":"This paper sets out to show that automated prompt optimization works best when the optimizer is grounded in concrete, per-example failures rather than in abstract instructions alone. The proposed method, FLARE, repeatedly diagnoses why a model got each validation example wrong and rewrites the prompt to fix those specific error patterns, helped by a small set of few-shot reference examples. Against a genetic-Pareto reflective-evolution baseline, the authors report winning on all ten task-model combinations they evaluate: up to +14.2 points on HotPotQA, 87.0% versus 81.0% on tool calling, and +15.3 micro-F1 on GoEmotions. They also report that FLARE reaches its best emotion-classification score with only 100 validation examples and is more stable across random seeds. The claim, if correct, is that reflective instruction evolution without few-shot grounding is leaving large, accessible gains on the table.","feed_headline":"FLARE beats the prior optimizer on every task-model pair","feed_subtitle":"Grounded in per-example failure diagnosis, it lifts HotPotQA by 14 points and peaks on just 100 validation examples.","key_machinery":"The load-bearing mechanism is the error-aware reflective feedback loop. For each validation example the optimizer receives a signal CORRECT or WRONG($y_i$, $\\hat{y}_i$) with the true label and the model's wrong prediction, and a strong LLM meta-optimizer performs a thinking pass to diagnose the root cause and rewrite the prompt surgically. The loop is stabilized by a sliding window of the last three iterations' scores and prompt summaries, and by few-shot reference examples held fixed. For tool calling, multi-tool ground-truth sequences are decomposed into individual tool-call examples so the optimizer can tell whether failure is in tool selection, argument extraction, or formatting. This machinery is what grounds abstract reflection in concrete failures.","core_discovery":"The central claim, stated on the paper's own terms, is that reflective prompt optimization should be driven by explicit per-instance error signals. At each iteration, a frontier model acting as meta-optimizer looks at the current best prompt, the complete training set, the model's prediction and the ground-truth label for every validation example, the current validation score, and a sliding history of recent prompt-score pairs, then writes a corrected prompt that directly targets the diagnosed failure modes. The paper claims this error-aware, few-shot-grounded loop consistently outperforms an instruction-only Pareto-evolution baseline on every task-model pair tested, across retrieval-augmented reasoning, tool calling, and multi-label emotion classification, and that the advantages come from the quality of the search rather than from scale: a heavier baseline budget does not close the gap, and peak performance is reached with as little as 100 validation examples.","pith_inferences":["Inference: the appendix's special override cases show the optimizer memorizing exact validation strings, such as 'BLOCK BLOCK BLOCK him!', into the final prompt; how much of the headline gain comes from such memorization versus transferable rules is left unmeasured, and a paraphrased re-test would settle it.","Inference: the decomposition of multi-tool traces into per-tool examples is a general trick that could isolate failure types in other agentic pipelines, not just customer-service tool calling.","Inference: the error-aware loop's success with 100 examples suggests a practical recipe for quickly adapting prompts in new domains where labels are scarce, though the paper only demonstrates this on emotion classification.","Inference: FLARE's per-iteration full validation rescoring costs roughly 2.3 times the baseline's evaluation calls; if prompts are deployed widely, the authors' one-time offline cost framing invites a cost-benefit study of prompt optimization in long-lived production systems."],"forward_implications":["If FLARE's results hold, instruction-only reflective prompt evolution is leaving substantial accuracy on the table across RAG, tool calling, and classification.","Prompt optimization can be highly data-efficient: on GoEmotions the best prompt comes from 100 validation examples, so low-label settings are a natural application target.","Throwing more optimization budget at an evolutionary search does not substitute for a better search signal, since the heavier-budget baseline still trails FLARE at nearly every validation size.","The same steep-rise-then-plateau efficiency profile appears on two different GPT-5-series backbones, so the data-efficiency property is attributed to the method rather than the model.","Smaller gains on multi-hop reasoning suggest that errors deep in the retrieval chain cannot be fully repaired by prompt editing alone, pointing toward hybrid prompt-plus-retriever optimization."],"supporting_citations":[{"why":"Supplies the genetic-Pareto reflective prompt-evolution baseline that FLARE is designed to beat on every task.","marker":"[1]"},{"why":"Provides the DSPy framework used to implement modular pipelines, signatures, and evaluation.","marker":"[9]"},{"why":"Supplies the GoEmotions multi-label emotion dataset used for the headline classification results and the validation-size sweep.","marker":"[4]"},{"why":"Supplies the HotPotQA multi-hop RAG benchmark where FLARE reports its largest gain of +14.2 points.","marker":"[19]"},{"why":"Supplies the tau2-bench tool-calling dataset reformulated for tool-selection evaluation.","marker":"[3]"},{"why":"Supplies the 2WikiMultiHopQA benchmark where gains are smallest, supporting the paper's claim about prompt optimization limits.","marker":"[5]"},{"why":"Supplies the MedQA medical RAG benchmark for the reasoning evaluation.","marker":"[8]"}],"fun_headline_variants":["FLARE beats GEPA on every task with 100 examples","Error-aware reflection outdoes evolutionary prompt search","FLARE: per-instance failures drive prompt gains","FLARE peaks on 100 examples, beats GEPA on all"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains are assumed to come from improved, generalizable prompt rules rather than from the optimizer memorizing the specific validation examples it saw during optimization.","fun_headline_variants_meta":{"raw":{"variants":["FLARE beats GEPA on every task with 100 examples","Error-aware reflection outdoes evolutionary prompt search","FLARE: per-instance failures drive prompt gains","FLARE peaks on 100 examples, beats GEPA on all"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3127,"prompt_tokens":1044,"completion_tokens":2083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":2017}},"tokens_in":660,"tokens_out":2083,"duration_ms":14833,"temperature":1.0,"reasoning_tokens":2017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:55:02.962494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the final optimized prompt documented in the appendix and run it on a held-out set that excludes or paraphrases the exact strings the prompt memorizes, such as 'BLOCK BLOCK BLOCK him!' and 'downvoted heavily but i understand your experience. salute, soldier.'. If the GoEmotions gain collapses toward the baseline on those paraphrases, the advantage is memorization; if the gain persists, it is a generalizable rule.","supporting_citations":[{"cited_title":"Cohen, Ruslan Salakhutdinov, and Christopher D","cited_arxiv_id":null,"evidence_quote":"Supplies the HotPotQA multi-hop RAG benchmark where FLARE reports its largest gain of +14.2 points."}],"review_version":2}