{"id":"055d37ad-122b-4d8f-95aa-327b3bc41653","arxiv_id":"2506.19773","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Testing three automatic prompt optimizers on triple extraction shows consistent gains over simple handcrafted prompts, with the largest gains at high schema complexity and long text.","lead":"This paper compares three automatic prompt optimizers on the task of extracting subject-relation-object triples from text. Optimized prompts beat simple handcrafted prompts, with the largest gains when the schema has many relations or the input text is long.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline prompts contain no in-context examples while DSPy/APE/TextGrad optimized prompts include few-shot demonstrations (Fig. 2, §4.3), so the reported gains may reflect example selection rather than prompt optimization.","rationale":"The reader's conditional verdict is reasonable, but their weakest assumption was split disjointness. That is a genuine reproducibility risk, since Section 4.2 does not document how the 1500-example test set was sampled or whether it overlaps the optimizer's validation/example data. However, the paper contains a more direct confound: the comparison between baseline and optimized prompts also changes the number and content of in-context demonstrations. Because DSPy explicitly co-optimizes instructions and few-shot examples, and because Figure 2 shows few-shot examples inside the optimized prompt, the observed gains cannot be attributed to prompt optimization without an ablation. This concern is internal to the experimental design, not a matter of disagreeing with external consensus or questioning author integrity. It is also testable at modest cost. A clean ablation on the existing setup would settle it. If the ablation shows that few-shot selection alone reproduces the gains, the central claim should be scaled back; if it shows that instruction optimization matters even without examples, the current conclusions stand. Until that ablation is reported, a conditional acceptance is the right posture, with the condition being explicit isolation of instructions from demonstrations. I agree with the reader that code and data release would help resolve secondary questions about splits and variance, but the few-shot confound is the load-bearing issue.","tokens_in":19201,"tokens_out":3370,"duration_ms":36866,"concrete_test":"On the same SynthIE 1500-example test set used in Table 2, run DSPy MIPROv2 under four conditions: (a) the base Predict (E-R-T) prompt with no examples; (b) the base prompt plus the optimized few-shot set selected by MIPROv2, with the base instruction restored; (c) the optimized instruction with zero demonstrations; and (d) the full optimized prompt. Repeat for APE and TextGrad. If triple F1 satisfies (d) ≈ (b) and (c) ≈ (a), the reported gain is few-shot selection, not prompt optimization. If (d) ≈ (c) > (a), the prompt text itself carries the benefit. Report the same comparison for at least the 100-relation and 800-relation settings to test whether the schema-complexity claim survives the ablation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that automatic prompt optimization generates human-like prompts that improve triple extraction. The evaluation, however, does not isolate the effect of the optimized prompt text from the effect of the few-shot examples that the optimizers also select. Section 4.3 states that DSPy uses MIPROv2 to optimize 'instructions and few-shot examples simultaneously,' and Figure 2 shows the optimized prompt explicitly appends 'selected few-shot examples from training data' to the baseline instruction. APE randomly picks 10 validation examples, and TextGrad's setup includes an optional one-shot example. The baseline prompts in Table 1 contain no such demonstrations. Therefore the +0.08 to +0.10 triple-F1 improvements in Table 2 could be caused entirely by adding in-context examples, independent of any improvement in the prompt wording. Section 5.4 even attributes part of the benefit to 'their selection as few-shot examples,' confirming that this confound is active in the pipeline. Without an ablation that holds demonstrations fixed, the headline conclusion that optimized prompts outperform handcrafted prompts is not uniquely supported. This is a more direct threat to the central claim than the unresolved test-split question, because a test-set overlap would inflate absolute numbers but this confound would invalidate the comparative conclusion itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of automatic prompt optimization (APO) for triple extraction in knowledge graph construction. The authors evaluate three optimizers—DSPy (with MIPROv2), APE, and TextGrad—across two datasets (SynthIE and REBEL) and seven research questions that vary prompting strategy, LLM used for generation and execution, schema complexity (number of allowed relations), input text length and diversity, optimization metric, and cross-dataset transfer. The main claim is that optimized prompts are comparable to human-crafted prompts in content and improve triple extraction F1 over handcrafted baselines, with the largest gains at higher schema complexity and longer text. A qualitative annotation of 22 generated prompts by three co-authors is also reported.","tokens_in":19437,"tokens_out":4110,"duration_ms":36439,"significance":"If the central comparison were supported, the study would provide useful evidence that automatic prompt optimization is a viable alternative to handcrafted prompts for KG construction, and the systematic study of seven factors is a contribution to the empirical literature. The paper has strengths: it covers three distinct optimizer families, uses open-source models, presents relation-type-level analyses (Figures 3 and 4), and reports a qualitative human evaluation. However, the headline comparative claim is undermined by a confound between prompt text and few-shot demonstrations (Major Comment 1), and the evaluation lacks evidence for test-split disjointness (Major Comment 2) and statistical reliability (Major Comment 3). These issues are fixable and do not appear to invalidate the descriptive parts of the study, such as the relation-type overlap analyses and the cost-performance trade-offs.","major_comments":[{"comment":"The main comparison in Table 2 and throughout is confounded: the baseline prompts in Table 1 contain no in-context demonstrations, while DSPy (with MIPROv2) optimizes instructions and few-shot examples jointly, APE randomly samples 10 validation examples as few-shots, and TextGrad uses an optional one-shot example. Figure 2 shows the optimized prompt appending 'selected few-shot examples from training data' to the baseline instruction, and Section 5.4 explicitly credits 'their selection as few-shot examples' for the improvement. Consequently, the reported gains of +0.08 to +0.10 in triple F1 could be entirely due to example selection rather than to the optimized prompt text. I recommend an ablation that fixes the demonstrations across baseline and optimized conditions (e.g., baseline prompt plus the same few-shot examples, or optimized prompt without examples) to isolate the prompt-wording effect. Without this, the headline conclusion that optimized prompts outperform handcrafted prompts is not uniquely supported.","section":"§4.3, Table 2, Figure 2"},{"comment":"The paper never documents the disjointness of the SynthIE/REBEL test split from the train/validation splits used by the optimizers for few-shot selection and prompt selection. Section 4.3 describes selecting few-shots from validation data and validating prompts on a validation set, and Section 5.5 (Table 6) uses train/val/test splits of the same datasets, but no statement explains how the 1500-example small test set was sampled or whether it overlaps with the examples used during optimization. If the test set contains passages or triples seen during optimization, the reported improvements are inflated and the cross-setting comparisons in Tables 2, 4, 5, 7 and 8 are not interpretable. Please specify the sampling procedure and verify non-overlap, or re-run with an explicitly disjoint test split.","section":"§4.2–§4.3, §5.5"},{"comment":"All results are single runs with no standard deviations, confidence intervals, or significance tests. Because many headline differences are small (Table 6 reports a +0.01 F1 gain in one cross-dataset condition, and Table 8 reports differences of 0.01–0.03 across optimization metrics), the reader cannot assess whether the observed improvements are stable or within run-to-run noise. At minimum, report multiple optimization seeds or bootstrap confidence intervals for the main comparisons in Tables 2, 4, 5, and 7.","section":"§5 (all tables)"}],"minor_comments":[{"comment":"The text says 'when DPSy is mentioned' but the correct acronym is 'DSPy'; please fix the typo.","section":"§4.3"},{"comment":"The paragraph beginning 'In DSPy, few-shot candidates are selected from a training set...' is repeated verbatim; one occurrence should be removed.","section":"§5.5"},{"comment":"There are minor formatting inconsistencies: 'F1-scores of0.54' is missing a space, and 'LLAMA 3.3 70B' in Section 5.3 is inconsistent with the 'Llama3.3-70B' spelling used elsewhere.","section":"§5.2, §5.3"},{"comment":"The word 'yieleded' should be 'yielded' in the paragraph about the two prompts receiving identical validation scores.","section":"§6"},{"comment":"The labels 'rel 100' through 'rel 800' are ambiguous; the caption should state that the number refers to the count of allowed relation types included in the prompt, not the number of relations in the input text.","section":"Table 4"},{"comment":"Figure 2 states that few-shot examples come from training data, while Section 4.3 says APE draws few-shots from validation data; clarify which split is used for few-shot selection in each optimizer.","section":"Figure 2 and §4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of an empirical NLP/AI venue, and the authors' qualitative analyses and descriptive findings are worth publishing once the comparison is made fair. The few-shot/prompt-wording confound is the key issue; test-split documentation and variance reporting are the other load-bearing gaps. I am not suggesting rejection, but the headline claim should be either re-scoped to 'APO that selects demonstrations improves over no-demonstration baselines' or supported by an ablation that isolates prompt text. Also, the 'first work' novelty claim for APO on triple extraction could be softened given the existing LLM-for-KG literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a broad, clearly organized empirical comparison of three automatic prompt optimizers (DSPy, APE, TextGrad) applied to triple extraction. The headline result—optimized prompts beat handcrafted ones—is currently undercut by a confound: the baselines in Table 1 contain no in-context examples, while the optimized prompts include few-shot demonstrations (Fig. 2, Section 4.3). That means the +0.08 to +0.10 F1 gains could come entirely from adding demonstrations, not from the improved instruction text.\n\nWhat is good: it is the first systematic comparison of these APO methods on triple extraction, and the experiment grid is sensible—prompting strategy, schema complexity, context length, inference versus prompt-generation LLM, and cross-dataset transfer. The finding that the inference LLM matters more than the prompt-generation LLM is interesting and plausible. The schema-complexity result, where optimized prompts degrade more gracefully from 100 to 800 relation types, is the most useful single insight. The cross-model transfer experiments add practical value. The paper is honestly written; the limitations section acknowledges the cross-dataset transfer drop and the dominance of the inference model.\n\nThe soft spots are real. The few-shot confound is the main one. Section 5.4 itself credits 'their selection as few-shot examples' for the longer-context gains, confirming that demonstrations are doing work. Without an ablation that adds the same selected few-shot examples to the baseline prompt, the comparative claim is not supported. The split-disjointness worry from the reader is secondary: a split overlap would inflate absolute numbers, but this confound would invalidate the comparison itself.\n\nAlso soft: single runs without error bars or significance tests, code and data not released, and the prompt-quality evaluation is self-referential. These are all fixable.\n\nWho is this for? People building practical KG construction pipelines who want to know whether APO is worth trying, and researchers working on prompt optimization evaluation. The paper deserves a serious referee, but it needs major revision: run the few-shot ablation, repeat runs with seeds, document how the test split was created, and release code and data.\n\nRecommendation: send to peer review. The topic is relevant and the experimental design is close to usable; a serious referee should engage with it, not desk reject.","headline":"A useful first comparison of automatic prompt optimizers for triple extraction, but the headline claim is currently confounded by few-shot examples in the optimized prompts; fix the ablation and it becomes a solid empirical contribution.","tokens_in":19953,"tokens_out":3354,"would_cite":false,"duration_ms":32688,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatic prompt optimization can replace handcrafted prompts for knowledge-graph triple extraction.","keywords":["Automatic Prompt Optimization","Prompt Engineering","Relation Extraction","Knowledge Graph Construction","Triple Extraction","Large Language Model","In-Context Learning"],"falsifier":"Take the SynthIE small test set used in the default experiments and check each passage against the training, few-shot, and validation examples fed to DSPy, APE, and TextGrad; if any overlap exists, rerun the default comparison on a provably disjoint holdout and see whether the optimized-prompt F1 advantage shrinks.","tokens_in":19034,"feed_emoji":"🔗","tokens_out":9460,"duration_ms":85869,"temperature":0.7,"pith_summary":"This paper asks whether automatic prompt optimization—having an LLM propose and score candidate task prompts from input-output examples—can replace handcrafted prompting for the triple extraction step of knowledge graph construction, in which text is turned into (subject, relation, object) facts. It reports that three optimizers (DSPy, APE, and TextGrad) generate prompts that consistently beat a fixed baseline prompt on entity, relation, and triple F1, across six prompting strategies and two datasets. The gains grow with schema complexity (100 to 800 allowed relations) and input text length (up to 10x), exactly where the baseline degrades most. A manual review found the majority of generated prompts interpretable and aligned with human intuition: they state the task, the output format, and how to use the allowed relations. The implication is that automatic prompt optimization is a viable way to build KG construction pipelines that do not require expensive, brittle handcrafted prompts.","feed_headline":"Prompt optimizers beat handcrafted prompts for knowledge graphs","feed_subtitle":"Across DSPy, APE, and TextGrad, optimized prompts beat baselines, with the largest gains on complex schemas and long text.","key_machinery":"The load-bearing mechanism is the prompt-optimization loop: an LLM proposes candidate instruction texts from input-output demonstrations, candidate prompts are executed against a validation set, scored by a metric (triple F1 by default), and the best candidate is retained. The three optimizers instantiate this loop differently—DSPy/MIPROv2 jointly searches instructions and few-shot demonstrations with Bayesian optimization, APE generates instruction candidates from a few examples without requiring an initial prompt, and TextGrad iteratively revises a prompt using natural-language feedback that minimizes $1-\\text{F1}$. The task formulation is equally load-bearing: the prompt always carries the input text plus a list of allowed canonical relations, and the optimizer searches over instruction phrasings around that fixed contract.","core_discovery":"On the paper's own terms, the central discovery is empirical: automatic prompt optimization is a viable strategy for triple extraction, with optimized prompts outperforming baseline prompts in precision, recall, and F1 for entities, relations, and triples under every prompting strategy tested—Predict, Chain-of-Thought, and Extract-Critique-Refine, in both triple-only and entity-relation-triple output forms. The benefit is not just an aggregate score: relation-type-level analysis shows more relation types improve than degrade after optimization, and relation types the baseline already handled correctly are largely retained. The largest relative gains appear where the task is hardest, as the number of allowed canonical relations grows from 100 to 800 and as passages are lengthened by concatenating related, random, or adversarial text. Among the three optimizers DSPy has a small edge, but all three beat the baseline; cross-dataset experiments show the gains shrink sharply when optimization examples come from a different dataset.","pith_inferences":["A testable extension of the schema-complexity finding is to push beyond 800 allowed relations and see whether APO's advantage continues to grow; the monotone trend in the paper suggests it would, but that is not measured.","The cross-dataset transfer result implies that enterprise KG builders should optimize prompts on their own in-domain text; a cheap pilot would measure how many in-domain few-shot examples are needed to recover same-dataset gains.","The adversarial-augmentation result points to passage coherence as a separate difficulty axis; an optimizer that explicitly teaches segmentation or focused extraction might extend gains to very long documents, which this paper does not test."],"forward_implications":["KG construction pipelines can drop manual prompt tuning for triple extraction and re-optimize prompts automatically whenever the underlying LLM changes.","On hard inputs—large schemas and long passages—the advantage of optimized prompts over a fixed baseline grows, so high-complexity extraction workloads benefit most.","Because the task-execution LLM drives most of the final performance, teams can spend optimization budget on a strong inference model while using a cheaper model to generate the prompt.","Optimized prompts do not transfer well across datasets: using a different dataset's training examples for optimization cuts the triple-F1 gain from about +8% to about +1%, so in-domain few-shot examples are essential for real deployments.","Optimization cost has diminishing returns: relation and triple F1 stabilize after a relatively small number of LLM calls, so heavy hyperparameter search is not needed."],"supporting_citations":[{"why":"Supplies the DSPy optimizer used in the main experiments, including the declarative prompt-as-program formulation.","marker":"[44]"},{"why":"Provides the MIPROv2 algorithm that jointly optimizes instructions and few-shot demonstrations inside DSPy.","marker":"[47]"},{"why":"Describes APE, which generates instruction candidates from examples with no initial prompt; one of the three optimizers compared.","marker":"[41]"},{"why":"Describes TextGrad, which revises prompts through natural-language gradient feedback; one of the three optimizers compared.","marker":"[37]"},{"why":"Provides the SynthIE dataset, the default training/validation/test source and the provenance of the example task in Figure 1.","marker":"[45]"},{"why":"Provides the REBEL dataset used for the cross-dataset transfer and second-dataset experiments.","marker":"[46]"},{"why":"Supplies the approach of passing the canonical relation schema inside the prompt, the task formulation on which all optimization experiments rest.","marker":"[32]"}],"fun_headline_variants":["Auto-optimized prompts beat handcrafted for triple extraction","Prompt optimizers shine on complex knowledge graph schemas","DSPy, APE, TextGrad all outdo manual prompts on KGs","Optimized prompts win big on long text and large schemas","KG triple extraction: auto prompt tuning beats hand tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume the 1500-example test set contains text the optimizers never saw during prompt search; if test passages double as few-shot or validation examples, the improvements could reflect memorization rather than genuinely better instructions.","fun_headline_variants_meta":{"raw":{"variants":["Auto-optimized prompts beat handcrafted for triple extraction","Prompt optimizers shine on complex knowledge graph schemas","DSPy, APE, TextGrad all outdo manual prompts on KGs","Optimized prompts win big on long text and large schemas","KG triple extraction: auto prompt tuning beats hand tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1499,"prompt_tokens":1074,"completion_tokens":425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":337}},"tokens_in":690,"tokens_out":425,"duration_ms":4567,"temperature":1.0,"reasoning_tokens":337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:24:55.102227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the SynthIE small test set used in the default experiments and check each passage against the training, few-shot, and validation examples fed to DSPy, APE, and TextGrad; if any overlap exists, rerun the default comparison on a provably disjoint holdout and see whether the optimized-prompt F1 advantage shrinks.","supporting_citations":[{"cited_title":"Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs","cited_arxiv_id":null,"evidence_quote":"Provides the MIPROv2 algorithm that jointly optimizes instructions and few-shot demonstrations inside DSPy."},{"cited_title":"Large Language Models are Human-Level Prompt Engineers","cited_arxiv_id":null,"evidence_quote":"Describes APE, which generates instruction candidates from examples with no initial prompt; one of the three optimizers compared."},{"cited_title":"Optimizing generative ai by backpropagating language model feedback","cited_arxiv_id":null,"evidence_quote":"Describes TextGrad, which revises prompts through natural-language gradient feedback; one of the three optimizers compared."},{"cited_title":"Exploiting asymmetry for synthetic training data generation: SynthIE and the case of information extraction","cited_arxiv_id":null,"evidence_quote":"Provides the SynthIE dataset, the default training/validation/test source and the provenance of the example task in Figure 1."},{"cited_title":"REBEL: Relation extraction by end-to-end language generation","cited_arxiv_id":null,"evidence_quote":"Provides the REBEL dataset used for the cross-dataset transfer and second-dataset experiments."},{"cited_title":"Enguix, and Kusum Lata","cited_arxiv_id":null,"evidence_quote":"Supplies the approach of passing the canonical relation schema inside the prompt, the task formulation on which all optimization experiments rest."}],"review_version":2}