Pith. sign in

REVIEW 3 major objections 6 minor 48 references

Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Study

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automatic prompt optimization can replace handcrafted prompts for knowledge-graph triple extraction.

desk verdict A useful first comparison of automatic prompt optimizers for triple extraction, but the headline claim is currently confounded by few-shot examples in the optimized prompts; fix the ablation and it becomes a solid empirical contribution. read the letter →

arxiv 2506.19773 v2 pith:OS7UYPHZ submitted 2025-06-24 cs.AI

Nandana Mihindukulasooriya , Niharika S. D'Souza , Faisal Chowdhury , Horst Samulowitz This is my paper · ORCID
classification cs.AI
keywords AutomaticPromptOptimizationEngineeringRelationExtractionKnowledgeGraphConstructionTripleLargeLanguageModelIn-ContextLearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether automatic prompt optimization—having an LLM propose and score candidate task prompts from input-output examples—can replace handcrafted prompting for the triple extraction step of knowledge graph construction, in which text is turned into (subject, relation, object) facts. It reports that three optimizers (DSPy, APE, and TextGrad) generate prompts that consistently beat a fixed baseline prompt on entity, relation, and triple F1, across six prompting strategies and two datasets. The gains grow with schema complexity (100 to 800 allowed relations) and input text length (up to 10x), exactly where the baseline degrades most. A manual review found the majority of generated prompts interpretable and aligned with human intuition: they state the task, the output format, and how to use the allowed relations. The implication is that automatic prompt optimization is a viable way to build KG construction pipelines that do not require expensive, brittle handcrafted prompts.

What carries the argument

The load-bearing mechanism is the prompt-optimization loop: an LLM proposes candidate instruction texts from input-output demonstrations, candidate prompts are executed against a validation set, scored by a metric (triple F1 by default), and the best candidate is retained. The three optimizers instantiate this loop differently—DSPy/MIPROv2 jointly searches instructions and few-shot demonstrations with Bayesian optimization, APE generates instruction candidates from a few examples without requiring an initial prompt, and TextGrad iteratively revises a prompt using natural-language feedback that minimizes $1-\text{F1}$. The task formulation is equally load-bearing: the prompt always carries the input text plus a list of allowed canonical relations, and the optimizer searches over instruction phrasings around that fixed contract.

What would settle it

Take the SynthIE small test set used in the default experiments and check each passage against the training, few-shot, and validation examples fed to DSPy, APE, and TextGrad; if any overlap exists, rerun the default comparison on a provably disjoint holdout and see whether the optimized-prompt F1 advantage shrinks.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is empirical: automatic prompt optimization is a viable strategy for triple extraction, with optimized prompts outperforming baseline prompts in precision, recall, and F1 for entities, relations, and triples under every prompting strategy tested—Predict, Chain-of-Thought, and Extract-Critique-Refine, in both triple-only and entity-relation-triple output forms. The benefit is not just an aggregate score: relation-type-level analysis shows more relation types improve than degrade after optimization, and relation types the baseline already handled correctly are largely retained. The largest relative gains appear where the task is hardest, as the number of allowed canonical relations grows from 100 to 800 and as passages are lengthened by concatenating related, random, or adversarial text. Among the three optimizers DSPy has a small edge, but all three beat the baseline; cross-dataset experiments show the gains shrink sharply when optimization examples come from a different dataset.

Load-bearing premise

The reported gains assume the 1500-example test set contains text the optimizers never saw during prompt search; if test passages double as few-shot or validation examples, the improvements could reflect memorization rather than genuinely better instructions.

Editorial extensions

If this is right

  • KG construction pipelines can drop manual prompt tuning for triple extraction and re-optimize prompts automatically whenever the underlying LLM changes.
  • On hard inputs—large schemas and long passages—the advantage of optimized prompts over a fixed baseline grows, so high-complexity extraction workloads benefit most.
  • Because the task-execution LLM drives most of the final performance, teams can spend optimization budget on a strong inference model while using a cheaper model to generate the prompt.
  • Optimized prompts do not transfer well across datasets: using a different dataset's training examples for optimization cuts the triple-F1 gain from about +8% to about +1%, so in-domain few-shot examples are essential for real deployments.
  • Optimization cost has diminishing returns: relation and triple F1 stabilize after a relatively small number of LLM calls, so heavy hyperparameter search is not needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the schema-complexity finding is to push beyond 800 allowed relations and see whether APO's advantage continues to grow; the monotone trend in the paper suggests it would, but that is not measured.
  • The cross-dataset transfer result implies that enterprise KG builders should optimize prompts on their own in-domain text; a cheap pilot would measure how many in-domain few-shot examples are needed to recover same-dataset gains.
  • The adversarial-augmentation result points to passage coherence as a separate difficulty axis; an optimizer that explicitly teaches segmentation or focused extraction might extend gains to very long documents, which this paper does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents an empirical study of automatic prompt optimization (APO) for triple extraction in knowledge graph construction. The authors evaluate three optimizers—DSPy (with MIPROv2), APE, and TextGrad—across two datasets (SynthIE and REBEL) and seven research questions that vary prompting strategy, LLM used for generation and execution, schema complexity (number of allowed relations), input text length and diversity, optimization metric, and cross-dataset transfer. The main claim is that optimized prompts are comparable to human-crafted prompts in content and improve triple extraction F1 over handcrafted baselines, with the largest gains at higher schema complexity and longer text. A qualitative annotation of 22 generated prompts by three co-authors is also reported.

Significance. If the central comparison were supported, the study would provide useful evidence that automatic prompt optimization is a viable alternative to handcrafted prompts for KG construction, and the systematic study of seven factors is a contribution to the empirical literature. The paper has strengths: it covers three distinct optimizer families, uses open-source models, presents relation-type-level analyses (Figures 3 and 4), and reports a qualitative human evaluation. However, the headline comparative claim is undermined by a confound between prompt text and few-shot demonstrations (Major Comment 1), and the evaluation lacks evidence for test-split disjointness (Major Comment 2) and statistical reliability (Major Comment 3). These issues are fixable and do not appear to invalidate the descriptive parts of the study, such as the relation-type overlap analyses and the cost-performance trade-offs.

major comments (3)
  1. [§4.3, Table 2, Figure 2] The main comparison in Table 2 and throughout is confounded: the baseline prompts in Table 1 contain no in-context demonstrations, while DSPy (with MIPROv2) optimizes instructions and few-shot examples jointly, APE randomly samples 10 validation examples as few-shots, and TextGrad uses an optional one-shot example. Figure 2 shows the optimized prompt appending 'selected few-shot examples from training data' to the baseline instruction, and Section 5.4 explicitly credits 'their selection as few-shot examples' for the improvement. Consequently, the reported gains of +0.08 to +0.10 in triple F1 could be entirely due to example selection rather than to the optimized prompt text. I recommend an ablation that fixes the demonstrations across baseline and optimized conditions (e.g., baseline prompt plus the same few-shot examples, or optimized prompt without examples) to isolate the prompt-wording effect. Without this, the headline conclusion that optimized prompts outperform handcrafted prompts is not uniquely supported.
  2. [§4.2–§4.3, §5.5] The paper never documents the disjointness of the SynthIE/REBEL test split from the train/validation splits used by the optimizers for few-shot selection and prompt selection. Section 4.3 describes selecting few-shots from validation data and validating prompts on a validation set, and Section 5.5 (Table 6) uses train/val/test splits of the same datasets, but no statement explains how the 1500-example small test set was sampled or whether it overlaps with the examples used during optimization. If the test set contains passages or triples seen during optimization, the reported improvements are inflated and the cross-setting comparisons in Tables 2, 4, 5, 7 and 8 are not interpretable. Please specify the sampling procedure and verify non-overlap, or re-run with an explicitly disjoint test split.
  3. [§5 (all tables)] All results are single runs with no standard deviations, confidence intervals, or significance tests. Because many headline differences are small (Table 6 reports a +0.01 F1 gain in one cross-dataset condition, and Table 8 reports differences of 0.01–0.03 across optimization metrics), the reader cannot assess whether the observed improvements are stable or within run-to-run noise. At minimum, report multiple optimization seeds or bootstrap confidence intervals for the main comparisons in Tables 2, 4, 5, and 7.
minor comments (6)
  1. [§4.3] The text says 'when DPSy is mentioned' but the correct acronym is 'DSPy'; please fix the typo.
  2. [§5.5] The paragraph beginning 'In DSPy, few-shot candidates are selected from a training set...' is repeated verbatim; one occurrence should be removed.
  3. [§5.2, §5.3] There are minor formatting inconsistencies: 'F1-scores of0.54' is missing a space, and 'LLAMA 3.3 70B' in Section 5.3 is inconsistent with the 'Llama3.3-70B' spelling used elsewhere.
  4. [§6] The word 'yieleded' should be 'yielded' in the paragraph about the two prompts receiving identical validation scores.
  5. [Table 4] The labels 'rel 100' through 'rel 800' are ambiguous; the caption should state that the number refers to the count of allowed relation types included in the prompt, not the number of relations in the input text.
  6. [Figure 2 and §4.3] Figure 2 states that few-shot examples come from training data, while Section 4.3 says APE draws few-shots from validation data; clarify which split is used for few-shot selection in each optimizer.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: the empirical comparison is self-contained, with only minor non-load-bearing self-citations and unresolved evaluation confounds that are not circularity.

full rationale

The paper's central claim is that automatic prompt optimizers generate human-like prompts that outperform handcrafted baselines for triple extraction. This is evaluated by comparing baseline prompts against optimizer-produced prompts on held-out test examples from SynthIE and REBEL, using standard precision/recall/F1 metrics. No result is defined in terms of the thing it claims to demonstrate: the optimizer-selected prompt is not assumed to be better by construction, and the test metric is not used as the optimization objective on the same examples. The self-citation to Text2KGBench [32] is used only to justify providing the relation schema in the prompt, which is a design choice rather than a load-bearing premise; other self-citations are contextual related-work references. The paper does not invoke any author-uniqueness theorem, and no ansatz is smuggled in via citation. The most plausible concerns are empirical rather than circular: (1) the DSPy/MIPROv2/APE/TextGrad optimized prompts include few-shot demonstrations while the baseline prompts do not, so part of the reported gains may be due to in-context example selection rather than prompt wording; the paper itself attributes part of the benefit to 'their selection as few-shot examples' in Section 5.4. This is a confound, not a definitional reduction. (2) The paper does not document whether the small test set is disjoint from the optimizer's validation/few-shot examples, which would be a data-leakage risk rather than circularity. Neither issue makes the derivation equivalent to its inputs, so the circularity score is minimal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No theory-derived free parameters exist; APO hyperparameters are examined in Section 5.8 and shown to have small effect. The main assumptions are about dataset splits, evaluation validity, and the strength of the baselines.

assumptions (4)
  • domain assumption SynthIE and REBEL ground truths are accurate for entity, relation, and triple extraction.
    All reported P/R/F1 numbers are computed against these labels; no error analysis of the datasets is provided. See Section 4.2.
  • domain assumption Train, validation, and test splits are disjoint, with no overlap between prompt-selection data and test data.
    The paper never describes split construction; the validity of reported test improvements relies on this assumption. See Sections 4.2, 4.3, and 5.
  • domain assumption Evaluation via exact string match against ground truth is a meaningful measure of extraction quality.
    Section 4.5 defines metrics as direct comparison of extracted values to ground truth; gains may partly reflect formatting alignment with canonical relation strings rather than semantic extraction quality.
  • domain assumption The handcrafted baseline prompts are representative of typical human prompt engineering effort.
    The comparison baseline in Table 1 consists of simple instructions; the 'similar to humans' claim depends on this comparison plus co-author annotation in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Study." pith.science (2026). https://pith.science/paper/OS7UYPHZ

@misc{pith2026250619773,
  author       = {Pith},
  title        = {Pith review of: Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OS7UYPHZ}},
  note         = {Machine review of arXiv:2506.19773}
}
read the original abstract

A KG represents a network of entities and illustrates relationships between them. KGs are used for various applications, including semantic search and discovery, reasoning, decision-making, natural language processing, machine learning, and recommendation systems. Triple (subject-relation-object) extraction from text is the fundamental building block of KG construction and has been widely studied, for example, in early benchmarks such as ACE 2002 to more recent ones, such as WebNLG 2020, REBEL and SynthIE. While the use of LLMs is explored for KG construction, handcrafting reasonable task-specific prompts for LLMs is a labour-intensive exercise and can be brittle due to subtle changes in the LLM models employed. Recent work in NLP tasks (e.g. autonomy generation) uses automatic prompt optimization/engineering to address this challenge by generating optimal or near-optimal task-specific prompts given input-output examples. This empirical study explores the application of automatic prompt optimization for the triple extraction task using experimental benchmarking. We evaluate different settings by changing (a) the prompting strategy, (b) the LLM being used for prompt optimization and task execution, (c) the number of canonical relations in the schema (schema complexity), (d) the length and diversity of input text, (e) the metric used to drive the prompt optimization, and (f) the dataset being used for training and testing. We evaluate three different automatic prompt optimizers, namely, DSPy, APE, and TextGrad and use two different triple extraction datasets, SynthIE and REBEL. Through rigorous empirical evaluation, our main contribution highlights that automatic prompt optimization techniques can generate reasonable prompts similar to humans for triple extraction. In turn, these optimized prompts achieve improved results, particularly with increasing schema complexity and text size.

Figures

Figures reproduced from arXiv: 2506.19773 by the authors.

Figure 1
Figure 1. An example of the triple extraction task with inputs and expected outputs (from the SynthIE dataset). [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. An example of an automatically optimized prompt using DSPy with the Llama 3.3-70B model and the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Number of relations types with +ve / -ve / 0 differences in accuracy after prompt optimization. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overlap of fully correct relation types, i.e., mean accuracy 1.0 (Upper, Green) and fully incorrect relation [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Difference in Macro F1 Scores (Optimized - Baseline) at different number of allowed relations. F1 is taken [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Number of relation types with +ve / -ve / 0 differences in mean accuracy after prompt optimization at each [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Training computation cost (in LLM calls) vs the relation and triple extraction F1. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 38 canonical work pages

  1. [1]

    Knowledge Graphs

    Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard de Melo, Claudio Gutierrez, Sab- rina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. Knowledge Graphs. ACM Computing Surveys (CSUR), 54(4):1–37, 2021

    work page 2021

  2. [2]

    A Comprehensive Survey on Automatic Knowl- edge Graph Construction

    Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. A Comprehensive Survey on Automatic Knowl- edge Graph Construction. ACM Computing Surveys, 56(4):1–62, 2023

    work page 2023

  3. [3]

    A survey of current approaches for mapping of relational databases to rdf

    Satya S Sahoo, Wolfgang Halb, Sebastian Hellmann, Kingsley Idehen, Ted Thibodeau Jr, Sören Auer, Juan Sequeda, and Ahmed Ezzat. A survey of current approaches for mapping of relational databases to rdf. W3C RDB2RDF Incubator Group Report, 1:113–130, 2009

    work page 2009

  4. [4]

    Rml: A generic language for integrated rdf mappings of heterogeneous data

    Anastasia Dimou, Miel Vander Sande, Pieter Colpaert, Ruben Verborgh, Erik Mannens, and Rik Van de Walle. Rml: A generic language for integrated rdf mappings of heterogeneous data. LDOW 2014, 1184, 2014

    work page 2014

  5. [5]

    Wikidata: A Free Collaborative Knowledge Base

    Denny Vrande ˇci´c and Markus Krötzsch. Wikidata: A Free Collaborative Knowledge Base. Communications of the ACM, 57(10):78–85, 2014

    work page 2014

  6. [6]

    Research trends for the interplay between large language models and knowledge graphs

    Hanieh Khorashadizadeh, Fatima Zahra Amara, Morteza Kamaladdini Ezzabady, Frédéric Ieng, Sanju Tiwari, Nandana Mihindukulasooriya, Jinghua Groppe, Soror Sahri, Farah Benamara, and Sven Groppe. Research trends for the interplay between large language models and knowledge graphs. In Proceedings of Workshops at the 50th International Conference on V ery Larg...

    work page 2024

  7. [7]

    Rodriguez-Méndez

    Edlira Vakaj, Nandana Mihindukulasooriya, Sanju Tiwari, and Sergio J. Rodriguez-Méndez. 4th international workshop on natural language processing for knowledge graph construction. WWW ’25, page 2545–2548. Association for Computing Machinery, 2025

    work page 2025

  8. [8]

    Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 2463–2473, Hong Kong, China, November 2019

    Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 2463–2473, Hong Kong, China, November 2019. 18 Automatic Prompt Optimization for Knowledge Graph Construction

    work page 2019

Show all 48 references
  1. [9]

    Knowledge graph generation from text

    Igor Melnyk, Pierre Dognin, and Payel Das. Knowledge graph generation from text. In Findings of the As- sociation for Computational Linguistics: EMNLP 2022 , pages 1610–1622, Abu Dhabi, United Arab Emirates, December 2022

    2022

  2. [10]

    Faisal Mahbub Chowdhury, Nandana Mihindukulasooriya, Owen Cornec, and Al- fio Massimiliano Gliozzo

    Gaetano Rossiello, Md. Faisal Mahbub Chowdhury, Nandana Mihindukulasooriya, Owen Cornec, and Al- fio Massimiliano Gliozzo. Knowgl: Knowledge generation and linking from text. In AAAI, pages 16476–16478. AAAI Press, 2023

    2023

  3. [11]

    Ex- ploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Text

    Hanieh Khorashadizadeh, Nandana Mihindukulasooriya, Sanju Tiwari, Jinghua Groppe, and Sven Groppe. Ex- ploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Text. In Text2KG at ESWC 2025, volume 3447, pages 132–153, 2023

    2025

  4. [12]

    Re2G: Retrieve, rerank, generate

    Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. Re2G: Retrieve, rerank, generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te...

    2022

  5. [13]

    From Local to Global: A Graph RAG Approach to Query-Focused Summarization

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv preprint arXiv:2404.16130, 2024

    2024 arXiv

  6. [14]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. InAdvances in Neural ...

    2020

  7. [15]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  8. [16]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [17]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  10. [18]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. 55(9), January 2023

    2023

  11. [19]

    PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Gong, and Xing Xie. PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. In Proceedings of the 1st ACM workshop on large ...

    2023

  12. [20]

    Ask me anything: A simple strategy for prompting language models

    Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christo- pher Re. Ask me anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  13. [21]

    Beyond english: The impact of prompt translation strategies across languages and tasks in multilingual llms

    Itai Mondshine, Tzuf Paz-Argaman, and Reut Tsarfaty. Beyond english: The impact of prompt translation strategies across languages and tasks in multilingual llms. LoResMT 2025 at NAACL 2025 , 2025

    2025

  14. [22]

    A systematic survey of automatic prompt optimization tech- niques

    Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang, et al. A systematic survey of automatic prompt optimization tech- niques. arXiv preprint arXiv:2502.16923, 2025

    2025

  15. [23]

    A survey of automatic prompt engineering: An optimiza- tion perspective

    Wenwu Li, Xiangfeng Wang, Wenhao Li, and Bo Jin. A survey of automatic prompt engineering: An optimiza- tion perspective. arXiv preprint arXiv:2502.11560, 2025

    2025 arXiv

  16. [24]

    Open information extraction from the web

    Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. Open information extraction from the web. Communications of the ACM, 51(12):68–74, 2008

    2008

  17. [25]

    Knowledge Graph Embedding with Hierar- chical Relation Structure

    Zhao Zhang, Fuzhen Zhuang, Meng Qu, Fen Lin, and Qing He. Knowledge Graph Embedding with Hierar- chical Relation Structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3198–3207, 2018

    2018

  18. [26]

    Open knowl- edge graphs canonicalization using variational autoencoders

    Sarthak Dash, Gaetano Rossiello, Nandana Mihindukulasooriya, Sugato Bagchi, and Alfio Gliozzo. Open knowl- edge graphs canonicalization using variational autoencoders. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP 2021) , pa...

    2021

  19. [27]

    Generative relation linking for question answering over knowledge bases

    Gaetano Rossiello, Nandana Mihindukulasooriya, Ibrahim Abdelaziz, Mihaela Bornea, Alfio Gliozzo, Tahira Naseem, and Pavan Kapanipathi. Generative relation linking for question answering over knowledge bases. In Proceedings of the 20th International Semantic Web Conference (ISW...

    2021

  20. [28]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:277...

    2022

  21. [29]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017

    2017

  22. [30]

    Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP 2022, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP 2022, 2022

    2022

  23. [31]

    An Explanation of In-context Learning as Implicit Bayesian Inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An Explanation of In-context Learning as Implicit Bayesian Inference. ICLR 2022, 2022

    2022

  24. [32]

    Enguix, and Kusum Lata

    Nandana Mihindukulasooriya, Sanju Tiwari, Carlos F. Enguix, and Kusum Lata. Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text. In 22nd International Semantic Web Conference ISWC 2023, volume 14266, pages 247–265, 2023

    2023

  25. [33]

    Automatic prompt optimization via heuristic search: A survey

    Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley Malin, and Sricharan Kumar. Automatic prompt optimization via heuristic search: A survey. arXiv preprint arXiv:2502.18746, 2025

    2025 arXiv

  26. [34]

    Fairness-guided few-shot prompting for large language models

    Huan Ma, Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu, Qinghua Hu, and Bingzhe Wu. Fairness-guided few-shot prompting for large language models. In NeurIPS 2023, pages 43136–43155, 2023

    2023

  27. [35]

    Gradient Descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic Prompt Optimiza- tion with “Gradient Descent” and Beam Search. EMNLP 2023, pages 7957–7968, 2023

    2023

  28. [36]

    Introducing MAPO: Momentum-Aided Gradient Descent Prompt Optimization

    Anthony Cui, Pranav Nandyalam, and Kevin Zhu. Introducing MAPO: Momentum-Aided Gradient Descent Prompt Optimization. 2025

    2025

  29. [37]

    Optimizing generative ai by backpropagating language model feedback

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639:609–616, 2025

    2025

  30. [38]

    Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery. Advances in Neural Information Processing Systems, 36:51008–51025, 2023

    2023

  31. [39]

    Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré

    Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Instructzero: Efficient instruction optimization for black-box large language models. 2024

    2024

  32. [40]

    Evoprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Evoprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers. 2023

    2023

  33. [41]

    Large Language Models are Human-Level Prompt Engineers

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large Language Models are Human-Level Prompt Engineers. In ICLR 2023, 2023

    2023

  34. [42]

    Query-dependent prompt evaluation and optimization with offline inverse rl

    Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl. In The Twelfth International Conference on Learning Representations , 2024

    2024

  35. [43]

    Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding

    Mirac Suzgun and Adam Tauman Kalai. Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding. arXiv:2401.12954, 2024

    2024 arXiv

  36. [44]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christo- pher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christo- pher Potts. Dspy: Compiling declarative language model calls into self-imp...

    arXiv

  37. [45]

    Exploiting asymmetry for synthetic training data generation: SynthIE and the case of information extraction

    Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. Exploiting asymmetry for synthetic training data generation: SynthIE and the case of information extraction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 1555...

    2023

  38. [46]

    REBEL: Relation extraction by end-to-end language generation

    Pere-Lluís Huguet Cabot and Roberto Navigli. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2370–2381, November 2021

    2021

  39. [47]

    Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. EMNLP 2024, pages 9340–9366, 2024. 20 Automatic Prompt Optimization for Knowle...

    2024

  40. [48]

    Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages

    Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages. In EMNLP 2023, pages 2695–2709, 2023. 21

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.