Pith. sign in

REVIEW 5 major objections 5 minor 45 references

Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that IPOMP, a two-stage evaluation-data selection method that combines semantic clustering, boundary sampling, and real-time model-performance-guided replacement of redundant samples, makes automated prompt optimization…

desk verdict A plausible and novel evaluation-selection method for prompt optimization, undermined by internal inconsistencies and an unvalidated core mechanism; worth a serious referee and a careful revision. read the letter →

arxiv 2505.10736 v3 pith:GOOEFXFP submitted 2025-05-15 cs.CL

Ximing Dong , Shaowei Wang , Dayi Lin , Ahmed E. Hassan This is my paper · ORCID
classification cs.CL
keywords promptoptimizationevaluationdataselectioncoresetlargelanguagemodelssemanticclusteringboundarysamplingmodelperformanceguidanceiterativerefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the evaluation subsets used during automated prompt optimization should not be chosen at random or by semantics alone. It introduces IPOMP, a two-stage approach that first selects a diverse seed set by semantic clustering plus boundary samples, then iteratively swaps out redundant samples identified by correlated logits across candidate prompts. On BIG-bench and LIAR with GPT-3.5 and GPT-4o-mini, IPOMP reports accuracy gains of 1.6% to 3.1% over the best baseline and at least 50% lower standard deviation, at under 1% extra runtime. If correct, it supplies a cheap, model-agnostic layer that makes any iterative prompt optimizer more reliable.

What carries the argument

The central object is the per-sample performance vector across candidate prompts, a matrix of logits of size |S| by (|output| times |candP|). Two samples are redundant when their logit rows are highly correlated across candidate prompts; hierarchical clustering with threshold 0.9 groups them, and a fraction beta = 0.5 is replaced by the most semantically dissimilar training examples via HNSW search. That mechanism converts a static semantic selection into a dynamic one, and the ablation shows it is responsible for most of the stability gain, with standard deviation dropping by a factor of 2.83 when stage 2 is removed.

What would settle it

On a held-out task, compare the correlation between evaluation-set accuracy and full-test accuracy across candidate prompts for the stage-1 set before and after stage-2 refinement; if refinement does not raise that correlation, the redundancy signal is not doing the claimed work. A second check: swap in randomly chosen replacements instead of least-similar ones; if final prompt accuracy matches IPOMP, the dissimilarity rule is not the active ingredient.

Watch

Extended reading notes

Core claim

The central claim is that evaluation data for prompt optimization can be selected and continuously refined by combining semantic diversity with real-time model behavior, and that this beats both random sampling and existing coreset methods designed for model benchmarking. Stage 1 builds a small 20-sample set from K-means semantic clusters and from farthest-apart boundary pairs. Stage 2 uses logits produced by candidate prompts during optimization: samples whose performance vectors are highly correlated (threshold 0.9) are treated as redundant, and a fraction beta = 0.5 are replaced each iteration by the least semantically similar training samples. The paper reports that this raises accuracy by 1.6% to 3.1% over Anchor-Point, the strongest baseline, cuts standard deviation by at least 50%, adds less than 1% overhead, and that the stage-2 refinement alone improves the Random, Boundary, Clustering, and Prediction-based baselines when attached to them.

Load-bearing premise

The load-bearing premise is that when two evaluation examples show nearly identical confidence scores across candidate prompts, one of them is redundant for judging prompts, and swapping it for a very different example makes the small evaluation set a better predictor of which prompt will win on the full test set.

Editorial extensions

If this is right

  • Any iterative prompt optimizer can carry the cost of IPOMP: the added runtime is under 1% of the optimizer's execution time, so the method can be layered onto APE, APO, or EVOPROMPT without a separate warm-up stage.
  • Small evaluation sets become viable: IPOMP outperforms all baselines even with only 5 samples, and the paper's sweep suggests 20 samples is the sweet spot before returns flatten.
  • Existing coreset selectors inherit the stability gain: attaching stage-2 refinement to Random, Boundary, Clustering, Anchor-Point, and Prediction-based improves their accuracy by 0.3% to 2.3% and reduces their standard deviation by up to 60% in the reported configurations.
  • The method avoids the cost of prior performance data: unlike Anchor-Point, which needs a preliminary stage evaluating the whole training set on about 10 prompts, IPOMP collects its performance signal during optimization itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension not explored in the paper: the same redundancy logic could be applied to any loop that scores many candidate configurations on a fixed sample set, such as few-shot example selection or hyperparameter search, as long as per-sample confidence scores are available.
  • The paper fixes the replacement target as the least semantically similar sample; an alternative that follows directly from its own logit-correlation logic would be to swap in samples whose confidence vectors are least correlated with the retained set, which could outperform semantic dissimilarity on tasks where semantics and model behavior diverge.
  • The 0.9 correlation threshold is static, yet the paper's case study shows redundancy drops from 19% to 10% after one round; adapting the threshold as the prompt set evolves is a natural way to squeeze out further stability gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes IPOMP, a two-stage method for selecting evaluation data subsets to be used during automated prompt optimization. Stage 1 selects representative and diverse samples by combining semantic clustering with boundary-case selection. Stage 2 then iteratively replaces samples that are deemed redundant based on high logit correlations across candidate prompts, substituting the most semantically dissimilar training samples. The method is evaluated on BIG-bench and LIAR using GPT-3.5 and GPT-4o-mini across three prompt optimizers (APE, APO, EVOPROMPT), with accuracy and standard deviation reported. The authors claim effectiveness gains of 1.6% to 3.1% over the best baseline, stability improvements of at least 50%, and computational overhead below 1%, and they further claim that Stage 2 can be used as a universal enhancement to other data-selection baselines.

Significance. If the results hold, IPOMP addresses a practical gap in prompt optimization: evaluation subset selection. The paper's empirical scope is reasonable for a first study, with two datasets, two LLMs, three prompt optimizers, multiple baselines, an ablation study, sensitivity analysis, and cost/overhead measurements. The idea of using real-time model performance to refine evaluation samples is interesting and potentially useful, especially because it does not require pre-collected performance data. However, several central numerical claims are not backed by the reported tables, and the mechanism that distinguishes Stage 2 from generic resampling is not directly validated. These issues materially affect the paper's conclusions but are fixable with additional analyses and corrections.

major comments (5)
  1. [Abstract, Section 5.1] The numeric claims are internally inconsistent and not fully supported by Table 1. The abstract states '1.6% to 5.3%' and 'at least 57%' improvement, while the full-text abstract and introduction state '1.6% to 3.1%' and '50% to 55.5%'. More importantly, row-level differences against Anchor-Point in Table 1 are as small as 0.2% (EVOPROMPT on GPT-3.5 BIG-bench: 0.776 vs 0.774; EVOPROMPT on GPT-4o-mini BIG-bench: 0.758 vs 0.756), and the GPT-3.5 BIG-bench average difference is 1.2% (0.757 vs 0.745). The claim of 'at least 1.6% to 3.1%' needs to be reconciled with these numbers or the statistic must be redefined.
  2. [Section 5.4, Table 3] The paper claims computational overhead of less than 1%, but Table 3 shows APE with IPOMP taking 120.23 seconds versus 109.84 seconds for random selection, which is a 9.5% overhead. The average overhead across APO, APE, and EVOPROMPT is 1.34% (401.61 vs 396.31 seconds). The claim should be corrected or the aggregation clarified.
  3. [Section 5.2, Table 2, Appendix A.7] The ablation removes Stage 2 as a whole, so it does not test whether the logit-correlation redundancy signal causes the observed gains. Since Stage 1 selects only 20 samples from training sets of 392 to 10,240 examples, any resampling, including random replacement, expands coverage. A control that replaces the same beta fraction of samples with randomly selected or minimally similar samples and reports final held-out accuracy is needed to support the mechanism claim and the 'universally applied' claim. Table 8 only reports pairwise logit correlations of replacement strategies, not the resulting prompt accuracy or stability.
  4. [Section 4.4, Section A.5, Figure 4] The sensitivity analysis states that accuracy 'consistently improves' as alpha increases from 0.1 to 0.9, yet the default configuration uses alpha=0.5. No justification is given for choosing alpha=0.5, and the main results may therefore not reflect the better-performing configuration. This is important because alpha and other hyperparameters are selected on the same datasets used for evaluation.
  5. [Tables 1 and 2] The stability claims are based on standard deviations computed over five runs, but the paper does not report confidence intervals or significance tests for either accuracy or standard-deviation differences. Given that several differences are small (e.g., 0.2% in Table 1), the reader cannot assess whether the reported stability improvements are robust rather than noise.
minor comments (5)
  1. [Section 5.2] There is a typo: 'sage 2' should be 'stage 2'.
  2. [Conclusion] The conclusion says evaluation was on the BIG-bench dataset, but the experiments also use LIAR; the conclusion should mention both.
  3. [Abstract] The phrase 'at least 50% to 55.5%' is ambiguous; consider using 'from 50% to 55.5%' or a similar explicit range.
  4. [Section 3.2, Algorithm 2] The definition of the runtime performance matrix MP_runtime is not fully specified for open-ended or multi-label outputs; please clarify how the logit-based representation is constructed in those cases.
  5. [Section 5.1, Figure 1] The caption of Figure 1 does not state which dataset, model, and prompt optimizer are shown; the text refers to APE but the caption should be self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IPOMP is an empirical data-selection method evaluated on held-out test sets, with no reported metric reduced to a fitted parameter or self-citation chain.

full rationale

The paper's central claims are empirical: IPOMP selects evaluation subsets and its effectiveness and stability are measured on held-out test portions of BIG-bench and LIAR using Accuracy and standard deviation. There is no equation in which a predicted quantity is defined in terms of the fitted input, and no fitted parameter is renamed as a prediction. Stage 2 replaces samples based on logit correlations across candidate prompts, but the reported 1.6% to 3.1% accuracy gains and standard-deviation reductions are computed on held-out testing data, not on the samples used to drive the replacement logic. The self-citations to Wang et al. (2018) and Rajbahadur et al. (2017) only motivate the use of hierarchical clustering for grouping highly correlated samples; they are not invoked to establish the method's effectiveness or to forbid alternative mechanisms. The ablation comparing IPOMP with IPOMPStage1 and IPOMPRandom addresses component contributions, and any weakness in isolating the redundancy signal from generic resampling is a correctness or experimental-design concern, not a circularity of the derivation. Hyperparameters such as CT and beta are tuned on the studied datasets, but this is standard model selection rather than a circular reduction of the reported test-set results. No self-definitional, fitted-input-as-prediction, or self-citation-load-bearing step is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

IPOMP's central claim rests on the validity of semantic embeddings as a diversity signal, logit-correlation as a redundancy signal, and the usual coreset proxy assumption. The only free constants are method hyperparameters, and no invented entities appear.

free parameters (5)
  • alpha (cluster-sample fraction) = 0.5
    Portion of the N selected samples drawn by semantic clustering in Stage 1; the remainder are boundary samples. Chosen by hand; ablation (Figure 4) shows performance varies with it.
  • beta (replacement rate) = 0.5
    Fraction of redundant samples replaced in each Stage 2 iteration. Chosen by hand; ablation shows sensitivity, for example APE degrades in one range and improves later.
  • correlation threshold CT = 0.9
    Samples with pairwise logit correlation above 0.9 are treated as redundant. This threshold directly determines which samples are replaced in Stage 2.
  • cluster count K for KMeans = 5
    Number of semantic clusters in Stage 1. The authors report low sensitivity to K, but it still shapes the sampling.
  • evaluation subset size N = 20
    Number of examples used to evaluate prompts during optimization. Selected as a cost-effectiveness balance; sample-size experiments show performance stabilizes around 20.
assumptions (4)
  • domain assumption Semantic embedding distance using Sentence-BERT captures task-relevant sample diversity for prompt evaluation.
    Stage 1 clusters by embeddings and selects boundary cases as least similar pairs; if embeddings miss prompt-relevant dimensions, the selected subset is not diverse for the actual optimization (Section 3.1).
  • domain assumption Samples whose logits are highly correlated across candidate prompts are redundant for evaluation.
    Stage 2 replaces high-correlation samples. The paper provides correlation statistics (Figure 1, Table 8) but no formal proof that logit-space redundancy transfers to prompt-quality estimation (Section 3.2).
  • domain assumption The best prompt on the selected small evaluation set is the best prompt on the held-out test set.
    The entire method assumes that optimizing against a 20-example selected subset improves test accuracy; this is the usual coreset proxy assumption (Sections 2.2 and 5.1).
  • standard math KMeans and HNSW behave as expected on the embedding spaces used.
    Used for clustering and approximate nearest-neighbor search; unstated implementation details could affect which samples are selected.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization." pith.science (2026). https://pith.science/paper/GOOEFXFP

@misc{pith2026250510736,
  author       = {Pith},
  title        = {Pith review of: Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GOOEFXFP}},
  note         = {Machine review of arXiv:2505.10736}
}
read the original abstract

Optimizing Large Language Model (LLM) performance requires well-crafted prompts, but manual prompt engineering is labor-intensive and often ineffective. Automated prompt optimization techniques address this challenge but the majority of them rely on randomly selected evaluation subsets, which fail to represent the full dataset, leading to unreliable evaluations and suboptimal prompts. Existing coreset selection methods, designed for LLM benchmarking, are unsuitable for prompt optimization due to challenges in clustering similar samples, high data collection costs, and the unavailability of performance data for new or private datasets. To overcome these issues, we propose IPOMP, an Iterative evaluation data selection for effective Prompt Optimization using real-time Model Performance. IPOMP is a two-stage approach that selects representative and diverse samples using semantic clustering and boundary analysis, followed by iterative refinement with real-time model performance data to replace redundant samples. Evaluations on the BIG-bench dataset show that IPOMP improves effectiveness by 1.6% to 5.3% and stability by at least 57% compared with SOTA baselines, with minimal computational overhead below 1%. Furthermore, the results demonstrate that our real-time performance-guided refinement approach can be universally applied to enhance existing coreset selection methods.

Figures

Figures reproduced from arXiv: 2505.10736 by the authors.

Figure 1
Figure 1. Correlation among the samples selected in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Comparison of the effectiveness of original baselines and Baseline+IPOMP [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The impact of different sample sizes selected [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The impact of different values of hyper [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Correlation among samples before and after the first round of real-time model performance-guided [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 13 canonical work pages

  1. [1]

    Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. 2020. Contextual diversity for active learning. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XVI 16, pages 137--153. Springer

    work page 2020

  2. [2]

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. 2024. A survey on data selection for language models. arXiv preprint arXiv:2402.16827

    arXiv 2024

  3. [3]

    Fabrizio Angiulli and Clara Pizzuti. 2002. Fast outlier detection in high dimensional spaces. In European conference on principles of data mining and knowledge discovery, pages 15--27. Springer

    work page 2002

  4. [4]

    BIG bench authors. 2023. https://openreview.net/forum?id=uyTL5Bvosj Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Transactions on Machine Learning Research

    2023

  5. [5]

    Chengliang Chai, Jiayi Wang, Nan Tang, Ye Yuan, Jiabin Liu, Yuhao Deng, and Guoren Wang. 2023. Efficient coreset selection with cluster-based methods. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 167--178

    work page 2023

  6. [6]

    Yutian Chen, Max Welling, and Alex Smola. 2012. Super-samples from kernel herding. arXiv preprint arXiv:1203.3472

    arXiv 2012

  7. [7]

    Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. 2019. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv:1906.11829

    arXiv 2019

  8. [8]

    Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. 2022. Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12548

    arXiv 2022

Show all 45 references
  1. [9]

    Melanie Ducoffe and Frederic Precioso. 2018. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09841

    2018 arXiv

  2. [10]

    Promptbreeder: Self-referential self-improvement via prompt evolution (2023)

    Chrisantha Fernando, D Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt \"a schel. Promptbreeder: Self-referential self-improvement via prompt evolution (2023). arXiv preprint arXiv:2309.16797

    2023 arXiv

  3. [11]

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2023. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532

    2023 arXiv

  4. [12]

    Sariel Har-Peled and Soham Mazumdar. 2004. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing, pages 291--300

    2004

  5. [13]

    Yihao Huang, Chong Wang, Xiaojun Jia, Qing Guo, Felix Juefei-Xu, Jian Zhang, Geguang Pu, and Yang Liu. 2024. Semantic-guided prompt organization for universal goal hijacking against llms. arXiv preprint arXiv:2405.14189

    2024 arXiv

  6. [14]

    Gurusha Juneja, Nagarajan Natarajan, Hua Li, Jian Jiao, and Amit Sharma. 2024. Task facet learning: A structured approach to prompt optimization. arXiv preprint arXiv:2406.10504

    2024 arXiv

  7. [15]

    Seong Tae Kim, Farrukh Mushtaq, and Nassir Navab. 2020. Confident coreset for active learning in medical image analysis. arXiv preprint arXiv:2004.02200

    2020 arXiv

  8. [16]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  9. [17]

    Yong Lin, Chen Liu, Chenlu Ye, Qing Lian, Yuan Yao, and Tong Zhang. 2023. Optimal sample selection through uncertainty estimation and its application in deep learning. arXiv preprint arXiv:2309.02476

    2023 arXiv

  10. [18]

    Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. 2021. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781--6792. PMLR

    2021

  11. [19]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1--35

    2023

  12. [20]

    Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824--836

    2018

  13. [21]

    Katerina Margatina, Giorgos Vernikos, Lo \" c Barrault, and Nikolaos Aletras. 2021. Active learning by acquiring contrastive examples. arXiv preprint arXiv:2109.03764

    2021 arXiv

  14. [22]

    Lorenzo Pacchiardi, Lucy G Cheke, and Jos \'e Hern \'a ndez-Orallo. 2024. 100 instances is all you need: predicting the success of a new llm on unseen data by testing on a few instances. arXiv preprint arXiv:2409.03563

    2024 arXiv

  15. [23]

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. 2021. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596--20607

    2021

  16. [24]

    Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992

    2024 arXiv

  17. [25]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495

    2023 arXiv

  18. [26]

    Gopi Krishnan Rajbahadur, Shaowei Wang, Yasutaka Kamei, and Ahmed E Hassan. 2017. The impact of using regression models to build defect classifiers. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), pages 135--145. IEEE

    2017

  19. [27]

    N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  20. [28]

    Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489

    2017 arXiv

  21. [29]

    Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Augustus Odena. 2020. Small-gan: Speeding up gan training using core-sets. In International Conference on Machine Learning, pages 9005--9015. PMLR

    2020

  22. [30]

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. 2018. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159

    2018 arXiv

  23. [31]

    Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. 2023. Anchor points: Benchmarking models with much fewer examples. arXiv preprint arXiv:2309.08638

    2023 arXiv

  24. [32]

    Shaowei Wang, Tse-Hsun Chen, and Ahmed E Hassan. 2018. Understanding the factors for fast answers in technical q&a websites: An empirical study of four stack exchange websites. Empirical Software Engineering, 23:1552--1593

    2018

  25. [33]

    liar, liar pants on fire

    William Yang Wang. 2017. " liar, liar pants on fire": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648

    2017 arXiv

  26. [34]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  27. [35]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. https://arxiv.org/abs/2309.03409 Large language models as optimizers . Preprint, arXiv:2309.03409

    2024 arXiv

  28. [36]

    Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, et al. 2023. Retroformer: Retrospective large language agents with policy gradient optimization. arXiv preprint arXiv:2308.02151

    2023 arXiv

  29. [37]

    Bianca Zadrozny. 2004. Learning and evaluating classifiers under sample selection bias. In Proceedings of the twenty-first international conference on Machine learning, page 114

    2004

  30. [38]

    Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. 2022. Tempera: Test-time prompting via reinforcement learning. arXiv preprint arXiv:2211.11890

    2022 arXiv

  31. [39]

    Zhihan Zhang, Shuohang Wang, Wenhao Yu, Yichong Xu, Dan Iter, Qingkai Zeng, Yang Liu, Chenguang Zhu, and Meng Jiang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.659 Auto-instruct: Automatic instruction generation and ranking for black-box language models . In Finding...

    2023 doi

  32. [40]

    Lexin Zhou, Fernando Mart \' nez-Plumed, Jos \'e Hern \'a ndez-Orallo, C \`e sar Ferri, and Wout Schellaert. 2022 a . Reject before you run: Small assessors anticipate big language models. In EBeM@ IJCAI

    2022

  33. [41]

    Lexin Zhou, Pablo A Moreno-Casares, Fernando Mart \' nez-Plumed, John Burden, Ryan Burnell, Lucy Cheke, C \`e sar Ferri, Alexandru Marcoci, Behzad Mehrbakhsh, Yael Moros-Daval, et al. 2023. Predictable artificial intelligence. arXiv preprint arXiv:2310.06167

    2023 arXiv

  34. [42]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022 b . Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910

    2022 arXiv

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  37. [45]

    Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung Ju Hwang, and Cho-Jui Hsieh. 2024. One prompt is not enough: Automated construction of a mixture-of-expert prompts. In International Conference on Machine Learning

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.