REVIEW 4 major objections 5 minor 32 references
The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that selecting in-context learning exemplars adaptively — one at a time, based on the model's uncertainty given the examples already chosen — outperforms selecting all exemplars at once on reasoning tasks.
desk verdict An incremental but genuinely new exemplar-selection mechanism is undermined by an off-by-one in the algorithm and effect sizes within API noise. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the conditional uncertainty score $u(q \mid E)$, defined as the disagreement ratio $t/l$ (unique responses over total repeated queries) or the entropy of the answer distribution when the model is shown the current exemplar set $E$ together with question $q$. The selection rule $q_j = \arg\max_{q \in Q} u(q \mid E)$ turns this score into a greedy sequential algorithm: after each annotation, $E$ grows and every remaining candidate is rescored, which is exactly the step that distinguishes Adaptive-Prompt from one-shot Active-Prompt.
What would settle it
On a single dataset, record for each candidate question both its uncertainty score $u(q \mid E)$ and the actual change in held-out accuracy caused by annotating and adding it; if the two are not positively correlated, the selection rule's justification fails.
Extended reading notes
Core claim
In its own terms, the paper's central claim is that conditional uncertainty is a better guide for exemplar selection than unconditional uncertainty. Starting with an empty exemplar set, the algorithm evaluates every candidate question under the prompt formed by the current exemplars plus that question, computes a disagreement ratio or an entropy score over repeated responses, and picks the highest-uncertainty question for human annotation. The chosen question is added to the exemplar set and removed from the candidate pool, and the scores are recomputed, so each selection is conditioned on all previous ones. On GPT-3.5 Turbo the entropy variant averages 76.0 percent accuracy versus 75.3 for the one-shot Active-Prompt baseline; on GPT-4o mini the corresponding numbers are 86.9 and 86.7 percent, with the adaptive method ahead on most individual datasets.
Load-bearing premise
The method assumes that a question the model currently answers with high response divergence is the question whose annotation most improves test accuracy, and the paper does not measure that link directly.
Editorial extensions
If this is right
- For a fixed annotation budget $k$, adaptively choosing exemplars gives higher or equal average accuracy than selecting the $k$ most uncertain questions at once.
- The gains appear on arithmetic and commonsense reasoning and on both tested GPT models, while shrinking on the stronger model, so adaptive selection behaves as a complementary enhancement of the model's existing capability.
- On datasets where zero-shot chain-of-thought is already strong, such as GPT-4o mini on letter concatenation, adaptive selection narrows the gap between few-shot and zero-shot performance rather than closing it.
- At very small exemplar counts the method behaves like one-shot selection because there are not enough iterations to exploit feedback; at very large counts all methods converge, so the adaptive advantage sits at intermediate budgets.
Reading between the lines
- A natural extension the paper does not test is to replace the uncertainty proxy with a direct estimate of marginal information gain. If the proxy is doing the work, both should rank candidates similarly.
- The redundancy argument implies a measurable side effect: adaptive exemplar sets should contain less pairwise redundancy than one-shot top-k sets. Measuring embedding similarity or answer-type overlap of the chosen sets would test that mechanism directly.
- The same greedy conditional-selection idea should carry over to other budgeted-choice settings for LLMs, such as selecting documents for retrieval-augmented generation or examples for few-shot classification, wherever redundant selections are known to hurt.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Adaptive-Prompt, an iterative exemplar-selection method for in-context learning. Starting from an empty exemplar set, the method repeatedly computes an uncertainty score u(q | E) for each remaining unlabeled question given the current exemplar set, selects the most uncertain question, annotates it, and adds it to the exemplar set. The authors compare this adaptive procedure with non-adaptive baselines, especially Active-Prompt, on six reasoning datasets using GPT-3.5 Turbo and GPT-4o mini, and report accuracy gains on most datasets. They also report experiments on annotation variability, exemplar-set size, and a weaker model (LLaMA3-8B).
Significance. If the adaptive-selection mechanism is validated with a correctly enforced annotation budget, the paper would provide a plausible and practical improvement over one-shot uncertainty-based exemplar selection: greedy feedback-driven selection can reduce redundancy among exemplars and use a fixed annotation budget more effectively. The manuscript includes clear pseudocode, multiple public benchmarks, human-annotation consistency checks, and additional analyses of k and annotator effects, which are useful contributions. However, the current evidence is not yet persuasive because the reported gains are small, no statistical significance is established, the algorithm pseudocode appears to permit one extra exemplar, and the load-bearing uncertainty proxy is not directly validated.
major comments (4)
- [Section 4, Algorithm 1] The stopping condition 'while |E| ≤ k' in Algorithm 1 causes the loop to execute one additional time after |E| becomes k, yielding an exemplar set of size k+1. This contradicts Section 4.1, which states that the process 'continues until the exemplar set reaches the desired size k.' If the implementation follows the pseudocode, Adaptive-Prompt uses a strictly larger prompt than all baselines, which are budgeted at k exemplars. Because the reported gains over Active-Prompt are only 0.1–0.8 percentage points on GPT-3.5 and 0.2–0.6 on GPT-4o mini, the extra exemplar alone could account for the advantage. Please correct the condition to 'while |E| < k' or explicitly confirm that the implementation enforces a size-k budget, and report whether the results in Tables 1 and 2 change under the corrected budget.
- [Section 6, Tables 1 and 2] The tables report only mean accuracies, with no error bars, confidence intervals, or significance tests, even though Section 5 states that the entire selection process was repeated three times. The abstract's claim that Adaptive-Prompt 'significantly enhances' performance is therefore unsupported by the reported evidence. The differences are often within a few tenths of a percentage point; for example, on GPT-4o mini, Adaptive-Prompt (D) reaches an average of 86.9 versus 86.7 for Active-Prompt (E), and it is worse than Active-Prompt on AQuA, SVAMP, and Letter Concat. Please provide per-run values, standard deviations, and a paired significance test or effect-size measure, and temper the significance language accordingly.
- [Sections 4.1 and 4.2] The central mechanism rests on the assumption that a high uncertainty score u(q | E) identifies a question whose addition to E most improves downstream test accuracy. The paper provides no direct evidence for this link: there is no correlation analysis between u(q | E) and the actual marginal accuracy contribution of q, and no ablation comparing the greedy uncertainty rule with alternative adaptive rules (e.g., random adaptive selection, diversity-aware adaptive selection, or re-computed top-k selection). Adding such an analysis, or at least a sensitivity check with a different uncertainty metric, would greatly strengthen the claim that the reported gains come from adaptive feedback rather than from the specific uncertainty estimator.
- [Section 6, Evaluation with Weaker Models] The paper states that on LLaMA3-8B, 'in most cases, Zero-Shot CoT outperforms both baselines and our method.' This is a direct counterexample to the general claim that Adaptive-Prompt enhances LLM performance, and it should be treated as a scope condition on the method rather than a side observation. Please discuss the implications for the central claim and qualify the abstract and conclusions accordingly, or provide an explanation supported by additional experiments.
minor comments (5)
- [Throughout] There are several typos and formatting issues: 'T able 1' in the table caption, 'SV AMP' should be 'SVAMP', 'examplars' should be 'exemplars', and 'Random-Prompt' and 'Random-CoT' are used inconsistently in the text and figure captions.
- [Section 5] The paper says the selection process was repeated three times, but it does not report the random seeds, API sampling temperature, or other details needed to reproduce the three runs. Please add these details or clarify which sources of randomness were controlled.
- [Algorithm 1] Algorithm 1 does not specify how ties in the arg max over uncertainty scores are broken. A deterministic tie-breaking rule (e.g., lowest index or random tie-break) should be stated for reproducibility.
- [Section 4.2] Reference [6] is cited for the uncertainty score, but the cited paper appears to be about pool-based batch active learning; please verify that this is the correct citation for the disagreement and entropy metrics.
- [Section 6] Figures 2 and 3 report results from 'an experiment run' without error bars or multiple runs, which limits the interpretability of the k-sensitivity analysis; please clarify or add variability information.
Circularity Check
No circularity found: Adaptive-Prompt selects exemplars from unlabeled training questions using model uncertainty and is evaluated on a held-out test set; neither test labels nor fitted parameters enter the construction, so the claimed accuracy gains are independent empirical outcomes.
full rationale
I walked the paper's claimed derivation chain. The chain is: define uncertainty u(q | E) as disagreement or entropy over l independent LLM responses (Section 4.2); select the question with the highest u(q | E), annotate it, append it to the exemplar set (Algorithm 1, Section 4.1); repeat until the exemplar set reaches the desired size; finally evaluate accuracy on the test set P with Self-Consistency (Section 5). The selection signal is computed entirely from the unlabeled training pool and the LLM's own responses conditioned on the current exemplar set; it never uses test labels or test-set accuracy, and no parameter is fitted to the evaluation metric. Therefore the reported gains are not equivalent to the inputs by construction. The uncertainty metric is explicitly borrowed from the external baseline Active-Prompt [3], not from prior work by the present authors, so there is no self-citation chain carrying the argument. The heuristic that high-uncertainty questions are the best next exemplars is an empirical premise, not a definition of the outcome; the paper itself reports datasets where Adaptive-Prompt does not beat baselines (AQuA and SVAMP on GPT-4o mini), which shows the result is not forced. I also noted an internal inconsistency between Algorithm 1's 'while |E| ≤ k' and the text's 'continues until the exemplar set reaches the desired size k': as written, the loop would add a (k+1)-th exemplar. That is a potential experimental confound or correctness bug, but it is not a circularity, because even a k+1-shot comparison is still an out-of-sample empirical measurement rather than an identity between the derivation and its inputs. Accordingly, no circular step can be exhibited, and the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- number of repeated queries l =
10
- candidate pool size s =
50 x k (200 to 400)
- self-consistency voting samples =
6
- per-dataset exemplar budget k =
4 to 8
assumptions (3)
- domain assumption The disagreement or entropy of l independent model responses is a valid measure of whether the current exemplar set covers a question's required knowledge.
- ad hoc to paper Greedy selection of the currently most uncertain question leads to an exemplar set that is near-optimal for test accuracy.
- domain assumption The random candidate subset of size s = 50 x k is representative of the full training distribution.
Cite this review
Pith. "Pith review of The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting." pith.science (2026). https://pith.science/paper/XXJ2UYJL
@misc{pith2026241217891,
author = {Pith},
title = {Pith review of: The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting},
year = {2026},
howpublished = {\url{https://pith.science/paper/XXJ2UYJL}},
note = {Machine review of arXiv:2412.17891}
}
read the original abstract
Large Language Models (LLMs) have demonstrated exceptional abilities across a broad range of language-related tasks, including generating solutions to complex reasoning problems. An effective technique to enhance LLM performance is in-context learning, which encourages a step-by-step reasoning process by including explanatory examples to guide the model's responses. However, selecting appropriate exemplars for the model poses a challenge, as each dataset demands a distinct set of exemplars to enable the LLM to learn effectively and perform well on the test set. Current studies often rely on uncertainty- or diversity-based selection strategies to select exemplars for annotation and to improve model learning. However, these studies typically employ a non-adaptive approach, selecting a set of exemplars all at once. We argue that this non-adaptive strategy may result in a set of exemplars with high redundancy in terms of the knowledge covered, ultimately reducing their overall informativeness. To address this limitation, we propose \textsc{Adaptive-Prompt}, a novel method that adaptively selects exemplars by leveraging model feedback from previously chosen exemplars. Experimental results show that \textsc{Adaptive-Prompt} significantly enhances LLM performance across a variety of reasoning tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2303.08774 (2023)
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
arXiv 2023
-
[2]
arXiv preprint arXiv:2110.14168 (2021)
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)
arXiv 2021
-
[3]
arXiv preprint arXiv:2302.12246 (2023)
Diao, S., Wang, P., Lin, Y., Zhang, T.: Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246 (2023)
arXiv 2023
-
[4]
Association for Computational Linguistics (2019)
Erdmann, A., Wrisley, D.J., Brown, C., Cohen-Bod´ en` es, S., Elsner, M., Feng, Y., Joseph, B., Joyeux-Prunel, B., de Marneffe, M.C.: Practical, efficient, and cus- tomizable active learning for named entity recognition in the digital humanities. Association for Computational Linguistics (2019)
work page 2019
-
[5]
In: The Eleventh International Conference on Learning Representations (2022)
Fu, Y., Peng, H., Sabharwal, A., Clark, P., Khot, T.: Complexity-based prompting for multi-step reasoning. In: The Eleventh International Conference on Learning Representations (2022)
work page 2022
-
[6]
Journal of Machine Learning Research 25(262), 1–42 (2024)
Gentile, C., Wang, Z., Zhang, T.: Fast rates in pool-based batch active learning. Journal of Machine Learning Research 25(262), 1–42 (2024)
work page 2024
-
[7]
Transactions of the Association for Computational Linguistics 9, 346–361 (2021)
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., Berant, J.: Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics 9, 346–361 (2021)
work page 2021
-
[8]
Advances in neural information processing systems 35, 22199–22213 (2022)
Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. Advances in neural information processing systems 35, 22199–22213 (2022)
2022
Show all 32 references
-
[9]
arXiv preprint arXiv:2211.08358 (2022)
K¨ oksal, A., Schick, T., Sch¨ utze, H.: Meal: stable and active learning for few-shot prompting. arXiv preprint arXiv:2211.08358 (2022)
2022 arXiv
-
[10]
arXiv preprint arXiv:1705.04146 (2017)
Ling, W., Yogatama, D., Dyer, C., Blunsom, P.: Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146 (2017)
2017 arXiv
-
[11]
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., Chen, W.: What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804 (2021)
2021 arXiv
-
[12]
Transactions of the Association for Computational Linguistics 12, 157–173 (2024)
Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12, 157–173 (2024)
2024
-
[13]
Advances in Neural Information Processing Systems 36, 43136–43155 (2023)
Ma, H., Zhang, C., Bian, Y., Liu, L., Zhang, Z., Zhao, P., Zhang, S., Fu, H., Hu, Q., Wu, B.: Fairness-guided few-shot prompting for large language models. Advances in Neural Information Processing Systems 36, 43136–43155 (2023)
2023
-
[14]
arXiv preprint arXiv:2109.03764 (2021) 12 S
Margatina, K., Vernikos, G., Barrault, L., Aletras, N.: Active learning by acquiring contrastive examples. arXiv preprint arXiv:2109.03764 (2021) 12 S. Cai et al
2021 arXiv
-
[15]
arXiv preprint arXiv:2310.20046 (2023)
Mavromatis, C., Srinivasan, B., Shen, Z., Zhang, J., Rangwala, H., Faloutsos, C., Karypis, G.: Which examples to annotate for in-context learning? towards effective and efficient selection. arXiv preprint arXiv:2310.20046 (2023)
2023 arXiv
-
[16]
OpenAI: Openai models documentation (2024), https://platform.openai.com/ docs/models
2024
-
[17]
Patel, A., Bhattamishra, S., Goyal, N.: Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 (2021)
2021 arXiv
-
[18]
arXiv preprint arXiv:2008.07267 (2020)
Schr¨ oder, C., Niekler, A.: A survey of active learning for text classification using deep neural networks. arXiv preprint arXiv:2008.07267 (2020)
2020 arXiv
-
[19]
arXiv preprint arXiv:2107.05687 (2021)
Schr¨ oder, C., Niekler, A., Potthast, M.: Revisiting uncertainty-based query strate- gies for active learning with transformers. arXiv preprint arXiv:2107.05687 (2021)
2021 arXiv
-
[20]
Settles, B.: Active learning literature survey (2009)
2009
-
[21]
In: International Conference on Machine Learning
Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., Chen, W.: Synthetic prompt- ing: Generating chain-of-thought demonstrations for large language models. In: International Conference on Machine Learning. pp. 30706–30775. PMLR (2023)
2023
-
[22]
arXiv preprint arXiv:2209.01975 (2022)
Su, H., Kasai, J., Wu, C.H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N.A., et al.: Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975 (2022)
2022 arXiv
-
[23]
In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Talmor, A., Herzig, J., Lourie, N., Berant, J.: Commonsenseqa: A question an- swering challenge targeting commonsense knowledge. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ...
2019
-
[24]
arXiv preprint arXiv:2307.09288 (2023)
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
2023 arXiv
-
[25]
arXiv preprint arXiv:2203.11171 (2022)
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)
2022 arXiv
-
[26]
Transactions on Machine Learning Research (2022)
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. Transactions on Machine Learning Research (2022)
2022
-
[27]
Advances in neural information processing systems 35, 24824–24837 (2022)
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824–24837 (2022)
2022
-
[28]
arXiv preprint arXiv:2210.03493 (2022)
Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 (2022)
2022 arXiv
-
[29]
arXiv preprint arXiv:2303.18223 (2023)
Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)
2023 arXiv
-
[30]
In: Findings of the Association for Computational Linguistics: EMNLP 2020
Zhao, Y., Zhang, R.H., Zhou, S., Zhang, Z.: Active learning approaches to enhanc- ing neural machine translation. In: Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 1796–1806 (2020)
2020
-
[31]
In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Zhou, C., You, W., Li, J., Ye, J., Chen, K., Zhang, M.: Inform: Information entropy based multi-step reasoning for large language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 3565–3576 (2023)
2023
-
[32]
In: 22nd International Conference on Computational Linguistics, Coling 2008
Zhu, J., Wang, H., Yao, T., Tsou, B.K.: Active learning with sampling by uncer- tainty and density for word sense disambiguation and text classification. In: 22nd International Conference on Computational Linguistics, Coling 2008. pp. 1137– 1144 (2008)
2008
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.