{"id":"2817d666-6e88-41b4-903a-6140f2ca52f6","arxiv_id":"2412.17891","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sequentially choosing the most uncertain training question given previously chosen exemplars improves few-shot chain-of-thought accuracy by about 0.7 points on average over non-adaptive active prompting.","lead":"Adaptive-Prompt selects chain-of-thought examples one at a time, using the model's uncertainty about the remaining questions after each pick to decide which example to annotate next. On reasoning benchmarks with GPT-3.5 Turbo and GPT-4o Mini, it beats non-adaptive selection baselines by small but consistent margins.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's loop 'while |E| ≤ k' selects k+1 exemplars for Adaptive-Prompt, so Tables 1–2 may compare against baselines using k exemplars while Adaptive-Prompt uses k+1. If so, the reported gains could be an artifact of the extra exemplar.","rationale":"The most load-bearing condition for the central claim is that Adaptive-Prompt and Active-Prompt are compared under the same annotation budget. The pseudocode's while-loop selects k+1 exemplars, contradicting the stated objective and the 'desired size k' in the text. Since all reported improvements are ≤0.8 pp and no error bars are given, even a single extra exemplar could fully explain them. This is a more direct threat than the uncertainty-proxy question: if the budget is unequal, the comparison is unfair regardless of how well u(q|E) correlates with marginal utility. The reader's weakest assumption (uncertainty proxy) is important but secondary; it would only matter after establishing a fair comparison. The fix is straightforward and verifiable, so the appropriate verdict is unchanged from the reader's CONDITIONAL: accept only if the authors confirm/correct the stopping condition and re-report comparisons at equal k. I disagree with the reader's identification of the weakest assumption because the off-by-one is not mentioned in the verdict.","tokens_in":9608,"tokens_out":7087,"duration_ms":68921,"concrete_test":"Re-run the full experimental protocol (Tables 1 and 2) with the loop condition changed to 'while |E| < k' so that Adaptive-Prompt outputs exactly k exemplars, identical in number to Active-Prompt, Random-CoT, and CoT. Report per-seed accuracies (or at least means over ≥10 runs). If the accuracy differences vs Active-Prompt remain at the same magnitude, the off-by-one concern is resolved; if they shrink to near zero, the paper's central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 in Section 4 (line 4) uses 'while |E| ≤ k do' as the stopping condition. Starting from E=∅, this loop also executes when |E|=k, so it adds a (k+1)-th exemplar before terminating. The textual description in Section 4.1 step 4 explicitly says the process 'continues until the exemplar set reaches the desired size k', so the intended condition should be |E| < k. If the implementation follows the pseudocode, Adaptive-Prompt receives a strictly larger prompt (e.g., 9 exemplars for GSM8K instead of 8) than Active-Prompt and all other baselines. Reported gains are only 0.1–0.8 pp on GPT-3.5 and GPT-4o mini, which is exactly the magnitude one extra exemplar could produce. The central claim that adaptive selection improves in-context learning is therefore not cleanly supported until the budget is verified. This is not about statistical noise or missing baselines; it is an internal inconsistency between the algorithm and its stated objective of producing an exemplar set of size k.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adaptive-Prompt, an iterative exemplar-selection method for in-context learning. Starting from an empty exemplar set, the method repeatedly computes an uncertainty score u(q | E) for each remaining unlabeled question given the current exemplar set, selects the most uncertain question, annotates it, and adds it to the exemplar set. The authors compare this adaptive procedure with non-adaptive baselines, especially Active-Prompt, on six reasoning datasets using GPT-3.5 Turbo and GPT-4o mini, and report accuracy gains on most datasets. They also report experiments on annotation variability, exemplar-set size, and a weaker model (LLaMA3-8B).","tokens_in":9869,"tokens_out":3608,"duration_ms":35394,"significance":"If the adaptive-selection mechanism is validated with a correctly enforced annotation budget, the paper would provide a plausible and practical improvement over one-shot uncertainty-based exemplar selection: greedy feedback-driven selection can reduce redundancy among exemplars and use a fixed annotation budget more effectively. The manuscript includes clear pseudocode, multiple public benchmarks, human-annotation consistency checks, and additional analyses of k and annotator effects, which are useful contributions. However, the current evidence is not yet persuasive because the reported gains are small, no statistical significance is established, the algorithm pseudocode appears to permit one extra exemplar, and the load-bearing uncertainty proxy is not directly validated.","major_comments":[{"comment":"The stopping condition 'while |E| ≤ k' in Algorithm 1 causes the loop to execute one additional time after |E| becomes k, yielding an exemplar set of size k+1. This contradicts Section 4.1, which states that the process 'continues until the exemplar set reaches the desired size k.' If the implementation follows the pseudocode, Adaptive-Prompt uses a strictly larger prompt than all baselines, which are budgeted at k exemplars. Because the reported gains over Active-Prompt are only 0.1–0.8 percentage points on GPT-3.5 and 0.2–0.6 on GPT-4o mini, the extra exemplar alone could account for the advantage. Please correct the condition to 'while |E| < k' or explicitly confirm that the implementation enforces a size-k budget, and report whether the results in Tables 1 and 2 change under the corrected budget.","section":"Section 4, Algorithm 1"},{"comment":"The tables report only mean accuracies, with no error bars, confidence intervals, or significance tests, even though Section 5 states that the entire selection process was repeated three times. The abstract's claim that Adaptive-Prompt 'significantly enhances' performance is therefore unsupported by the reported evidence. The differences are often within a few tenths of a percentage point; for example, on GPT-4o mini, Adaptive-Prompt (D) reaches an average of 86.9 versus 86.7 for Active-Prompt (E), and it is worse than Active-Prompt on AQuA, SVAMP, and Letter Concat. Please provide per-run values, standard deviations, and a paired significance test or effect-size measure, and temper the significance language accordingly.","section":"Section 6, Tables 1 and 2"},{"comment":"The central mechanism rests on the assumption that a high uncertainty score u(q | E) identifies a question whose addition to E most improves downstream test accuracy. The paper provides no direct evidence for this link: there is no correlation analysis between u(q | E) and the actual marginal accuracy contribution of q, and no ablation comparing the greedy uncertainty rule with alternative adaptive rules (e.g., random adaptive selection, diversity-aware adaptive selection, or re-computed top-k selection). Adding such an analysis, or at least a sensitivity check with a different uncertainty metric, would greatly strengthen the claim that the reported gains come from adaptive feedback rather than from the specific uncertainty estimator.","section":"Sections 4.1 and 4.2"},{"comment":"The paper states that on LLaMA3-8B, 'in most cases, Zero-Shot CoT outperforms both baselines and our method.' This is a direct counterexample to the general claim that Adaptive-Prompt enhances LLM performance, and it should be treated as a scope condition on the method rather than a side observation. Please discuss the implications for the central claim and qualify the abstract and conclusions accordingly, or provide an explanation supported by additional experiments.","section":"Section 6, Evaluation with Weaker Models"}],"minor_comments":[{"comment":"There are several typos and formatting issues: 'T able 1' in the table caption, 'SV AMP' should be 'SVAMP', 'examplars' should be 'exemplars', and 'Random-Prompt' and 'Random-CoT' are used inconsistently in the text and figure captions.","section":"Throughout"},{"comment":"The paper says the selection process was repeated three times, but it does not report the random seeds, API sampling temperature, or other details needed to reproduce the three runs. Please add these details or clarify which sources of randomness were controlled.","section":"Section 5"},{"comment":"Algorithm 1 does not specify how ties in the arg max over uncertainty scores are broken. A deterministic tie-breaking rule (e.g., lowest index or random tie-break) should be stated for reproducibility.","section":"Algorithm 1"},{"comment":"Reference [6] is cited for the uncertainty score, but the cited paper appears to be about pool-based batch active learning; please verify that this is the correct citation for the disagreement and entropy metrics.","section":"Section 4.2"},{"comment":"Figures 2 and 3 report results from 'an experiment run' without error bars or multiple runs, which limits the interpretability of the k-sensitivity analysis; please clarify or add variability information.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The off-by-one issue in Algorithm 1 is the most serious concern: if the implementation actually uses k+1 exemplars, the central comparison is invalid. The paper is salvageable if the authors confirm the correct budget and find that the gains persist, but the lack of any significance testing and the very small effect sizes make the current evidence borderline. I would also encourage the authors to consider whether the contribution, once the budget and statistics are fixed, is substantial enough for a full-length archival paper or better suited to a workshop-length presentation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2412.17891. The core idea is genuinely a small step forward: unlike Active-Prompt, which scores all candidate questions once with an empty exemplar set, Adaptive-Prompt recalculates uncertainty conditioned on the growing set, so the second and later picks respond to what was already chosen. That is a real distinction, not a relabeling. The paper also does some things right in the evaluation: three independent runs of the whole selection process, self-consistency voting on the test side, and a second-annotator robustness check.\n\nThe problem is that the paper cannot support its own headline. Algorithm 1's stopping condition is \"while |E| ≤ k\", which from E=∅ yields k+1 exemplars, while the text and the output specification say the set should have exactly k. If the experiments followed the pseudocode, every Adaptive-Prompt result was produced with a larger prompt than every baseline. The reported gains average about 0.7 points, and on the best case, GSM8K, the gain is 0.8 points with the entropy variant. That is exactly the size of effect an extra exemplar could plausibly produce. So the central \"adaptive selection helps\" comparison is not clean until this is resolved. The paper must either fix the pseudocode to \"< k\" and confirm that is what the implementation used, or acknowledge that the comparison used k+1 and rerun baselines with k+1.\n\nEven setting the off-by-one aside, the evidence is thin. There are no error bars or significance tests, and the three runs are averaged without reporting variance. Several numbers are within 0.1–0.5 points of the baselines, a range where API nondeterminism alone could flip the ranking. On GPT-4o mini the method is flat or negative on half the datasets; on LLaMA3-8B it provides no benefit. The closely related ADAICL baseline is cited but not run. The uncertainty score is never validated as a proxy for marginal information gain; an ablation pitting it against, say, random or diversity-based selection within the same adaptive loop would have addressed that.\n\nWho this is for: people working on exemplar selection or active learning for LLMs. The adaptive-conditioning idea is plausible and worth discussing; someone could build something solid on it. But as submitted, the claims outrun the evidence. I would send it to review, because the idea deserves a serious look, but the referee should require the pseudocode fix, variance estimates, the missing ADAICL comparison, and code release before anything is accepted. Do not let the \"significantly\" in the abstract survive as is.","headline":"An incremental but genuinely new exemplar-selection mechanism is undermined by an off-by-one in the algorithm and effect sizes within API noise.","tokens_in":10398,"tokens_out":2865,"would_cite":false,"duration_ms":25099,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that selecting in-context learning exemplars adaptively — one at a time, based on the model's uncertainty given the examples already chosen — outperforms selecting all exemplars at once on reasoning tasks.","keywords":["in-context learning","chain-of-thought prompting","adaptive exemplar selection","active prompting","uncertainty estimation","large language models","reasoning tasks","annotation budget"],"falsifier":"On a single dataset, record for each candidate question both its uncertainty score $u(q \\mid E)$ and the actual change in held-out accuracy caused by annotating and adding it; if the two are not positively correlated, the selection rule's justification fails.","tokens_in":9393,"feed_emoji":"🧠","tokens_out":7025,"duration_ms":59376,"temperature":0.7,"pith_summary":"The paper claims that the exemplars used for in-context learning should be selected sequentially, not in one batch. Its method, Adaptive-Prompt, repeatedly asks a language model to answer each remaining unlabeled question using the exemplars already chosen, scores the question by how much the model's responses disagree, and annotates the most uncertain one. Across arithmetic, commonsense, and symbolic reasoning datasets, this iterative procedure gets higher average accuracy than selecting the most uncertain questions once and annotating them together. The reason to care is practical: if the claim is right, a fixed budget of human annotation yields more reasoning accuracy when the selection itself is model-aware and sequential.","feed_headline":"Adaptive prompting beats one-shot exemplar selection","feed_subtitle":"Picking exemplars one by one from model feedback beats one-shot selection across six reasoning datasets.","key_machinery":"The load-bearing object is the conditional uncertainty score $u(q \\mid E)$, defined as the disagreement ratio $t/l$ (unique responses over total repeated queries) or the entropy of the answer distribution when the model is shown the current exemplar set $E$ together with question $q$. The selection rule $q_j = \\arg\\max_{q \\in Q} u(q \\mid E)$ turns this score into a greedy sequential algorithm: after each annotation, $E$ grows and every remaining candidate is rescored, which is exactly the step that distinguishes Adaptive-Prompt from one-shot Active-Prompt.","core_discovery":"In its own terms, the paper's central claim is that conditional uncertainty is a better guide for exemplar selection than unconditional uncertainty. Starting with an empty exemplar set, the algorithm evaluates every candidate question under the prompt formed by the current exemplars plus that question, computes a disagreement ratio or an entropy score over repeated responses, and picks the highest-uncertainty question for human annotation. The chosen question is added to the exemplar set and removed from the candidate pool, and the scores are recomputed, so each selection is conditioned on all previous ones. On GPT-3.5 Turbo the entropy variant averages 76.0 percent accuracy versus 75.3 for the one-shot Active-Prompt baseline; on GPT-4o mini the corresponding numbers are 86.9 and 86.7 percent, with the adaptive method ahead on most individual datasets.","pith_inferences":["A natural extension the paper does not test is to replace the uncertainty proxy with a direct estimate of marginal information gain. If the proxy is doing the work, both should rank candidates similarly.","The redundancy argument implies a measurable side effect: adaptive exemplar sets should contain less pairwise redundancy than one-shot top-k sets. Measuring embedding similarity or answer-type overlap of the chosen sets would test that mechanism directly.","The same greedy conditional-selection idea should carry over to other budgeted-choice settings for LLMs, such as selecting documents for retrieval-augmented generation or examples for few-shot classification, wherever redundant selections are known to hurt."],"forward_implications":["For a fixed annotation budget $k$, adaptively choosing exemplars gives higher or equal average accuracy than selecting the $k$ most uncertain questions at once.","The gains appear on arithmetic and commonsense reasoning and on both tested GPT models, while shrinking on the stronger model, so adaptive selection behaves as a complementary enhancement of the model's existing capability.","On datasets where zero-shot chain-of-thought is already strong, such as GPT-4o mini on letter concatenation, adaptive selection narrows the gap between few-shot and zero-shot performance rather than closing it.","At very small exemplar counts the method behaves like one-shot selection because there are not enough iterations to exploit feedback; at very large counts all methods converge, so the adaptive advantage sits at intermediate budgets."],"supporting_citations":[{"why":"Supplies the one-shot Active-Prompt baseline and the uncertainty-score definitions that Adaptive-Prompt reuses and extends.","marker":"[3]"},{"why":"Supplies the diversity-based Auto-CoT baseline that clusters questions by embedding similarity.","marker":"[28]"},{"why":"Supplies the chain-of-thought prompting setup and the per-dataset exemplar counts used in the experiments.","marker":"[27]"},{"why":"Supplies the zero-shot CoT prompting baseline used in all comparisons.","marker":"[8]"},{"why":"Supplies the Self-Consistency voting scheme used to determine final test answers.","marker":"[25]"},{"why":"Supplies the GSM8K arithmetic dataset used in the main evaluation and in the exemplar-size study.","marker":"[2]"},{"why":"Supplies the StrategyQA commonsense dataset used in the main evaluation.","marker":"[7]"},{"why":"Supplies the CSQA commonsense dataset used in the main evaluation and in the exemplar-size study.","marker":"[23]"}],"fun_headline_variants":["Conditional uncertainty picks better in-context examples","Adaptive exemplar selection improves LLM reasoning","Pick prompts one by one for stronger LLM performance","Model feedback guides superior exemplar selection","Adaptive-Prompt outperforms one-shot selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a question the model currently answers with high response divergence is the question whose annotation most improves test accuracy, and the paper does not measure that link directly.","fun_headline_variants_meta":{"raw":{"variants":["Conditional uncertainty picks better in-context examples","Adaptive exemplar selection improves LLM reasoning","Pick prompts one by one for stronger LLM performance","Model feedback guides superior exemplar selection","Adaptive-Prompt outperforms one-shot selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1494,"prompt_tokens":920,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":504}},"tokens_in":536,"tokens_out":574,"duration_ms":5882,"temperature":1.0,"reasoning_tokens":504,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:17:06.897283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a single dataset, record for each candidate question both its uncertainty score $u(q \\mid E)$ and the actual change in held-out accuracy caused by annotating and adding it; if the two are not positively correlated, the selection rule's justification fails.","supporting_citations":[{"cited_title":"Transactions of the Association for Computational Linguistics 9, 346–361 (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the StrategyQA commonsense dataset used in the main evaluation."},{"cited_title":"In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","cited_arxiv_id":null,"evidence_quote":"Supplies the CSQA commonsense dataset used in the main evaluation and in the exemplar-size study."}],"review_version":1}