Pith. sign in

REVIEW 4 major objections 5 minor 63 references

Sample Efficient Demonstration Selection for In-Context Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CASE finds near-optimal few-shot demonstration subsets using up to 7x fewer calls to a large language model—an 87% reduction—without losing task accuracy.

desk verdict CASE delivers credible LLM-call savings for exemplar selection, but the theory rests on an unvalidated linear reward model and a lemma that assumes its own conclusion. read the letter →

arxiv 2506.08607 v1 pith:7DXGQFXP submitted 2025-06-10 cs.LG

Kiran Purohit , V Venktesh , Sourangshu Bhattacharya , Avishek Anand This is my paper
classification cs.LG
keywords in-contextlearningexemplarselectionfew-shotpromptingtop-marmidentificationstochasticlinearbanditschallenger-armsamplingsampleefficiencylargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make task-level exemplar selection for in-context learning substantially cheaper: choosing the k demonstration examples that go into a prompt currently costs many calls to a large language model, because candidate subsets are scored by running the model on a validation set. The authors formulate this as a top-m arm identification problem in a stochastic linear bandit, where each arm is a k-sized subset of exemplars and the reward is a linear function of the mean sentence-similarity between the subset and the validation examples. Their algorithm, CASE, maintains a shortlist of challenger arms and pulls only one arm per iteration, cutting the number of LLM evaluations by up to 7x (an 87% reduction) and runtime by up to 7x while matching or exceeding the task accuracy of state-of-the-art methods like LENS and EXPLORA on five reasoning benchmarks. If correct, this makes high-quality few-shot prompting affordable for black-box models and enables reusing exemplars selected by small models on larger ones.

What carries the argument

The central objects are the challenger-arm shortlist $N_t$ and the gap-index rule. Each arm is a $k$-sized subset of exemplars; its feature vector is $x_a(i)=a(i)\cdot \frac{1}{n'}\sum_{j=1}^{n'} \mathrm{Sim}_{i,j}$, the mean Sentence-BERT similarity of exemplar $i$ to the validation set, so the modeled reward is $\rho(a;\alpha)=\alpha^\top x_a$ (Eq. 3). Each round CASE uniformly samples $m'$ arms from the complement of $U_t \cup N_{t-1}$, keeps the top $m'$ into the shortlist $N_t$, and then selects exactly one arm to pull by minimizing the variance of the gap between the most ambiguous top-m arm $b_t$ and the most ambiguous challenger $s_t$, with gap index $B_t(i,j)=\hat\rho_t(i)-\hat\rho_t(j)+W_t(i,j)$. The pulled arm's validation accuracy is received as the reward and used in a least-squares update of $\alpha$ via the Sherman-Morrison formula, so the cost scales with the number of pulls, not with the exponential number of subsets.

What would settle it

Compute the linear score $\hat\alpha^\top x_a$ (after fitting $\hat\alpha$ with CASE's least-squares update on a training run) for a large pool of $k$-exemplar subsets of a held-out task, then obtain the true validation accuracy of each subset by direct LLM scoring; if the rank correlation between predicted and measured scores is near zero or negative, the linear surrogate (and the top-m guarantees built on it) fails for that task. Alternatively, run CASE on synthetic linear bandits with known top-m arms and check whether the returned set equals the true top-m with frequency $\geq 1-\delta$; a systematic miss would falsify the assumed average-regret bound.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the quality of a prompt built from k exemplars can be captured by a linear score: the reward of a k-subset is the mean sentence-similarity between its exemplars and the validation set, weighted by learned coefficients. That turns the search over exponentially many subsets into a top-m arm identification problem in a stochastic linear bandit. CASE then identifies the m best subsets with a challenger-arm shortlist: in each round it uniformly samples a small set of candidate subsets, keeps the highest-scoring ones as challengers, and pulls exactly one arm per iteration according to a gap-index selection rule. The paper reports that this reduces LLM evaluations by up to 7x (87% fewer) and runtime by up to 7x relative to state-of-the-art exemplar selection methods, while matching or exceeding their accuracy on GSM8K, AquaRat, TabMWP, FinQA, and StrategyQA, and that exemplars chosen by CASE with small models transfer to larger models.

Load-bearing premise

Everything hinges on the modeling choice that a subset's validation accuracy equals a linear function of the mean sentence-similarity between its exemplars and the validation examples (Eq. 3), an assumption the paper does not test independently; a second, explicitly deferred premise is that the selective-exploration set $U_T \cup N_T$ achieves average regret at most $\epsilon$ (Section 3.4).

Editorial extensions

If this is right

  • On the five evaluated reasoning benchmarks (GSM8K, AquaRat, TabMWP, FinQA, StrategyQA), CASE with GPT-3.5-turbo improves over LENS by up to 15.19% and matches or edges EXPLORA while making roughly one seventh of the LLM calls, e.g., 480 versus 3619 on GSM8K.
  • Task-level exemplars selected by CASE on smaller open models (Llama2-7b, Mistral-7b) transfer to larger models such as GPT-3.5-turbo and GPT-4o-mini, so prompt selection can be done once on a cheap model and reused.
  • Because CASE selects a static task-level set of subsets, the resulting prompts are compatible with prompt caching of key-value attention states, avoiding per-test retrieval overhead at inference time.
  • Hybrid variants that run KNN or MMR on top of CASE's top-m subsets (with self-consistency decoding) reach the highest reported scores, e.g., 87.49 on GSM8K and 64.25 on FinQA, improving over EXPLORA by up to 19.85% on AquaRat.
  • Theorem 1 gives an $(\epsilon, m, \delta)$-PAC sample-complexity bound on the event $E$, certifying that with high probability the algorithm needs at most $O(H_\epsilon(\mu) C_{\delta,\cdot}^2)$ pulls to return the $\epsilon$-optimal top-m set from the explored set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear similarity surrogate holds beyond these five datasets, the same challenger-arm bandit could be applied to other prompt-design decisions with scalar validation feedback—instruction wording, reasoning format, or demonstration length—treating each discrete choice as a bandit arm.
  • The 7x efficiency gain is demonstrated against LENS and EXPLORA on math and commonsense reasoning tasks; on tasks where the linear surrogate is a worse fit the bandit may need more pulls to certify the top-m set, so the gain is likely task-dependent.
  • The paper's theoretical story has an explicit gap: the average-regret bound for the selective-exploration set $U_T \cup N_T$ is assumed, not derived (Section 3.4), and Lemma 1's proof posits that the shortlist contains at least one true top-m arm; a rigorous regret analysis would turn CASE's empirical efficiency into a guarantee.
  • A natural extension is to use CASE's top-m subsets as a fixed retrieval pool for per-test instance selection, which the hybrid KNN/MMR variants already approximate; this would connect task-level coreset selection with dynamic retrieval while keeping inference-time overhead low.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CASE, a method for task-level exemplar selection in in-context learning that formulates the problem as top-m identification in a stochastic linear bandit. To cope with the exponential number of exemplar subsets, CASE maintains a shortlist of 'challenger' arms, samples from the unexplored set, and applies a gap-index-based selection rule, claiming up to 7x fewer LLM calls and runtime reductions relative to LENS and EXPLORA while remaining competitive in task accuracy. The paper also states an (epsilon, m, delta)-PAC sample-complexity bound for CASE and reports experiments on GSM8K, AquaRat, TabMWP, FinQA, and StrategyQA, plus synthetic bandit experiments.

Significance. If the empirical efficiency claim is correct, CASE would be practically valuable: it reduces LLM query cost for task-level demonstration selection by an order of magnitude while keeping accuracy comparable to state-of-the-art methods. The idea of using a small challenger shortlist inside a linear-bandit top-m algorithm is reasonable and the qualitative analysis of selected exemplars is a useful contribution. The paper also makes a positive practical move in demonstrating exemplar reuse from smaller to larger LLMs and releases code and data. However, the central reward model in Eq. (3) is not validated on real tasks, the theoretical guarantee relies on an unproven regret assumption and on an assumption inside Lemma 1 that essentially presupposes the target property, and some headline comparisons are confounded by the use of self-consistency in hybrid variants. These issues currently prevent full support for the paper's strongest claims.

major comments (4)
  1. [§3.2, Eq. (3)] The linear reward model is load-bearing but never validated on real tasks. Eq. (3) asserts that validation accuracy of an exemplar subset is a linear function of mean Sentence-BERT similarity, and every component of CASE (least-squares estimation, gap indices, selection rule, Theorem 1) is derived for this surrogate. The synthetic experiments in §4.2 and A.9 generate rewards from a known linear model, so they validate the bandit algorithm but not the surrogate's fidelity on actual LLM rewards. If Eq. (3) is misspecified, the identified 'top-m' arms maximize the wrong objective and the claimed efficiency could trade away accuracy on other datasets. The authors should report a direct validation on real tasks (e.g., correlation or R^2 between predicted linear scores and observed validation accuracy per dataset) or compare against a non-linear surrogate; without this, the empirical claim is not fully supported.
  2. [§3.4 and Appendix A.1] The theoretical guarantee is conditional on an unproven and explicitly postponed assumption. Section 3.4 states that the algorithm 'assumes' the average regret of U_T ∪ N_T is bounded by epsilon and postpones a rigorous derivation, yet this low-regret property is exactly what ensures that the true top-m arms are contained in the set from which CASE selects. Theorem 1 only certifies identification within U_T ∪ N_T. Moreover, Lemma 1's proof in case (iii) 'posits' that N_t contains at least one true top-m arm, which is essentially the conclusion needed for the global guarantee. The authors should either prove these properties, or explicitly state them as conjectures and weaken the claimed (epsilon, m, delta)-PAC result accordingly. As written, the PAC statement overstates what is established.
  3. [Table 1 and Section 4.3] The main accuracy results lack variance or repeated-run information for the central comparison. Table 1 reports a single number per method per dataset, and although the appendix reports standard deviations over evaluation-set partitions, this does not quantify run-to-run variability of the selection procedure. On StrategyQA, CASE scores 84.49 versus EXPLORA's 85.71, a drop that directly contradicts the headline 'without sacrificing performance' claim. The significance markers in Table 1 do not support a claim that CASE itself outperforms EXPLORA on the five datasets. The authors should provide multiple seeds, pairwise significance tests for CASE versus EXPLORA, and revise the performance claim to 'competitive' where appropriate.
  4. [Table 1 and Section 4.3] The comparison between hybrid variants and EXPLORA is confounded by the use of self-consistency. CASE+KNN+SC and CASE+MMR+SC add self-consistency decoding, while EXPLORA and LENS are reported without SC. Since self-consistency is known to improve accuracy substantially, the reported improvements of up to 19.85% over EXPLORA may be largely due to SC rather than the exemplar-selection method. This makes the percentage improvements in Table 1 misleading. The authors should add EXPLORA+SC and LENS+SC baselines, or report hybrid variants without SC, before claiming superiority over these methods.
minor comments (5)
  1. [Algorithm 1, line 7] The while condition reads 'B_t(s_t, b_t) ≤ epsilon', but the stopping rule described in §3.3 is to stop when B_t(s_t, b_t) ≤ epsilon. The pseudocode condition should be '> epsilon' to match the text and the intended behavior of running until convergence.
  2. [Theorem 1] The notation 'inf{u ∈ ρ*+ : u > 1 + Hε(µ)C_{δ,u}^2 + O(K)}' is not defined and is hard to parse. The set ρ*+ should be defined, or the expression should be replaced with a standard fixed-confidence sample-complexity statement.
  3. [Appendix A.1] The proof of Lemma 1 has several notation slips, for example W_t(b_t, c_t) appears where W_t(b_t, s_t) is clearly intended, and Property 1 is asserted without proof even though it underlies the argument. These should be corrected for the appendix to be checkable.
  4. [Table 3] The ablation table lists 'One-time sampling' and 'CASE (-exploration)' without clarifying the exact difference between them. The text says the former fits the linear model once and selects the highest mean, while the latter is a version without exploration; these should be defined precisely so that readers can interpret the comparison.
  5. [Figure 2 and Figure captions] The figure captions refer to 'N=3' without specifying whether N denotes the challenger-set size or the number of arms pulled per round; this is ambiguous and should be spelled out.

Circularity Check

2 steps flagged · score 4.0 of 10

The empirical efficiency claim is independently benchmarked, but the theoretical derivation is conditional on assumed low-regret behavior and on a 'posit' that the challenger shortlist contains a true top-m arm, which is close to the target property being proved.

  1. other [Appendix A.1, proof of Lemma 1, case (iii)]
    "Due to the above sampling approach adopted for Nt which captures the next m' arms with the highest means, we posit that Nt captures at least one arm in S⋆m."

    The proof of the gap-index bound in case (iii) assumes, rather than establishes, that the challenger shortlist Nt contains a true top-m arm. This is the very property needed for the algorithm to correct an erroneous Ut and to certify that the final top-m set is epsilon-optimal. The sampling rule in Algorithm 1 selects Nt using estimated rewards, so the presence of a true top-m arm inside Nt is not guaranteed by construction; it is introduced as an unproved input. The lemma therefore proves the needed bound only conditional on a property that is essentially part of the target guarantee, making the derivation circular at this step.

  2. other [Section 3.4, Sample Complexity bounds for the top-m selection]
    "Specifically, we assume the average regret (total regret / #iterations) of the set UT ∪ NT to upper bounded by epsilon. While we postpone a rigorous derivation of the regret bound for CASE to a later study, we justify our assumption by using the SETC Algorithm (Algo 13 in (Lattimore & Szepesvari, 2020))."

    Theorem 1's sample-complexity guarantee is conditional on the explicit assumption that the selective-exploration set UT ∪ NT achieves average regret at most epsilon. Because the derivation of this regret bound is postponed, the theorem does not derive the central near-optimality property from the algorithm's dynamics; it takes that property as an input. The bound then describes the cost of a procedure that is already assumed to be near-optimal, rather than proving that CASE attains the low-regret condition. This is a transparently stated omitted proof, but it means the theoretical chain includes the target behavior as an assumption rather than as a derived consequence.

full rationale

CASE's headline empirical claim—up to 7x fewer LLM calls without sacrificing task accuracy—is supported by held-out accuracy on five reasoning datasets and is not circular: the evaluation uses actual LLM rewards and external baselines, and the self-citation to EXPLORA is a comparison baseline, not a load-bearing justification. The circularity risk is concentrated in the theory. The paper explicitly concedes that the low-regret property of UT ∪ NT is assumed and that a rigorous regret bound is postponed. In addition, Lemma 1's case (iii) posits that the challenger shortlist contains a true top-m arm, which is essentially the correctness property the algorithm needs to establish. These are stated assumptions rather than hidden equivalences, but they make Theorem 1 conditional on the target behavior instead of a first-principles derivation. The linear reward model of Eq. 3 is an unvalidated ansatz for real validation accuracy; because the synthetic experiments generate rewards from that same linear model, they validate the bandit machinery but not the surrogate's fidelity to LLM accuracy. This is a modeling-risk issue rather than an equivalence-by-definition, so it does not raise the circularity score further. Overall, the practical contribution is independently benchmarked, so the paper is not globally circular; the score reflects the proof-level circularity and conditional structure in the theoretical claims.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim (7x fewer LLM calls at comparable accuracy) is an empirical algorithmic claim. It rests on a linear surrogate model that is fitted to the same validation rewards used for selection, and on a theoretical analysis that assumes a regret bound and a challenger-set property rather than proving them. The hyperparameters (m, m', |V|, epsilon, cluster count) are hand-chosen.

free parameters (6)
  • Linear reward coefficients alpha_i = Estimated online by ridge regression from LLM validation rewards
    The surrogate rho(a)=alpha^T x_a (Eq. 3) is fitted to noisy validation accuracies during the bandit loop; the algorithm's arm rankings and the final top-m selection depend entirely on this fit, which is never validated against a ground-truth reward model.
  • Number of training clusters = 5
    The candidate arm set S is built by sampling one exemplar from each of 5 clusters with replacement (Section 4.1); this ad hoc choice determines which exemplar subsets are ever considered.
  • Top-m size m = 10
    The number of exemplar subsets selected (Section 4.1, Hyperparameters (CASE)).
  • Challenger shortlist size m' (|N_t|) = 5
    The number of next-best arms kept as challengers in each iteration; a key complexity knob (Section 4.1).
  • Validation set size |V| = 20
    Rewards A(pi(a), V) are estimated on only 20 validation examples; this yields high-variance binary accuracy rewards (Section 4.1).
  • Stopping threshold epsilon = 0.1
    The convergence criterion B_t(s_t,b_t) <= epsilon; chosen by hand (Section 4.1).
assumptions (4)
  • ad hoc to paper Reward rho(a) for an exemplar subset is a linear function alpha^T x_a of the mean BERT similarity between selected exemplars and validation examples, plus subgaussian noise.
    Introduces Eq. (3) in Section 3.2; no empirical check of linearity; complex reasoning accuracy is unlikely to be linear in averaged similarities.
  • ad hoc to paper The average regret of the selected set U_T union N_T is bounded by epsilon via an analogy to the SETC algorithm.
    Stated in Section 3.4 as an assumption with the derivation postponed; used to argue U_T union N_T is low-regret before gap-index identification.
  • ad hoc to paper The challenger shortlist N_t contains at least one arm from the true top-m set S*_m at every round.
    Used in the proof of Lemma 1, case (iii); the paper says 'we posit' without proof, making the sample-complexity guarantee conditional on the algorithm already working.
  • standard math High-probability event E: all confidence intervals contain the true gaps (rho_i - rho_j).
    Standard concentration event in pure-exploration linear bandits (Reda et al., 2021); the theorem holds only on this event.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sample Efficient Demonstration Selection for In-Context Learning." pith.science (2026). https://pith.science/paper/7DXGQFXP

@misc{pith2026250608607,
  author       = {Pith},
  title        = {Pith review of: Sample Efficient Demonstration Selection for In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7DXGQFXP}},
  note         = {Machine review of arXiv:2506.08607}
}
read the original abstract

The in-context learning paradigm with LLMs has been instrumental in advancing a wide range of natural language processing tasks. The selection of few-shot examples (exemplars / demonstration samples) is essential for constructing effective prompts under context-length budget constraints. In this paper, we formulate the exemplar selection task as a top-m best arms identification problem. A key challenge in this setup is the exponentially large number of arms that need to be evaluated to identify the m-best arms. We propose CASE (Challenger Arm Sampling for Exemplar selection), a novel sample-efficient selective exploration strategy that maintains a shortlist of "challenger" arms, which are current candidates for the top-m arms. In each iteration, only one of the arms from this shortlist or the current topm set is pulled, thereby reducing sample complexity and, consequently, the number of LLM evaluations. Furthermore, we model the scores of exemplar subsets (arms) using a parameterized linear scoring function, leading to stochastic linear bandits setting. CASE achieves remarkable efficiency gains of up to 7x speedup in runtime while requiring 7x fewer LLM calls (87% reduction) without sacrificing performance compared to state-of-the-art exemplar selection methods. We release our code and data at https://github.com/kiranpurohit/CASE

Figures

Figures reproduced from arXiv: 2506.08607 by the authors.

Figure 1
Figure 1. Overview of CASE for selection of top-m best exemplar subsets (arms). LLMs to perform tasks by conditioning on a context that includes demonstration examples or instructions, without the need for additional fine-tuning, making it flexible and adaptable. COT facilitates stepwise problem-solving by employing rationales. However, one key challenge in maxi￾mizing the effectiveness of ICL is the careful selection of few-… view at source ↗
Figure 2
Figure 2. Top-m arm identification by CASE, LinGIFA and LinGapE for K=20, m=3, N=3. GSM Aqua Tab Fin Strat 50 100 150 200 Avg LLM calls / iteration EXPLORA CASE (a) Average number of LLM calls/iteration. GSM Aqua Tab Fin Strat 0 1 2 3 4 ·104 LLM Calls (in 10 4 ) LENS EXPLORA CASE (b) LLM calls LENS, EXPLORA vs CASE GSM Aqua Tab Fin Strat 0 20 40 60 Time (in hours) LENS EXPLORA CASE (c) Runtime LENS, EXPLORA vs CASE [PITH_FUL… view at source ↗
Figure 3
Figure 3. Sample efficiency of CASE compared to LENS and EXPLORA. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Reuse, Llama2 to gpt-3.5-turbo. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Top-m arm identification by CASE, LinGIFA and LinGapE for K=7, m=3, N=3. (a) Average number of comparisons across simulations (b) Average runtime (in seconds) (c) Gap Index (Bt(st, bt)) comparison and (d) Simple regret comparison for each round across simulations CASE …
Figure 6
Figure 6. Figure 6: Top-m arm identification by CASE, LinGIFA and LinGapE for K=10, m=3, N=3. (a) Average number of comparisons across simulations (b) Average runtime (in seconds) (c) Gap Index (Bt(st, bt)) comparison and (d) Simple regret comparison for each round across simulations by t…
Figure 7
Figure 7. Figure 7: Prompt for GSM8K AQUA Prompt Instruction:You are a helpful, respectful and honest assistant helping solve math word tasks requiring reasoning. Follow given examples and solve the tasks in step by step manner. Exemplars : [Question]: John found that the average of 15 nu…
Figure 8
Figure 8. Figure 8: Prompt for AquaRat FinQA Prompt Instruction:You are a helpful, respectful and honest assistant helping solve math word tasks requiring reasoning, using the information from given table and text. Exemplars : Read the following table, and then answer the question: [Table…
Figure 9
Figure 9. Figure 9: Prompt for FinQA 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Prompt for TabMWP StrategyQA Prompt Instruction:You are a helpful, respectful and honest assistant helping to solve commonsense problems requiring reasoning. Follow the given examples that use the facts to answer a question by decomposing into sub-questions first and …
Figure 11
Figure 11. Figure 11: Prompt for StrategyQA 19 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 46 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Improved algorithms for linear stochastic bandits

    Abbasi-Yadkori, Y., P \'a l, D., and Szepesv \'a ri, C. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011

    2011

  3. [3]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...

    work page 1901

  4. [4]

    Multiple identifications in multi-armed bandits

    Bubeck, S., Wang, T., and Viswanathan, N. Multiple identifications in multi-armed bandits. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp.\ 258--265, Atlanta, Georgia, USA, 17--19 Jun 2013. PMLR

    work page 2013

  5. [5]

    Nearly Instance Optimal Sample Complexity Bounds for Top-k Arm Selection

    Chen, L., Li, J., and Qiao, M. Nearly Instance Optimal Sample Complexity Bounds for Top-k Arm Selection . In Singh, A. and Zhu, J. (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp.\ 101--110. PMLR, 20--22 Apr 2017

    work page

  6. [6]

    Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2022 a

    work page 2022

  7. [7]

    Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., and Wang, W. Y. Finqa: A dataset of numerical reasoning over financial data, 2022 b

    work page 2022

  8. [8]

    Training verifiers to solve math word problems, 2021

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021

    2021

Show all 63 references
  1. [9]

    Gamification of pure exploration for linear bandits

    Degenne, R., Menard, P., Shang, X., and Valko, M. Gamification of pure exploration for linear bandits. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 2432--2...

    2020

  2. [10]

    Active prompting with chain-of-thought for large language models

    Diao, S., Wang, P., Lin, Y., Pan, R., Liu, X., and Zhang, T. Active prompting with chain-of-thought for large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2024 doi

  3. [11]

    Sequential experimental design for transductive linear bandits, 2019

    Fiez, T., Jain, L., Jamieson, K., and Ratliff, L. Sequential experimental design for transductive linear bandits, 2019

    2019

  4. [12]

    Complexity-based prompting for multi-step reasoning

    Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  5. [13]

    Best arm identification: A unified approach to fixed budget and fixed confidence

    Gabillon, V., Ghavamzadeh, M., and Lazaric, A. Best arm identification: A unified approach to fixed budget and fixed confidence. In Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, ...

    2012

  6. [14]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies

    Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9: 0 346--361, 2021

    2021

  7. [15]

    Prompt cache: Modular attention reuse for low-latency inference, 2024

    Gim, I., Chen, G., seob Lee, S., Sarda, N., Khandelwal, A., and Zhong, L. Prompt cache: Modular attention reuse for low-latency inference, 2024

    2024

  8. [16]

    and De Rijke, M

    Grotov, A. and De Rijke, M. Online learning to rank for information retrieval: Sigir 2016 tutorial. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pp.\ 1215--1218, 2016

    2016

  9. [17]

    Deepcore: A comprehensive library for coreset selection in deep learning, 2022

    Guo, C., Zhao, B., and Bai, Y. Deepcore: A comprehensive library for coreset selection in deep learning, 2022

    2022

  10. [18]

    Iyer, R. K. and Bilmes, J. A. Submodular optimization with submodular cover and submodular knapsack constraints. Advances in neural information processing systems, 26, 2013

    2013

  11. [19]

    and Proutiere, A

    Jedra, Y. and Proutiere, A. Optimal best-arm identification in linear bandits. Advances in Neural Information Processing Systems, 33: 0 10007--10017, 2020

    2020

  12. [20]

    and Stone, P

    Kalyanakrishnan, S. and Stone, P. Efficient selection of multiple bandit arms: Theory and practice. In ICML, volume 10, pp.\ 511--518, 2010

    2010

  13. [21]

    Pac subset selection in stochastic multi-armed bandits

    Kalyanakrishnan, S., Tewari, A., Auer, P., and Stone, P. Pac subset selection in stochastic multi-armed bandits. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, pp.\ 227–234, Madison, WI, USA, 2012. Omnipress. ISBN 9...

    2012

  14. [22]

    and Kalyanakrishnan, S

    Kaufmann, E. and Kalyanakrishnan, S. Information complexity in bandit subset selection. In Shalev-Shwartz, S. and Steinwart, I. (eds.), Proceedings of the 26th Annual Conference on Learning Theory, volume 30 of Proceedings of Machine Learning Research, pp.\ 228--251, Princeton...

    2013

  15. [23]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners, 2023

    2023

  16. [24]

    and Szepesv \'a ri, C

    Lattimore, T. and Szepesv \'a ri, C. Bandit algorithms. Cambridge University Press, 2020

    2020

  17. [25]

    Online learning to rank with features

    Li, S., Lattimore, T., and Szepesv \'a ri, C. Online learning to rank with features. In International Conference on Machine Learning, pp.\ 3856--3865. PMLR, 2019

    2019

  18. [26]

    and Qiu, X

    Li, X. and Qiu, X. Finding support examples for in-context learning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  19. [27]

    Optimal exploration is no harder than thompson sampling, 2023

    Li, Z., Jamieson, K., and Jain, L. Optimal exploration is no harder than thompson sampling, 2023

    2023

  20. [28]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems

    Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 158--167...

    2017 doi

  21. [29]

    N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan, A

    Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan, A. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In ICLR, 2023

    2023

  22. [30]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association ...

    2022 doi

  23. [31]

    Flashcheck: Exploration of efficient evidence retrieval for fast fact-checking

    Nanekhan, K., Martin, E., Vatndal, H., Setty, V., and Anand, A. Flashcheck: Exploration of efficient evidence retrieval for fast fact-checking. In European Conference on Information Retrieval, 2025

    2025

  24. [32]

    A., and Lewis, M

    Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., and Lewis, M. Measuring and narrowing the compositionality gap in language models, 2023

    2023

  25. [33]

    M., Bhattacharya, S., and Anand, A

    Purohit, K., V, V., Devalla, R., Yerragorla, K. M., Bhattacharya, S., and Anand, A. EXPLORA : Efficient exemplar subset selection for complex reasoning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural La...

    2024 doi

  26. [34]

    Guiding retrieval using llm-based listwise rankers

    Rathee, M., MacAvaney, S., and Anand, A. Guiding retrieval using llm-based listwise rankers. In European Conference on Information Retrieval, pp.\ 230--246. Springer Nature Switzerland Cham, 2025 a

    2025

  27. [35]

    Quam: Adaptive retrieval through query affinity modelling

    Rathee, M., MacAvaney, S., and Anand, A. Quam: Adaptive retrieval through query affinity modelling. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, pp.\ 954--962, 2025 b

    2025

  28. [36]

    Breaking the lens of the telescope: Online relevance estimation over large retrieval sets

    Rathee, M., Venktesh, V., MacAvaney, S., and Anand, A. Breaking the lens of the telescope: Online relevance estimation over large retrieval sets. Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, 2025 c

    2025

  29. [37]

    Top-m identification for linear bandits

    R \'e da, C., Kaufmann, E., and Delahaye-Duriez, A. Top-m identification for linear bandits. In International Conference on Artificial Intelligence and Statistics, pp.\ 1108--1116. PMLR, 2021

    2021

  30. [38]

    Knowledge-aware language model pretraining, 2021

    Rosset, C., Xiong, C., Phan, M., Song, X., Bennett, P., and Tiwary, S. Knowledge-aware language model pretraining, 2021

    2021

  31. [39]

    Roy, R. S. and Anand, A. Question answering for the curated web: Tasks and methods in qa over knowledge bases and text collections, 2022

    2022

  32. [40]

    Learning to retrieve prompts for in-context learning

    Rubin, O., Herzig, J., and Berant, J. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 2655--2671, Seattle, United State...

    2022 doi

  33. [41]

    Top-m identification for linear bandits, 2021

    Réda, C., Kaufmann, E., and Delahaye-Duriez, A. Top-m identification for linear bandits, 2021

    2021

  34. [42]

    Seer : A knapsack approach to exemplar selection for in-context hybridqa, 2023

    Tonglet, J., Reusens, M., Borchert, P., and Baesens, B. Seer : A knapsack approach to exemplar selection for in-context hybridqa, 2023

    2023

  35. [44]

    Quantemp: A real-world open-domain benchmark for fact-checking numerical claims

    Venktesh, V., Anand, A., Anand, A., and Setty, V. Quantemp: A real-world open-domain benchmark for fact-checking numerical claims. arXiv preprint arxiv:2403.17169, 2024

    2024 arXiv

  36. [45]

    Sunar: Semantic uncertainty based neighborhood aware retrieval for complex qa

    Venktesh, V., Rathee, M., and Anand, A. Sunar: Semantic uncertainty based neighborhood aware retrieval for complex qa. arXiv preprint arXiv:2503.17990, 2025

    2025 arXiv

  37. [46]

    A study into investigating temporal robustness of llms

    Wallat, J., Abdallah, A., Jatowt, A., and Anand, A. A study into investigating temporal robustness of llms. In Findings of the Association for Computational Linguistics: ACL, 2025

    2025

  38. [47]

    K.-W., and Lim, E.-P

    Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., and Lim, E.-P. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the As...

    2023 doi

  39. [48]

    V., Chi, E

    Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023 b

    2023

  40. [49]

    H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models, 2022

    2022

  41. [50]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  42. [51]

    DQ -lore: Dual queries with low rank approximation re-ranking for in-context learning

    Xiong, J., Li, Z., Zheng, C., Guo, Z., Yin, Y., Xie, E., YANG, Z., Cao, Q., Wang, H., Han, X., Tang, J., Li, C., and Liang, X. DQ -lore: Dual queries with low rank approximation re-ranking for in-context learning. In The Twelfth International Conference on Learning Representat...

    2024

  43. [52]

    Fully adaptive algorithm for pure exploration in linear bandits, 2017

    Xu, L., Honda, J., and Sugiyama, M. Fully adaptive algorithm for pure exploration in linear bandits, 2017

    2017

  44. [53]

    A fully adaptive algorithm for pure exploration in linear bandits

    Xu, L., Honda, J., and Sugiyama, M. A fully adaptive algorithm for pure exploration in linear bandits. In International Conference on Artificial Intelligence and Statistics, pp.\ 843--851. PMLR, 2018

    2018

  45. [54]

    J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J

    Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022

    2022

  46. [55]

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. H otpot QA : A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ 2369--2380, B...

    2018 doi

  47. [56]

    Compositional exemplars for in-context learning, 2023 a

    Ye, J., Wu, Z., Feng, J., Yu, T., and Kong, L. Compositional exemplars for in-context learning, 2023 a

    2023

  48. [57]

    Complementary explanations for effective in-context learning

    Ye, X., Iyer, S., Celikyilmaz, A., Stoyanov, V., Durrett, G., and Pasunuru, R. Complementary explanations for effective in-context learning. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 4469--4484, Toronto, Canada, July 2023 b . Association for ...

    2023

  49. [58]

    Zhang, R., Frei, S., and Bartlett, P. L. Trained transformers learn linear models in-context, 2023

    2023

  50. [59]

    Active example selection for in-context learning

    Zhang, Y., Feng, S., and Tan, C. Active example selection for in-context learning. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 9134--9148, Abu Dhabi, United Arab Emirates, Dec...

    2022 doi

  51. [60]

    Z., Wallace, E., Feng, S., Klein, D., and Singh, S

    Zhao, T. Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models, 2021

    2021

  52. [61]

    Online learning to rank in stochastic click models

    Zoghi, M., Tunys, T., Ghavamzadeh, M., Kveton, B., Szepesvari, C., and Wen, Z. Online learning to rank in stochastic click models. In International conference on machine learning, pp.\ 4199--4208. PMLR, 2017

    2017

  53. [62]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  54. [63]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  55. [64]

    Yes" or

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2007 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.