{"id":"efebf5cc-1c9c-41ac-9044-0e6b21d43d1d","arxiv_id":"2505.19426","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Diversity-aware selection of in-context examples improves performance on complex and out-of-distribution tasks, though effect sizes are often modest.","lead":"This paper compares ways of choosing example prompts for large language models, testing whether a diverse set of examples beats only the most similar ones. It finds diversity helps on harder tasks, out-of-distribution queries, and harder questions, and it offers a simplified model of when this helps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.3's beyond-coverage result requires α ≥ 1 − 1/l, but all TopK-Div experiments use α = 1/2; the theory never covers the experimental regime it is invoked to justify.","rationale":"The reader's identified weakest assumption, that 'complexity' is confounded with other task properties and that per-example difficulty from fine-tuned 0-shot accuracy may be unreliable, is a legitimate secondary concern. I do not dispute it. However, the sharper and more checkable weakness is the parameter mismatch between the theoretical justification and the experiments. The paper's central novelty includes a 'beyond coverage' mechanism, and Section 3.1 explicitly invokes Section 4 as providing theoretical justification for it. Yet the only theorem formalizing beyond-coverage behavior (Theorem 4.3) requires α ≥ 1 − 1/l, while the experiments fix α = 1/2 for all TopK-Div results. This is internally inconsistent: either the theorem is being applied outside its stated regime, or the experiments do not test the regime the theory explains. The Appendix D simulations do not resolve this, because they use uniform subset distributions rather than the Example II construction. A direct simulation of Example II at α = 0.5 would settle the question. Because the empirical claims are extensive, multi-model, and reported with standard deviations in the appendix, and because the reader's CONDITIONAL verdict already flags limited theoretical support, I do not see grounds to move away from CONDITIONAL. The paper should be accepted only if the authors either extend Theorem 4.3 to α = 1/2 (or prove it is unnecessary) or soften the claimed theoretical support for the beyond-coverage interpretation.","tokens_in":36039,"tokens_out":14486,"duration_ms":133251,"concrete_test":"Run the Appendix D simulation on the exact Example II distributions of Theorem 4.3 (DE draws |T1| = l−1 from [2l] and |T2| = 1 from {2l+1,...,4l}; QE draws |T| = l from [2l]) with K = 2 and α = 0.5, comparing expected loss of TopK vs TopK-Div over random θT and queries. If TopK-Div does not beat TopK at α = 0.5, the theorem's condition is necessary and the theory does not cover the experiments; if it still beats TopK, the condition is sufficient but not necessary, and the concern reduces to a statement gap rather than a substantive failure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4's Theorem 4.3, the paper's only 'beyond coverage' justification, proves L > L′ only under the condition α ≥ 1 − 1/l (for l ≥ 3, this is at least 2/3; in the l=200 variant it is ≈0.995). Throughout the experiments, however, TopK-Div is run with α = 1/2 (Section 3, hyperparameters). Thus the theoretical result is not instantiated in any of the experiments it is claimed to explain. The gap is not merely cosmetic: Table 3's reading-task rows show the largest gains at high diversity (low α), the opposite of the theorem's parameter regime. Since the abstract and Section 3.1 state that the theory 'supports' the empirical beyond-coverage finding, this is an internal inconsistency in the argument, not an outside-consensus disagreement. The empirical findings could still hold, but the paper's stated explanation for them is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether diversity-aware selection of in-context examples improves LLM performance relative to similarity-only selection. It introduces TopK-Div, a method that greedily balances query similarity and pairwise diversity with a hyperparameter α, and compares it with random selection (Rand), TopK similarity retrieval, and a diversity-then-similarity coreset method (Div) across sentiment classification, commonsense reasoning, math, text-to-SQL, and reading comprehension, using Llama-3.1/3.2, Gemma-2, and Mistral-v0.3 models. The main empirical findings are that diversity-aware methods help on challenging tasks, on out-of-distribution queries, and on individually harder examples. The paper also develops a theoretical framework based on min-norm linear regression, with two constructed distributions (Examples I and II), proving that TopK-Div can beat TopK either by improving coverage (Theorem 4.2) or beyond coverage (Theorem 4.3).","tokens_in":36222,"tokens_out":7998,"duration_ms":67877,"significance":"If the empirical findings hold, the paper provides a useful and fairly broad map of when diversity in demonstration selection matters, with experiments spanning multiple model families, model sizes, shot counts, embeddings, and decoding strategies. The paper is transparent about its core assumptions, reports standard deviations for the main table in the appendix, and includes extensive ablations. The theoretical section is self-contained and the proofs are detailed, but it is a stylized illustration rather than a general theory; its value depends on how tightly it connects to the experimental claims. The main contribution is therefore the empirical characterization, and the theoretical framework is a supporting element that currently needs to be reconciled with the experimental parameter regime.","major_comments":[{"comment":"Theorem 4.3 proves L > L' only under the condition α ≥ 1 − 1/l (with l ≥ 3, i.e., α ≥ 2/3), whereas all main experiments fix α = 1/2, and Table 3 shows the largest diversity gains on reading tasks at low α (negative Δ). The theory's 'beyond-coverage' result is therefore not instantiated in the experimental regime it is invoked to support; please either restrict the theoretical claim, add experiments in the α ≥ 2/3 regime, or provide a separate argument connecting the high-α regime to the observed low-α gains.","section":"Section 4 (Theorem 4.3) and Section 3 (hyperparameters)"},{"comment":"The 'Average' row in Table 5 is numerically inconsistent with the per-model rows: for PRM800K, the row-by-row averages are approximately Rand 46.4, TopK 48.9, Div 46.7, TopK-Div 48.7, but the table reports 61.86, 64.97, 62.21, and 64.87. As printed, the average row shows TopK beating TopK-Div, which contradicts the text's claim that diversity-aware methods are more robust in this OOD setting; please correct the computation and re-evaluate the OOD conclusion accordingly.","section":"Table 5"},{"comment":"Tables 4, 5, 6, and 7 report no standard deviations or confidence intervals, even though Table 10 in Appendix B shows that typical per-method stds in the main table are 0.3–1.5%. Several load-bearing differences in these tables are below 1% (e.g., Table 4, SST-2 with Amazon demonstrations: TopK 89.50 vs. TopK-Div 89.60), so without error bars it is impossible to distinguish signal from noise; please report variability or significance tests for these tables.","section":"Tables 4-7"},{"comment":"Finding 1 states that 'Div and TopK-Div perform better' on challenging tasks, but Table 1 contains multiple cells where TopK-Div underperforms TopK on reading and math (e.g., Llama-3.1-8B SQuAD 4-shot: 73.28 vs. 73.51; GSM8K 4-shot: 81.74 vs. 81.99). The claim should be qualified to the specific method (Div) or supported by a significance test over the aggregated results, rather than asserted uniformly for both diversity-aware methods.","section":"Section 3.1, Finding 1 and Table 1"}],"minor_comments":[{"comment":"The title and abstract contain missing spaces ('THEROLE OFDIVERSITY ININ-CONTEXTLEARNING FOR LARGELANGUAGEMODELS'); please fix the typography.","section":"Title / Abstract"},{"comment":"The affiliation line contains a typo: 'Nanjing Universtiy' should be 'Nanjing University'.","section":"Author affiliation"},{"comment":"The sentence 'their improvement margins are significantly larger than that of Div (though still more than 1% lower than Div)' is ambiguous about whether 'lower than Div' refers to absolute accuracy or improvement margin; please rephrase.","section":"Section 3.1, Diversity v.s. coverage"},{"comment":"The Δ definition in Eq. (4) implies α values 0.1, ..., 1.0, but Section 3 states α = 1/2 for main experiments; please clarify which α values are used in the Table 3 sweep and whether any high-α (≥ 2/3) settings were tested.","section":"Section 3, hyperparameters / Table 3"},{"comment":"The paragraph referencing Figure 4 says 'the relative improvement from diversity-aware selection does not diminish significantly as model size increases', but Figure 4 also includes the shot-count ablation; the caption and text should be aligned to avoid confusion.","section":"Appendix B.2 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The Table 5 'Average' row problem is the most serious data-integrity issue; if it is a formatting artifact, it must be fixed, and the corrected numbers may weaken the OOD claim. The mismatch between Theorem 4.3's α condition and the experimental α = 1/2 is fixable by rewording the support claim or by adding high-α experiments. The empirical sweep is a useful contribution, but the missing error bars in several tables and the inconsistent per-method performance behind Finding 1 need to be addressed before the main claims can be accepted at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is the empirical picture: across three model families and five task types, diversity-aware selection (Div, TopK-Div) reliably beats similarity-only TopK on harder tasks, on out-of-distribution queries, and on individually harder examples. The direction of the effect is consistent, and the paper does a genuinely thorough job with ablations—model sizes from 1B to 70B, shot counts, different embeddings, decoding strategies, and a purely diversity-based K-Means baseline. The authors also say plainly where the mechanism is not understood, which I respect.\n\nThe real soft spot is the gap between the theory and the experiments. Theorem 4.3, the paper's only \"beyond coverage\" justification, requires α ≥ 1 − 1/l (at least 2/3 for l ≥ 3, essentially 1 for the l=200 variant). Every TopK-Div experiment in Section 3 uses α = 1/2. So the theorem is never instantiated in the experiments it is claimed to support. That is an internal inconsistency in the argument, and it undermines the abstract's claim that the theory explains the empirical beyond-coverage finding. The empirical result may still be true, and the toy-theory is honest as a toy, but the paper currently overclaims the link.\n\nSmaller issues: several key comparisons differ by less than 1%, and Tables 4–7 (OOD, hard-example, and perturbation results) omit error bars, so the magnitude of the \"diversity helps\" effect is sometimes hard to pin down. The difficulty-based explanation (Finding 3) rests on using a fine-tuned model's 0-shot accuracy as the difficulty label, which is reasonable but could be confounded with format or prior knowledge. No code is released, which matters for a paper whose value is largely a reusable practical rule.\n\nWho this is for: anyone actively working on demonstration selection for ICL will use this as a reference for when to reach for diversity. It deserves a serious referee—the empirical scope is above the bar for a workshop or a solid conference submission, but the theory–experiment mismatch should be addressed in revision, either by softening the theory claim or by running TopK-Div in the theorem's parameter regime. I would not desk-reject it.","headline":"Big empirical survey of diversity in ICL demonstration selection, with a theory that does not actually cover the regime it is invoked to explain.","tokens_in":36751,"tokens_out":1089,"would_cite":true,"duration_ms":13334,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that diversity-aware demonstration selection improves in-context learning on challenging tasks and out-of-distribution queries, and provides a linear-regression theory of why.","keywords":["in-context learning","demonstration selection","diversity","similarity-based retrieval","out-of-distribution generalization","LLM prompting","linear regression theory"],"falsifier":"Run the same comparison on sentiment or other simple tasks reformatted to be as generation-heavy as math and code, for example multi-step sentiment reasoning with exact-match grading: if TopK still beats TopK-Div under that controlled complexity, the complexity-based explanation of the gains would be refuted.","tokens_in":1542,"feed_emoji":"🧩","tokens_out":3106,"duration_ms":66258,"temperature":0.7,"pith_summary":"This paper argues that when choosing which examples to put in an LLM prompt, favoring diversity among the demonstrations — not just similarity to the query — improves in-context learning on demanding tasks and on out-of-distribution queries. Across sentiment, reasoning, reading comprehension, math, and SQL-code benchmarks on Llama-3.1, Gemma-2, and Mistral-v0.3, diversity-aware selection (Div and TopK-Div) beats similarity-only TopK on math, code, and reading comprehension, while TopK remains best on simple sentiment. The paper also reports that diversity helps most on individually harder examples and when demonstrations come from a different distribution than queries. A linear-regression model with binary \"skill\" embeddings is used to show why: diverse demonstrations can cover the query's required skills, and can help even when coverage is already complete.","feed_headline":"Diverse examples beat similar ones on hard LLM tasks","feed_subtitle":"Across Llama, Gemma, and Mistral, diversity-aware selection lifts math, code, and reading comprehension.","key_machinery":"The load-bearing object is the TopK-Div selection rule: examples are chosen greedily to maximize α·Similarity(E(x), query) + (1−α)·Diversity(E(x), already-selected set). Div is the special case α=0 after building a diverse \"coreset\". The theoretical mechanism is a linear-regression model of ICL in which the LLM is assumed to compute the min-norm solution over the selected demonstrations (Assumption 4.1), with inputs encoded as binary vectors whose 1-entries are \"skills\" or local structures. Two constructed distributions then show TopK-Div has lower expected loss than TopK: one where diversity helps by covering the query's non-zero entries (coverage), and one where both methods fully cover the query yet diversity still wins (beyond coverage).","core_discovery":"The paper's central claim is that diversity in demonstration selection is a genuine performance lever for in-context learning, not a side effect of coverage: explicitly mixing similarity with diversity (TopK-Div, with α=1/2) or selecting from a diverse coreset (Div) consistently improves over pure TopK on harder tasks, on harder examples, and under distribution shift, across open-weight model families and scales from 1B to 70B. On simple sentiment classification the ordering reverses, which the paper reads as evidence that task complexity controls when diversity pays. The paper further claims a \"beyond coverage\" effect: diversity helps even when all of a query's relevant features are already covered by the selected demonstrations, and it supports this claim with SQuAD noise-trimming experiments and with Theorem 4.3 for α≥1−1/l.","pith_inferences":["If task complexity is the true driver, diversity-aware selection should also help on other structured generation tasks such as formal proofs or tool-use planning, which the paper does not test.","The beyond-coverage result suggests a testable extension: corrupting support passages with noise should degrade TopK more than TopK-Div, because diversity may protect against overfitting to noisy surface patterns.","The fixed defaults (α=1/2 and Div coreset size 100) are likely dataset-dependent, and an adaptive α that rises with estimated task difficulty or distribution shift could yield further gains beyond the paper's experiments.","The theory assumes binary skill embeddings and min-norm regression; extending it to non-binary or compositional skill vectors could predict the boundary where added diversity stops helping."],"forward_implications":["On text-to-SQL (GeoQuery), TopK-Div outperforms TopK by at least 7 absolute accuracy points in the standard split and by larger margins in out-of-distribution splits.","On math (GSM8K, GSM-Plus-Mini, PRM800K), diversity-aware methods match or beat TopK, with the largest gains when the query set is out-of-distribution.","On reading comprehension, diversity-aware methods beat TopK, and the benefit concentrates on examples a fine-tuned model cannot answer zero-shot.","Scaling models from 1B to 70B does not erase the diversity advantage on GeoQuery and SCIQ, so the effect is not limited to small models.","Random and k-means selections can beat TopK on math and reading, showing the diversity advantage is not tied to one specific implementation."],"supporting_citations":[{"why":"Supplies the TopK similarity-based demonstration selection baseline that the diversity-aware methods are compared against.","marker":"Liu et al., 2021"},{"why":"Provides the Div method, which builds a diverse coreset via greedy diversity maximization before applying TopK.","marker":"Su et al., 2023"},{"why":"Prior evidence that diverse demonstrations improve compositional generalization, used as the coverage-based comparison point.","marker":"Levy et al., 2023"},{"why":"Empirical basis for Assumption 4.1 that transformers learn the min-norm solution for linear regression in-context.","marker":"Akyürek et al., 2023"},{"why":"Coverage-based example selection approach contrasted with explicit diversity throughout the paper.","marker":"Gupta et al., 2023"},{"why":"Motivates the difficulty measure that labels examples as easy or hard based on a fine-tuned model's zero-shot accuracy.","marker":"Swayamdipta et al., 2020"},{"why":"Another coverage-focused method referenced when explaining why diversity can go beyond feature coverage.","marker":"Ye et al., 2023"}],"fun_headline_variants":["Diversity beats similarity on hard in-context learning","For math and code, diverse examples outperform similar ones","Diverse ICL examples boost robustness and complex tasks","Why diversity in prompts beats pure similarity on hard tasks"],"cache_read_input_tokens":38912,"weakest_assumption_plain":"That the difference between \"easy\" and \"hard\" tasks is really task complexity rather than confounds like output format, answer length, or dataset noise, and that per-example difficulty can be measured by a fine-tuned model's zero-shot accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Diversity beats similarity on hard in-context learning","For math and code, diverse examples outperform similar ones","Diverse ICL examples boost robustness and complex tasks","Why diversity in prompts beats pure similarity on hard tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3177,"prompt_tokens":855,"completion_tokens":2322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":2259}},"tokens_in":471,"tokens_out":2322,"duration_ms":14501,"temperature":1.0,"reasoning_tokens":2259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:14:47.730443+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same comparison on sentiment or other simple tasks reformatted to be as generation-heavy as math and code, for example multi-step sentiment reasoning with exact-match grading: if TopK still beats TopK-Div under that controlled complexity, the complexity-based explanation of the gains would be refuted.","supporting_citations":[{"cited_title":"Diverse demonstrations improve in-context compositional generalization","cited_arxiv_id":null,"evidence_quote":"Prior evidence that diverse demonstrations improve compositional generalization, used as the coverage-based comparison point."},{"cited_title":"Coverage-based example selection for in-context learning","cited_arxiv_id":null,"evidence_quote":"Coverage-based example selection approach contrasted with explicit diversity throughout the paper."}],"review_version":1}