{"id":"26cf8bc5-0ad0-4d58-a72c-2a5149a525fc","arxiv_id":"2505.19966","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A generative preference-learning method trains a latent demonstration selector from LLM feedback and improves few-shot in-context learning performance on most of 19 benchmark datasets.","lead":"GenICL is a method that learns, from a large language model's own feedback, which few-shot examples to place in the prompt when asking the model to answer new questions. On 19 datasets it selects prompts that outperform standard retrieval-based baselines on most tasks, making few-shot prompting more reliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GenICL's set-level goal (Eq. 1) is optimized via item-level scores (Eqs. 5, 11), a gap the Limitations concede; Table 5 tests order only, not set composition.","rationale":"The reader's weakest assumption already identifies the item-level versus set-level mismatch, and the paper's own Limitations confirm it. This is the most load-bearing concern because it attacks the core claim of direct optimization: even if every single-demonstration score is well learned, nothing guarantees that top-K by Eq. (11) approximates the argmax in Eq. (1). The order experiments in Table 5 are the only direct test of prompt-level effects, and they deliberately hold the set fixed, so they cannot detect composition effects. The Bayesian derivation issues flagged by the reader are real but secondary: a flawed derivation weakens the theoretical narrative, whereas the item-level gap directly undermines the stated optimization target. Given the consistent empirical gains and the ablations showing both loss terms matter, the concern warrants a conditional verdict rather than rejection. No code, seeds, or error bars are provided, which further supports keeping the verdict conditional pending release of artifacts and the proposed set-level test.","tokens_in":16169,"tokens_out":5771,"duration_ms":73420,"concrete_test":"On BoolQ and CommonGen, fix 20 test queries and the E5-filtered candidate pool (~50 items). For each query, draw 200 random sets of K=8 demonstrations plus GenICL's selected set, and measure ICL accuracy with the frozen LLM. Compute the Spearman correlation between the sum of GenICL's item scores from Eq. (11) over a set and that set's measured accuracy. If the correlation is not significantly positive, or if GenICL's set does not rank in the top quartile of the 200 random sets, the item-level selection rule is not optimizing the set-level objective in Eq. (1).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that GenICL directly optimizes demonstration selection for ICL. The formal objective in Eq. (1) is over sets of K demonstrations, but the method scores and selects demonstrations independently: Eq. (5) labels candidates by single-demonstration likelihood P(y | (x_k, y_k), x), Eq. (6) trains on each candidate in isolation, and inference ranks by P(z | (x_k, y_k), x_test) and takes the top-K (Eq. 11). For a set-level utility, item-level optimality is not sufficient because the contribution of a demonstration depends on the other K-1 demonstrations and their order. The paper's Limitations explicitly state that 'the combination of individually optimal demonstrations does not necessarily result in the overall best combination.' Table 5 only shuffles the order of a fixed selected set; it does not vary the set composition. Figure 4 shows non-monotonic performance as K changes, which is at least consistent with set-level interactions. The claimed equivalence between Eq. (11) and Eq. (1) is therefore unsupported, and the 'direct optimization' framing is not established by the experiments. The method may still be an effective heuristic, but the strongest version of the central claim is load-bearing on an assumption the authors themselves disclaim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GenICL, a generative preference-learning framework for selecting in-context demonstrations. A latent variable z is introduced to represent the LLM's demonstration preference; the method derives an ELBO-style objective from the ICL marginal likelihood, trains z with KTO-style preference losses at both the demonstration level and the answer level, and at inference scores each candidate by the probability of generating z given the candidate and the test query, selecting the top-K demonstrations. Experiments on 19 datasets across classification, multi-choice, and generation tasks report consistent gains over retrieval and learned-retriever baselines, with additional results on multiple backbone LLMs and cross-model transfer of selected demonstrations.","tokens_in":1298,"tokens_out":1395,"duration_ms":65579,"significance":"If the central claim were fully established, GenICL would be a useful contribution: a lightweight trainable selector that, together with a frozen LLM, improves few-shot ICL across many task types. The paper deserves credit for its broad evaluation, ablations of the two loss terms, generalization across several LLMs, and the cross-model transfer experiment in Table 4. However, the theoretical derivation and the relationship between the set-level objective and the item-level scoring procedure need substantial clarification before the 'directly optimizes demonstration selection' claim can be accepted.","major_comments":[{"comment":"Equation (2) is presented as a transformation, but it assumes PM(Y | z, {(Xk,Yk)}_K, X) = PM(Y | z, X), i.e., that the target output is conditionally independent of the demonstrations once z is known. This is an unstated modeling assumption, not an identity. The ELBO derivation in Appendix D then uses q(z)=delta(z-z*) and discards delta(0) as a constant; a Dirac measure is not a density with respect to Lebesgue measure, the ratio PM(...)/delta(0) is not well-defined, and optimizing the location of a point-mass variational distribution does not produce the stated objective. These issues are load-bearing for the claim that Eq. (4) is derived from Eq. (1).","section":"Section 3.2 and Appendix D"},{"comment":"The formal objective in Eq. (1) is over sets of K demonstrations, but the method scores and selects demonstrations independently: Eq. (4) replaces the set posterior with a single-demonstration term, Eq. (6) trains on each candidate in isolation, and Eq. (11) selects the top-K by individual scores. The paper's own Limitations state that 'the combination of individually optimal demonstrations does not necessarily result in the overall best combination.' Table 5 only tests the order of a fixed selected set and does not vary set composition, while Figure 4 shows non-monotonic performance as K changes, which is at least consistent with set-level interactions. The experiments therefore do not establish the set-level 'direct optimization' claim; at present the method should be described as an item-level scoring heuristic.","section":"Section 3.1-3.4 and Limitations"},{"comment":"Appendix A states that EPR results are 'directly adopt[ed]' from the LLM-R paper, but Section 4.1 claims that 'our method and all baselines use the same template.' Borrowed numbers may come from different prompt templates, evaluation subsets, or decoding settings, so the head-to-head comparisons with EPR in Table 1 are not controlled. EPR should be rerun under the same protocol as the other methods, or the EPR column should be removed from the comparison.","section":"Section 4.1 and Appendix A"},{"comment":"All experimental tables report single point estimates without standard errors, number of seeds, or significance tests. Some reported gains are small (e.g., Table 3, Qwen2.5-3B on E2E NLG: 37.9 vs. 37.0; Table 1, AGNews: 92.6 vs. 92.4), so the claim of consistent, significant superiority is not statistically supported. The authors should report mean and variance over multiple seeds or at least provide significance tests for the main comparisons.","section":"Tables 1-4"}],"minor_comments":[{"comment":"The text calls s_k the 'log-likelihood' of the LLM generation, but the displayed formula contains no logarithm.","section":"Section 3.3, Eq. (5)"},{"comment":"There is a typo in the first sentence: 'th generalizability' should be 'the generalizability'.","section":"Section 4.4"},{"comment":"The latent variable z is called a 'task-specific description' with trainable parameters, but the paper does not specify how probabilities such as PM(z | (xk,yk), x) are computed; please clarify whether z is a token sequence and whether these are token-level log-probabilities.","section":"Section 3.2 and 3.4"},{"comment":"The figure caption says 'predicted probability of the ground truth' but does not explain how the distribution is computed or over what set of queries; please add this information.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is potentially useful, but the paper's strongest framing—that GenICL directly optimizes the set-level ICL objective—is not supported by the derivation or the experiments. The Dirac-delta ELBO and the unstated conditional independence in Eq. (2) should be presented as approximations or removed, and the set-level versus item-level gap acknowledged in the Limitations should be reflected in the main text. The borrowed EPR numbers and the absence of variance reporting need to be addressed before the comparative claims can be relied upon."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GenICL is a useful, lightweight demonstration selector that consistently beats stronger baselines across many tasks and models. The theory section is the weak part, and the paper's \"direct optimization\" claim is stronger than what the method actually does.\n\nThe actual new thing is the combination of the CBDS-style latent variable with KTO-style preference learning on paired preferred and non-preferred demonstrations, plus the two-stage (demonstration-level and answer-level) objective. That combination is new relative to the cited work, and the empirical evaluation is the paper's strength: 19 datasets, 11 task categories, multiple LLMs (Llama-7B, Vicuna-13B, GPT-Neo 2.7B, LLaMA-3.2-3B, Qwen2.5-3B), with ablations showing both loss terms and non-preferred data matter. The paper also honestly lists its limitations. That is real value.\n\nThe soft spots are mostly in the formal framing. Eq. (2) is a modeling assumption about a latent variable z, not a derivation; the Dirac-delta ELBO in Appendix D discards an infinite constant, which is hand-waving. More importantly, the optimization target in Eq. (1) is over sets of K demonstrations, but the method scores each demonstration independently (Eq. 5) and selects the top-K (Eq. 11). The paper's own Limitations concede that the combination of individually optimal demonstrations is not necessarily the overall best combination. Table 5 only shuffles order; it does not vary set composition. So the \"directly optimizes\" phrasing should be softened to something like \"a preference-learning heuristic that works empirically.\" That is a framing issue, not a load-bearing flaw: the experiments are consistent and the method clearly helps.\n\nThe reproducibility side is thinner than I'd like. No code, no seeds, no error bars, no significance tests, and EPR numbers are borrowed from the LLM-R paper. Those are all fixable in revision, and a referee should ask for them.\n\nWho should read this: anyone working on ICL demonstration selection or retrieval-based prompting. It is a good baseline to compare against, and the cross-model transfer result (Table 4) is a nice practical finding. Bring it to a reading group if you want a discussion about how much \"Bayesian\" framing is load-bearing versus decorative.\n\nMy recommendation: send it to peer review, but the authors should be asked to (1) fix or reframe the derivation, (2) acknowledge the set-level gap explicitly in the main text, and (3) release code and run significance tests.","headline":"GenICL is a solid empirical contribution with a shaky theoretical derivation and an acknowledged set-level gap; worth a serious referee but needs honest reframing and reproducibility artifacts.","tokens_in":17001,"tokens_out":3127,"would_cite":true,"duration_ms":31498,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GenICL claims that demonstration selection for in-context learning can be directly optimized as generative preference learning from LLM feedback, outperforming retrieval-based baselines across 19 datasets.","keywords":["in-context learning","demonstration selection","preference learning","LLM feedback","latent variable","generative Bayesian optimization","KTO","few-shot prompting"],"falsifier":"On a single dataset, take a pool of about 20 candidate demonstrations, compute GenICL's per-example score for each, then exhaustively evaluate all K-combinations, with a fixed order, under the frozen LLM. If the set formed by the top-K individual scores does not match the highest-accuracy combination, the independence assumption fails; the paper's Limitations predict this will happen for some datasets. A simpler observation-level check is to compare accuracy when the top-K set is replaced by a lower-scoring set that is more diverse in gold answers.","tokens_in":15952,"feed_emoji":"🎯","tokens_out":10885,"duration_ms":95752,"temperature":0.7,"pith_summary":"This paper argues that the usual way of choosing in-context demonstrations for large language models—retrieving examples by embedding similarity, or training a retriever with contrastive losses—optimizes a surrogate objective that is misaligned with what actually helps the model. It proposes GenICL, a generative preference-learning method that treats demonstration selection as a Bayesian inference problem with a latent variable representing the LLM's preference, and optimizes that variable directly against LLM feedback. The training signal is a pair of losses, one at the demonstration level and one at the answer level, both formulated with preference learning on preferred and non-preferred examples. At inference, candidates are scored by the probability they generate the learned preference variable, and the top K are placed in the prompt. If the claim holds, a modestly sized trainable module plus a frozen LLM can consistently outperform retrieval-based selectors, including on tasks where most candidate demonstrations are useless.","feed_headline":"Trained selector beats retrievers at choosing LLM few-shot examples","feed_subtitle":"GenICL learns what demonstrations the model prefers and lifts accuracy across 19 datasets","key_machinery":"The load-bearing object is the latent demonstration-preference variable $z$, realized in implementation as a task-specific description with low-rank adapter (LoRA) parameters. It is introduced through the Bayesian identity $P_{\\mathcal{M}}(Y\\mid \\{(X_k,Y_k)\\}_{k=1}^K, X) = \\int_z P_{\\mathcal{M}}(Y\\mid z, X) P_{\\mathcal{M}}(z\\mid \\{(X_k,Y_k)\\}_{k=1}^K, X)\\,dz$, and an ELBO argument turns optimization into two terms: an answer-level loss on $P_{\\mathcal{M}}(Y\\mid z, X)$ and a demonstration-level loss on $P_{\\mathcal{M}}(z\\mid (X_k,Y_k), X)$. Both losses are KTO-style preference objectives—utility-based updates that push up the utility of preferred demonstrations and answers while pushing down non-preferred ones relative to a reference model—and they are optimized alternately. At inference, each candidate $(x_k,y_k)$ is scored independently by $P_{\\mathcal{M}}(z\\mid (x_k,y_k), x_{\\text{test}})$ and the top $K$ are selected, so the latent variable is what converts the combinatorial set-selection problem into a per-example ranking problem.","core_discovery":"The paper's central claim is that the demonstration-selection problem for in-context learning, $\\arg\\max_{(x_k,y_k)\\in\\mathcal{P}} P_{\\mathcal{M}}(y \\mid \\{(x_k,y_k)\\}_{k=1}^K, x)$, can be reorganized into a generative Bayesian optimization problem by introducing a latent variable $z$ representing the LLM's demonstration preference. Using an ELBO argument, the objective separates into a term $-\\log P_{\\mathcal{M}}(Y\\mid z,X)$ and a term $-\\log P_{\\mathcal{M}}(z\\mid (X_k,Y_k), X)$, which GenICL optimizes alternately as an answer-level preference loss and a demonstration-level preference loss, both built with KTO-style updates on preferred and non-preferred demonstrations. The result is a trained latent variable, realized as a task-specific description with LoRA parameters, such that at inference, ranking demonstrations by $P_{\\mathcal{M}}(z\\mid (x_k,y_k), x_{\\text{test}})$ and taking the top $K$ yields demonstrations the LLM actually benefits from. The paper reports that this direct optimization outperforms retrieval and retriever-training baselines on 19 datasets across classification, multiple-choice, and generation tasks, and that demonstrations selected by a small model transfer to larger models.","pith_inferences":["The paper's own Limitations imply that a set-level reranking stage, one that scores combinations of demonstrations rather than individual ones, is the natural next step; GenICL does not test this, and such a stage could close the gap between Eq. (1) and the top-K approximation.","Because the preference signal is just the LLM's log-likelihood of the gold output conditioned on a candidate, the same training recipe could be applied to other prompt components such as instructions or output templates; the paper restricts the claim to demonstration examples.","The transfer result across model sizes suggests a cheap deployment path, namely training one small selector and reusing its ranked demonstrations as a frozen front-end for larger models, but the paper reports this on only four tasks, so broader transfer is an open extension.","The observed non-monotonic effect of K implies the number of demonstrations could be optimized per task as part of the same preference framework; the paper reports the effect but does not exploit it."],"forward_implications":["A moderately sized trainable module, about 79M parameters via LoRA, plus a frozen LLM can replace per-task retrieval pipelines and still improve few-shot accuracy.","Demonstrations selected on a small LLM transfer to larger LLMs without retraining, so one trained selector can serve multiple models.","High-quality demonstrations chosen by preference learning are less sensitive to ordering, weakening the common failure mode where random few-shot prompts depend heavily on example order.","More demonstrations is not uniformly better: the selected top-K can degrade accuracy when K grows, so K should be treated as a tunable per-task choice.","Directly optimizing the ICL objective gives gains over trained retrievers on generation metrics like ROUGE and exact match, where retrieval baselines plateau."],"supporting_citations":[{"why":"Introduces in-context learning, the inference-time adaptation paradigm this paper optimizes demonstration selection for.","marker":"Brown et al., 2020"},{"why":"Provides the Bayesian inference view of ICL that motivates the latent-variable reformulation of Eq. (2).","marker":"Xie et al., 2021"},{"why":"The CBDS baseline that learns a latent demonstration concept; GenICL positions its generative objective against this prior latent-variable approach.","marker":"Wang et al., 2024b"},{"why":"KTO supplies the preference-learning update used for both the demonstration-level and answer-level losses.","marker":"Ethayarajh et al., 2024"},{"why":"LLM-R is the trained-retriever baseline GenICL must outperform across datasets and model sizes.","marker":"Wang et al., 2023"},{"why":"EPR is the canonical LLM-feedback retriever baseline trained on positive and negative demonstrations.","marker":"Rubin et al., 2021"},{"why":"E5 base is both a baseline retriever and the pool-shrinking step that reduces the quadratic scoring cost before GenICL ranks candidates.","marker":"Wang et al., 2022"}],"fun_headline_variants":["GenICL: Generative selection beats retrievers for few-shot prompts","Direct preference learning picks better LLM demonstrations","Training a selector to choose examples LLMs truly prefer","LLM feedback-driven demonstration selection surpasses retrievers","GenICL: Learn which examples your LLM actually wants"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the best prompt can be assembled by ranking each candidate demonstration on its own and taking the top few; the paper's Limitations explicitly admit that individually optimal demonstrations need not form the overall best combination.","fun_headline_variants_meta":{"raw":{"variants":["GenICL: Generative selection beats retrievers for few-shot prompts","Direct preference learning picks better LLM demonstrations","Training a selector to choose examples LLMs truly prefer","LLM feedback-driven demonstration selection surpasses retrievers","GenICL: Learn which examples your LLM actually wants"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000486,"raw_usage":{"total_tokens":2397,"prompt_tokens":949,"completion_tokens":1448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1369}},"tokens_in":565,"tokens_out":1448,"duration_ms":10678,"temperature":1.0,"reasoning_tokens":1369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:03:12.401536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a single dataset, take a pool of about 20 candidate demonstrations, compute GenICL's per-example score for each, then exhaustively evaluate all K-combinations, with a fixed order, under the frozen LLM. If the set formed by the top-K individual scores does not match the highest-accuracy combination, the independence assumption fails; the paper's Limitations predict this will happen for some datasets. A simpler observation-level check is to compare accuracy when the top-K set is replaced by a lower-scoring set that is more diverse in gold answers.","supporting_citations":[],"review_version":1}