Pith. sign in

REVIEW 3 major objections 7 minor 80 references

Exploring Imbalanced Annotations for Effective In-Context Learning

T0 review · 3 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Class imbalance in the annotation pool degrades in-context learning across selection methods, and a score reweighting that adds an estimated conditional-bias term to class weights recovers most of the loss.

desk verdict Solid empirical paper with a useful reweighting fix for imbalanced ICL annotation pools, but the 'conditional bias' mechanism is not actually established by the experiments. read the letter →

arxiv 2502.04037 v2 pith:AGSOC5B6 submitted 2025-02-06 cs.CL cs.LG

Hongfu Gao , Feipeng Zhang , Hao Zeng , Deyu Meng , Bingyi Jing , Hongxin Wei This is my paper · ORCID
classification cs.CLcs.LG
keywords in-contextlearningclassimbalancelong-taileddistributiondemonstrationselectionconditionalbiasreweightinglargelanguagemodelsBayesianoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the class distribution of the annotation pool from which in-context demonstrations are drawn is a first-order determinant of ICL accuracy, not a nuisance. It claims that long-tailed annotation sets degrade ICL performance across tasks and across all tested demonstration-selection methods, and that classical rebalancing tricks fail because they correct only class priors while ignoring conditional bias, the shift in feature distributions inside each class. To fix this, the paper introduces RCB, which multiplies each candidate demonstration's selection score by a per-class weight combining class frequency and an estimated conditional-bias term. If the claim is right, any ICL pipeline sampling demonstrations from real-world imbalanced data is leaving recoverable accuracy on the table without fine-tuning.

What carries the argument

The load-bearing object is the identity $P_t(x,y)/P_c(x,y) = w + \beta$, which splits the distribution mismatch between test and annotation pools into class weights and conditional bias. RCB computes $w$ from effective numbers, draws a balanced subset $D_b$, and estimates one scalar $\beta_j$ per class by Bayesian optimization over the loss on $D_b$ when demonstrations are selected from the remaining imbalanced data with scores reweighted by $w + \beta$. This per-class scalar is then added to the class weight and multiplies the original selection score, making the mechanism a score-reweighting wrapper around any existing selection method.

What would settle it

Construct a long-tailed dataset whose tail classes each contain two distinct subpopulations with opposite distribution shifts, and compare RCB against vanilla selection and against oracle reweighting using the true conditional ratio $P_t(x|y)/P_c(x|y)$; if RCB underperforms the oracle by a large margin, or underperforms vanilla, the scalar approximation is the failing link.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that imbalanced annotations hurt ICL through two separable channels: class weights $w = P_t(Y)/P_c(Y)$, the mismatch of class priors between test and annotation distributions, and conditional bias $\beta = \frac{P_t(y)}{P_c(y)}(\frac{P_t(x|y)}{P_c(x|y)} - 1)$, the within-class feature shift that pure class rebalancing ignores. The paper derives this decomposition by importance sampling, verifies with KL divergence that $\beta$ is nonzero, and shows empirically that reweighting demonstration scores by $w + \beta$, with $\beta$ estimated on a small balanced subset via Bayesian optimization, lifts the average accuracy of six selection methods from 46.74% to 52.16% at imbalance ratio 100 and also improves generation tasks.

Load-bearing premise

The argument rests on assuming that a single per-class number, estimated on a small balanced subset, can capture the within-class feature shift between annotation and test data.

Editorial extensions

If this is right

  • At imbalance ratio 100, RCB raises the average accuracy of six selection methods from 46.74% to 52.16%, a direct 5.42-point gain with the same number of demonstrations.
  • The gain grows with imbalance: on AgNews and Yahoo, the improvement rises from 4.64 points at ratio 10 to 11.81 points at ratio 1000.
  • RCB also improves generation tasks, lifting exact match on Natural Questions from 21.20 to 22.93 at imbalance ratio 100.
  • Because RCB only accesses model outputs, it helps across open-weight models and black-box APIs.
  • Classical rebalancing methods cannot substitute for the correction: stratified sampling can even reduce accuracy because it overrides the selection method's ranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same two-component correction should apply to any retrieval pipeline drawing examples from an imbalanced pool, not just ICL; the paper does not test this extension.
  • Because the conditional-bias term is a per-class scalar, the method will be miscalibrated when within-class feature shift varies sharply across inputs, so a per-example or per-cluster estimator is the natural next experiment.
  • The balanced-subset requirement is the practical bottleneck: in extremely long-tailed pools where the smallest class has very few examples, holding out a balanced subset may starve the selection pool, a limitation the paper itself acknowledges.
  • Reweighting the selection pool and calibrating the model's output target different failure modes, so combining RCB with output calibration could recover more than either alone; the paper compares but does not combine them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies in-context learning (ICL) when the annotation pool is class-imbalanced. It reports that imbalance degrades ICL performance across models and selection methods, that classical rebalancing heuristics give limited or negative gains, and it proposes Reweighting with Conditional Bias (RCB), which reweights demonstration-selection scores by an effective-number class weight plus a per-class "conditional bias" scalar estimated by Bayesian optimization on a balanced subset. Experiments on four classification and two generation tasks show average gains of up to 5.42 points over six existing selection methods.

Significance. If the central empirical claim holds, the paper identifies a practically important and under-studied factor in ICL: the distribution of the annotation pool, not just the prompt composition, matters for downstream accuracy. The proposed RCB is a cheap, model-agnostic plug-in that improves several selection methods across multiple LLMs and APIs. The paper's strengths are its broad experimental coverage (seven datasets, five open-weight models plus two APIs, six selection methods, imbalance ratios up to 1000, standard deviations over three runs) and the availability of code and data in the supplement. However, the mechanistic interpretation of the fitted scalar as conditional bias is not established by the experiments, because the method is compared only against fixed rebalancing heuristics and not against a baseline that tunes class weights on the same balanced subset used to fit beta.

major comments (3)
  1. [Section 4.1, Eq. (8) vs. Section 4.2] The importance-sampling identity in Eq. (8) reweights the expected error M[...] by Pt(x,y)/Pc(x,y), i.e., it applies to the loss on a test point, not to the scores s(ci,xt) of candidate demonstrations. The proposed RCB, however, multiplies the demonstration scores by w+beta, and the paper does not derive why this score reweighting minimizes the reweighted error in Eq. (7). The connection is asserted rather than shown, so the theoretical motivation for RCB is incomplete. Please either supply the missing derivation or explicitly present RCB as a heuristic motivated by the decomposition in Eq. (8).
  2. [Section 4.2, Eq. (11) and Section 5.2, Table 1] The fitted quantity beta is a vector of per-class scalars, so w+beta is observationally just another per-class multiplicative weight on the selection score. Since beta is estimated on Db, a balanced subset, and the test sets are also class-balanced (Section 5.1, Appendix D.1), the optimization objective in Eq. (11) is aligned with the evaluation metric. The paper compares RCB only against fixed rebalancing heuristics (Section 3.2, Figure 1c), not against a baseline that tunes per-class weights on Db itself, either with or without the effective-number term w. Such a baseline is needed to distinguish the mechanistic claim that beta corrects the conditional-bias term Pt(x|y)/Pc(x|y)-1 from the alternative that the gain comes from validation-tuned class reweighting that transfers to balanced test sets. Please add this baseline, and/or constrain beta using feature-level estimates of the conditional distribution, or revise the claims to describe RCB as a tuned per-class reweighting method.
  3. [Section 4.2, Eq. (11)] The manuscript does not state whether beta is estimated once per dataset and then applied to all selection methods or re-estimated separately for each selection method. Since Eq. (11) uses TopK with a specific scoring function s, the fitted beta likely depends on the selection method, but this is not clarified. If beta is re-fit per selection method on Db, the comparison in Table 1 is closer to per-method validation tuning; if it is shared, the paper should say so explicitly and justify why a single beta works across different selection methods.
minor comments (7)
  1. [Abstract] The abstract contains the typo "condition bias" and should read "conditional bias".
  2. [Section 5.2] The sentence "the test accuracy of ICL using TopK is improved to 4.18" appears to be missing a percentage sign and the word "points"; based on Table 1, the intended statement is likely "improved by 4.18 points".
  3. [Table 3 and Section E.2] The vanilla Macro-F1 for Emotion is reported as 32.22 in Table 3 but as 32.12 in the text of Section 5.2 and Section E.2; please harmonize the values.
  4. [Appendix D.1] The appendix says "We randomly subsample examples from the test dataset to verify the performance of the imbalanced annotated dataset," which is inconsistent with the main text's description of creating imbalanced annotation pools from training data; please clarify whether the subsampling is from the train set.
  5. [Section 5.1 and Table 1] The baseline name is written inconsistently as "ByCS" in the text and "ByDC" in Table 1 and several appendix tables; please use one name consistently throughout.
  6. [Appendix C] The Bayesian optimization details omit numerical settings such as the number of initial points, the Expected Improvement exploration parameter epsilon, and the stopping criterion; please report these values or point to the code defaults.
  7. [Table 1] At imbalance ratio 1 (balanced data), RCB occasionally slightly decreases accuracy (e.g., TopK with K=8: 58.32 to 58.12), so the phrase "consistently enhance" overstates the results; please qualify the claim to imbalanced settings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RCB's per-class beta is tuned on a held-out balanced subset and evaluated on separate test sets, so the reported gains are not forced by construction.

full rationale

The paper's derivation chain is not circular. The theoretical decomposition in Eq. (8) is an exact importance-sampling identity: the ratio Pt(x,y)/Pc(x,y) is split into class weights w and a conditional-bias term beta. This identity is a mathematical equivalence, not a result that presupposes the method's success. The actual method, RCB, then estimates a per-class scalar beta by minimizing the ICL loss on a balanced subset Db (Eq. 11), and the main results in Table 1 are measured on separate balanced test sets. Thus the central claim that RCB improves accuracy is an empirical, held-out evaluation rather than a quantity that reduces to the definition of beta or to the fitting objective. The balanced subset is used for validation-style tuning, which is a legitimate experimental procedure; it does not make the test-set numbers circular. The paper's self-citations ([7], [42]) appear in contextual discussions of long-tailed data and ICL robustness, not as the load-bearing justification for RCB or for the claimed improvement. A possible weakness is that the tuned per-class beta may absorb effects other than the defined conditional bias, and the paper does not compare against a Db-tuned class-weight baseline; however, this concerns the mechanistic interpretation and experimental control, not circularity. No step in the derivation is equivalent to its inputs by construction, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central fitted object is the per-class conditional bias beta, tuned on a balanced subset. The method also relies on the ICL-as-Bayesian-inference framing from Xie et al. [18] and on the assumption that the test prior is class-balanced.

free parameters (4)
  • beta (conditional bias vector, per-class) = not reported in text
    Estimated via Bayesian optimization on Db (Eq. 11); controls the reweighting and is the central fitted object of RCB.
  • |Db| (balanced subset size) = 100 by default
    Chosen by hand (Section 5.1); paper shows insensitivity across sizes 50, 100, 200.
  • K' (candidate pool size) = 1600 by default
    Chosen by hand (Section 5.1); paper shows K' should grow with imbalance ratio.
  • Bayesian optimization hyperparameters = 30 iterations mentioned in Appendix E.6
    Required to define the fitting procedure; bounds and initialization are not specified.
assumptions (4)
  • domain assumption In-context learning approximates the posterior predictive of the data-generating model theta* (Remark 1, from Xie et al. [18]).
    Basis for Eq. (3) and the claim that class prior mismatch affects ICL predictions; cited, not proved.
  • domain assumption The test distribution has class-balanced priors while the annotation set is imbalanced.
    RCB's balanced subset Db is meant to align with the test composition; Section 2.2 and Eq. (9).
  • standard math The importance-sampling weights Pt(x,y)/Pc(x,y) are well-defined, i.e., Pc has support covering Pt.
    Needed for the Error decomposition in Eq. (8).
  • ad hoc to paper Conditional bias can be approximated by a per-class scalar.
    Eq. (10)-(11) replace the per-example beta in Eq. (8) with a scalar per class; not formally justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Imbalanced Annotations for Effective In-Context Learning." pith.science (2026). https://pith.science/paper/AGSOC5B6

@misc{pith2026250204037,
  author       = {Pith},
  title        = {Pith review of: Exploring Imbalanced Annotations for Effective In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGSOC5B6}},
  note         = {Machine review of arXiv:2502.04037}
}
read the original abstract

Large language models (LLMs) have shown impressive performance on downstream tasks through in-context learning (ICL), which heavily relies on the demonstrations selected from annotated datasets. However, these datasets often exhibit long-tailed class distributions in real-world scenarios, leading to biased demonstration selection. In this work, we show that such class imbalances significantly degrade the ICL performance across various tasks, regardless of selection methods. Moreover, classical rebalancing methods, which focus solely on class weights, yield poor performance due to neglecting condition bias--skewed feature distributions within classes. To address this, we propose Reweighting with Conditional Bias (dubbed RCB), a simple and complementary approach to enhance ICL performance under class imbalance. In particular, RCB estimates conditional bias using a balanced subset and re-weights demonstration scores based on both class weight and conditional bias. In effect, RCB prevents over-selection from dominant classes while preserving the efficacy of current selection methods. Extensive experiments on common benchmarks demonstrate the effectiveness of our method, improving the average accuracy of current selection methods by up to 5.42%.

Figures

Figures reproduced from arXiv: 2502.04037 by the authors.

Figure 1
Figure 1. The impact of imbalanced annotations on ICL performance. (a) Overall accuracy across [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The Kullback-Leibler (KL) divergence between [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Results for ablation studies. (a) Ablation on the different balanced subset sizes. (b) Ablation [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The impact of imbalanced annotations on the performance of In-Context Learning (ICL) in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: An example of imbalanced dataset. In Emotion [12], a few sentiments make a large contribution and data tend to show a long-tailed distribution. For example, “joy“ and “sadness“ are head classes, while most other classes, such as “love“ and “surprise“, are tail classes …
Figure 6
Figure 6. Figure 6: The Kullback-Leibler (KL) divergence between [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: The comparison of average accuracy for each class in the Amazon (a), Yelp (b) and Yahoo [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 58 canonical work pages

  1. [6]

    Bayesian example selection improves in-context learning for speech, text and visual modalities

    Siyin Wang, Chao-Han Huck Yang, Ji Wu, and Chao Zhang. Bayesian example selection improves in-context learning for speech, text and visual modalities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20812–20828, 2024

    work page 2024

  2. [1]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877–1901, 2020

    work page 1901

  3. [2]

    Few- shot fine-tuning vs

    Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few- shot fine-tuning vs. in-context learning: A fair comparison and evaluation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12284–12314, 2023

    work page 2023

  4. [3]

    In-context learning through the bayesian prism

    Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. In International Conference on Learning Representations, 2024

    work page 2024

  5. [4]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, 2022

    work page 2022

  6. [5]

    What makes multimodal in-context learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1539–1550, 2024

    Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Pi- wowarski. What makes multimodal in-context learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1539–1550, 2024

    work page 2024

  7. [7]

    Open-sampling: Exploring out-of-distribution data for re-balancing long-tailed datasets

    Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. Open-sampling: Exploring out-of-distribution data for re-balancing long-tailed datasets. In International Conference on Machine Learning, pages 23615–23630, 2022

    work page 2022

  8. [8]

    A survey of deep long-tail classification advance- ments

    Charika De Alvis and Suranga Seneviratne. A survey of deep long-tail classification advance- ments. arXiv preprint arXiv:2404.15593, 2024

    arXiv 2024

Show all 80 references
  1. [9]

    Class-balanced loss based on effective number of samples

    Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019

    2019

  2. [10]

    Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The multilingual amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020

    2020

  3. [11]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems, volume 28, 2015

    2015

  4. [12]

    CARER: Contextualized affect representations for emotion recognition

    Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687–3697, Brussels, Belgium, 2018

    2018

  5. [13]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  6. [14]

    Evaluating the state of semantic code search (2019)

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, Marc Brockschmidt, and Code-SearchNet Challenge. Evaluating the state of semantic code search (2019). doi: 10.48550. ARXIV, 2019

    2019

  7. [15]

    Revisiting demonstration selection strategies in in-context learning

    Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Revisiting demonstration selection strategies in in-context learning. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

    2024

  8. [16]

    Learning to retrieve prompts for in- context learning

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in- context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, 2022

    2022

  9. [17]

    Compositional ex- emplars for in-context learning

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional ex- emplars for in-context learning. In International Conference on Machine Learning , pages 39818—-39833, 2023

    2023

  10. [18]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022

    2022

  11. [19]

    OPT: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. O...

    2022 arXiv

  12. [20]

    Llama 3 model card, 2024

    AI@Meta. Llama 3 model card, 2024

    2024

  13. [21]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  14. [22]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  15. [23]

    Complementary explanations for effective in-context learning

    Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Veselin Stoyanov, Greg Durrett, and Ramakanth Pasunuru. Complementary explanations for effective in-context learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4469–4484, 2023

    2023

  16. [24]

    How re-sampling helps for long-tail learning? In Advances in Neural Information Processing Systems, pages 75669–75687, 2023

    Jiang-Xin Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How re-sampling helps for long-tail learning? In Advances in Neural Information Processing Systems, pages 75669–75687, 2023

    2023

  17. [25]

    Smote: synthetic minority over-sampling technique

    Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321– 357, 2002

    2002

  18. [26]

    Exploratory undersampling for class-imbalance learning

    Xu-Ying Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39:539– 550, 2008

    2008

  19. [27]

    Experiments with svm and stratified sampling with an imbalanced problem: Detection of intestinal contractions

    Fernando Vilariño, Panagiota Spyridonos, Jordi Vitrià, and Petia Radeva. Experiments with svm and stratified sampling with an imbalanced problem: Detection of intestinal contractions. In International Conference on Pattern Recognition and Image Analysis, pages 783–791, 2005

    2005

  20. [28]

    Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective

    Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2020

  21. [29]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2019

  22. [30]

    MetaICL: Learning to learn in context

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809, 2022. 11

    2022

  23. [31]

    Smith, and Tao Yu

    Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In International Conference on Learning Representations, 2023

    2023

  24. [32]

    Decoupling representation and classifier for long-tailed recognition

    Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020

    2020

  25. [33]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Representations, 2023

    2023

  26. [34]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9318—-9333, 2023

    2023

  27. [35]

    Many-shot in-context learning

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024

    2024 arXiv

  28. [36]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, 2024

    2024

  29. [37]

    Aleksandra Edwards and Jose Camacho-Collados. Language models for text classification: Is in-context learning enough? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10058–10072, 2024

    2024

  30. [38]

    Fabian Falck, Ziyu Wang, and Christopher C. Holmes. Is in-context learning in large language models bayesian? A martingale perspective. In International Conference on Machine Learning, pages 12784–12805, 2024

    2024

  31. [39]

    Demystifying prompts in language models via perplexity estimation

    Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10136–10148, 2023

    2023

  32. [40]

    Bayes’ power for explaining in-context learning generalizations

    Samuel Müller, Noah Hollmann, and Frank Hutter. Bayes’ power for explaining in-context learning generalizations. arXiv preprint arXiv:2410.01565, 2024

    2024 arXiv

  33. [41]

    Mitigating label biases for in-context learning

    Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. Mitigating label biases for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14014–14031, 2023

    2023

  34. [42]

    On the noise robustness of in-context learning for text generation

    Hongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu, Feng Zheng, and Hongxin Wei. On the noise robustness of in-context learning for text generation. In Advances in Neural Information Processing Systems, 2024

    2024

  35. [43]

    Icl markup: Structuring in-context learning using soft-token tags

    Marc-Etienne Brunet, Ashton Anderson, and Richard Zemel. Icl markup: Structuring in-context learning using soft-token tags. arXiv preprint arXiv:2312.07405, 2023

    2023 arXiv

  36. [44]

    Enhancing in-context learning via linear probe calibration

    Momin Abbas, Yi Zhou, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Salonidis, and Tianyi Chen. Enhancing in-context learning via linear probe calibration. In International Conference on Artificial Intelligence and Statistics, pages 307–315, 2024

    2024

  37. [45]

    Active example selection for in-context learning

    Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134–9148, 2022

    2022

  38. [46]

    In-context example selection with influences

    Tai Nguyen and Eric Wong. In-context example selection with influences. arXiv preprint arXiv:2302.11042, 2023. 12

    2023 arXiv

  39. [47]

    In-context learning with iterative demonstration selection

    Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demonstration selection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7441—-7455, 2024

    2024

  40. [48]

    Unifying demonstration selection and compression for in-context learning

    Jun Gao, Ziqiang Cao, and Wenjie Li. Unifying demonstration selection and compression for in-context learning. arXiv preprint arXiv:2405.17062, 2024

    2024 arXiv

  41. [49]

    In-context learning with retrieved demonstrations for language models: A survey

    Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. In-context learning with retrieved demonstrations for language models: A survey. arXiv preprint arXiv:2401.11624, 2024

    2024 arXiv

  42. [50]

    C-ICL: Contrastive in-context learning for information extraction

    Ying Mo, Jiahao Liu, Jian Yang, Qifan Wang, Shun Zhang, Jingang Wang, and Zhoujun Li. C-ICL: Contrastive in-context learning for information extraction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10099–10114, 2024

    2024

  43. [51]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2022

  44. [52]

    Let’s learn step by step: Enhancing in-context learning ability with curriculum learning

    Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, Yong Huang, and Wei Lu. Let’s learn step by step: Enhancing in-context learning ability with curriculum learning. arXiv preprint arXiv:2402.10738, 2024

    2024 arXiv

  45. [53]

    More is not always better? enhancing many-shot in-context learning with differentiated and reweighting objectives

    Xiaoqing Zhang, Ang Lv, Yuhan Liu, Flood Sung, Wei Liu, Shuo Shang, Xiuying Chen, and Rui Yan. More is not always better? enhancing many-shot in-context learning with differentiated and reweighting objectives. arXiv preprint arXiv:2501.04070, 2025

    2025 arXiv

  46. [54]

    Mind your format: Towards consistent evaluation of in-context learning improvements

    Anton V oronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evaluation of in-context learning improvements. arXiv preprint arXiv:2401.06766, 2024

    2024 arXiv

  47. [55]

    More samples or more prompts? exploring effective few-shot in-context learning for LLMs with in-context sampling

    Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, and Dakuo Wang. More samples or more prompts? exploring effective few-shot in-context learning for LLMs with in-context sampling. In Findings of the Association for...

    2024

  48. [56]

    Calibrate before use: Improving few-shot performance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706, 2021

    2021

  49. [57]

    How robust are llms to in-context majority label bias? arXiv preprint arXiv:2312.16549, 2023

    Karan Gupta, Sumegh Roychowdhury, Siva Rajesh Kasa, Santhosh Kumar Kasa, Anish Bhanushali, Nikhil Pattisapu, and Prasanna Srinivasa Murthy. How robust are llms to in-context majority label bias? arXiv preprint arXiv:2312.16549, 2023

    2023 arXiv

  50. [58]

    Fine-tune language models to approximate unbiased in-context learning

    Timothy Chu, Zhao Song, and Chiwun Yang. Fine-tune language models to approximate unbiased in-context learning. arXiv preprint arXiv:2310.03331, 2023

    2023 arXiv

  51. [59]

    Mixtures of in-context learners

    Giwon Hong, Emile van Krieken, Edoardo Ponti, Nikolay Malkin, and Pasquale Minervini. Mixtures of in-context learners. arXiv preprint arXiv:2411.02830, 2024

    2024 arXiv

  52. [60]

    Beyond performance: Quantifying and mitigating label bias in llms

    Yuval Reif and Roy Schwartz. Beyond performance: Quantifying and mitigating label bias in llms. arXiv preprint arXiv:2405.02743, 2024

    2024 arXiv

  53. [61]

    Large scale fine-grained categorization and domain-specific transfer learning

    Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4109–4118, 2018

    2018

  54. [62]

    Generalized test utilities for long-tail performance in extreme multi-label classification

    Erik Schultheis, Marek Wydmuch, Wojciech Kotlowski, Rohit Babbar, and Krzysztof Dem- bczynski. Generalized test utilities for long-tail performance in extreme multi-label classification. In Advances in Neural Information Processing Systems, volume 36, 2024

    2024

  55. [63]

    Equalization loss for long-tailed object recognition

    Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11662–11671, 2020. 13

    2020

  56. [64]

    Robust asymmetric loss for multi-label long-tailed learning

    Wongi Park, Inhyuk Park, Sungeun Kim, and Jongbin Ryu. Robust asymmetric loss for multi-label long-tailed learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2711–2720, 2023

    2023

  57. [65]

    Robust loss function for class imbalanced semantic segmentation and image classification

    S Divakar Bhat, Mudit Soni, and Yuji Yasui. Robust loss function for class imbalanced semantic segmentation and image classification. IFAC-PapersOnLine, 56(2):7934–7939, 2023

    2023

  58. [66]

    Stochastic smoothing of the top-k calibrated hinge loss for deep imbalanced classification

    Camille Garcin, Maximilien Servajean, Alexis Joly, and Joseph Salmon. Stochastic smoothing of the top-k calibrated hinge loss for deep imbalanced classification. InInternational Conference on Machine Learning, pages 7208–7222, 2022

    2022

  59. [67]

    Retrieval augmented classification for long-tail visual recognition

    Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2022

  60. [68]

    Lt-darts: An architectural approach to enhance deep long-tailed learning

    Yuhan Pan, Yanan Sun, and Wei Gong. Lt-darts: An architectural approach to enhance deep long-tailed learning. arXiv preprint arXiv:2411.06098, 2024

    2024

  61. [69]

    Bayesian optimization with inequality constraints

    Jacob R Gardner, Matt J Kusner, Zhixiang Eddie Xu, Kilian Q Weinberger, and John P Cun- ningham. Bayesian optimization with inequality constraints. In International Conference on Machine Learning, volume 2014, pages 937–945, 2014

    2014

  62. [70]

    Bayesian Optimization: Open source constrained global optimization tool for Python, 2014

    Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014

    2014

  63. [71]

    OpenICL: An open-source framework for in-context learning

    Zhenyu Wu, Yaoxiang Wang, Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, and Yu Qiao. OpenICL: An open-source framework for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), ...

    2023

  64. [72]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Ben- jamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. Peft: State-of-the- art parameter-efficient fine-tuning methods, 2022

    2022

  65. [73]

    A bayesian approach for prompt optimization in pre-trained language models

    Antonio Sabbatella, Andrea Ponti, Antonio Candelieri, Ilaria Giordani, and Francesco Archetti. A bayesian approach for prompt optimization in pre-trained language models. arXiv preprint arXiv:2312.00471, 2023

    2023 arXiv

  66. [74]

    Searching for optimal solutions with llms via bayesian optimization

    Dhruv Agarwal, Manoj Ghuhan Arivazhagan, Rajarshi Das, Sandesh Swamy, Sopan Khosla, and Rashmi Gangadharaiah. Searching for optimal solutions with llms via bayesian optimization. In International Conference on Learning Representations, 2025. 14 Appendix A Related Work In-conte...

    2025

  67. [75]

    Random selects demonstrations randomly from an example set without repetition [30]

  68. [76]

    TopK retrieves demonstrations that are semantically similar to a test query sample [4]

  69. [77]

    DPP uses the original BERT embeddings as mentioned above without fine-tuning, and adopts MAP inference for subset retrieval [17]

  70. [78]

    VoteK proposes an unsupervised and graph-based selective annotation method to select diverse and representative demonstrations [31]

  71. [79]

    ConE searches for demonstrations by minimizing the difference in cross-entropy between the test input and the demonstrations [15]

  72. [80]

    func NewMessage() Message{return Message{Context: context.Background(),Headers: map[string]string{},Data: render.Data{},moot:&sync.RWMutex,}}

    ByCS assumes that an accurate inverse likelihood probability will lead to an accurate posterior probability and selects demonstrations based on their inverse inference results [6]. D.3 Inference Perplexity For classification tasks, we compute the sentence perplexity for each s...

    1974

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.