RAG in in-context linear regression has an exact bias-variance tradeoff and a finite-sample bound revealing a generalization ceiling as retrieved examples grow.
NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In-Context Learning (ICL) is suffering from unsatisfactory performance and under-calibration due to high prior bias and unfaithful confidence. Some previous works fine-tuned language models for better ICL performance with enormous datasets and computing costs. In this paper, we propose NoisyICL, simply perturbing the model parameters by random noises to strive for better performance and calibration. Our experiments on two models and 12 downstream datasets show that NoisyICL can help ICL produce more accurate predictions. Our further analysis indicates that NoisyICL enables the model to provide more fair predictions, and also with more faithful confidence. Therefore, we believe that NoisyICL is an effective calibration of ICL. Our experimental code is uploaded to Github.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds
RAG in in-context linear regression has an exact bias-variance tradeoff and a finite-sample bound revealing a generalization ceiling as retrieved examples grow.