A diagnostic ladder shows that speech LMs lose emotion accuracy both at the decision rule over option logits and at the readout from hidden states, and that hidden-state emotion information exists but is rarely used in answers.
Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering
8 Pith papers cite this work. Polarity classification is still indexing.
8
Pith papers citing it
abstract
Prompting and in-context learning (ICL) have become efficient learning paradigms for large language models (LLMs). However, LLMs suffer from prompt brittleness and various bias factors in the prompt, including but not limited to the formatting, the choice verbalizers, and the ICL examples. To address this problem that results in unexpected performance degradation, calibration methods have been developed to mitigate the effects of these biases while recovering LLM performance. In this work, we first conduct a systematic analysis of the existing calibration methods, where we both provide a unified view and reveal the failure cases. Inspired by these analyses, we propose Batch Calibration (BC), a simple yet intuitive method that controls the contextual bias from the batched input, unifies various prior approaches, and effectively addresses the aforementioned issues. BC is zero-shot, inference-only, and incurs negligible additional costs. In the few-shot setup, we further extend BC to allow it to learn the contextual bias from labeled data. We validate the effectiveness of BC with PaLM 2-(S, M, L) and CLIP models and demonstrate state-of-the-art performance over previous calibration baselines across more than 10 natural language understanding and image classification tasks.
representative citing papers
Surprise Calibration uses the model's own surprise at each demonstration's label to dynamically correct class priors in in-context learning, improving accuracy on eight NLP benchmarks.
CausalAbstain filters multilingual self-feedback by comparing how much it changes the model's abstention decision, improving abstention accuracy over baselines on two benchmarks.
A one-second, training-free model (Jev) can correctly settle about a third of SCADA anomaly windows in a water network, leaving deliberate LLM review for the hard cases.
Context anxiety — abandoning solvable tasks over perceived token limits — is measurable and reducible by fine-tuning on anxiety-free reasoning traces, but the paper's causal mechanism is not actually tested.
PCS improves steering direction accuracy by adaptively sampling the intervention coefficient from a cosine-similarity-conditioned Gaussian, but its evaluation is partly circular because the optimal coefficient is chosen on the same logit-difference metric used for scoring.
Debiasing via fine-tuning can enhance LLM robustness to semantically neutral prompt perturbations by addressing perturbation-induced bias in neural network outputs.
citing papers explorer
-
Separating Decision-Rule Misalignment from Readout-Coverage Limitations in Speech Language Models
A diagnostic ladder shows that speech LMs lose emotion accuracy both at the decision rule over option logits and at the readout from hidden states, and that hidden-state emotion information exists but is rarely used in answers.
-
Surprise Calibration for Better In-Context Learning
Surprise Calibration uses the model's own surprise at each demonstration's label to dynamically correct class priors in in-context learning, improving accuracy on eight NLP benchmarks.
-
CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention
CausalAbstain filters multilingual self-feedback by comparing how much it changes the model's abstention decision, improving abstention accuracy over baselines on two benchmarks.
-
HydroJEV: A one-second, training-free screen for cyber-attack and fault attribution in water distribution networks
A one-second, training-free model (Jev) can correctly settle about a third of SCADA anomaly windows in a water network, leaving deliberate LLM review for the hard cases.
-
Lost in Context: Addressing Context Anxiety in Large Language Models
Context anxiety — abandoning solvable tasks over perceived token limits — is measurable and reducible by fine-tuning on anxiety-free reasoning traces, but the paper's causal mechanism is not actually tested.
-
Probabilistic Concept-Aware Steering for Trustworthy LLM Inference
PCS improves steering direction accuracy by adaptively sampling the intervention coefficient from a cosine-similarity-conditioned Gaussian, but its evaluation is partly circular because the optimal coefficient is chosen on the same logit-difference metric used for scoring.
-
Harnessing non-adversarial robustness in large language models
Debiasing via fine-tuning can enhance LLM robustness to semantically neutral prompt perturbations by addressing perturbation-induced bias in neural network outputs.
- From Words to Widgets for Controllable LLM Generation