Pith. sign in

REVIEW 9 cited by

Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.00793 v1 pith:VJU5GF6F submitted 2024-06-02 stat.ML cs.LG

Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective

Fabian Falck , Ziyu Wang , Chris Holmes This is my paper
classification stat.ML cs.LG
keywords bayesianmartingalelearningpropertydatahypothesisuncertaintyin-context
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

In-context learning (ICL) has emerged as a particularly remarkable characteristic of Large Language Models (LLM): given a pretrained LLM and an observed dataset, LLMs can make predictions for new data points from the same distribution without fine-tuning. Numerous works have postulated ICL as approximately Bayesian inference, rendering this a natural hypothesis. In this work, we analyse this hypothesis from a new angle through the martingale property, a fundamental requirement of a Bayesian learning system for exchangeable data. We show that the martingale property is a necessary condition for unambiguous predictions in such scenarios, and enables a principled, decomposed notion of uncertainty vital in trustworthy, safety-critical systems. We derive actionable checks with corresponding theory and test statistics which must hold if the martingale property is satisfied. We also examine if uncertainty in LLMs decreases as expected in Bayesian learning when more data is observed. In three experiments, we provide evidence for violations of the martingale property, and deviations from a Bayesian scaling behaviour of uncertainty, falsifying the hypothesis that ICL is Bayesian.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Variational predictive resampling

    stat.ME 2026-05 conditional novelty 7.0

    Variational predictive resampling iteratively imputes data from a variational predictive to produce posterior samples that converge to the exact Bayesian posterior in Gaussian models where mean-field VI retains a gap.

  2. Variational predictive resampling

    stat.ME 2026-05 unverdicted novelty 7.0

    Variational predictive resampling uses sequential imputation from variational predictives to generate samples whose distribution converges to the exact Bayesian posterior in Gaussian models and improves dependence cap...

  3. Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

    cs.LG 2026-05 unverdicted novelty 7.0

    Manifold steering along activation geometry induces behavioral trajectories matching the natural manifold of outputs, while linear steering produces off-manifold unnatural behaviors.

  4. Soft Head Selection for Injecting ICL-Derived Task Embeddings

    cs.CL 2025-07 conditional novelty 7.0

    SITE applies soft gradient-based head selection to inject ICL-derived task embeddings, outperforming prior embedding adaptation and few-shot ICL across generation, reasoning, and NLU tasks on 12 LLMs from 4B to 70B pa...

  5. Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs

    cs.AI 2026-04 conditional novelty 6.5

    An agentic forecaster with linguistic belief states, logit-space multi-trial shrinkage, and hierarchical Platt calibration achieves SOTA Brier Index on ForecastBench binary questions.

  6. Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Games

    cs.AI 2026-06 unverdicted novelty 6.0

    Equation-to-Behavior Prompting lets large LLMs match cognitive models like Bayesian updating in persuasion games; RL training cuts small-model belief error by 26.5% and improves diverse training outcomes by 2.5-12%.

  7. Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs

    cs.AI 2026-04 unverdicted novelty 6.0

    BLF achieves state-of-the-art binary forecasting on ForecastBench by using linguistic belief states updated in tool-use loops, hierarchical multi-trial logit averaging, and hierarchical Platt scaling calibration.

  8. Learning to Remember, Learn, and Forget in Attention-Based Models

    cs.LG 2026-02 conditional novelty 6.0

    Palimpsa adds a per-slot importance/precision state to gated linear attention, letting a fixed-size memory forget stale information and protect important information, and recovers Mamba2 as a high-forgetting limit.

  9. LILO: Bayesian Optimization with Natural Language Feedback

    cs.LG 2025-10 unverdicted novelty 6.0

    LILO integrates LLMs to translate natural language feedback into preference signals for Gaussian process-based Bayesian optimization, outperforming standard preference BO and LLM-only methods on benchmarks.