Pith. sign in

REVIEW 2 major objections 5 minor 18 references

Thinking makes language models use current evidence more cleanly, not seek more information.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 19:22 UTC pith:O5IPD6IF

load-bearing objection Clean multi-model result: thinking tightens evidence use and cuts unstructured noise, without a positive information-seeking signature on the measured γ/η probe. the 2 major comments →

arxiv 2607.26845 v1 pith:O5IPD6IF submitted 2026-07-29 cs.LG

Thinking Under Uncertainty: Evidence Use and Information-Seeking in Language Models

Hua-Dong Xiong , Xinyuan Yan , Ji-An Li , Jingming Xue , Marcelo G. Mattar , Robert C. Wilson This is my paper
classification cs.LG
keywords inference-time thinkingexploration-exploitationtwo-armed banditUCB-like explorationThompson-like explorationmetacognitive monitoringreported confidencelanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Inference-time thinking often raises language-model scores, but higher scores alone do not say whether models stick more tightly to what they already know or act to learn more. This paper separates those responses in a controlled two-armed bandit: ten open-weight models face matched uncertainty, choose an arm, report confidence, and optionally produce a private thinking trace. A cognitive decomposition splits value-guided choice and plain choice noise from two exploration signatures—a preference for the less-known arm and choice variability that grows with total uncertainty. On average, thinking tightened value-guided action and cut uncertainty-independent noise, without producing either exploration signature. Thinking length rose mainly after information-imbalanced histories, and reported confidence tracked decision difficulty and chosen evidence more tightly. Decoder changes altered noise and length but did not recreate the full pattern. The practical claim is that, in this setting, thinking improves use of available evidence rather than shifting policy toward information seeking.

Core claim

In matched horizon-style bandit trials, thinking strengthened value-guided action and reduced uncertainty-independent choice noise, while neither the UCB-like preference for the less-known arm nor Thompson-like uncertainty-driven variability supported a more information-seeking policy. Separately, information-imbalanced histories were linked to longer thinking traces, and reported confidence became more sensitive to decision difficulty and more strongly tied to chosen task evidence—patterns the authors treat as consistent with metacognitive control and monitoring without proving those processes.

What carries the argument

A continuous-logit hybrid choice model that puts action-logit differences in value-equivalent units: value sensitivity, a UCB-like relative-uncertainty shift γ, Thompson-like total-uncertainty variability η, and an uncertainty-independent noise floor λ. Fitted to the same controlled warm-up histories in thinking and non-thinking modes, it separates cleaner evidence use from measured exploration signatures.

Load-bearing premise

That two exploration signatures estimated from only the first free choice after a fixed warm-up—with a long horizon announced but later rounds never played—are enough to conclude models did not adopt a more information-seeking policy.

What would settle it

Run the same models through the full announced 100-round horizon with self-selected outcomes, refit γ and η on later choices under matched uncertainty, and check whether thinking then produces a clear preference for less-known arms or stronger uncertainty-driven choice variability that the initial-choice analysis missed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Evaluations of reasoning agents should score evidence-acquiring actions separately from answer quality, consistency, and thinking length.
  • Longer thinking traces cannot be treated as a proxy for information-seeking exploration.
  • Reported confidence under thinking is a distinct, difficulty-sensitive signal and should not be collapsed with action logits alone.
  • Decoder temperature can lengthen traces and raise downstream choice noise without recreating thinking’s joint effect on action, length, and confidence.
  • Claims that thinking yields better exploration need direct exploration signatures, not only higher aggregate performance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pattern holds beyond bandits, tool-use and question-asking agents may need explicit information-seeking objectives; chain-of-thought alone may mainly polish exploitation of the current context.
  • Matched human runs with the same dialogue prompts would test whether models under-explore relative to people who show directed exploration in classic horizon tasks.
  • Training or prompting that rewards reducing future uncertainty—not only immediate reward—may be required to move γ or η in the information-seeking direction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper asks whether inference-time thinking helps language models use available evidence more effectively or adopt a more information-seeking policy. Ten open-weight models complete matched horizon-style two-armed bandit trials in thinking and non-thinking modes after controlled warm-up histories. A continuous-logit hybrid model separates value-guided action, an uncertainty-independent noise floor (λ), a UCB-like relative-uncertainty shift (γ), and Thompson-like variability that scales with total uncertainty (η). On average, thinking strengthens value tracking and lowers λ, while mean γ moves only from negative toward zero (not into a less-known-arm preference) and mean η decreases. Separately, information-imbalanced histories are associated with longer thinking traces, and reported confidence becomes more sensitive to decision difficulty and chosen task evidence; decoder sweeps alter noise and length but do not reproduce the joint cross-output pattern. Claims are scoped to the controlled initial free choice and treat UCB/Thompson labels and metacognitive readings as behavioral analogies rather than internal algorithms.

Significance. The central distinction—better use of current evidence versus actions that change the information state—is load-bearing for how the field evaluates reasoning-time compute, and the paper operationalizes it with a design that is unusually clean for LLM behavioral work: matched histories across modes, counterbalancing of imbalance side and value sign, graded action and confidence logits rather than parsed free-form text, and a hybrid decomposition that separates structured exploration from unstructured noise. The null on information-seeking signatures is informative precisely because the announced H=100 supplies a long-horizon incentive at the measured decision while holding the evidence state fixed. Careful scoping (no claim of internal UCB/Thompson implementations; metacognition only as consistency, not process identification) and the decoder-control sweeps strengthen the contribution relative to aggregate accuracy or trace-length reports alone. If the result holds under the stated scope, evaluations of reasoning agents should measure evidence acquisition separately from answer quality, consistency, and thinking length.

major comments (2)
  1. [§3.1, Fig. 3, Fig. 12, App. B.2] §3.1 and Fig. 3 report mean Δγ = +1.38, Δη = −0.22, Δλ = −2.21 over nine paired models, with non-thinking estimates described as heterogeneous and one model (GPT-OSS 120B) dropped because non-thinking βV = 0 leaves normalized γ, η, λ unidentified (App. B.2; Fig. 12). The headline “neither measured signature supported a more information-seeking policy” is defensible as a mean statement, but the manuscript should make the paired, model-level pattern load-bearing rather than optional appendix material: how many of the nine models move γ across zero versus only toward zero, how many show Δη < 0, and whether the mean Δγ is driven by correcting large negative non-thinking γ in a minority of checkpoints. Without that, the aggregate bars can overstate uniformity of the exploration-null even though the scoped claim remains directionally supported.
  2. [§3.2, Methods §2.1, App. A.2, Limitations] §3.2 regresses thinking length on |RU|, |V|, and total uncertainty and interprets the |RU| association as consistent with metacognitive control. In this L-shaped design the (2,6) cells display eight rewards versus six in the matched-TU (3,3) cells (Methods §2.1; App. A.2; Limitations), so |RU| is not an isolated information-imbalance contrast. The abstract and limitations already flag the confound, but §3.2 and the Discussion still lean on this coefficient as the main non-action signature of uncertainty-responsive control. Either add a length-matched or observation-count control contrast, or relegate the metacognitive-control reading more firmly to a condition-level association (as the abstract does) so it cannot be read as a clean test that thinking allocates compute to relative uncertainty.
minor comments (5)
  1. [§2.2, Eqs. (1)–(5)] Eq. (1)/(3) present the choice-level Φ form while reported estimates come from the continuous-logit Gaussian in Eq. (4)–(5). A one-sentence pointer in §2.2 that main-text γ, η, λ are the normalized continuous-logit quantities would prevent readers from treating Fig. 2’s schematic Φ curves as the fitted object.
  2. [§3.2, Fig. 5] Fig. 5’s “weak” difficulty–length trend is described as suggestive with substantial bin overlap; consider moving it to the supplement or stating explicitly that no reliable |V| effect is claimed, so it does not compete with the clearer |RU|/condition association.
  3. [Figs. 3–4, App. B.5] App. B.5 clarifies that error bars are between-model SEM, not hierarchical posteriors. Repeating that once in the main-text figure captions (Figs. 3–4 especially) would reduce over-reading of precision.
  4. [Table 1] Table 1 lists evaluated variants; stating parameter counts or MoE vs dense status in one column would help readers interpret heterogeneity in Fig. 12 without leaving the paper.
  5. [passim] Minor prose/spacing artifacts appear throughout (e.g., “uncertainty-independentchoicenoise”, “information-imbalancedhistory”), likely from PDF extraction; clean tokenization in the camera-ready text.

Circularity Check

0 steps flagged

Empirical fit-and-report study; no derivation reduces the headline claim to its inputs by construction.

full rationale

The paper is a controlled behavioral experiment on ten open-weight LMs in a horizon-style two-armed bandit. The hybrid choice model (Eq. 1 / continuous-logit Eq. 4–5) is taken from Gershman (2018) and Wilson et al. (2014); γ, η, and λ are maximum-likelihood estimates reported as mode differences (Δγ, Δη, Δλ), not parameter-free predictions. Exploration labels are explicitly behavioral analogies. Thinking-length and confidence regressions are ordinary associations with task variables. Decoder sweeps are empirical controls. Mild self-citations (Ji-An et al. 2025; Xiong 2026) appear only as related-work framing for metacognition and cognitive modeling and do not force the central claim. No equation equates a claimed prediction to a fitted input by construction, and no uniqueness theorem is imported. Score 1 reflects only non-load-bearing author-overlap citations; the derivation chain is self-contained empirical reporting under a stated single-choice scope.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 1 invented entities

The central claim rests on a standard explore–exploit behavioral decomposition applied to LLM action logits, plus the experimental premise that a single controlled initial choice under an announced long horizon can detect information-seeking. Fitted hybrid-model and regression parameters are free; no new physical entities are introduced. Process labels (metacognitive control/monitoring) are interpretive glosses the authors explicitly do not establish.

free parameters (5)
  • Hybrid choice parameters βV, βRU, λd, ηd (normalized to γ, η, λ) = Mean thinking-mode shifts reported as Δγ=+1.38, Δη=−0.22, Δλ=−2.21 (nine models)
    Maximum-likelihood fits per model/mode/condition on action-logit differences; headline Δγ, Δη, Δλ are functions of these fits.
  • Behavioral regression coefficients on |V|, |RU|, TU for |d|, confidence, log thinking length
    Linear (and right-censored length) regressions supply the non-action cross-output claims.
  • Reported-confidence baseline b and slope s on |V|
    Per-model Ci=b+s|Vi| fits support the difficulty-sensitivity claim.
  • Decoder settings (temperature, top-p, top-k) and default thinking generation hyperparameters = Defaults T=1.00, top-p=0.95; sweeps T∈{0.5..1.5}, etc.
    Chosen experimental knobs; defaults include T=1.0, top-p=0.95, family-specific top-k, max 4096 thinking tokens.
  • Standard-error proxy scale (reward SD = 10 in sa=10/√na) = 10
    Fixed scale entering RU and TU definitions and thus all exploration estimates.
axioms (6)
  • domain assumption UCB-like γ (preference for less-observed arm) and Thompson-like η (noise rising with total uncertainty) are adequate behavioral signatures of information-seeking exploration in this task.
    Imported from Gershman 2018 / Wilson et al. 2014 horizon-bandit analyses; paper states these are analogies, not internal algorithms (§2.2).
  • domain assumption Analyzing only the initial free choice after controlled warm-up, with later H=100 rounds not executed, still tests whether the model acts to improve its information state.
    Core design choice in §2.1 and Limitations; rules out trajectory learning but is required for matched evidence states.
  • domain assumption Action-logit difference di is a valid graded preference measure, modeled as heteroskedastic Gaussian in V and RU.
    Continuous-logit estimator in Technical Supplement B.2; avoids collapsing to sampled choices.
  • ad hoc to paper Normalized parameters γ=βRU/βV, η=ηd/|βV|, λ=λd/|βV| put effects in value-equivalent units and remain interpretable when βV≠0.
    Normalization choice; forces omission of GPT-OSS 120B non-thinking when βV=0.
  • domain assumption Thinking length and verbal confidence can be read as behavioral observables consistent with metacognitive control and monitoring without establishing those processes.
    Explicit interpretive stance in Abstract and §3.2; cites Hay et al., Lee & Daunizeau.
  • standard math Standard probability and MLE/BIC model comparison machinery.
    Used for hybrid vs nested choice models and regressions.
invented entities (1)
  • None beyond labeled behavioral components (UCB-like / Thompson-like signatures as fit parameters) no independent evidence
    purpose: Decompose action under uncertainty into interpretable coefficients.
    These are renamed/fitted effects from prior cognitive models, not new latent objects claimed to exist inside the networks.

pith-pipeline@v1.2.0-daily-grok45 · 17975 in / 3731 out tokens · 75823 ms · 2026-07-30T19:22:36.355177+00:00 · methodology

0 comments
read the original abstract

Inference-time thinking improves the performance of large language models, but aggregate outcomes do not reveal whether models use available evidence more effectively or seek information that could improve future decisions. We distinguish these responses by measuring action preference, thinking length, and reported confidence under matched uncertainty. Ten open-weight models completed matched horizon-style two-armed bandit trials in thinking and non-thinking modes. A cognitive model separated value-guided action and uncertainty-independent choice noise from two behavioral signatures of exploration: a UCB-like preference for the less-known arm and Thompson-like choice variability that increases with total uncertainty. On average, thinking strengthened value-guided action and reduced uncertainty-independent choice noise, without producing UCB-like exploration or strengthening Thompson-like exploration. Outside action, the information-imbalanced history condition, which also displayed more observations than the matched balanced condition, was associated with greater thinking length. Reported confidence became more sensitive to decision difficulty and more strongly associated with chosen task evidence. We interpret these thinking-length and reported-confidence patterns as consistent with metacognitive control and metacognitive monitoring, respectively, without establishing either process. Decoder sweeps, especially temperature, altered choice noise and thinking length but did not reproduce the joint cross-output pattern. In this controlled decision setting, thinking improved how models acted on current evidence, while neither measured signature supported a shift toward a more information-seeking policy.

Figures

Figures reproduced from arXiv: 2607.26845 by Hua-Dong Xiong, Ji-An Li, Jingming Xue, Marcelo G. Mattar, Robert C. Wilson, Xinyuan Yan.

Figure 1
Figure 1. Figure 1: Horizon bandit task. The model observed controlled [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Schematic behavioral signatures of UCB-like and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Task variables are associated more clearly with thinking length and reported confidence, whereas value dominates [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Thinking length increases only weakly with deci [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Thinking increases reported-confidence sensitivity [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Thinking lowers baseline reported confidence and [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: Reported confidence is largely insensitive to de [PITH_FULL_IMAGE:figures/full_fig_p006_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Thinking length responds more to temperature and [PITH_FULL_IMAGE:figures/full_fig_p006_11.png] view at source ↗
Figure 9
Figure 9. Figure 9: Decoding settings move choice noise more than [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 12
Figure 12. Figure 12: Individual-model exploration estimates in non-thinking (left) and default-thinking (right) modes. Each point shows one [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Individual-model reported confidence fits in non-thinking (left) and default-thinking (right) modes. For each model [PITH_FULL_IMAGE:figures/full_fig_p012_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 9 linked inside Pith

  1. [1]

    Binz, M.; and Schulz, E. 2023a. Turning large language models into cognitive models. ArXiv:2306.03917 [cs]. Binz,M.;andSchulz,E.2023b. Usingcognitivepsychology tounderstandGPT-3.Proceedings of the National Academy of Sciences, 120(6): e2218523120. Brown,T.;Mann,B.;Ryder,N.;Subbiah,M.;Kaplan,J.D.; Dhariwal,P.;Neelakantan,A.;Shyam,P.;Sastry,G.;Askell, A.;Ag...

    Pith/arXiv arXiv

  2. [7]

    ArXiv:2603.17839 [cs]

    How do LLMs Com- pute Verbal Confidence. ArXiv:2603.17839 [cs]. Lee, D. G.; and Daunizeau, J

    Pith/arXiv arXiv

  3. [8]

    ArXiv:2603.25052 [cs]

    Closing the Confidence-Faithfulness Gap in Large Language Models. ArXiv:2603.25052 [cs]. Murthy, S. K.; Zhao, R.; Hu, J.; Kakade, S. M.; Wulfmeier, M.; Qian, P.; and Ullman, T

    arXiv

  4. [9]

    ArXiv:2512.20856 [cs]

    NVIDIA Nemotron 3: Efficient and Open Intelligence. ArXiv:2512.20856 [cs]. OpenAI; et al

    Pith/arXiv arXiv

  5. [10]

    ArXiv:2508.10925 [cs.CL]

    gpt-oss-120b & gpt-oss-20b Model Card. ArXiv:2508.10925 [cs.CL]. Qwen, T

    Pith/arXiv arXiv

  6. [11]

    Original-date: 2025-09-11T05:32:39Z

    Qwen3.5: Towards Native Multimodal Agents. Original-date: 2025-09-11T05:32:39Z. Shenhav, A.; Botvinick, M.; and Cohen, J

    2025

  7. [13]

    Xiong, H.-D

    Humans use directed and random ex- ploration to solve the explore–exploit dilemma.Journal of Experimental Psychology: General, 143(6): 2074–2081. Xiong, H.-D

    2074

  8. [14]

    ArXiv:2605.05851 [cs.LG]

    Hypothesis generation and updating in large language models. ArXiv:2605.05851 [cs.LG]. Xiong, M.; Hu, Z.; Lu, X.; Li, Y.; Fu, J.; He, J.; and Hooi, B

    Pith/arXiv arXiv

  9. [15]

    ArXiv:2306.13063 [cs]

    Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ArXiv:2306.13063 [cs]. Yoon,D.;Kim,S.;Yang,S.;Kim,S.;Kim,S.;Kim,Y.;Choi, E.; Kim, Y.; and Seo, M

    Pith/arXiv arXiv

  10. [16]

    ArXiv:2505.14489 [cs.AI]

    Reasoning Models Better Express Their Confidence. ArXiv:2505.14489 [cs.AI]. Zhao, R.; Meterez, A.; Kakade, S.; Pehlevan, C.; Jelassi, S.; and Malach, E

    arXiv

  11. [17]

    Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining. Model family Evaluated variants GPT-OSS 20B, 120B Gemma 4 E2B, E4B, A4B Nemotron 3 Nano A3B, Super A12B Qwen 3.5 4B, 3.5 9B, 3.6 A3B Table1:Modelfamiliesandvariantsincludedinthereported analyses. A Experimental Details A.1 Models and Thinking Modes We evaluated ten open-weight mode...

    2025

  12. [100]

    Each game starts withNwarm-up rounds

    Each machine has a fixed average payoff, but individual payoffs vary randomly. Each game starts withNwarm-up rounds. In each warm-up round, the game selects which machine you play, and you respond exactly: I play <LETTER> These rounds let you observe both machines, but their points do not count toward your score. After the warm-up rounds, you have 100 fre...

    2025

  13. [2014]

    ArXiv:1408.2048 [cs.AI]

    Selecting Computations: Theory and Applications. ArXiv:1408.2048 [cs.AI]. Ji-An, L.; Xiong, H.; Wilson, R.; Mattar, M. G.; and Benna, M. K

    Pith/arXiv arXiv 2048

  14. [2020]

    InAdvances in Neural Information Processing Systems, volume 33, 1877–1901

    Language Models are Few-Shot Learners. InAdvances in Neural Information Processing Systems, volume 33, 1877–1901. Coda-Forno, J.; Binz, M.; Wang, J. X.; and Schulz, E

    1901

  15. [2023]

    ArXiv:2203.11171 [cs]

    Self-Consistency ImprovesChainofThoughtReasoninginLanguageModels. ArXiv:2203.11171 [cs]. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; ichter, b.; Xia, F.; Chi, E.; Le, Q. V.; and Zhou, D

    Pith/arXiv arXiv

  16. [2024]

    InProceedings of the 41st International Conference on Machine Learning, 9076–9108

    CogBench: a large language model walks into a psychology lab. InProceedings of the 41st International Conference on Machine Learning, 9076–9108. PMLR. Gershman,S.J.2018. Deconstructingthehumanalgorithms for exploration.Cognition, 173: 34–42. Guiomar, G.; Torre, E.; Moure, P.; Shavina, V.; Giulianelli, M.; Liu, S.-C.; and Mante, V

    2018

  17. [2025]

    InAdvances in Neural Information Processing Systems, vol- ume 38, 60073–60108

    Language Models Are Capable of Metacogni- tive Monitoring and Control of Their Internal Activations. InAdvances in Neural Information Processing Systems, vol- ume 38, 60073–60108. Curran Associates, Inc. Kadavath, S.; Conerly, T.; Askell, A.; Henighan, T.; Drain, D.; Perez, E.; Schiefer, N.; Hatfield-Dodds, Z.; DasSarma, N.; Tran-Johnson, E.; Johnston, S....

    Pith/arXiv arXiv 2022

  18. [2026]

    ArXiv:2602.08693 [cs.LG] version:

    Reasoning aligns language models to human cognition. ArXiv:2602.08693 [cs.LG] version:

    arXiv