What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

Christopher J. Earls; Eric Ho; Pratyush Ranjan Tiwari; Rapha\"el Sarfati; Siddharth Boppana; Srikar Varadaraj

REVIEW 4 major objections 7 minor 20 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Probes Catch What LLM Forecasters Won't Say: Confidence, Lies, and Pre-Set Answers

2026-07-10 01:00 UTC pith:LTJMHRWX

load-bearing objection Probes calibrate LLM forecasters better than verbalized confidence, but the main comparison omits recalibrated baselines the 4 major comments →

arxiv 2607.08046 v1 pith:LTJMHRWX submitted 2026-07-09 cs.CL cs.AI

What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

Rapha\"el Sarfati , Pratyush Ranjan Tiwari , Siddharth Boppana , Christopher J. Earls , Srikar Varadaraj , Eric Ho This is my paper

classification cs.CL cs.AI

keywords reasoninginternalrepresentationsactivationsanswerbettercalibrationevidence

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models fine-tuned for forecasting can be accurate yet badly overconfident, and their chain-of-thought reasoning often fails to reflect the evidence that actually drove their prediction. This paper argues that the model's internal activations contain a more faithful record of both its confidence and its reasoning process than anything it verbalizes. By training small linear probes on pooled intermediate activations, the authors recover a confidence signal that is substantially better calibrated than the model's own stated probability (ECE 0.044 vs 0.093), and they show this holds across multiple model families and even into out-of-distribution math reasoning. The same probes serve as lie detectors: when evidence is removed or fabricated and the model's forecast shifts but its written reasoning stays silent, the probe's internal state still registers the shift, predicting the direction of change 84% of the time. Finally, by forcing the model to answer before it reasons, the authors show that the forecast and its confidence are largely fixed before the chain of thought even begins, and that a single cheap pre-reasoning pass can route questions to commit early, reason, or retrieve more evidence, saving 30-47% of generated tokens with no accuracy loss.

Core claim

The central discovery is that a language model forecaster's internal representations carry well-calibrated confidence, detect hidden evidence influence, and reveal pre-committed answers, all of which the model's verbalized output distorts or conceals. A lightweight probe trained on intermediate activations reads this internal state directly, yielding a practical instrument for calibration, auditing, and inference triage that requires no changes to the model itself.

What carries the argument

The key mechanism is a pooling probe: a small linear classifier trained on frozen intermediate activations (residual-stream vectors) from a large language model. Three pooling architectures are compared: mean-pooling (averaging activations across token positions), attention-pooling (trained attention heads that weight token positions), and covariance-pooling (compressing second-order co-fluctuation statistics of activations into a low-dimensional bilinear bottleneck). The covariance probe at a mid-to-late layer (layer 21 of 36) achieves the best calibration. The probe outputs a single probability of answer correctness, trained with binary cross-entropy. The same probe activations, read over原

Load-bearing premise

The probe's calibration advantage is assumed to reflect a genuine model-internal correctness signal rather than dataset-specific priors learned from the training distribution. The headline calibration result is evaluated in-distribution (trained on OpenForesight-train, tested on OpenForesight-test); if the probe learns question-level or context-level patterns specific to this dataset rather than model-internal correctness structure, the advantage may not transfer to truly out

What would settle it

Train the probe on one forecasting dataset and evaluate calibration on a completely different forecasting benchmark with different question types, evidence formats, and resolution criteria. If the probe's calibration advantage disappears, the signal is dataset-specific rather than model-internal. Alternatively, if probes trained on random labels or shuffled activations achieve similar calibration, the advantage is an artifact of the training procedure rather than the activations themselves.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If probes can reliably read calibrated confidence from frozen models, then any deployed reasoning model could be retrofitted with calibrated confidence without retraining, simply by training a small probe on its activations.
If forecasts are largely pre-committed before reasoning begins, then chain-of-thought is functioning more as a post-hoc rationalization than a search process for some questions, and the value of long reasoning traces may be concentrated in a narrow band of question difficulty.
If the probe detects silent evidence shifts that the CoT conceals, it could serve as a real-time auditing tool for deployed AI systems, flagging cases where the model's stated reasoning diverges from the evidence that actually moved its answer.
If pre-reasoning answer spread predicts when reasoning will help vs. when retrieval is needed, then inference routing could become a standard pre-pass that cuts compute cost on easy questions and redirects effort toward gathering evidence on hard ones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the model's confidence is fixed before reasoning and the probe can read it, one could train the model to verbalize the probe's internal confidence estimate directly, potentially closing the gap between internal and stated confidence without an external probe at inference time.
The finding that stealth cases produce smaller probe shifts than faithful cases of equal behavioral impact suggests the model processes concealed and disclosed evidence differently at the representation level, which could be exploited to build a continuous faithfulness score rather than a binary stealth/faithful classifier.
If the pre-reasoning answer distribution's entropy sorts questions into commit/reason/retrieve regimes, then a single forward pass could also serve as a difficulty estimator for curriculum learning or active learning, directing training effort toward questions in the middle regime where reasoning actually changes the answer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Probes calibrate LLM forecasters better than verbalized confidence, but the main comparison omits recalibrated baselines

read the letter

The paper you sent me — Sarfati et al. on probing internal representations of LLM forecasters — has a genuinely useful core. Three findings stand out: (1) lightweight probes on intermediate activations achieve much better calibration than the model's verbalized confidence (ECE 0.044 vs 0.093), (2) the same probes detect when the chain-of-thought silently shifts in response to evidence perturbation (rho=0.565 vs 0.215 for the CoT text itself), and (3) forecasts are largely committed before reasoning begins, enabling a token-saving triage procedure that cuts 30-47% of generated tokens with no accuracy loss. The forced-answering experiment is the cleanest part — the finding that confidence is reasoning-independent (Spearman rho=0.90 between forced and free confidence) is striking and well-demonstrated across three splits. The lie-detector application to CoT faithfulness is a nice extension of the probing framework to a new audit setting, and the evidence ablation/injection experiments are carefully designed with appropriate noise-floor controls. The GLM probe-only experiments on frozen models provide reasonable evidence that the effect doesn't depend on the underlying model being trained for forecasting. Now the soft spots. The stress-test concern about missing recalibrated baselines is the real issue. The headline ECE comparison pits a BCE-trained probe (which naturally produces calibrated outputs by construction) against raw verbalized confidence with no post-hoc recalibration. The paper itself shows in the math experiment (Table 2) that isotonic-recalibrated verbalized confidence beats the probe on ECE (0.125 vs 0.143). If the same holds on the forecasting task — and the near-identical AUROC (0.756 vs 0.758) suggests it would — the calibration advantage shrinks to 'the probe is naturally calibrated without an extra step,' which is a convenience claim, not a fundamental information advantage. The paper should have run isotonic or temperature scaling on the verbalized baseline on the main task. The in-distribution concern is also valid but less damaging: the OOD results on skysports hold up (ECE 0.064 vs 0.089), though aljazeera attenuates (0.140 vs 0.178). The proprietary EF-8B model and non-public improved-context dataset limit reproducibility, but the GLM experiments partially offset this. The probe layer and architecture selection involves enough free parameters (layer, pooling type, read site, bottleneck dim) that some optimistic selection is likely, though the sweep is systematic and reported. Overall: the paper does several things well, the practical token-saving procedure has immediate value, and the lie-detector framing is genuinely new. The missing recalibrated baseline is the one gap that materially weakens the headline claim. A serious referee should require that comparison. Beyond that, the work is solid and worth engaging with.

Referee Report

4 major / 7 minor

Summary. The paper investigates whether internal representations of LLM forecasters provide a more reliable signal for confidence calibration and chain-of-thought faithfulness than verbalized outputs. Using Eternis-Forecaster 8B (EF-8B) on the OpenForesight dataset, the authors train pooling probes (mean, attention, covariance) on intermediate activations and compare their calibration against the model's verbalized confidence. They find probes achieve lower ECE (0.044 vs 0.093), that CoT reasoning is only partially faithful under evidence ablation (23% stealth rate), that probes track behavioral shifts better than CoT text (Spearman ρ=0.565), and that forecasts are largely pre-committed before reasoning begins, enabling 30–47% token savings. Results are partially extended to GLM models and to a math reasoning setting.

Significance. The paper addresses a timely and practically important question: whether internal probes can serve as calibration and auditing tools for reasoning LLMs. The faithfulness experiments (evidence ablation, diversionary injection, probe-based lie detection) are well-designed and the forced-answering analysis yielding token savings is a nice practical contribution. The paper ships falsifiable predictions (direction-of-change accuracy, token savings bounds) and includes leakage controls for the GLM experiments. The multi-model generalization (EF-8B, GLM-4.7-Flash, GLM-4.5-Air, Qwen3-8B math) is commendable. However, the central calibration claim has a methodological asymmetry that needs to be addressed before publication.

major comments (4)

§4.2.2, Figure 1: The headline calibration comparison (ECE 0.044 vs 0.093) compares the probe's raw BCE-trained probabilities against the model's raw verbalized confidence, with no post-hoc recalibration applied to either signal. This is a consequential asymmetry. BCE training directly optimizes for matching empirical frequencies, so the probe is naturally calibrated by construction of its loss function; raw verbalized confidence has no such guarantee. The paper itself demonstrates the importance of this point: in the math experiment (Table 2, §4.4), isotonic-recalibrated verbalized confidence achieves ECE 0.125, beating the probe's 0.143 on the DCPO-recipe model. Since the probe and verbalized confidence have nearly identical AUROC on the forecasting task (0.756 vs 0.758), applying isotonic regression (or temperature scaling) to verbalized confidence would likely close most of the ECE差距
gap. The paper should either (a) apply the same post-hoc recalibration to verbalized confidence on the forecasting task and report the resulting ECE, or (b) reframe the calibration claim from 'substantially better calibration' to something like 'the probe is naturally calibrated without requiring a held-out recalibration set,' which is a valid practical advantage but a weaker and more precise claim. As it stands, the abstract's 'substantially better calibration' overstates what the experimental design can support.
§4.2.2 and Appendix C: The probe is trained on OpenForesight-train rollouts and evaluated on OpenForesight-test rollouts from the same dataset. While the OOD splits (skysports, aljazeera) provide partial transfer evidence, the OOD calibration advantage attenuates substantially on aljazeera (ECE 0.140 vs 0.178, a 0.038 gap compared to the 0.049 in-distribution gap). The paper should explicitly discuss the risk that the probe learns dataset-specific correctness priors rather than model-internal correctness signals, and clarify whether the OOD evaluation uses a probe trained on the in-distribution train split (which would make the OOD result a genuine transfer test) or retrained on each OOD split.
§4.5.3, Figure 7a: The probe-based lie detector result (ρ=0.565) is one of the paper's strongest contributions, but the comparison baseline is the CoT-text correlation (ρ=0.215). This is not the only possible baseline: a natural alternative would be to measure how well the change in verbalized confidence (|Δprob| itself, or some function of it) predicts the behavioral shift. Since |Δprob| is the behavioral change by definition, using it as the predictor would be circular, but the paper should clarify why the probe's |Δprobe| is a non-trivial predictor beyond what is already available from the verbalized probability shift. The claim that the probe 'predicts the direction of change in 84% of cases' (§4.5.3) needs clarification: is this the direction of the signed behavioral shift, and what is the chance baseline for this task?

minor comments (7)

§3.2: The 'improved context-building scheme' for the non-public compact-context build is mentioned but not described in sufficient detail for reproducibility. What specifically was improved over the semantic-search contexts used in Chandak et al. [2026]?
§4.1.1: The temperature sweep uses T=1.0 for the rest of the paper, but the ECE is reported as 0.110–0.150 across temperatures while the main-text calibration comparison in §4.2.2 uses ECE 0.093 for verbalized confidence. Please clarify whether the 0.093 figure uses a different evaluation set or conditions.
Table 1: The 'Clean ΔAUROC' column for GLM-4.7-Flash has a 95% CI of [−0.006, 0.111], which includes zero. The text correctly notes this ('the leakage-controlled ranking delta is positive but not significant'), but the figure caption for Figure 2 says 'Internal probes improve AUROC on both full test sets' without qualification for GLM-4.7-Flash. Consider adding a qualifier.
§4.6: The forced-answering experiments use EF-32B while the rest of the paper primarily uses EF-8B. The switch is noted but not motivated. Why is EF-32B used here rather than EF-8B? Are the forced-answering findings expected to transfer to EF-8B?
Figure 6a: The x-axis label 'behavioral impact |Δprob|' uses a symbol (¡) that appears to be a rendering artifact. Please check all figure labels for encoding issues.
§4.7: The entropy gate classifier is described as trained on 'cheap signals read off a single forced-answer pass,' but the training data and evaluation protocol for this classifier are not specified. Is it trained on the test split with cross-validation, or on a separate train split?
Appendix A, Figure 11: The data audit is useful but the claim that all 429 shared-qid instances are 'collisions rather than duplicated questions' is verified only by 'exact match and LLM-as-a-judge.' A brief note on the judge's reliability for this specific task would strengthen the audit.

Circularity Check

0 steps flagged

No circularity found: empirical paper with proper train/test splits and externally falsifiable claims

full rationale

This paper is an empirical study with no formal derivation chain that could be circular. The central claims are measured on held-out or OOD data: (1) The calibration probes are trained on OpenForesight-train rollouts and evaluated on the test split — the ECE 0.044 vs 0.093 result is an empirical measurement on held-out data, not a tautological consequence of the training procedure. While BCE loss has calibration-encouraging properties, good test-set calibration is not guaranteed by training-set loss minimization. (2) The lie-detector claim (ρ=0.565) uses perturbation data the probe was never trained on. (3) The forced-answering and token-saving results are direct empirical measurements comparing two decoding conditions. (4) The GLM and math experiments use frozen models with probes trained on separate splits and evaluated OOD. The self-citations present (e.g., Boppana et al. 2026 for attention-pooling, Dooms et al. 2026 for covariance pooling) are method citations that do not bear the central claims — the claims stand on the paper's own experiments. The skeptic's concern about recalibration asymmetry (BCE naturally calibrates the probe while verbalized confidence gets no post-hoc recalibration) is a legitimate fairness/correctness critique, but it is not circularity: the paper is not defining its output in terms of its input, nor invoking a self-cited theorem to force its conclusion. The paper is self-contained against external benchmarks and its results are independently falsifiable.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The paper introduces no new entities or particles. It relies on standard ML concepts (probes, activations, ECE) and applies them to existing LLM architectures. The free parameters are standard hyperparameters selected on validation data, not fitted constants in a derivation.

free parameters (4)

Probe layer selection = L21 (covariance) for EF-8B, L18 (attention) for GLM-4.5-Air
Selected on validation data; the specific layer and pooling architecture are chosen post-hoc by best validation performance.
High-impact threshold = |delta_prob| > 0.05
Used to define 'high-impact' ablation pairs for the stealth rate calculation; chosen by hand.
Covariance bottleneck dimension = 64
Hyperparameter for covariance probes, chosen ad hoc.
Entropy gate cutoff = Single cutoff on entropy H
Used for the commit-early token-saving procedure; fixed but not specified how.

axioms (3)

domain assumption Linear probes on pooled residual-stream activations can faithfully read out a model's internal confidence.
The entire approach assumes that a linear classifier on intermediate activations captures a meaningful confidence signal that is not available in the model's verbalized output. This is supported by the results but is a foundational assumption.
domain assumption LLM-as-a-judge provides reliable correctness labels for forecasting answers.
All correctness labels used for probe training and evaluation are generated by an LLM judge. If the judge is systematically biased, the probes learn that bias.
domain assumption Evidence ablation and diversionary injection are valid proxies for reasoning faithfulness.
The faithfulness claims depend on the assumption that removing or injecting articles and measuring CoT change is a valid test of faithfulness, following Gur-Arieh et al. 2026.

pith-pipeline@v1.1.0-glm · 27320 in / 2148 out tokens · 252302 ms · 2026-07-10T01:00:28.410954+00:00 · methodology

0 comments

read the original abstract

Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with Eternis-Forecaster 8B on OpenForesight, we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM-4.7-Flash and GLM-4.5-Air. We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model's forecast while leaving the reasoning trace untouched. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84% of cases, including when the CoT conceals the perturbation's influence. Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens, with no loss of accuracy. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly.

Figures

Figures reproduced from arXiv: 2607.08046 by Christopher J. Earls, Eric Ho, Pratyush Ranjan Tiwari, Rapha\"el Sarfati, Siddharth Boppana, Srikar Varadaraj.

**Figure 1.** Figure 1: Reliability diagram for the layer-21 covariance probe (ember) and EF-8B ’s verbalized confidence from the final <probability> tag (slate), on the OpenForesight-test split (N = 3,020 rollouts each). 10 equal-width bins; marker size ∝ bin count; diagonal = perfect calibration. The probe tracks the diagonal (ECE = 0.044) while verbalized confidence is overconfident above 0.5 (ECE = 0.093). 9 [PITH_FULL_IMAG… view at source ↗

**Figure 2.** Figure 2: Probe-only confidence from GLM internals. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

**Figure 3.** Figure 3: GLM-4.5-Air reliability. The layer-18 attention probe is substantially better calibrated than the model’s stated probability on the test split. Linear Attn-1 Attn-4 Cov 0.6 0.65 0.7 0.75 0.8 Best-layer test AUROC Matched 24–40 harvest GLM-4.5-Air GLM-4.7-Flash [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

**Figure 4.** Figure 4: Matched-depth cross-model comparison. Best-layer test AUROC by probe family for GLM-4.5-Air and GLM-4.7-Flash on the matched 24–40 harvest, with each model’s verbalized-confidence AUROC as a reference line. The calibration effect is the common result. On GLM-4.7-Flash , the practical mean-pool probe has ECE 0.054 versus 0.287 for verbalized confidence; the leakage-controlled ranking delta is positive but n… view at source ↗

**Figure 5.** Figure 5: Probe vs. token-logprob confidence on pooled OOD math. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

**Figure 6.** Figure 6: Chain-of-thought faithfulness under evidence perturbation. (a) [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

**Figure 7.** Figure 7: Internal probe as a lie detector for unfaithful reasoning. (a) [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

**Figure 8.** Figure 8: What chain of thought does and does not change. (a) [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗

**Figure 9.** Figure 9: Triaging forecasting questions by the spread of the model’s pre-reasoning answer [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗

**Figure 10.** Figure 10: Cross-split qid overlap in OpenForesight . Off-diagonal cells give the number of question IDs (qids) shared by both splits; the diagonal (split sizes, listed below) is omitted from the color scale. Nine of fifteen split pairs share qids (429 sharedqid instances in total), yet every shared qid is a collision: in all 429 cases the two questions differ in both title and gold answer (same-question rate = 0).… view at source ↗

**Figure 11.** Figure 11: Prompt token length by split in OpenForesight . Bars show the mean token count of the retrieval-augmented prompt field, tokenized with the Qwen3 tokenizer with special tokens excluded; whiskers span the 2.5th–97.5th percentiles, i.e. the range containing the central 95% of prompts in each split. Estimates use every question for splits with n ≤ 3, 000 and a uniform random sample of 3,000 for train. Two reg… view at source ↗

**Figure 12.** Figure 12: Temperature sweep for EF-8B on the OpenForesight-test split. Each curve summarizes 10 rollouts per question across all 296 test questions (29,600 generations) as a function of sampling temperature T ∈ {0.2, . . . , 2.0}. Self-consistency (fraction of a question’s rollouts in its modal answer cluster) falls monotonically from 65.9% (T=0.2) to 30.1% (T=2.0), while per-rollout accuracy (Claude-judged answer … view at source ↗

**Figure 13.** Figure 13: Reliability of EF-8B ’s verbalized confidence across temperature. Reliability diagrams for T = 0.4, 0.8, 1.2, 1.6 on the OpenForesight-test split. In each panel the model’s stated probabilities are grouped into ten equal-width confidence bins; bar height is the empirical accuracy within the bin and the dashed diagonal marks perfect calibration, so bars below the diagonal indicate overconfidence. The model… view at source ↗

**Figure 14.** Figure 14: Probe AUROC across layers and reasoning sites [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗

**Figure 15.** Figure 15: Probe Brier score across layers and reasoning sites [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗

**Figure 16.** Figure 16: GLM-4.5-Air probe versus verbalized confidence. [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗

**Figure 17.** Figure 17: GLM-4.5-Air depth profile. Forecast-correctness decodability across residual-stream layers and probe families, showing a broad layer 18–23 plateau. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_17.png] view at source ↗

**Figure 18.** Figure 18: Negative-control probes sit at chance. The dashed line marks AUROC = 0.5. Random-label and shuffled-activation probes are trained identically to the real probe, with 5 seeds each. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_18.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · 10 internal anchors

[1]

URLhttps://arxiv.org/abs/1610.01644. Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Olek- sandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and B...

work page internal anchor Pith review Pith/arXiv arXiv
[2]

Probing Classifiers: Promises, Shortcomings, and Advances

URLhttps://arxiv. org/abs/2102.12452. Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, and Jack Merullo. Reasoning theater: Disentangling model beliefs from chain-of-thought,

work page internal anchor Pith review Pith/arXiv arXiv
[3]

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

URL https://arxiv.org/abs/2603.05488. Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. Scaling open-ended reasoning to predict the future,

work page internal anchor Pith review Pith/arXiv arXiv
[4]

Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas

URLhttps://arxiv.org/abs/2512.25070. Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond binary rewards: Training lms to reason about their uncertainty,

work page arXiv
[5]

Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

URLhttps://arxiv.org/ abs/2507.16806. Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. InInternational Conference on Machine Learning (ICML),

work page internal anchor Pith review Pith/arXiv arXiv
[6]

GLM Team and Zhipu AI

Blog post, accessed 2026-06-11. GLM Team and Zhipu AI. GLM-4.5: Agentic, reasoning, and coding foundation models. Technical report, Zhipu AI,

work page 2026
[7]

FutureSim: Replaying World Events to Evaluate Adaptive Agents

Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, and Jonas Geiping. Futuresim: Replaying world events to evaluate adaptive agents. CoRR, abs/2605.15188,

work page internal anchor Pith review Pith/arXiv arXiv
[8]

FutureSim: Replaying World Events to Evaluate Adaptive Agents

doi: 10.48550/ARXIV.2605.15188. URLhttps://doi.org/10.48550/arXiv. 2605.15188. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. InAdvances in Neural Information Processing Systems (NeurIPS),

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.15188
[9]

A Survey on LLM-as-a-Judge

URLhttps://arxiv.org/abs/2411.15594. Yoav Gur-Arieh, Ana Marasović, and Mor Geva. Faithfulness metrics don’t measure faithfulness: A meta- evaluation with ground truth,

work page internal anchor Pith review Pith/arXiv arXiv
[10]

Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth

URLhttps://arxiv.org/abs/2605.25052. Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level forecasting with language models. InAdvances in Neural Information Processing Systems (NeurIPS),

work page internal anchor Pith review Pith/arXiv arXiv
[11]

Toni J.B

URLhttps://iclr.cc/virtual/2025/poster/28507. Toni J.B. Liu, Nicolas Boullé, Raphaël Sarfati, and Christopher Earls. LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15097–15117, Miami, Florida, USA,

work page 2025
[12]

URLhttps://aclanthology.org/2024.emnlp-main.842/

Association for Computational Linguistics. URLhttps://aclanthology.org/2024.emnlp-main.842/. Zhengzhao Ma, Xueru Wen, Boxi Cao, Yaojie Lu, Hongyu Lin, Jinglin Yang, Min He, Xianpei Han, and Le Sun. Decoupling reasoning and confidence: Resurrecting calibration in reinforcement learning from verifiable rewards,

work page 2024
[13]

Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

URLhttps://arxiv.org/abs/2603.09117. Michael T. Pearce, Thomas Dooms, Ryo Yamamoto, Joshua Meehl, Carl Molnar, Mark Bissell, Dron Hazra, Ching Fang, Nam Nguyen, Michael Anderson, Collin Osborne, Patrick Duffy, Bridget Toomey, Eric Klee, Elena Myasoedova, Alexander J. Ryu, Shant Ayanian, Panos Korfiatis, Matt Redlon, Archa Jain, Daniel Balsam, and Nicholas...

work page internal anchor Pith review Pith/arXiv arXiv
[14]

URLhttps://www.biorxiv.org/ content/early/2026/04/16/2026.04.10.717844

doi: 10.64898/2026.04.10.717844. URLhttps://www.biorxiv.org/ content/early/2026/04/16/2026.04.10.717844. Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Rafael Valdece Sousa Bastos, and Philip E. Tetlock. Wisdom of the silicon crowd: Llm ensemble prediction capabilities rival human crowd accuracy.Science Advances, 10(45):eadp1528,

work page doi:10.64898/2026.04.10.717844 2026
[15]

URLhttps://www.science.org/doi/abs/ 10.1126/sciadv.adp1528

doi: 10.1126/sciadv.adp1528. URLhttps://www.science.org/doi/abs/ 10.1126/sciadv.adp1528. Mingtian Tan, Mike A. Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language mod- els actually useful for time series forecasting? InAdvances in Neural Information Processing Systems (NeurIPS),

work page doi:10.1126/sciadv.adp1528
[16]

Qwen3 Technical Report

URLhttps://arxiv.org/abs/2505.09388. Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track,

work page internal anchor Pith review Pith/arXiv arXiv
[17]

5) and are considerably longer.OpenForesightrows are from the publictest split (qids 645, 159);MATH-500from itstestsplit; AIME rows from AIME24 and AMC rows from AMC23 (the AIME25 and AMC24 splits share an identical format). Prompt / Problem Ground truth OpenForesight — forecasting(testsplit) continued on next page 20 Table 3 (continued) Prompt / Problem ...

work page 2025
[18]

Your answer will be evaluated using the BRIER SCORING RULE which is basically (- (1 - p)^2) if your answer is correct and (- 1 - p^2) if your answer is incorrect

You will be rewarded based on the probability (p) you assign to your answer. Your answer will be evaluated using the BRIER SCORING RULE which is basically (- (1 - p)^2) if your answer is correct and (- 1 - p^2) if your answer is incorrect. For example, if p = 0.5, and your answer is incorrect, then your score will be (-1 - 0.5^2) = (-1 - 0.25) = -1.25 whe...

work page 2025
[19]

Then the value of log2(x4y3z2) is m n wheremandnare relatively prime positive integers

14 3 AIME24 — math reasoning Letx, yandzbe positive real numbers that satisfy the following system of equations: log2 x yz = 1 2 ,log 2 y xz = 1 3 ,log 2 z xy = 1 4 . Then the value of log2(x4y3z2) is m n wheremandnare relatively prime positive integers. Findm+n.(2024-II-4) 33 continued on next page 22 Table 3 (continued) Prompt / Problem Ground truth Jen...

work page 2024
[20]

Models and decode.We compare the untrained Qwen3-8B base model with ourQwen3-8B model trained according to the DCPO recipe of Ma et al

This appendix records the supporting setup. Models and decode.We compare the untrained Qwen3-8B base model with ourQwen3-8B model trained according to the DCPO recipe of Ma et al. [2026]. The decode settings are matched across cells: temperature 0.7, top-p0.8, top-k20, presence penalty1.5, maximum3,000generated tokens, logprobs enabled, no thinking mode, ...

work page 2026

[1] [1]

URLhttps://arxiv.org/abs/1610.01644. Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Olek- sandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and B...

work page internal anchor Pith review Pith/arXiv arXiv

[2] [2]

Probing Classifiers: Promises, Shortcomings, and Advances

URLhttps://arxiv. org/abs/2102.12452. Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, and Jack Merullo. Reasoning theater: Disentangling model beliefs from chain-of-thought,

work page internal anchor Pith review Pith/arXiv arXiv

[3] [3]

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

URL https://arxiv.org/abs/2603.05488. Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. Scaling open-ended reasoning to predict the future,

work page internal anchor Pith review Pith/arXiv arXiv

[4] [4]

Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas

URLhttps://arxiv.org/abs/2512.25070. Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond binary rewards: Training lms to reason about their uncertainty,

work page arXiv

[5] [5]

Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

URLhttps://arxiv.org/ abs/2507.16806. Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. InInternational Conference on Machine Learning (ICML),

work page internal anchor Pith review Pith/arXiv arXiv

[6] [6]

GLM Team and Zhipu AI

Blog post, accessed 2026-06-11. GLM Team and Zhipu AI. GLM-4.5: Agentic, reasoning, and coding foundation models. Technical report, Zhipu AI,

work page 2026

[7] [7]

FutureSim: Replaying World Events to Evaluate Adaptive Agents

Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, and Jonas Geiping. Futuresim: Replaying world events to evaluate adaptive agents. CoRR, abs/2605.15188,

work page internal anchor Pith review Pith/arXiv arXiv

[8] [8]

FutureSim: Replaying World Events to Evaluate Adaptive Agents

doi: 10.48550/ARXIV.2605.15188. URLhttps://doi.org/10.48550/arXiv. 2605.15188. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. InAdvances in Neural Information Processing Systems (NeurIPS),

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.15188

[9] [9]

A Survey on LLM-as-a-Judge

URLhttps://arxiv.org/abs/2411.15594. Yoav Gur-Arieh, Ana Marasović, and Mor Geva. Faithfulness metrics don’t measure faithfulness: A meta- evaluation with ground truth,

work page internal anchor Pith review Pith/arXiv arXiv

[10] [10]

Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth

URLhttps://arxiv.org/abs/2605.25052. Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level forecasting with language models. InAdvances in Neural Information Processing Systems (NeurIPS),

work page internal anchor Pith review Pith/arXiv arXiv

[11] [11]

Toni J.B

URLhttps://iclr.cc/virtual/2025/poster/28507. Toni J.B. Liu, Nicolas Boullé, Raphaël Sarfati, and Christopher Earls. LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15097–15117, Miami, Florida, USA,

work page 2025

[12] [12]

URLhttps://aclanthology.org/2024.emnlp-main.842/

Association for Computational Linguistics. URLhttps://aclanthology.org/2024.emnlp-main.842/. Zhengzhao Ma, Xueru Wen, Boxi Cao, Yaojie Lu, Hongyu Lin, Jinglin Yang, Min He, Xianpei Han, and Le Sun. Decoupling reasoning and confidence: Resurrecting calibration in reinforcement learning from verifiable rewards,

work page 2024

[13] [13]

Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

URLhttps://arxiv.org/abs/2603.09117. Michael T. Pearce, Thomas Dooms, Ryo Yamamoto, Joshua Meehl, Carl Molnar, Mark Bissell, Dron Hazra, Ching Fang, Nam Nguyen, Michael Anderson, Collin Osborne, Patrick Duffy, Bridget Toomey, Eric Klee, Elena Myasoedova, Alexander J. Ryu, Shant Ayanian, Panos Korfiatis, Matt Redlon, Archa Jain, Daniel Balsam, and Nicholas...

work page internal anchor Pith review Pith/arXiv arXiv

[14] [14]

URLhttps://www.biorxiv.org/ content/early/2026/04/16/2026.04.10.717844

doi: 10.64898/2026.04.10.717844. URLhttps://www.biorxiv.org/ content/early/2026/04/16/2026.04.10.717844. Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Rafael Valdece Sousa Bastos, and Philip E. Tetlock. Wisdom of the silicon crowd: Llm ensemble prediction capabilities rival human crowd accuracy.Science Advances, 10(45):eadp1528,

work page doi:10.64898/2026.04.10.717844 2026

[15] [15]

URLhttps://www.science.org/doi/abs/ 10.1126/sciadv.adp1528

doi: 10.1126/sciadv.adp1528. URLhttps://www.science.org/doi/abs/ 10.1126/sciadv.adp1528. Mingtian Tan, Mike A. Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language mod- els actually useful for time series forecasting? InAdvances in Neural Information Processing Systems (NeurIPS),

work page doi:10.1126/sciadv.adp1528

[16] [16]

Qwen3 Technical Report

URLhttps://arxiv.org/abs/2505.09388. Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track,

work page internal anchor Pith review Pith/arXiv arXiv

[17] [17]

5) and are considerably longer.OpenForesightrows are from the publictest split (qids 645, 159);MATH-500from itstestsplit; AIME rows from AIME24 and AMC rows from AMC23 (the AIME25 and AMC24 splits share an identical format). Prompt / Problem Ground truth OpenForesight — forecasting(testsplit) continued on next page 20 Table 3 (continued) Prompt / Problem ...

work page 2025

[18] [18]

Your answer will be evaluated using the BRIER SCORING RULE which is basically (- (1 - p)^2) if your answer is correct and (- 1 - p^2) if your answer is incorrect

You will be rewarded based on the probability (p) you assign to your answer. Your answer will be evaluated using the BRIER SCORING RULE which is basically (- (1 - p)^2) if your answer is correct and (- 1 - p^2) if your answer is incorrect. For example, if p = 0.5, and your answer is incorrect, then your score will be (-1 - 0.5^2) = (-1 - 0.25) = -1.25 whe...

work page 2025

[19] [19]

Then the value of log2(x4y3z2) is m n wheremandnare relatively prime positive integers

14 3 AIME24 — math reasoning Letx, yandzbe positive real numbers that satisfy the following system of equations: log2 x yz = 1 2 ,log 2 y xz = 1 3 ,log 2 z xy = 1 4 . Then the value of log2(x4y3z2) is m n wheremandnare relatively prime positive integers. Findm+n.(2024-II-4) 33 continued on next page 22 Table 3 (continued) Prompt / Problem Ground truth Jen...

work page 2024

[20] [20]

Models and decode.We compare the untrained Qwen3-8B base model with ourQwen3-8B model trained according to the DCPO recipe of Ma et al

This appendix records the supporting setup. Models and decode.We compare the untrained Qwen3-8B base model with ourQwen3-8B model trained according to the DCPO recipe of Ma et al. [2026]. The decode settings are matched across cells: temperature 0.7, top-p0.8, top-k20, presence penalty1.5, maximum3,000generated tokens, logprobs enabled, no thinking mode, ...

work page 2026