Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Toward Better Generalisation in Uncertainty Estimators: Leveraging Data-Agnostic Features

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding probability and entropy features improves LLM uncertainty probes' cross-task generalization in most settings, but gains depend on the probe giving those features enough weight.

desk verdict Honest empirical map, but the central claim is not yet established because every number is a single point estimate with no variance. read the letter →

arxiv 2507.03998 v1 pith:HJWYPZDX submitted 2025-07-05 cs.AI

Thuy An Ha , Bao Quoc Vo This is my paper
classification cs.AI
keywords uncertaintyquantificationhidden-stateprobesdata-agnosticfeaturesout-of-domaingeneralisationlargelanguagemodelsfeatureselectionSHAPvalueshallucinationdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models can be confidently wrong, so a reliable estimate of whether an answer is factually correct matters. The paper asks whether a cheap, task-independent feature augmentation can make uncertainty probes—classifiers trained on the model's internal hidden states—generalize to new tasks and datasets. It trains random-forest probes on hidden states, with and without data-agnostic features (token probabilities and entropy), and measures cross-dataset accuracy, ROC AUC, and calibration. The central finding is that adding data-agnostic features improves out-of-domain generalization in most transfer settings, particularly on short-form question answering, but degrades performance in a minority of cases. The paper attributes those failures to the probe underweighting the data-agnostic features relative to hidden-state features, and shows that pruning hidden states to the most informative dimensions tends to amplify the contribution of the data-agnostic signals.

What carries the argument

The machinery is the hybrid feature set fed to a random forest regressor: the 1-layer (or 5-layer) hidden-state vector of the last answer token, plus a small set of data-agnostic features that do not depend on the dataset—for multiple-choice questions, the four softmax probabilities over A/B/C/D sorted descending and their entropy; for short-form answers, average and maximum per-token negative log-probability and average and maximum per-token entropy. The paper also uses top-300 hidden-feature selection by absolute Pearson correlation with the correctness label, and Tree SHAP values to measure how much each feature contributes to the probe's predictions. The data-agnostic features are the lever: the paper's hypothesis is that they carry task-independent signal, so when the probe assigns them high weight they improve out-of-domain transfer.

What would settle it

Re-run the short-form transfer experiments with a stated, validated Rouge-L threshold (or with human-annotated correctness labels) and compare the accuracy gaps between hidden-state-only and hybrid probes; if the gaps disappear or reverse, the central claim is an artifact of the label proxy. A second, mechanism-specific test: on a failing commonsense pair like SWAG-to-RACE under Llama2, train a probe on the data-agnostic features alone; if underweighting is the cause, this probe should generalize at least as well as the hybrid probe on that pair.

Watch

Extended reading notes

Core claim

The paper's claim is that the generalization failure of hidden-state uncertainty probes is partly a feature-competition problem, not an information problem. When a random forest probe is trained on 4,096 hidden-unit features plus a handful of data-agnostic statistics (sorted token probabilities and entropy for multiple-choice; average and maximum negative log-probability and entropy for short-form answers), the hybrid probe usually transfers better to an unseen dataset than the hidden-state-only probe: in the reported tables, 7/9 to 9/9 of transfer pairs improve, with the largest accuracy gains on short-form benchmarks. The improvement is not universal—some commonsense-reasoning pairs such as SWAG-to-RACE get worse—and the paper identifies the cause in SHAP feature-importance values: in failing cases the data-agnostic features are not ranked at the top, or are intermixed with lower-ranked hidden features, so the probe underweights them. Selecting the top-300 hidden features by absolute Pearson correlation with the labels generally amplifies the data-agnostic features' importance and yields the expected ordering of gains (selected greater than one-layer greater than five-layer) in most, but not all, transfer pairs. The paper concludes that probability- and entropy-based features significantly enhance generalization, especially on short-form QA, while noting that the probe still underperforms on commonsense transfer pairs where data-agnostic features are not prioritized.

Load-bearing premise

The load-bearing premise is that the correctness labels for short-form answers, obtained by binarizing Rouge-L similarity scores, are accurate enough to train on; the paper never states the cutoff used to turn a Rouge-L score into a 0/1 label or checks those labels against human judgment, so noisy targets could drive the measured generalization differences.

Editorial extensions

If this is right

  • If the finding holds, uncertainty probes can be made more portable without collecting additional training data from multiple tasks, since data-agnostic features are computed from the model's own outputs.
  • On short-form question answering benchmarks, the hybrid feature set yields the largest generalization gains, so this is a promising direction for open-ended generation where labels are harder to obtain.
  • Feature selection that keeps only the most label-correlated hidden dimensions amplifies the relative importance of data-agnostic features and usually, though not always, produces larger gains than using the full hidden-state vector.
  • Commonsense-reasoning transfer pairs remain the failure mode: when data-agnostic features do not rank at the top of SHAP importance, adding them can hurt, so the benefit is conditional on feature weighting rather than guaranteed.
  • The reported accuracy improvements are not calibration improvements; ECE values stay in a narrow range, so the probe's reliability ordering changes without materially changing calibration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit, testable recipe follows: before deploying a hybrid probe on a new target domain, one could inspect SHAP rankings of the data-agnostic features; if they are underweighted, reweighting them or increasing their number might recover the generalization gain the paper reports.
  • The paper leaves the Rouge-L binarization threshold unspecified; a natural extension is to sweep that threshold or replace the automatic labels with human judgments to see whether the reported generalization differences persist under cleaner targets.
  • The asymmetric transfer pattern (e.g., RACE-to-SWAG improves while SWAG-to-RACE degrades) suggests that data-agnostic features interact with the training distribution; an extension would be to test whether the direction of the asymmetry flips when the source and target datasets are swapped systematically across all three tasks.
  • Because the mechanism is feature weighting rather than new data, one could try to make the probe explicitly regularize hidden-state features (e.g., by penalizing their variance) to force the data-agnostic features to carry more weight, which the paper does not attempt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether supplementing hidden-state features with probability- and entropy-based "data-agnostic" features improves the cross-dataset generalization of supervised LLM uncertainty probes. It trains random forest regressors on hidden states from Llama2-7B and Mistral-7B across three tasks (multiple-choice QA and short-form QA), and compares hidden-state-only probes against hybrid feature sets, including a top-k feature-selection variant and a 5-layer variant. Across Tables 5-8 the paper reports that the hybrid features improve accuracy on most, but not all, transfer pairs, and a SHAP analysis suggests that data-agnostic features receive high feature importance in successful cases. The authors openly acknowledge limitations such as baseline drift, high misclassification rates, and the arbitrary choice of data-agnostic features, and they release code and complementary materials.

Significance. If the central empirical claim held, the paper would provide a low-cost, practical recipe for improving the cross-task generalization of hidden-state uncertainty probes, which is a recognized limitation in the literature. The paper has clear strengths: it releases code, reports both accuracy and ROC AUC, provides a feature-contribution analysis with SHAP, and explicitly acknowledges several limitations that other papers often omit. However, the headline claim rests on single point estimates without any measure of variability, and the labeling pipeline for short-form datasets is underspecified, so the significance of the reported effects cannot currently be assessed.

major comments (3)
  1. [§3.6, §3.7, Tables 5-8, §6] The central claim that adding data-agnostic features 'significantly enhances' generalization is based on single point estimates with no error bars, confidence intervals, repeated seeds, or significance tests. The RandomForestRegressor is stochastic unless a random state is fixed, and the manuscript does not state whether random_state was set or how the train/test split was chosen. Many reported deltas are tiny, e.g., SWAG–MMLU with Llama2-7B in Table 5 changes from 0.5255 to 0.5270 (+0.0015), and MMLU–SWAG changes by 0.0000; these are counted in the 'improved' or 'not degraded' totals. The word 'significantly' in the Conclusion therefore has no statistical backing. I would require repeated runs over multiple seeds, reporting of mean and variance (or confidence intervals), and ideally paired significance tests over transfer pairs before the qualitative 'generally improves in most cases' conclusion can be accepted.
  2. [§3.4, §3.8] The binary labels for the short-form datasets (TriviaQA, SQuAD, Winogrande) are derived from Rouge-L, but the threshold used to convert a Rouge-L similarity score into a 0/1 label is never stated. Section 3.4 says only 'the results of Rouge-L will be the labels,' and Section 3.8 refers to a 'predefined threshold' without distinguishing the 0.5 accuracy threshold from the Rouge-L binarization threshold. Since these labels are used both to train the probes and to compute all reported metrics, an arbitrary or ill-documented threshold could materially change every number in Tables 7-8 and the conclusions drawn from them. The authors should specify the exact binarization procedure and, ideally, validate the proxy against human judgments or at least a labeled subset.
  3. [§3.9] The summary statistics in the text are inconsistent with the reported tables. The text says '7 improved out of 9 for Llama2 – 7B on multiple choice question answering datasets; 8 improved out of 9 for Mistral – 7B on short-form question answering datasets; 9 improved out of 9 for Llama2 – 7B on multiple choice question answering datasets; and 9 improved out of 9 for Mistral – 7B on short-form question answering datasets.' However, Table 8 (Mistral-7B short-form) shows all 9 transfer pairs improving, not 8, and Table 6 (Mistral-7B multiple-choice) shows 8 improvements, which appears to be the intended '8 out of 9' entry. As written, the reader cannot verify the counts that support the abstract's 'generally enhances in most cases' claim; this needs correction and verification.
minor comments (5)
  1. [§3.9] The notation paragraph says a probe trained on dataset A and tested on dataset B is denoted 'B-A', but the table headers use the reverse form (e.g., 'MMLU–RACE' in Table 1), which is confusing. The direction convention should be stated unambiguously and used consistently.
  2. [Table 16] The rows in Table 16 appear not to be sorted in descending order as in the other SHAP tables: feature 1924 has mean SHAP 0.000260, feature 1233 has 0.000092, and feature 2172 has 0.000894, which breaks the monotonic ordering. Please check the table construction.
  3. [§7] The first bullet in the Limitation section says the authors 'randomly select the data-agnostic features,' which is misleading because the features are actually predetermined by the method (probabilities and entropies). I suggest rewording to 'arbitrarily selected' or 'chosen without an ablation.'
  4. [§3.3] The citation 'Liu et al. (2024)' in the sentence about using the last token should be 'Liu et al. (2024b)' to disambiguate from Liu et al. (2024a) in the reference list.
  5. [§4.2] There is a typo in the subsection heading: 'Compreshension' should be 'Comprehension.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the generalization comparison is an empirical test of independently defined feature sets.

full rationale

This paper makes an empirical claim about generalization, not a derivation that reduces to its inputs. The pipeline defines D=(theta_i,c_i) and D'=(theta_i,{Delta_i},c_i); the data-agnostic features are computed from LLM output logits, token probabilities, and entropies, while labels are obtained by comparison to ground truth or by Rouge-L. Probes are trained on one dataset's training split and evaluated on another dataset's test split, so the reported accuracy is an out-of-sample measurement of a feature set, not a fitted parameter renamed as a prediction. The top-300 feature selection using absolute Pearson correlation with labels is performed on the training split and is applied to both the baseline and the hybrid condition; this is standard supervised feature selection and does not make the comparison circular, because the quantity being compared (cross-dataset generalization) is not defined in terms of the selected features or the probe's own outputs. The paper relies on prior work for its feature definitions (He et al. 2024; Manakul et al. 2023; Liu et al. 2024b), but the authors of the present paper are not the authors of those references, so there is no self-citation chain bearing the argument. Section 7's limitations--baseline drift across configurations, relatively high misclassification rates, and underweighting of data-agnostic features on Commonsense Reasoning--are threats to robustness and interpretation, not circularity. The unspecified Rouge-L threshold in Section 3.4 is a label-validity concern, not a definitional equivalence between input and output. No step was found in which a claimed prediction is equivalent by construction to its inputs, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

All numbers listed are design choices or thresholds that the experimental results depend on but that are not fitted with uncertainty. The axioms are background assumptions inherited from prior work or introduced without validation; none are mathematically derived.

free parameters (5)
  • top-k for feature selection = 300
    Chosen by hand to reduce the 4096 hidden features; results vary with k and no sensitivity analysis is provided.
  • accuracy threshold = 0.5
    Used to binarize predicted confidence scores into correct/incorrect; not tuned, though prior work uses it.
  • hidden layer selection = layer 15 (0-indexed) for 1-layer; layers 13-17 for 5-layer
    Chosen based on prior work; the choice affects which hidden states are used and therefore the results.
  • random forest hyperparameters = unspecified (likely scikit-learn defaults)
    The paper does not report hyperparameters, so results may depend on default settings.
  • Rouge-L binarization threshold = not stated
    Short-form labels are produced by binarizing Rouge-L scores, but the threshold is omitted, leaving reproducibility incomplete.
assumptions (5)
  • domain assumption Hidden states of LLMs encode information correlated with the truthfulness of responses.
    Adopted from Azaria and Mitchell (2023), Liu et al. (2024b), etc.; the paper relies on this to justify hidden-state features.
  • domain assumption Data-agnostic probability and entropy features are task-independent and can improve generalization.
    Adopted from He et al. (2024); the paper's RQ1 assumes this, though results are mixed.
  • domain assumption Pearson correlation between features and labels in the training split identifies the most informative hidden features.
    Used in Section 3.7 to select top-300 features; no justification beyond correlation ranking.
  • domain assumption Rouge-L scores can serve as valid binary correctness labels for short-form answers.
    Section 3.4 uses Rouge-L without threshold specification or validation, which is load-bearing for the short-form experiments.
  • standard math SHAP values computed on a random forest give a faithful estimate of feature contribution.
    Relies on TreeSHAP (Lundberg and Lee, 2017) for the importance analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Better Generalisation in Uncertainty Estimators: Leveraging Data-Agnostic Features." pith.science (2026). https://pith.science/paper/HJWYPZDX

@misc{pith2026250703998,
  author       = {Pith},
  title        = {Pith review of: Toward Better Generalisation in Uncertainty Estimators: Leveraging Data-Agnostic Features},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJWYPZDX}},
  note         = {Machine review of arXiv:2507.03998}
}
read the original abstract

Large Language Models (LLMs) often generate responses that are factually incorrect yet expressed with high confidence, which can pose serious risks for end users. To address this, it is essential for LLMs not only to produce answers but also to provide accurate estimates of their correctness. Uncertainty quantification methods have been introduced to assess the quality of LLM outputs, with factual accuracy being a key aspect of that quality. Among these methods, those that leverage hidden states to train probes have shown particular promise, as these internal representations encode information relevant to the factuality of responses, making this approach the focus of this paper. However, the probe trained on the hidden states of one dataset often struggles to generalise to another dataset of a different task or domain. To address this limitation, we explore combining data-agnostic features with hidden-state features and assess whether this hybrid feature set enhances out-of-domain performance. We further examine whether selecting only the most informative hidden-state features, thereby discarding task-specific noise, enables the data-agnostic features to contribute more effectively. The experiment results indicate that although introducing data-agnostic features generally enhances generalisation performance in most cases, in certain scenarios their inclusion degrades performance. A similar pattern emerges when retaining only the most important hidden-state features - adding data-agnostic features does not consistently further enhance performance compared to using the full set of hidden-state features. A closer analysis reveals that, in some specific cases, the trained probe underweights the data-agnostic features relative to the hidden-state features, which we believe is the main reason why the results are inconclusive.

Figures

Figures reproduced from arXiv: 2507.03998 by the authors.

Figure 1
Figure 1. Accuracy gap between not including data-agnostic features and including data-agnostic [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Accuracy gap between not including data-agnostic features and including data-agnostic [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Accuracy gap between not including data-agnostic features and including data-agnostic [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Accuracy gap between not including data-agnostic features and including data-agnostic [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: PCA of the hidden states for multiple choice question answering (left) and short-form [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: PCA of the hidden states for multiple choice question answering (left) and short-form [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 9 canonical work pages

  1. [1]

    L., McGrew, B., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., McGrew, B., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    and Mitchell, T

    Azaria, A. and Mitchell, T. (2023). The internal state of an llm knows when it’s lying. arXiv preprint arXiv:2304.13734

    arXiv 2023

  3. [3]

    Beigi, M., Shen, Y., Yang, R., Lin, Z., Wang, Q., Mohan, A., Huang, L., et al. (2024). Internalinspector \(i^2\): Robust confidence estimation in llms through internal states. arXiv preprint arXiv:2406.12053

    work page Pith review arXiv 2024

  4. [4]

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Fiedel, N., et al. (2023). Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1--113

    work page 2023

  5. [5]

    Duan, H., Yang, Y., and Tam, K. Y. (2024). Do llms know about hallucination? an empirical investigation of llm’s hidden states. arXiv preprint arXiv:2402.09733

    arXiv 2024

  6. [6]

    Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. (2023). Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. arXiv preprint arXiv:2307.01379

    arXiv 2023

  7. [7]

    Farquhar, S., Kossen, J., Kuhn, L., and Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature , 630(8017):625--630

    2024

  8. [8]

    He, J., Gong, Y., Lin, Z., Zhao, Y., and Chen, K. (2024). Llm factoscope: Uncovering llms’ factual discernment through measuring inner states. In Findings of the Association for Computational Linguistics (ACL) , pages 10218--10230

    work page 2024

Show all 35 references
  1. [9]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  2. [10]

    Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Liu, T., et al. (2023a). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232

    2023 arXiv

  3. [11]

    Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., and Ma, L. (2023b). Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236

    2023 arXiv

  4. [12]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D., Bressand, F., et al. (2023). Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  5. [13]

    S., and Zettlemoyer, L

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

    2017 arXiv

  6. [14]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  7. [15]

    Kossen, J., Han, J., Razzak, M., Schut, L., Malik, S., and Gal, Y. (2024). Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927

    2024 arXiv

  8. [16]

    Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. (2017). Race: Large‐scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683

    2017 arXiv

  9. [17]

    Lin, C. (2004). Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74--81

    2004

  10. [18]

    Lin, S., Hilton, J., and Evans, O. (2022). Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334

    2022 arXiv

  11. [19]

    Liu, J., Chen, S., Cheng, Y., and He, J. (2024a). On the universal truthfulness hyperplane inside llms. arXiv preprint arXiv:2407.08582

    2024 arXiv

  12. [20]

    Liu, L., Pan, Y., Li, X., and Chen, G. (2024b). Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint arXiv:2404.15993

    2024 arXiv

  13. [21]

    Lundberg, S. M. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems , volume 30

    2017

  14. [22]

    Mahaut, M., Aina, L., Czarnowska, P., Hardalov, M., Müller, T., and Màrquez, L. (2024). Factual confidence of llms: On reliability and robustness of current estimators. arXiv preprint arXiv:2406.13415

    2024 arXiv

  15. [23]

    Manakul, P., Liusie, A., and Gales, M. J. F. (2023). Selfcheckgpt: Zero‐resource black‐box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896

    2023 arXiv

  16. [24]

    Ni, S., Bi, K., Yu, L., and Guo, J. (2024). Are large language models more honest in their probabilistic or verbalized confidence? In China Conference on Information Retrieval , pages 124--135. Springer Nature Singapore

    2024

  17. [25]

    Orgad, H., Toker, M., Gekhman, Z., Reichart, R., Szpektor, I., Kotek, H., and Belinkov, Y. (2024). Llms know more than they show: On the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707

    2024 arXiv

  18. [26]

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250

    2016 arXiv

  19. [27]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. (2021). Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99--106

    2021

  20. [28]

    S., and Gerli, A

    Salvagno, M., Taccone, F. S., and Gerli, A. G. (2023). Artificial intelligence hallucinations. Critical Care , 27:180

    2023

  21. [29]

    Z., and Majumdar, A

    Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., and Majumdar, A. (2024). A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. arXiv preprint arXiv:2412.05563

    2024 arXiv

  22. [30]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Scialom, T., et al. (2023). Llama 2: Open foundation and fine‐tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  23. [31]

    Ulmer, D., Gubri, M., Lee, H., Yun, S., and Oh, S. (2024). Calibrating large language models using their generations only. arXiv preprint arXiv:2403.05973

    2024 arXiv

  24. [32]

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., and Zhou, D. (2022). Self‐consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  25. [33]

    Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2023). Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063

    2023 arXiv

  26. [34]

    Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. (2018). Swag: A large‐scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326

    2018 arXiv

  27. [35]

    Zhang, F., Yu, P., Yi, B., Zhang, B., Li, T., and Liu, Z. (2024). Prompt‐guided internal states for hallucination detection of large language models. arXiv preprint arXiv:2411.04847

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.