Pith. sign in

REVIEW 4 major objections 6 minor 50 references

Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that samples an LLM already knows are what ruin calibration during fine-tuning, and shows that applying a confidence penalty selectively to those samples cuts calibration error by over half without hurting accuracy.

desk verdict A useful empirical recipe for better fine-tuning calibration, but the headline causal claim about prior knowledge is not actually proven by the experiments. read the letter →

arxiv 2505.20903 v1 pith:7BZQ53QR submitted 2025-05-27 cs.CL

Ziming Wang , Zeyu Shi , Haoyi Zhou , Shiqi Gao , Qingyun Sun , Jianxin Li This is my paper
classification cs.CL
keywords LLMcalibrationfine-tuningpriorknowledgeoverconfidencebiasexpectederrorknownversusunknowndataCogCalib
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the model's own prior knowledge, not the fine-tuning process itself, is the main reason fine-tuned LLMs end up poorly calibrated. Samples the base model can already answer correctly ('known data') make confidence grow faster than accuracy during training, producing overconfidence, whereas genuinely new knowledge ('unknown data') keeps the two aligned. On that basis the paper proposes CogCalib, which scores each training sample's negative log-likelihood, flags the known ones with an adaptive threshold, and applies a confidence-penalizing regularization term to exactly those samples while leaving new knowledge to be learned with plain cross-entropy. Across seven QA tasks, four instruction-tuned models, and both LoRA and full fine-tuning, the method reports expected calibration error reductions averaging 57% on Llama3-8B with accuracy maintained, including on out-of-distribution tests. If the claim holds, calibration for fine-tuned LLMs is a data-overlap problem fixable during training rather than a post-hoc rescaling problem.

What carries the argument

The load-bearing mechanism is knowledge bias, the gap between what the base model already knows and what the fine-tuning data teaches, and a gating rule that acts on it. Offline, data is labelled known or unknown with the SliCK procedure: 10 different 4-shot prompt templates and 16 sampling passes at temperatures 0 and 0.5, with samples the base model always answers correctly counted as known. During training, CogCalib replaces this expensive procedure with the sample's negative log-likelihood $L_{\mathrm{NLL}} = -\sum_k q_k \log p_k$, comparing it to an adaptively updated threshold $t$ (grid-searched on a held-out calibration set each epoch to maximize true-positive plus true-negative rates). The binary indicator $I(p,q) = 1$ for known samples gates the loss $L = L_{\mathrm{CE}} + I(p,q)\cdot\alpha L_{\mathrm{cal}}$, where $L_{\mathrm{cal}}$ is label smoothing, margin-based label smoothing, or ECP. A brief style-adaptation phase before training makes NLL values comparable across the model's output style and the task's label format. The gating identity is what carries the argument: it routes each sample into one of two learning regimes, confidence moderation for already-known material and full task learning for new material.

What would settle it

Construct two matched fine-tuning sets: 'known' samples the base model answers correctly, and 'unknown' samples it cannot answer but that are equally learnable, so both reach the same accuracy in the same small number of training steps. The paper's claim predicts the accuracy-confidence gap still diverges for the known set while staying synchronized for the difficult-but-learnable set; if both sets show the same divergence, the effect tracks difficulty rather than prior knowledge.

Watch

Extended reading notes

Core claim

The paper's central claim is stated plainly: data aligned with the model's prior knowledge tend to cause overconfidence, while data exhibiting knowledge bias contribute to better calibration. Known data are samples the base model already answers correctly; unknown data are samples it cannot answer before fine-tuning. During fine-tuning on known data, test accuracy plateaus early (around step 200 in the paper's tracking) while confidence keeps climbing, widening the calibration gap; on unknown data, accuracy and confidence rise in step, and the model develops separated confidence levels for correct and incorrect predictions, which also improves out-of-distribution detection (AUROC 0.85 vs 0.77 at step 600). The effect is directional in the known/unknown ratio, appears across every model family tested, and persists under full fine-tuning. The paper further shows that simply deleting known data is not a reliable cure, because it improves calibration on ARC-C but worsens it on OBQA, motivating the selective-regularization design of CogCalib.

Load-bearing premise

The whole argument depends on identifying 'known data' with samples the base model answers correctly before fine-tuning; if that correctness reflects question difficulty, lucky guessing, or surface wording patterns rather than stored knowledge, the claimed causal link between prior knowledge and overconfidence is confounded.

Editorial extensions

If this is right

  • Calibration can be improved during fine-tuning itself: CogCalib introduces no extra parameters and no added cost at inference, unlike post-hoc methods such as temperature scaling or ensembles.
  • Because the mechanism is knowledge overlap, calibration pressure grows as base models become more capable: a larger prior means more of any domain corpus counts as 'known' and therefore confidence-inflating.
  • The selective treatment is essential to the result: applying the calibration loss to all data, or to randomly chosen data, degrades downstream accuracy, because unknown data are the samples that actually align the model with the task.
  • The benefits transfer out of distribution: models fine-tuned on OBQA with CoLS, CoMbLS, or CoECP show lower ECE on ARC-C, ARC-E, and MMLU subjects than vanilla SFT, temperature scaling, MC-dropout, and deep ensembles.
  • The framework is agnostic to the calibration term and the tuning method: consistent ECE reductions appear with label smoothing, margin-based label smoothing, and ECP, under both LoRA and full-parameter fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: The paper's NLL-based known/unknown signal could be fed back into data selection or curriculum design, not just loss design, for example by ordering known samples early and interleaving novel knowledge; the paper only explores this direction destructively by deleting known data.
  • Editorial: The paper states its scope as models up to 13B parameters; if the mechanism strengthens with prior knowledge, larger base models should show an even larger 'known' fraction in any domain corpus and hence a stronger version of the effect, but that scaling is untested.
  • Editorial: The causal story would be sharpened by dissociating 'known' from 'easy,' since zero-shot correctness and sample difficulty are entangled in the SliCK operationalization; a matched-difficulty comparison would settle whether stored knowledge or general easiness drives the confidence inflation.
  • Editorial: The same gating logic may apply beyond supervised fine-tuning, to preference optimization or continual learning, where overlap with prior knowledge also dominates the training signal; the paper does not test those settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies how an LLM's prior knowledge affects confidence calibration during supervised fine-tuning. It defines 'known' data as examples the base model already answers correctly under the SliCK procedure and 'unknown' data as examples it never answers correctly. Through ratio sweeps and learning-dynamics plots, the authors conclude that known data induce overconfidence while unknown data improve calibration. They then propose CogCalib, which uses an adaptive NLL threshold to detect known examples during training and applies a calibration regularizer (label smoothing, margin-based label smoothing, or entropy penalization) only to those examples. Experiments across seven QA tasks and four models report large ECE reductions relative to vanilla SFT, temperature scaling, MC-dropout, and deep ensembles, with ablations against vanilla and random calibration supporting the gating mechanism.

Significance. If the central causal claim survives scrutiny, the paper identifies a practically important and previously underemphasized driver of miscalibration in fine-tuned LLMs: the overlap between fine-tuning data and the model's pretraining knowledge. The proposed CogCalib framework is attractive because it adds no inference-time cost, is compatible with multiple training-based calibration losses, and is evaluated across model families and task formats. The paper also ships extensive appendices with per-model and per-dataset results, and the ablation against vanilla and random calibration is a useful control. The main risk is that the known/unknown dichotomy is operationalized through zero-shot correctness, which may conflate 'prior knowledge' with 'initial solvability' or 'example difficulty.' This conflation is load-bearing because the paper's mechanistic claims and the design of CogCalib both rest on that dichotomy.

major comments (4)
  1. [§3.1 and Appendix G.1] The central causal claim—that data aligned with the model's prior knowledge cause overconfidence—is not established because the known/unknown split is defined by zero-shot correctness. Under SliCK (Table 21), Known examples satisfy P(T=0)=1 and Unknown examples satisfy P(T≥0)=0, so the two sets differ by construction in initial accuracy. The learning-dynamics evidence in Figure 3a (accuracy plateau with rising confidence for known data; synchronized accuracy and confidence for unknown data) is exactly what one would expect for any easy-versus-hard split, regardless of whether the ease comes from stored knowledge or from surface cues such as question format or lexical overlap. To support the mechanistic interpretation, the authors should control for difficulty, for example by matching Known and Unknown subsets on base-model accuracy under a different prompt format, or by measuring prior knowledge with a held-out knowledge probe that is not the same task used for fine-tuning.
  2. [§4.1, Algorithm 1] The online knowledge-bias evaluator uses correctness as the label for threshold selection. In Algorithm 1, ci=1 if the model answers correctly and the threshold is chosen to maximize TPR+TNR, where the 'true' labels are these correctness indicators. This means the gating signal is 'currently answerable,' not 'in the model's prior knowledge' as established by SliCK. Table 1 validates the NLL-based classifier against SliCK only in aggregate accuracy, TPR, and TNR; the actual training-time algorithm never consults SliCK. The authors should clarify whether the calibration-set labels are SliCK labels or correctness labels, and if the latter, justify why the method is not simply 'regularize easy examples.'
  3. [Appendix A] The calibration set used for threshold updates is described as 'randomly selected from the training set.' Because the threshold t is tuned on samples that are also used for model training, the reported CogCalib gains may be optimistic and the chosen threshold may not transfer to held-out data. A held-out calibration set, or a cross-validation scheme that excludes calibration samples from the training update, should be used to ensure the threshold-selection procedure is not overfitting the training distribution.
  4. [Tables 2, 3, 7–16] The main results report only three-seed averages without standard deviations or significance tests. The only seed-level variation reported is in Appendix Table 5, which covers confidence values for Figure 3b but not the ECE or accuracy numbers in the main tables. Given that the paper claims 'significant' calibration improvements and that some ECE gaps between CogCalib and baselines are large, the authors should report standard deviations or confidence intervals for the primary ECE and accuracy results, or at least for the headline Llama3-8B results in Tables 2 and 3.
minor comments (6)
  1. [Abstract and §1] The abstract states an 'average 57% reduction in ECE,' while the introduction says '55.92%' (compared to TS) and '65.02%' (compared to SFT) for Llama3-8B on multiple-choice tasks. These numbers should be reconciled, and the aggregation method (per-dataset average percentage reduction versus pooled reduction) should be stated explicitly.
  2. [Figure 3b] The AUROC values 0.85 and 0.77 are reported without error bars. Appendix Table 5 provides standard deviations only for confidence values, not for the AUROC metric, so the reader cannot assess the stability of the OOD-detection claim.
  3. [Appendix F.7] The heading 'Comparision to Vanilla and Random Calibration' contains a typo; it should read 'Comparison.'
  4. [Table 5] The abbreviations 'CS' and 'IS' in Table 5 are undefined; the table caption should explain what these terms denote.
  5. [§5.1 and Appendix G.3] The open-ended and multiple-choice tasks use different calibration-loss hyperparameters (Tables 22 and 23), but the sensitivity analysis in Appendix F.9 covers only ARC-C and WG-S. Reporting sensitivity for at least one open-ended task, such as HotpotQA or MedMCQA, would strengthen the claim of robustness.
  6. [§5.2.1] Table 1 reports NLL-based known/unknown classification accuracy for Llama3-8B only. Since the method is claimed to be model-agnostic, reporting the same validation for at least one other model would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: known/unknown labels are externally defined and the calibration improvements are empirical.

full rationale

The paper's derivation chain is not circular. Section 3 adopts the external SliCK taxonomy (Gekhman et al., 2024) to label data as known/unknown, where the labels are defined by base-model greedy correctness (Appendix G.1), not by the calibration outcome being predicted. The NLL-based knowledge-bias evaluator in Section 4.1 is a proxy whose accuracy is validated against SliCK labels (Table 1), so it is not a fitted parameter relabeled as a prediction. The threshold t is a practical hyperparameter tuned on a held-out calibration set and does not presuppose the ECE conclusions. CogCalib's loss (Eq. 3) applies established calibration regularizers (LS/MbLS/ECP) to the known subset; its ECE improvements are empirical results, not identities. The main methodological caveat—that 'known' may be confounded with initial solvability or example difficulty—is a construct-validity concern, not a reduction of the claim to its inputs. There are no load-bearing self-citations (SliCK is external), and no uniqueness theorem is imported from the authors' own prior work. Therefore no significant circularity is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are introduced. The known/unknown categories are operational labels for existing data, and the free parameters are the regularization strengths and the NLL threshold.

free parameters (5)
  • NLL threshold t = Updated each epoch by grid search over calibration set, maximizing TPR+TNR (Algorithm 1)
    Determines which samples receive the calibration loss; a fitted value per epoch, not derived from theory.
  • Grid size M for threshold search = Not specified in the paper
    Number of candidate thresholds in Algorithm 1's linspace; undocumented, affecting threshold precision.
  • Label smoothing epsilon (LS) = 0.1 for multiple-choice, 0.15 for open-ended (Tables 22-23)
    Hand-set regularization strength for the calibration loss.
  • MbLS gamma and margin = gamma 0.1/0.15, margin 0/10
    Hand-set parameters for margin-based label smoothing.
  • ECP beta = 0.1 for multiple-choice, 0.15 for open-ended
    Hand-set strength for the entropy penalty calibration loss.
assumptions (4)
  • domain assumption SliCK's zero-shot correctness classification is a valid measure of an LLM's prior knowledge (Appendix G.1).
    The entire known/unknown distinction in Section 3 relies on this. The paper validates NLL against it, but treats SliCK itself as ground truth.
  • domain assumption Negative log-likelihood (NLL) is a reliable real-time proxy for the SliCK categories during fine-tuning (Section 4.1).
    Thresholding NLL is the core of CogCalib; the paper reports high agreement (Table 1) but this is an empirical, not formal, equivalence.
  • domain assumption The calibration set is representative of the training distribution and remains so as the model updates (Algorithm 1).
    Threshold t is re-tuned on a fixed calibration set each epoch; if the set is small or unrepresentative, the gate misclassifies known/unknown data.
  • domain assumption Applying calibration loss only to known data preserves learning of unknown data (Section 5.3, Appendix F.6).
    The method's design depends on this; the ablation supports it empirically, but it is not guaranteed for other datasets or model scales.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?." pith.science (2026). https://pith.science/paper/7BZQ53QR

@misc{pith2026250520903,
  author       = {Pith},
  title        = {Pith review of: Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BZQ53QR}},
  note         = {Machine review of arXiv:2505.20903}
}
read the original abstract

Fine-tuned Large Language Models (LLMs) often demonstrate poor calibration, with their confidence scores misaligned with actual performance. While calibration has been extensively studied in models trained from scratch, the impact of LLMs' prior knowledge on calibration during fine-tuning remains understudied. Our research reveals that LLMs' prior knowledge causes potential poor calibration due to the ubiquitous presence of known data in real-world fine-tuning, which appears harmful for calibration. Specifically, data aligned with LLMs' prior knowledge would induce overconfidence, while new knowledge improves calibration. Our findings expose a tension: LLMs' encyclopedic knowledge, while enabling task versatility, undermines calibration through unavoidable knowledge overlaps. To address this, we propose CogCalib, a cognition-aware framework that applies targeted learning strategies according to the model's prior knowledge. Experiments across 7 tasks using 3 LLM families prove that CogCalib significantly improves calibration while maintaining performance, achieving an average 57\% reduction in ECE compared to standard fine-tuning in Llama3-8B. These improvements generalize well to out-of-domain tasks, enhancing the objectivity and reliability of domain-specific LLMs, and making them more trustworthy for critical human-AI interaction applications.

Figures

Figures reproduced from arXiv: 2505.20903 by the authors.

Figure 1
Figure 1. LLMs’ prior knowledge leads to poor calibra [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Accuracy and ECE of Llama3-8B fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) Accuracy and confidence of Llama3-8B [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 5
Figure 5. Figure 5: CogCalib’s framework. CogCalib dynami￾cally assesses knowledge bias during training through NLL, employing customized learning strategies with distinct loss functions to enhance calibration. Addition￾ally, CogCalib incorporates a style adaptation process to improve the…
Figure 6
Figure 6. Figure 6: Comparison of our method’s performance against baseline approaches on OOD datasets is presented. The [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparison of CogCalib against baselines [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Sensitivity to Hyperparameters. We adjust [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Accuracy and ECE of Llama3-8B fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Accuracy and ECE of Llama3-8B fine-tuned with different knowledge biases in the open-ended dataset [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: ECE of Llama3-8B fine-tuned with dif￾ferent knowledge biases in MathQA using Full Fine￾Tuning (FFT). The ratio varies from 5:0 to 0:5 (unknown data:known data), with equal dataset sizes. Calibration deteriorates as the knowledge bias lowers, while higher knowledge bia…
Figure 16
Figure 16. Figure 16: ECE of Llama3-8B after fine-tuning on unknown (high-bias), mixed, and known (low-bias) datasets, where the mixed dataset is randomly sampled from OBQA with an equal size. ECE curve for models fine-tuned on the mixed dataset maintains an intermedi￾ate position, indicat…
Figure 14
Figure 14. Figure 14: ECE of Qwen2.5-7B fine-tuned with differ [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: ECE of Mistral-7B fine-tuned with different [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 17
Figure 17. Figure 17: Reliability diagrams of models fine-tuned on OBQA known data or unknown data, evaluated on both [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Comparison of accuracy on downstream tasks with and without calibration methods when fine-tuning on [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Comparsion between CogECP, vanilla ECP, and dynamic random ECP. Our proposed method, Co￾ECP, consistently outperforms all baseline methods in terms of both calibration and accuracy across multiple datasets. Dataset Metrics ECP factor 0.05 0.075 0.1 0.125 ARC-C ACC 82.…
Figure 20
Figure 20. Figure 20: Sensitivity to Threshold Computation. Our [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 20 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. 2024. A survey on data selection for language models. arXiv preprint arXiv:2402.16827

    arXiv 2024

  3. [3]

    Christoph Berger, Magdalini Paschali, Ben Glocker, and Konstantinos Kamnitsas. 2021. Confidence-based out-of-distribution detection: A comparative study and analysis. In Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis, pages 122--132, Cham. Springer International Publishing

    work page 2021

  4. [4]

    Jochen Br \"o cker and Leonard A Smith. 2007. Increasing the reliability of reliability diagrams. Weather and forecasting, 22(3):651--661

    work page 2007

  5. [5]

    Dingshuo Chen, Zhixun Li, Yuyan Ni, Guibin Zhang, Ding Wang, Qiang Liu, Shu Wu, Jeffrey Xu Yu, and Liang Wang. 2024. Beyond efficiency: Molecular data pruning for enhanced generalization. arXiv preprint arXiv:2409.01081

    work page Pith review arXiv 2024

  6. [6]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044

    arXiv 2019

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  8. [8]

    Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050--1059. PMLR

    work page 2016

Show all 50 references
  1. [9]

    Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. 2023. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513--1589

    2023

  2. [10]

    Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904

    2024 arXiv

  3. [11]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017 a . On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321--1330. PMLR

    2017

  4. [12]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017 b . On calibration of modern neural networks. In International conference on machine learning, pages 1321--1330. PMLR

    2017

  5. [13]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964

    2020 arXiv

  6. [14]

    Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. 2021. https://doi.org/10.18653/v1/2021.eacl-main.95 Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models . In Proceedings of the 16th Conference of...

    2021 doi

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  8. [16]

    Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.01244

    2024 arXiv

  9. [17]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. S...

    2025

  10. [18]

    Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398

    2023 arXiv

  11. [19]

    Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng. 2023. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks. arXiv preprint arXiv:2311.00288

    2023 arXiv

  12. [20]

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2016. Simple and scalable predictive uncertainty estimation using deep ensembles. Cornell University - arXiv,Cornell University - arXiv

    2016

  13. [21]

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30

    2017

  14. [22]

    Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. 2022. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 80--88

    2022

  15. [23]

    Xin Liu, Muhammad Khalifa, and Lu Wang. 2023. Litcab: Lightweight calibration of language models on outputs of varied lengths. arXiv preprint arXiv:2310.19208

    2023 arXiv

  16. [24]

    Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Vardaan Pahuja, Tanya Berger-Wolf, Song Gao, Charles Stewart, Yu Su, et al. 2024. Fine-tuning is fine, if calibrated. arXiv preprint arXiv:2409.16223

    2024 arXiv

  17. [25]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP

    2018

  18. [26]

    Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. 2020. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15288--15299

    2020

  19. [27]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Lear...

    2022

  20. [28]

    Gabriel Pereyra, George Tucker, Jan Chorowski, ukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548

    2017 arXiv

  21. [29]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  22. [30]

    Sarah Sarabadani. 2019. Detection of adverse drug reaction mentions in tweets using elmo. In Proceedings of the Fourth Social Media Mining for Health Applications (\# SMM4H) Workshop & Shared Task, pages 120--122

    2019

  23. [31]

    Nabeel Seedat, Nicolas Huynh, Boris van Breugel, and Mihaela van der Schaar. 2023. Curated llm: Synergy of llms and data curation for tabular augmentation in ultra low-data regimes. arXiv preprint arXiv:2312.12112

    2023 arXiv

  24. [32]

    Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. 2024. Thermometer: Towards universal calibration for large language models. arXiv preprint arXiv:2403.08819

    2024 arXiv

  25. [33]

    Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. 2024. Continual learning of large language models: A comprehensive survey. arXiv preprint arXiv:2404.16789

    2024 arXiv

  26. [34]

    Smith, and Yejin Choi

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. https://aclanthology.org/2020.emnlp-main.746/ Dataset cartography: Mapping and diagnosing datasets with training dynamics . In Proceedings of the 2020 Con...

    2020

  27. [35]

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818--2826

    2016

  28. [36]

    Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. 2022. Mitigating neural network overconfidence with logit normalization. In International conference on machine learning, pages 23631--23644. PMLR

    2022

  29. [37]

    Pijing Wei, Qianqian Wang, Zhen Gao, Ruifen Cao, and Chunhou Zheng. 2024. Dmfvae: mirna-disease associations prediction based on deep matrix factorization method with variational autoencoder. Frontiers of Computer Science, 18(6):186912

    2024

  30. [38]

    Zishan Xu, Linlin Song, Shichao Liu, and Wen Zhang. 2024. Deepcrbp: improved predicting function of circrna-rbp binding sites with deep feature learning. Frontiers of Computer Science, 18(2):182907

    2024

  31. [39]

    Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. 2023. Bayesian low-rank adaptation for large language models. arXiv preprint arXiv:2308.13111

    2023 arXiv

  32. [40]

    Bufang Yang, Siyang Jiang, Lilin Xu, Kaiwei Liu, Hai Li, Guoliang Xing, Hongkai Chen, Xiaofan Jiang, and Zhenyu Yan. 2024 a . Drhouse: An llm-empowered diagnostic reasoning system through harnessing outcomes from sensor data and expert knowledge. Proceedings of the ACM on Inte...

    2024

  33. [41]

    Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.58 Self-distillation bridges distribution gap in language model fine-tuning . In Proceedings of the 62nd Annual Meeting of the Association ...

    2024 doi

  34. [42]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA : A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing ( EMNLP )

    2018

  35. [43]

    Qiang Zhang, Juan Liu, Wen Zhang, Feng Yang, Zhihui Yang, and Xiaolei Zhang. 2024. A multi-stream network for retrosynthesis prediction. Frontiers of Computer Science, 18(2):182906

    2024

  36. [44]

    Xiao Zhang and Ji Wu. 2024. https://openreview.net/forum?id=tmsqb6WpLz Dissecting learning and forgetting in language model finetuning . In The Twelfth International Conference on Learning Representations

    2024

  37. [45]

    Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. 2023. On the calibration of large language models and alignment. arXiv preprint arXiv:2311.13240

    2023 arXiv

  38. [46]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36:50117--50143

    2023

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  41. [49]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.