{"id":"a90fd19d-2aac-42bf-b0af-9d4955777e20","arxiv_id":"2505.20903","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning on data aligned with an LLM's prior knowledge induces overconfidence, and CogCalib mitigates this by gating a calibration loss to known data.","lead":"Fine-tuning on examples the model already knows makes large language models overconfident, while newly learned examples improve calibration. The authors propose CogCalib, which identifies known examples during training and applies a calibration loss only to them, lowering calibration error without sacrificing accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The known/unknown split is built from zero-shot correctness, so the observed overconfidence may be driven by initial solvability or difficulty rather than prior knowledge per se.","rationale":"The reader's weakest assumption is the same one I would stress: the SliCK operationalization conflates prior knowledge with zero-shot correctness and difficulty. This is load-bearing because the paper's headline contribution is the causal story, not just CogCalib. CogCalib itself is clearly specified, the gating ablation (Figure 7) supports applying calibration loss only to known/low-NLL data, and the method's OOD results are suggestive; those strengths should be credited. However, none of the CogCalib results discriminates between 'prior knowledge causes overconfidence' and 'easy or initially-confident examples cause overconfidence.' The proposed concrete test uses the paper's own SliCK taxonomy and is cheap to run; if it fails, the central claim must be weakened to a difficulty/confidence-based statement, which would substantially reduce novelty. I therefore keep the reader's CONDITIONAL verdict but make the condition explicit: acceptance should require the four-category (or matched-confidence) control experiment. The paper should not be rejected outright because the method and ablations have independent value, and the required experiment is feasible.","tokens_in":23516,"tokens_out":9556,"duration_ms":109481,"concrete_test":"Re-run the Section 3.1 experiment using the four SliCK categories already defined in Table 21 (HighlyKnown, MaybeKnown, WeaklyKnown, Unknown) instead of the collapsed Known/Unknown split, on OBQA with Llama3-8B. Construct equal-size fine-tuning sets for each category and plot accuracy/confidence trajectories and ECE against the category as in Figure 3a. If MaybeKnown and WeaklyKnown—which are not always zero-shot correct but carry partial prior knowledge—show the same early-plateau/overconfidence pattern as HighlyKnown, then the effect tracks initial solvability rather than knowledge overlap, and the central claim needs reframing. To further separate initial confidence from knowledge, also compare a subset of Known and Unknown examples matched on pre-fine-tuning target-token NLL or confidence; if the ECE gap persists in matched samples, the prior-knowledge interpretation is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central causal claim—that data aligned with the model's prior knowledge induce overconfidence—rests entirely on the SliCK dichotomy (Appendix G.1). By construction, Known examples are those the base model answers correctly under all ten greedy 4-shot prompts (P(T=0)=1), while Unknown examples are those it never answers correctly (P(T>=0)=0). The two sets therefore differ by construction in zero-shot correctness (100% vs 0%) and, presumably, in initial confidence on the target token. Section 3's key evidence (Figure 3a) shows that Known fine-tuning produces an early accuracy plateau with rising confidence, while Unknown fine-tuning keeps accuracy and confidence synchronized. But this is exactly the expected trajectory for any initially-solvable versus initially-unsolvable set: easy examples start near ceiling so accuracy saturates while confidence keeps being reinforced; hard examples start at zero and confidence rises only as learning occurs. The Figure 2 ratio sweep only varies the proportion of initially-correct examples; it does not control for example difficulty, topic, length, or the fact that Known and Unknown subsets are different samples. Moreover, the proposed mechanism is confidence inflation, yet SliCK labels are defined by correctness; confidently-held but wrong beliefs (which should also inflate confidence) are classified as Unknown. The CogCalib results are equally compatible with the weaker statement that applying calibration regularization to low-NLL (easy) examples improves ECE. Without a control that separates prior knowledge from initial solvability or initial confidence, the paper's core causal claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how an LLM's prior knowledge affects confidence calibration during supervised fine-tuning. It defines 'known' data as examples the base model already answers correctly under the SliCK procedure and 'unknown' data as examples it never answers correctly. Through ratio sweeps and learning-dynamics plots, the authors conclude that known data induce overconfidence while unknown data improve calibration. They then propose CogCalib, which uses an adaptive NLL threshold to detect known examples during training and applies a calibration regularizer (label smoothing, margin-based label smoothing, or entropy penalization) only to those examples. Experiments across seven QA tasks and four models report large ECE reductions relative to vanilla SFT, temperature scaling, MC-dropout, and deep ensembles, with ablations against vanilla and random calibration supporting the gating mechanism.","tokens_in":23782,"tokens_out":5091,"duration_ms":55605,"significance":"If the central causal claim survives scrutiny, the paper identifies a practically important and previously underemphasized driver of miscalibration in fine-tuned LLMs: the overlap between fine-tuning data and the model's pretraining knowledge. The proposed CogCalib framework is attractive because it adds no inference-time cost, is compatible with multiple training-based calibration losses, and is evaluated across model families and task formats. The paper also ships extensive appendices with per-model and per-dataset results, and the ablation against vanilla and random calibration is a useful control. The main risk is that the known/unknown dichotomy is operationalized through zero-shot correctness, which may conflate 'prior knowledge' with 'initial solvability' or 'example difficulty.' This conflation is load-bearing because the paper's mechanistic claims and the design of CogCalib both rest on that dichotomy.","major_comments":[{"comment":"The central causal claim—that data aligned with the model's prior knowledge cause overconfidence—is not established because the known/unknown split is defined by zero-shot correctness. Under SliCK (Table 21), Known examples satisfy P(T=0)=1 and Unknown examples satisfy P(T≥0)=0, so the two sets differ by construction in initial accuracy. The learning-dynamics evidence in Figure 3a (accuracy plateau with rising confidence for known data; synchronized accuracy and confidence for unknown data) is exactly what one would expect for any easy-versus-hard split, regardless of whether the ease comes from stored knowledge or from surface cues such as question format or lexical overlap. To support the mechanistic interpretation, the authors should control for difficulty, for example by matching Known and Unknown subsets on base-model accuracy under a different prompt format, or by measuring prior knowledge with a held-out knowledge probe that is not the same task used for fine-tuning.","section":"§3.1 and Appendix G.1"},{"comment":"The online knowledge-bias evaluator uses correctness as the label for threshold selection. In Algorithm 1, ci=1 if the model answers correctly and the threshold is chosen to maximize TPR+TNR, where the 'true' labels are these correctness indicators. This means the gating signal is 'currently answerable,' not 'in the model's prior knowledge' as established by SliCK. Table 1 validates the NLL-based classifier against SliCK only in aggregate accuracy, TPR, and TNR; the actual training-time algorithm never consults SliCK. The authors should clarify whether the calibration-set labels are SliCK labels or correctness labels, and if the latter, justify why the method is not simply 'regularize easy examples.'","section":"§4.1, Algorithm 1"},{"comment":"The calibration set used for threshold updates is described as 'randomly selected from the training set.' Because the threshold t is tuned on samples that are also used for model training, the reported CogCalib gains may be optimistic and the chosen threshold may not transfer to held-out data. A held-out calibration set, or a cross-validation scheme that excludes calibration samples from the training update, should be used to ensure the threshold-selection procedure is not overfitting the training distribution.","section":"Appendix A"},{"comment":"The main results report only three-seed averages without standard deviations or significance tests. The only seed-level variation reported is in Appendix Table 5, which covers confidence values for Figure 3b but not the ECE or accuracy numbers in the main tables. Given that the paper claims 'significant' calibration improvements and that some ECE gaps between CogCalib and baselines are large, the authors should report standard deviations or confidence intervals for the primary ECE and accuracy results, or at least for the headline Llama3-8B results in Tables 2 and 3.","section":"Tables 2, 3, 7–16"}],"minor_comments":[{"comment":"The abstract states an 'average 57% reduction in ECE,' while the introduction says '55.92%' (compared to TS) and '65.02%' (compared to SFT) for Llama3-8B on multiple-choice tasks. These numbers should be reconciled, and the aggregation method (per-dataset average percentage reduction versus pooled reduction) should be stated explicitly.","section":"Abstract and §1"},{"comment":"The AUROC values 0.85 and 0.77 are reported without error bars. Appendix Table 5 provides standard deviations only for confidence values, not for the AUROC metric, so the reader cannot assess the stability of the OOD-detection claim.","section":"Figure 3b"},{"comment":"The heading 'Comparision to Vanilla and Random Calibration' contains a typo; it should read 'Comparison.'","section":"Appendix F.7"},{"comment":"The abbreviations 'CS' and 'IS' in Table 5 are undefined; the table caption should explain what these terms denote.","section":"Table 5"},{"comment":"The open-ended and multiple-choice tasks use different calibration-loss hyperparameters (Tables 22 and 23), but the sensitivity analysis in Appendix F.9 covers only ARC-C and WG-S. Reporting sensitivity for at least one open-ended task, such as HotpotQA or MedMCQA, would strengthen the claim of robustness.","section":"§5.1 and Appendix G.3"},{"comment":"Table 1 reports NLL-based known/unknown classification accuracy for Llama3-8B only. Since the method is claimed to be model-agnostic, reporting the same validation for at least one other model would be helpful.","section":"§5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the empirical scope is substantial, but the referee report's central concern is real: the known/unknown operationalization conflates prior knowledge with initial solvability. This is not a circularity in the 'fitted parameter' sense, but it is a confound that directly affects the paper's main mechanistic claim. If the authors can add a difficulty-controlled experiment or otherwise disentangle prior knowledge from example easiness, the paper could become a solid contribution. The current version should not be accepted without that evidence. There is also a minor novelty concern: the paper builds directly on SliCK's known/unknown taxonomy, so the framing should more clearly separate the new calibration findings from the existing data-selection results of Gekhman et al."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ziming et al. report something practically useful: during LLM fine-tuning, examples the base model already answers correctly (SliCK-known) drive confidence inflation, and gating a calibration loss to those examples roughly halves ECE on several benchmarks with no inference overhead. The CogCalib recipe is simple, reproducible in principle, and the NLL-based known/unknown proxy is validated against SliCK with high accuracy. That is a real contribution, especially for practitioners who need better-calibrated fine-tuned models.\n\nThe main caveat is that the causal claim in the title and Section 1 is stronger than the evidence. Known data are defined by zero-shot correctness; unknown data are, by construction, initially unsolvable. So comparing them conflates 'aligned with prior knowledge' with 'easy and already solved.' The observed learning dynamics—accuracy plateaus while confidence keeps rising on known data—are exactly what you'd expect for any initially-solvable set, regardless of whether the knowledge is stored as an encyclopedic fact or a lexical shortcut. The paper does not control for difficulty, topic, or initial confidence. The CogCalib results are equally compatible with the weaker, well-known statement that applying calibration regularization to low-NLL (easy) examples improves ECE. That should be acknowledged.\n\nThat said, the weaker statement is still valuable, and the paper largely delivers on it. The ablation against vanilla and random gating shows that targeting low-NLL examples is what matters. The OOD generalization is a nice plus. The main tables report only seed-averaged numbers without error bars; the abstract's 57% reduction and the introduction's 55.92% vs TS and 65.02% vs SFT are different baselines, but the presentation invites confusion. The 'ubiquitous known data' premise is asserted, not measured. No code or data is released, which is a practical obstacle.\n\nThe honest summary: treat the paper as an empirical method paper, not as a settled mechanistic explanation. The method is likely to transfer because it is essentially a principled version of 'regularize on easy examples.' With a difficulty-matched control and error bars, the causal story could be made much more solid. As it stands, I would send it to review. The finding is useful, the method is clean, and the confound is addressable rather than fatal.","headline":"A useful empirical recipe for better fine-tuning calibration, but the headline causal claim about prior knowledge is not actually proven by the experiments.","tokens_in":24367,"tokens_out":2531,"would_cite":true,"duration_ms":26919,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that samples an LLM already knows are what ruin calibration during fine-tuning, and shows that applying a confidence penalty selectively to those samples cuts calibration error by over half without hurting accuracy.","keywords":["LLM calibration","fine-tuning","prior knowledge","overconfidence","knowledge bias","expected calibration error","known versus unknown data","CogCalib"],"falsifier":"Construct two matched fine-tuning sets: 'known' samples the base model answers correctly, and 'unknown' samples it cannot answer but that are equally learnable, so both reach the same accuracy in the same small number of training steps. The paper's claim predicts the accuracy-confidence gap still diverges for the known set while staying synchronized for the difficult-but-learnable set; if both sets show the same divergence, the effect tracks difficulty rather than prior knowledge.","tokens_in":23316,"feed_emoji":"⚖️","tokens_out":11528,"duration_ms":101183,"temperature":0.7,"pith_summary":"This paper claims that the model's own prior knowledge, not the fine-tuning process itself, is the main reason fine-tuned LLMs end up poorly calibrated. Samples the base model can already answer correctly ('known data') make confidence grow faster than accuracy during training, producing overconfidence, whereas genuinely new knowledge ('unknown data') keeps the two aligned. On that basis the paper proposes CogCalib, which scores each training sample's negative log-likelihood, flags the known ones with an adaptive threshold, and applies a confidence-penalizing regularization term to exactly those samples while leaving new knowledge to be learned with plain cross-entropy. Across seven QA tasks, four instruction-tuned models, and both LoRA and full fine-tuning, the method reports expected calibration error reductions averaging 57% on Llama3-8B with accuracy maintained, including on out-of-distribution tests. If the claim holds, calibration for fine-tuned LLMs is a data-overlap problem fixable during training rather than a post-hoc rescaling problem.","feed_headline":"Fine-tuning on data the model already knows breeds overconfidence","feed_subtitle":"New knowledge improves calibration; a targeted regularizer cut ECE by 57% on Llama3-8B.","key_machinery":"The load-bearing mechanism is knowledge bias, the gap between what the base model already knows and what the fine-tuning data teaches, and a gating rule that acts on it. Offline, data is labelled known or unknown with the SliCK procedure: 10 different 4-shot prompt templates and 16 sampling passes at temperatures 0 and 0.5, with samples the base model always answers correctly counted as known. During training, CogCalib replaces this expensive procedure with the sample's negative log-likelihood $L_{\\mathrm{NLL}} = -\\sum_k q_k \\log p_k$, comparing it to an adaptively updated threshold $t$ (grid-searched on a held-out calibration set each epoch to maximize true-positive plus true-negative rates). The binary indicator $I(p,q) = 1$ for known samples gates the loss $L = L_{\\mathrm{CE}} + I(p,q)\\cdot\\alpha L_{\\mathrm{cal}}$, where $L_{\\mathrm{cal}}$ is label smoothing, margin-based label smoothing, or ECP. A brief style-adaptation phase before training makes NLL values comparable across the model's output style and the task's label format. The gating identity is what carries the argument: it routes each sample into one of two learning regimes, confidence moderation for already-known material and full task learning for new material.","core_discovery":"The paper's central claim is stated plainly: data aligned with the model's prior knowledge tend to cause overconfidence, while data exhibiting knowledge bias contribute to better calibration. Known data are samples the base model already answers correctly; unknown data are samples it cannot answer before fine-tuning. During fine-tuning on known data, test accuracy plateaus early (around step 200 in the paper's tracking) while confidence keeps climbing, widening the calibration gap; on unknown data, accuracy and confidence rise in step, and the model develops separated confidence levels for correct and incorrect predictions, which also improves out-of-distribution detection (AUROC 0.85 vs 0.77 at step 600). The effect is directional in the known/unknown ratio, appears across every model family tested, and persists under full fine-tuning. The paper further shows that simply deleting known data is not a reliable cure, because it improves calibration on ARC-C but worsens it on OBQA, motivating the selective-regularization design of CogCalib.","pith_inferences":["Editorial: The paper's NLL-based known/unknown signal could be fed back into data selection or curriculum design, not just loss design, for example by ordering known samples early and interleaving novel knowledge; the paper only explores this direction destructively by deleting known data.","Editorial: The paper states its scope as models up to 13B parameters; if the mechanism strengthens with prior knowledge, larger base models should show an even larger 'known' fraction in any domain corpus and hence a stronger version of the effect, but that scaling is untested.","Editorial: The causal story would be sharpened by dissociating 'known' from 'easy,' since zero-shot correctness and sample difficulty are entangled in the SliCK operationalization; a matched-difficulty comparison would settle whether stored knowledge or general easiness drives the confidence inflation.","Editorial: The same gating logic may apply beyond supervised fine-tuning, to preference optimization or continual learning, where overlap with prior knowledge also dominates the training signal; the paper does not test those settings."],"forward_implications":["Calibration can be improved during fine-tuning itself: CogCalib introduces no extra parameters and no added cost at inference, unlike post-hoc methods such as temperature scaling or ensembles.","Because the mechanism is knowledge overlap, calibration pressure grows as base models become more capable: a larger prior means more of any domain corpus counts as 'known' and therefore confidence-inflating.","The selective treatment is essential to the result: applying the calibration loss to all data, or to randomly chosen data, degrades downstream accuracy, because unknown data are the samples that actually align the model with the task.","The benefits transfer out of distribution: models fine-tuned on OBQA with CoLS, CoMbLS, or CoECP show lower ECE on ARC-C, ARC-E, and MMLU subjects than vanilla SFT, temperature scaling, MC-dropout, and deep ensembles.","The framework is agnostic to the calibration term and the tuning method: consistent ECE reductions appear with label smoothing, margin-based label smoothing, and ECP, under both LoRA and full-parameter fine-tuning."],"supporting_citations":[{"why":"Supplies the SliCK procedure that classifies fine-tuning samples as known versus unknown, the operational definition of prior knowledge on which the entire analysis rests.","marker":"(Gekhman et al., 2024)"},{"why":"Provides temperature scaling, the main post-hoc calibration baseline, and the reliability-diagram style calibration framework the paper measures with ECE.","marker":"(Guo et al., 2017b)"},{"why":"Established negative log-likelihood overfitting as a cause of miscalibration in models trained from scratch, the finding this paper extends to LLMs with prior knowledge.","marker":"(Mukhoti et al., 2020)"},{"why":"Provides label smoothing, the first calibration term that CogCalib gates onto known data.","marker":"(Szegedy et al., 2016)"},{"why":"Provides margin-based label smoothing (MbLS), the second calibration term used in the gating framework.","marker":"(Liu et al., 2022)"},{"why":"Provides ECP, the entropy-penalty regularizer that forms the third CogCalib variant.","marker":"(Pereyra et al., 2017)"},{"why":"Supports the claim that LLMs adapt rapidly to downstream task syntax and that known and unknown data follow different learning dynamics, motivating the style-adaptation step.","marker":"(Zhang and Wu, 2024)"},{"why":"Provides the geometric-mean confidence measure used to evaluate calibration on open-ended generation tasks.","marker":"(Liu et al., 2023)"},{"why":"Supplies OpenBookQA, the dataset on which the core knowledge-bias ratio experiments and main calibration results are built.","marker":"(Mihaylov et al., 2018)"}],"fun_headline_variants":["Fine-tuning on known data breeds LLM overconfidence","CogCalib cuts ECE by 57% by targeting prior knowledge","Known data in fine-tuning hurts calibration; new knowledge helps","Why fine-tuning on known data harms LLM calibration","Targeting prior knowledge in fine-tuning improves calibration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument depends on identifying 'known data' with samples the base model answers correctly before fine-tuning; if that correctness reflects question difficulty, lucky guessing, or surface wording patterns rather than stored knowledge, the claimed causal link between prior knowledge and overconfidence is confounded.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning on known data breeds LLM overconfidence","CogCalib cuts ECE by 57% by targeting prior knowledge","Known data in fine-tuning hurts calibration; new knowledge helps","Why fine-tuning on known data harms LLM calibration","Targeting prior knowledge in fine-tuning improves calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1645,"prompt_tokens":963,"completion_tokens":682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":601}},"tokens_in":579,"tokens_out":682,"duration_ms":6669,"temperature":1.0,"reasoning_tokens":601,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:44:29.412459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct two matched fine-tuning sets: 'known' samples the base model answers correctly, and 'unknown' samples it cannot answer but that are equally learnable, so both reach the same accuracy in the same small number of training steps. The paper's claim predicts the accuracy-confidence gap still diverges for the known set while staying synchronized for the difficult-but-learnable set; if both sets show the same divergence, the effect tracks difficulty rather than prior knowledge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides margin-based label smoothing (MbLS), the second calibration term used in the gating framework."}],"review_version":1}