{"id":"78e45070-d065-4d87-ae8b-09d37cc3705b","arxiv_id":"2501.01336","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An LLM is fine-tuned with DPO to make the strength of its stance in conversation match its self-estimated confidence, improving resistance to misleading arguments and receptiveness to corrections.","lead":"AFICE is a training framework that estimates an LLM's confidence in its own answers using internal states and sampling statistics, then turns those confidence scores into preference data that teaches the model when to defend its answer and when to accept a user's opposing argument. The paper reports accuracy gains over several baselines on reasoning benchmarks, but the evaluation protocol for the baselines is not fully specified.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core assumption that BCE confidence reliably proxies answer correctness on evaluation tasks is unvalidated; confidently-wrong answers could be trained to persist, so the reported gains may not reflect a general mechanism.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the preference labels in §3.3 rely on the model's self-estimated confidence as a proxy for answer correctness, and the paper does not analyze the confidently-wrong case. I agree with that assessment. The central claim is that aligning a model to its own confidence estimates via DPO improves accuracy in argumentative conversations; this is only true if confidence is well-calibrated on the tasks where the method is deployed. The paper shows calibration evidence on MMLU (Figure 3), but MMLU is both the training distribution for the regressor and a multiple-choice format, whereas evaluation tasks are open-ended and out-of-distribution. The reported aggregate improvements are consistent with confidence being informative, but they do not demonstrate that confidently-wrong answers are rare or that the fine-tuned model handles them properly. A direct calibration and accuracy-binning analysis on the evaluation datasets would settle whether the training signal is aligned with correctness. Because the reader's verdict is already CONDITIONAL and this concern is a refinement of the same assumption, no verdict change is needed; the paper should add this analysis before being accepted.","tokens_in":20274,"tokens_out":10535,"duration_ms":100702,"concrete_test":"Using the released base models (Vicuna-7B, LLaMA3-8B) and the BCE confidence estimator from §3.2, generate initial answer v1 for every question in the five evaluation datasets (GSM8K, PrOntoQA, StrategyQA, CSQA2, CREAK) and the nine BIG-Bench tasks. Bin v1 into the same three confidence groups as §3.3 (thresholds at 33.3/66.7 percentiles of the MMLU confidence distribution) and report: (i) the accuracy of v1 within each bin, (ii) the fraction of confidently-wrong answers (top bin but incorrect v1), and (iii) the same breakdown for the LLM-Correct and LLM-False scenario constructions. If the top-bin accuracy is not substantially above the bottom-bin accuracy, or if the confidently-wrong rate exceeds roughly 10%, the positive/negative set assignments in §3.3 are misaligned with correctness on the deployment tasks, and the mechanism claimed in §4.2 is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AFICE's preference construction in §3.3 labels persist-type responses (r1/r2) as positive when Confidence(q,a) is high and yield-type responses (r4/r5) as positive when it is low, using thresholds at the 33.3/66.7 percentiles of the MMLU training set. For this to improve final accuracy in the two evaluation scenarios, Confidence(q,a) must be positively correlated with correctness of the original answer a on the deployment tasks (GSM8K, PrOntoQA, StrategyQA, CSQA2, CREAK, BIG-Bench). The paper validates BCE only on MMLU (Figure 3, ECE/AUROC), which is multiple-choice and in-distribution for the regressor; evaluation tasks are open-ended and out-of-distribution, and no calibration or accuracy-vs-confidence breakdown is reported for them. Moreover, DPO optimizes labels derived from the base model's confidence, but after fine-tuning the deployed policy is a different model, so even base-model calibration does not guarantee fine-tuned behavior. If the base model is confidently wrong on a nontrivial fraction of its initial answers, DPO will reinforce exactly the stubborn errors that the LLM-False scenario is meant to correct. The high LLM-False accuracy in Table 15 (0.990) suggests the effect is limited, but aggregate averages can mask failure modes; the paper provides no direct evidence on the confidently-wrong rate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AFICE, a framework that combines Bilateral Confidence Estimation (BCE) with Direct Preference Optimization (DPO) to make LLMs more robust to opposing arguments in two-round conversations. BCE first estimates question-level confidence by regressing internal hidden states onto semantic entropy computed on a subset of MMLU, then adjusts this estimate for a specific answer using a cumulative probability ratio over sampled responses. These confidence scores are used to label five possible response types (persist, slight concession, neutral, lean toward opposing view, full agreement) as preferred or dispreferred, and DPO fine-tunes the model to follow the labels. The method is evaluated on math (GSM8K), first-order logic (PrOntoQA), commonsense (StrategyQA, CSQA2, CREAK), and generic reasoning (nine BIG-Bench tasks), in two settings: when the model's initial answer is correct and the user argues incorrectly, and when the initial answer is incorrect and the user argues correctly. Averaged results over both settings are reported for Vicuna-7B and LLaMA-3-8B, with AFICE obtaining the highest average accuracy on most categories.","tokens_in":20574,"tokens_out":6132,"duration_ms":56830,"significance":"If the reported results are reliable, the paper makes a useful practical contribution by demonstrating that a model's own internal-state-based confidence can be used as a training signal to influence whether the model persists in or revises its initial answer during argumentative conversations. The proposed BCE method is computationally cheaper than full-sequence semantic entropy, and the paper provides extensive evaluation tables across several reasoning benchmarks. The conceptual idea of aligning conversational behavior with self-estimated confidence is interesting and could be applicable beyond the specific two-round setting. However, the significance of the work depends on whether the evaluation is fair and whether the confidence signal is actually predictive of answer correctness on the deployment tasks; both points need substantial additional support before the empirical claims can be accepted.","major_comments":[{"comment":"The evaluation protocol for the black-box baselines (Verbalization and Semantic Entropy) is underspecified. The paper states that black-box methods are applied directly to the evaluation dataset and that the confidence score is used as an indication of the LLM's agreement with the initial answer, but it never explains how this score influences the final response in the two-round conversation. Without a precise mechanism — for example, a threshold above which the model is instructed to persist and below which it is instructed to yield — the accuracies reported for these baselines in Table 1 are not interpretable, and the comparison with AFICE, which uses a concrete DPO training procedure, may be unfair.","section":"§4.1, Appendix B.2"},{"comment":"The confidence estimator is trained and evaluated only on the MMLU dataset (Figure 3), while the evaluation tasks in Table 1 are open-ended and out-of-distribution relative to the multiple-choice MMLU regressor. The preference labels in §3.3 assume that Confidence(q,a) is positively correlated with the correctness of the initial answer a, but no calibration curve, AUROC, or accuracy-vs-confidence breakdown is reported for GSM8K, PrOntoQA, StrategyQA, CSQA2, CREAK, or BIG-Bench. The paper therefore does not establish that the high-confidence-is-correct assumption holds in the deployment setting, leaving open the possibility that confidently-wrong answers are reinforced by DPO.","section":"§3.2, §3.3, Figure 3"},{"comment":"All results are reported as single point estimates without error bars, standard deviations, or significance tests. Several of the reported improvements of AFICE over the strongest baseline are small, such as Vicuna on GSM8K (0.623 vs. 0.616 for Predictive Entropy) and LLaMA-3 on FOL (0.752 vs. 0.733 for Semantic Entropy). Because the underlying evaluation sets are relatively small (e.g., 464 GSM8K examples, 307 PrOntoQA examples), these differences may not be statistically reliable, and the abstract's claim of 'significant improvements' is not currently supported by appropriate statistical evidence.","section":"Table 1, Appendix D"},{"comment":"The DPO preference labels are derived from the base model's confidence, but after fine-tuning the deployed policy has different internal states; the paper itself acknowledges in C.1 that the regressor may need to be retrained post-fine-tuning. The paper does not verify whether the fine-tuned model's behavior is actually consistent with its own post-fine-tuning confidence, which is the mechanism that the title and framing claim. This gap leaves the causal chain from confidence estimation to improved final accuracy only partially demonstrated.","section":"Limitations C.1, §3.3"},{"comment":"There is a tension between the paper's definition of 'faithful integrity' — responding according to what the model believes to be true — and the evaluation metric, which is accuracy against ground-truth labels. If the base model is confidently wrong about an answer, persisting in that answer is faithful by the paper's definition but counts as an error in the LLM False scenario. The paper does not analyze the rate of confidently-wrong initial answers on the evaluation tasks, and thus does not resolve whether the observed LLM False improvements reflect genuine correction of low-confidence errors or are partly artifacts of the synthetic user arguments and dataset composition.","section":"§2.1, §4.2"}],"minor_comments":[{"comment":"There are typos in the prompt templates, such as 'Gengerating response candidates' instead of 'Generating response candidates', which should be corrected.","section":"Appendix B.1"},{"comment":"Mielke et al. 2022a and 2022b are the same paper and should be consolidated into a single reference to avoid confusion.","section":"References"},{"comment":"The description of the alpha and gamma sensitivity analysis is unclear: 'AUROC peaks near zero for both, declines between 0.5 and 1, and then gradually improves' needs a clearer explanation of what is being varied and what the observed trend means for the chosen default values.","section":"§4.3, Figure 4"},{"comment":"The column labeled 'Num.' in Table 2 is not defined in the text; a note explaining that it refers to the number of output sequences generated per input would improve readability.","section":"Table 2"},{"comment":"Equation (2) contains a stray closing brace at the end of the display, and the notation for the hyperparameters θ_h is not clearly separated from the model parameters θ in the conditional probability; this should be cleaned up.","section":"Equation (2)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and may be publishable if the evaluation concerns are properly addressed. The most pressing issues are the missing protocol for black-box baselines, the lack of validation that the confidence estimator predicts correctness on the out-of-distribution evaluation tasks, and the absence of statistical reliability measures. The self-training nature of the pipeline (training on labels derived from the model's own confidence) is not by itself disqualifying for this venue, but the paper should explicitly discuss and, where possible, test the confidently-wrong case. I would recommend asking the authors for a careful revision that adds the missing evaluation details and supporting analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the central idea is more interesting than the average alignment patch: instead of training the model to match external correctness labels, AFICE trains it to match its own estimated confidence, using a bilateral confidence estimator that combines internal-state regression with cumulative probability ratios. That combination is new, and the preference construction (five response types, thresholds at confidence percentiles, six DPO pairs) is concrete and reproducible. Second, the main table shows real, reasonably consistent gains across two base models and several benchmarks, with especially strong improvements on the LLM-False scenario. That's a solid empirical result worth engaging with.\n\nWhat the paper does well: BCE is efficient (early token truncation at K=60, N=20), and the comparison against Semantic Entropy, P(True), and Predictive Entropy is fair in spirit because all are used as confidence signals for the same alignment recipe. The appendix reporting LLM Correct vs. LLM False separately, plus Both/Either, is more informative than a single average. The limitations section is honest: C.1 explicitly flags that the regressor may need retraining after fine-tuning, which is exactly the soft spot the stress-test note hits.\n\nNow the soft spots, in proportion. The black-box baseline protocol is genuinely underspecified: it's unclear how the verbalization or semantic entropy score is turned into a final response decision, so part of the advantage could come from how the policy is adjusted rather than the confidence estimator itself. No error bars or significance tests either, and some margins are thin (Vicuna AFICE 0.623 vs. Predictive Entropy 0.616 on GSM8K). The deeper concern is the one the paper's own limitation C.1 gestures at: the confidence regressor is validated only on MMLU, which is multiple-choice and in-distribution, while the deployment tasks are open-ended and out-of-distribution. If the base model is confidently wrong on those tasks, DPO will reinforce exactly the stubborn errors the paper is trying to fix. The high LLM-False accuracy (0.990 for LLaMA-3 AFICE) suggests the effect is limited in practice, but we don't get a calibration or accuracy-versus-confidence breakdown on the eval tasks, so the worry is not directly retired.\n\nWho this is for: people working on LLM alignment, factuality, and conversational robustness. It deserves a serious referee, not a desk reject. The authors should be asked to clarify the black-box baseline protocol, add error bars or significance tests, and provide a calibration or accuracy-vs-confidence breakdown on at least one out-of-distribution eval task. If those come back clean, this could be a useful recipe.","headline":"A plausibly useful self-confidence alignment recipe with a genuinely new confidence estimator, but the evaluation's out-of-distribution calibration gap and underspecified baselines keep it short of a clean accept.","tokens_in":21059,"tokens_out":1925,"would_cite":true,"duration_ms":21362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligned to its own confidence estimates, a language model defends correct answers against opposing arguments and concedes when its answer is wrong, raising final-answer accuracy across debate-style reasoning benchmarks.","keywords":["faithful integrity","opposing argument","confidence estimation","Direct Preference Optimization","semantic entropy","LLM alignment","conversational robustness","bilateral confidence estimation"],"falsifier":"On a subset of MMLU where the base model's bilateral confidence is high but the answer is wrong, present the correct answer as the user's counter-argument; if the AFICE-tuned model concedes no more often than it defends correct high-confidence answers, the confidence-as-truth-proxy assumption is falsified.","tokens_in":20077,"feed_emoji":"⚖️","tokens_out":8334,"duration_ms":72865,"temperature":0.7,"pith_summary":"Large language models cave too easily when a user argues back, even when the original answer was right, and they dig in too hard when the user is right and they are wrong. The paper claims the two failure modes share one fix: align the model to its own confidence. The authors compute a two-part confidence score for each answer, build preference pairs that tell the model to persist when confident and yield when unsure, and fine-tune on those pairs. Across mathematics, logic, commonsense, and generic reasoning benchmarks, the final-answer accuracy after a two-round debate rises, from 0.598 to 0.742 in the generic category for LLaMA-3-8B. If the claim holds, conversational models can be made more reliable without retrieval, fact-checking, or external knowledge at inference time.","feed_headline":"Confidence-aligned tuning lifts LLM debate accuracy to 74.2%","feed_subtitle":"Fine-tuning models to trust their own calibrated confidence helps them defend correct answers and concede wrong ones.","key_machinery":"Bilateral Confidence Estimation (BCE), a two-stage confidence score. A regressor trained on the model's internal hidden states (the 26th layer output at the last token) approximates semantic entropy, giving a question-level confidence; a per-answer adjustment multiplies it by a cumulative probability ratio computed from the length-normalized log-likelihood of the answer among a multinomial beam sample. Confidence thresholds at the 66.7th and 33.3rd percentiles map five hand-written response stances (persist, slight concession, neutral, lean opposing, fully agree) into positive and negative sets for Direct Preference Optimization. The same score decides both persistence and concession, which is what makes a single alignment pass address both failure modes.","core_discovery":"The central claim is that a model aligned to its own calibrated confidence will behave with faithful integrity in argument: it defends an originally correct statement against an incorrect user counter-argument and corrects an originally incorrect statement when the user's counter-argument is faithful. The evidence is that AFICE fine-tuning raises the average proportion of final answers matching the correct answer across all four question categories and both base models, with the confidence-based method beating verbalization, semantic entropy, P(True), and predictive entropy baselines. The paper presents the gain as coming from aligning response stance to confidence rather than from injecting correct content, since the preference data are built from the model's own answers and confidence scores.","pith_inferences":["The paper's own limitation note predicts that fine-tuning shifts internal states, so the confidence regressor may need refitting after alignment; a concrete next experiment is to measure AUROC of the regressor on the fine-tuned model and see whether periodic re-estimation restores performance.","Because the preference labels come from self-estimated confidence, the method inherits any systematic overconfidence; a testable extension is to run AFICE on a deliberately miscalibrated model and check whether the accuracy gains shrink or reverse.","The user counter-arguments are machine-generated as either fully correct or fully incorrect; real-user arguments are often partially valid, so a harder extension would test AFICE against mixed-validity arguments and finer-grained concession thresholds.","The confidence score could be extended from a two-way persist-or-concede decision to a three-way decision that includes abstention ('I don't know') or a retrieval call at very low confidence."],"forward_implications":["A conversational LLM trained this way should show measurably fewer sycophantic U-turns when correct and more willingness to be corrected when wrong, without needing retrieval or fact-checking at inference time.","The cheap confidence estimate—using early-truncated tokens (K=60) and 20 samples—makes the alignment practical in interactive latency budgets where full-sequence semantic entropy is too slow.","Because one confidence score drives both persistence and concession, the method should transfer to any QA-style domain where internal states predict correctness, not just the four benchmark categories tested.","The consistent gain over P(True) and predictive entropy indicates that combining question-level internal-state certainty with answer-level likelihood rank is what lets the model decide when to yield."],"supporting_citations":[{"why":"Supplies the debate-style evaluation setup and the finding that strong LLMs concede to opposing arguments, the problem AFICE targets.","marker":"Wang, Yue, and Sun 2023"},{"why":"Semantic entropy is the reference-free uncertainty target that the internal-state regressor is trained to approximate.","marker":"Kuhn, Gal, and Farquhar 2023"},{"why":"P(True) serves as a white-box baseline, and its calibration-curve, AUROC, and ECE evaluation protocol is used to compare confidence methods.","marker":"Kadavath et al. 2022a"},{"why":"Verbalization is the black-box baseline that prompts the model to state its own uncertainty.","marker":"Mielke et al. 2022a"},{"why":"Predictive entropy with Monte Carlo sampling is the other white-box baseline for confidence estimation.","marker":"Malinin and Gales 2020"},{"why":"MMLU provides the 14,042-example preference dataset used to construct the DPO training pairs.","marker":"Hendrycks et al. 2020"},{"why":"Provides the DPO formulation and reference-policy setup used for fine-tuning.","marker":"Tian et al. 2023"},{"why":"LoRA is the efficient fine-tuning approach used in the DPO stage.","marker":"Hu et al. 2022"}],"fun_headline_variants":["Confidence-aligned tuning keeps LLMs faithful in arguments","AFICE: Aligning LLMs to their confidence for honest debates","LLMs trained to trust their own confidence resist bad arguments","New method helps LLMs defend correct answers and fix failures","Confidence-aware fine-tuning improves LLM argument integrity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's self-estimated confidence tracks actual correctness well enough to serve as the training signal, so confidently wrong answers would be trained to be stubbornly defended and the paper does not analyze that case.","fun_headline_variants_meta":{"raw":{"variants":["Confidence-aligned tuning keeps LLMs faithful in arguments","AFICE: Aligning LLMs to their confidence for honest debates","LLMs trained to trust their own confidence resist bad arguments","New method helps LLMs defend correct answers and fix failures","Confidence-aware fine-tuning improves LLM argument integrity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1570,"prompt_tokens":935,"completion_tokens":635,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":553}},"tokens_in":551,"tokens_out":635,"duration_ms":6742,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:29:49.387970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a subset of MMLU where the base model's bilateral confidence is high but the answer is wrong, present the correct answer as the user's counter-argument; if the AFICE-tuned model concedes no more often than it defends correct high-confidence answers, the confidence-as-truth-proxy assumption is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the debate-style evaluation setup and the finding that strong LLMs concede to opposing arguments, the problem AFICE targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Semantic entropy is the reference-free uncertainty target that the internal-state regressor is trained to approximate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Predictive entropy with Monte Carlo sampling is the other white-box baseline for confidence estimation."}],"review_version":1}