{"id":"90f2e91b-c290-4888-ab10-a4242820efd5","arxiv_id":"2411.13343","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper measures LLM confidence per atomic fact, weights correctness by relevance, and uses high-confidence facts from the same response to correct low-confidence facts.","lead":"This paper measures an AI model's confidence separately for each fact in a long answer, not just for the whole answer. It then uses the facts the model is confident about as clues to fix the facts it is unsure about, aiming to reduce made-up content without an external search.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ConFix's acceptance rule relies on confidence increases that may not reflect factual improvement; the central hallucination-mitigation claim lacks objective factuality evaluation.","rationale":"The reader's weakest assumption is essentially the same one I would stress-test: ConFix depends on self-reported confidence being reliable enough both to choose correct high-confidence reference facts and to accept corrections when confidence rises. My concern sharpens this into a testable claim: the acceptance rule in Section 5.2 S3 uses confidence increase as a proxy for factual improvement, but no objective factuality measure is applied to the before/after comparison. The paper's GPT-4 pairwise evaluation is a subjective preference judgment, not a hallucination detector, so the headline 'effectively mitigates hallucinations' is not directly supported. The paper's own numbers make the risk concrete: low recall in error detection means most hallucinations are never flagged, and the large regression on LLaMA-2-7b shows confidence shifts can be negatively correlated with correctness. The decisive check is to re-score revised facts with the same SAFE/retrieval-based factuality evaluator used for calibration in Section 4.1. If confidence increases do not correlate with SAFE improvements, the central self-correction claim fails even though the calibration framework and F-ECE remain useful contributions. I therefore keep the reader's CONDITIONAL verdict, since the concern is addressable with an additional objective evaluation and does not invalidate the calibration findings.","tokens_in":957,"tokens_out":814,"duration_ms":44368,"concrete_test":"Re-run the ConFix pipeline on the same five models and both datasets, but score every original fact and every accepted revised fact with the SAFE factuality evaluator already used in Section 4.1 (search-augmented, not GPT-4 pairwise). Compute (i) factual precision of the full response before and after ConFix; (ii) the correlation between confidence change (NewConf - OldConf) and SAFE correctness change; and (iii) the hallucination rate among accepted revisions. If SAFE-based factual precision does not improve, or if confidence increases are not positively correlated with SAFE improvements, the S3 acceptance rule is not a valid hallucination mitigator and the 'without external knowledge' claim for ConFix fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that a rise in the model's self-estimated confidence after revision indicates a genuinely more factual fact. This enters twice: in S1 the high-confidence facts are trusted as reference knowledge, and in S3 (Eq. 8) a revision is accepted only if the re-estimated confidence exceeds the old confidence. But the paper never measures whether confidence increases track objective factuality. Table 3 rates revisions with GPT-4 pairwise judgment (Appendix C.5), which is not a factuality verifier; the same GPT-4 could prefer fluent or more assertive wording. The paper's own data weaken the assumption: Table 2 shows error-detection recall is at most 30.6%, so many false facts are never low-confidence; Table 3 shows LLaMA-2-7b regresses on 80.7% of the 207 revised facts, and even the best model, Llama-2-13b, improves only 53.35% versus 27.07% regression. With no objective before/after factuality score, the central claim that ConFix 'effectively mitigates hallucinations without external knowledge' is unverified: a revision accepted because confidence rose could still introduce or preserve a hallucination. The calibration analysis and F-ECE metric are independent contributions, but the self-correction claim rests on this untested correlation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fact-level confidence calibration framework that decomposes long-form responses into atomic facts, estimates a confidence score for each fact, and introduces F-ECE, a fact-level expected calibration error that compares confidence to relevance-weighted correctness. Using this framework, the authors report three observations: fact-level calibration imposes a stricter standard than response-level calibration; response-level overconfidence arises from high-confidence implicit facts; and fact-level confidence has high variance within a single response. These observations motivate ConFix, a self-correction method that treats high-confidence facts in a response as internal reference knowledge for revising low-confidence facts, accepting a revision only when the model's re-estimated confidence for that fact increases. Experiments on ASQA and LongFact with five base models and three confidence estimators evaluate calibration and, for self-correction, use GPT-4 pairwise judgments to classify outcomes as improved, same, or regressed. The abstract claims that ConFix 'effectively mitigates hallucinations without requiring external knowledge sources.'","tokens_in":15228,"tokens_out":4988,"duration_ms":50129,"significance":"The calibration contribution is meaningful and reproducible: F-ECE is a natural extension of ECE to the fact level, the three observations are clearly stated and supported by the presented tables and figures, and the code is released, which strengthens reproducibility. If the self-correction claim were rigorously validated, ConFix would be an important result because it would show that a model can improve its own factuality using only its internal confidence signals, without retrieval or external knowledge. However, as submitted, the central self-correction claim rests on an unvalidated assumption that a rise in self-estimated confidence corresponds to genuine factual improvement, and the evaluation of correction outcomes uses GPT-4 as a judge rather than an objective factuality benchmark. The calibration analysis can stand on its own, but the hallucination-mitigation claim requires substantially stronger evidence.","major_comments":[{"comment":"The acceptance rule in S3 (Eq. 8) accepts a revision only if the re-estimated confidence exceeds the old confidence, but the paper never verifies that confidence increases track objective factuality. The outcome evaluation in Table 3 is based on GPT-4 pairwise judgments (Appendix C.5), not an external factuality verifier, so the improved/regressed labels themselves may reflect fluency or assertiveness rather than factual accuracy. The paper's own data underscore the risk: Llama-2-7b regresses on 80.68% of 207 revised facts, and even the best model, Llama-2-13b, regresses on 27.07% of revised facts while improving on 53.35%. Without an objective before/after factuality measurement (e.g., SAFE or human annotation) and a direct analysis of the confidence-factuality correlation, the abstract's claim that ConFix 'effectively mitigates hallucinations' is not supported.","section":"Section 5.3, Table 3, and Eq. (8)"},{"comment":"The high-confidence facts are used as reference knowledge to correct low-confidence facts, but the paper does not establish that high-confidence facts are actually factually correct. Table 2 shows that error-detection recall is at most 30.62% and as low as 0.15% (Vicuna-7b), which means that many false facts are not low-confidence and may be included in the high-confidence set f_h. If the reference set contains hallucinations, the correction step may propagate or reinforce errors. This is load-bearing because the entire ConFix mechanism depends on the trustworthiness of the high-confidence facts, and the paper provides no measurement of the factuality of f_h.","section":"Section 5.2 (S1) and Table 2"},{"comment":"The abstract claims experiments 'across four datasets and six models,' but Section 4.1 describes only two datasets (ASQA and LongFact) and five base models (LLaMA-2-7b, LLaMA-2-13b, Vicuna-7b, Vicuna-13b, GPT-3.5-turbo). No additional datasets or models appear in the main text or the appendix. If the experiments were run on only two datasets and five models, the claims of breadth should be corrected; if additional experiments exist, they should be described explicitly. This discrepancy affects the credibility of the reported scope and should be resolved.","section":"Abstract and Section 4.1"}],"minor_comments":[{"comment":"The first sentence is a sentence fragment ('Confidence calibration in LLMs, i.e., aligning their self-assessed confidence with the actual accuracy of their responses, enabling them to self-evaluate the correctness of their outputs.'); it needs a main verb and should be rewritten as a complete sentence.","section":"Abstract"},{"comment":"The summation in Eq. (3) is written as running from n=1 to N, but the terms use the bin index k and the bin set B_k; the summation should be over the number of bins, i.e., k=1 to B.","section":"Eq. (3)"},{"comment":"The row for Vicuna-13b in the Consistency block is labeled 'Vicuna-13bw'; this appears to be a typo for 'Vicuna-13b'.","section":"Table 1, Consistency"},{"comment":"The sentence 'Once facts are extracted, we evaluates both the correctness and relevance of each fact' contains a subject-verb agreement error; it should be 'we evaluate.'","section":"Section 3.2"},{"comment":"The stopping rule is ambiguous: after the maximum number of iterations N is reached, does the model accept the last candidate fact even if its confidence is not higher than the original, or does it revert to the original fact? The current wording says the model will 'return \"NoError\" and make no modifications,' which should be clarified with a precise algorithmic description.","section":"Section 5.2, S3"},{"comment":"Several references are duplicated: Kadavath et al. appears as both 2022a and 2022b, and Guo et al. appears as both 2017a and 2017b. These duplicates should be consolidated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk in this manuscript is the self-correction claim: the acceptance rule is internal to the model and the evaluation is another LLM, so the central contribution of hallucination mitigation is not yet established. An objective evaluation would significantly strengthen the paper. Also, the abstract's 'four datasets and six models' is not supported by the submitted text, which describes only two datasets and five models; please verify whether additional experiments were omitted from this version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fact-level calibration is the solid half; the self-correction claim is the soft half. The F-ECE metric and the three observations are a genuine contribution to calibration evaluation, but ConFix as presented doesn't demonstrate that confidence-guided correction reduces hallucinations, because the acceptance rule trusts confidence increases as a proxy for factual improvement and the evaluation is GPT-4 pairwise judgment rather than objective factuality.\n\nWhat's new: the paper moves calibration from response-level scalars to fact-level vectors, weights correctness by relevance, and defines F-ECE. The observations that fact-level calibration is stricter, that response-level confidence is dominated by the highest-confidence fact, and that within-response confidence variance is high are worth taking seriously. The empirical setup for calibration is standard (three confidence estimation methods, five models, two datasets), and the comparisons to response-level ECE are informative.\n\nSoft spots: the core problem is the self-correction loop. A revision is accepted when the model's re-estimated confidence is higher than the old confidence (Eq. 8). That assumes confidence increase tracks factual accuracy, but the paper never checks that correlation. Table 3 shows LLaMA-2-7b regresses on 80.7% of revised facts, and even the best model, Llama-2-13b, improves only 53.35% versus 27.07% regression. That is a modest effect at best. And the evaluation is GPT-4 pairwise, which can prefer fluent or assertive phrasing over factual correctness. Low recall in error detection (max 30.6%, often much lower) means most hallucinations are never low-confidence, so the method can only correct a small subset. The abstract says four datasets and six models, but the experiments use two datasets and five models; that mismatch should be fixed. There are also no comparisons to baselines like SelfCheckGPT or other self-correction approaches, so the claim that ConFix effectively mitigates hallucinations is unanchored.\n\nThe calibration half does not suffer from these issues. F-ECE is a clean extension of ECE to fact granularity, and the relevance weighting is a reasonable choice. I'd be happy to cite it for the metric and the observations.\n\nWho is this for: anyone working on calibration or hallucination evaluation in long-form generation. The paper deserves a serious referee, but the referee should insist on objective factuality evaluation and baseline comparisons for ConFix, and a corrected abstract.\n\nBottom line: the calibration work is solid enough to justify peer review; the self-correction claim needs more evidence before it is accepted.","headline":"The fact-level calibration metric and observations are a genuine contribution, but the self-correction claim is under-supported because confidence increases are not shown to track factuality and the evaluation is not objective.","tokens_in":15747,"tokens_out":2651,"would_cite":true,"duration_ms":26727,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a language model can use its own high-confidence facts as internal knowledge to correct its low-confidence facts, reducing hallucinations without external retrieval.","keywords":["confidence calibration","fact-level calibration","self-correction","hallucination mitigation","large language models","long-form generation","F-ECE","ConFix"],"falsifier":"Run ConFix on a well-calibrated model, verify every accepted rewrite against an external gold-standard fact base, and compare verified fact-level accuracy before and after; if accepted rewrites do not raise verified accuracy, the confidence-increase rule does not track correctness.","tokens_in":14749,"feed_emoji":"🤖","tokens_out":12404,"duration_ms":119417,"temperature":0.7,"pith_summary":"This paper argues that long-form answers should be calibrated fact by fact, because a single confidence number for the whole response hides which parts are uncertain. It introduces a fact-level calibration framework that scores each atomic fact's correctness and its relevance to the question, and defines F-ECE as the gap between confidence and relevance-weighted correctness. Using that lens, the paper reports three observations: fact-level calibration is a stricter test, response-level overconfidence is driven by a few highly confident facts, and confidence varies widely within a single response. The paper then proposes ConFix, which treats high-confidence facts in the same response as a reference knowledge base and rewrites low-confidence facts, accepting a rewrite only when the model's re-estimated confidence rises. The central claim is that ConFix reduces hallucinations without external knowledge sources such as retrieval systems.","feed_headline":"Confident facts in an answer can correct the uncertain ones","feed_subtitle":"ConFix cuts hallucination in long answers using only the model's own high-confidence facts, no retrieval.","key_machinery":"The load-bearing mechanism is the fact-level confidence vector and the three-step ConFix correction loop. Facts are split into a high-confidence set and a low-confidence set at the mean confidence threshold; the high-confidence set functions as an internal knowledge base. Step two extracts modifiable 'factors' from each low-confidence fact and asks the model to correct only those factors using the high-confidence facts as reference knowledge. Step three re-estimates confidence for the revised fact and accepts it only if new confidence exceeds old confidence, otherwise iterating until a maximum number of rounds. The evaluation metric F-ECE, a binning error between confidence and relevance-weighted correctness, is what lets the paper claim fact-level calibration is a stricter and more informative standard than response-level ECE.","core_discovery":"The central discovery is that a model's confidence, when measured at the granularity of individual facts rather than whole responses, exposes hallucination patterns that response-level calibration hides. The paper defines relevance-weighted correctness per fact and F-ECE, and shows that response-level confidence distributions track only the most confident fact in an answer, which explains overconfidence. It further observes that low-confidence facts are outliers inside otherwise confident responses, making them identifiable targets for correction. ConFix operationalises this by splitting a response into high- and low-confidence facts, rewriting the low-confidence facts with the high-confidence ones as in-context references, and accepting the revision only when the re-estimated confidence exceeds the original. The paper claims this loop improves fact-level accuracy for sufficiently well-calibrated models while requiring no external knowledge.","pith_inferences":["Editorial note: the abstract advertises four datasets and six models, while the body's main tables report two datasets and five base models, so the breadth of the empirical claim is not fully visible in this version.","Beyond the paper: a hybrid design could use the confidence split to decide which facts need external retrieval, reserving the ConFix loop for facts whose re-estimated confidence stays low instead of trusting the loop alone.","Beyond the paper: the accept-if-confidence-rises rule could reward rewrites that merely sound more fluent or generic; an external factuality check on accepted revisions would separate confidence gain from correctness gain."],"forward_implications":["If fact-level calibration is the right lens, response-level ECE scores understate hallucination risk and should be supplemented by F-ECE in model evaluation.","If ConFix works as claimed, long-form answers can be improved at generation time with no retrieval and no fine-tuning, which lowers the cost of hallucination mitigation.","The method's success is conditional on baseline calibration: models with poor fact-level calibration, such as LLaMA-2-7b in the paper's tables, can regress rather than improve, so calibration quality becomes a precondition for self-correction.","The high-confidence-facts-as-knowledge idea implies that a model's own response contains reusable internal knowledge, which can be exploited repeatedly as the correction loop iterates."],"supporting_citations":[{"why":"Supplies the ECE metric and reliability-histogram protocol that F-ECE extends to fact-level calibration.","marker":"Guo et al., 2017a"},{"why":"Provides the LongFact dataset and the SAFE search-augmented factuality evaluator used for correctness scoring.","marker":"Wei et al., 2024"},{"why":"Provides the ASQA dataset of ambiguous factoid questions used for long-form QA evaluation.","marker":"Stelmakh et al., 2022"},{"why":"Supplies the consistency-based confidence estimation method SelfCheckGPT used as one of the three confidence estimators.","marker":"Manakul et al., 2023"},{"why":"Supplies the isTrueLogit confidence estimation method used to derive fact confidence from a model's True/False logit.","marker":"Kadavath et al., 2022a"},{"why":"Supplies the verbalization-based confidence estimation method used to elicit per-fact confidence scores.","marker":"Tian et al., 2023"}],"fun_headline_variants":["Fact-level calibration lets LLMs self-correct","High-confidence facts erase hallucinated ones","ConFix: no retrieval needed to fix LLM lies","Per-fact confidence spots and fixes errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's self-reported confidence is reliable enough both to pick out facts that are actually correct as references and to signal genuine improvement when re-estimated confidence rises after a rewrite.","fun_headline_variants_meta":{"raw":{"variants":["Fact-level calibration lets LLMs self-correct","High-confidence facts erase hallucinated ones","ConFix: no retrieval needed to fix LLM lies","Per-fact confidence spots and fixes errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1129,"prompt_tokens":862,"completion_tokens":267,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":209}},"tokens_in":478,"tokens_out":267,"duration_ms":3170,"temperature":1.0,"reasoning_tokens":209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:31:27.464760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ConFix on a well-calibrated model, verify every accepted rewrite against an external gold-standard fact base, and compare verified fact-level accuracy before and after; if accepted rewrites do not raise verified accuracy, the confidence-increase rule does not track correctness.","supporting_citations":[],"review_version":1}