{"id":"37333f4c-8048-4899-8fad-9f0016fecd14","arxiv_id":"2509.10753","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HalluField flags LLM hallucinations using a hand-weighted temperature-perturbation of token-level 'free energy' (negative log-likelihood) and Shannon entropy.","lead":"This paper introduces HalluField, a hallucination detector that scores an LLM response by measuring how its negative log-likelihood and per-token entropy change when the sampling temperature is perturbed. It needs no auxiliary model or fine-tuning, and on question-answering benchmarks it matches or slightly beats semantic-entropy methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3 predicts low δU for hallucinations, but Appendix A reports hallucinated responses have higher ΔB, ΔP, and Δ(TH), so δU is higher — contradicting the paper's central theoretical mechanism.","rationale":"The reader's weakest assumption concerned hand-picked weights, λ, and lack of held-out validation. While that is a legitimate concern, the most load-bearing problem is more severe: the paper's own empirical appendix contradicts the theoretical mechanism that the central claim rests on. The abstract claims the method is 'grounded in a principled physical interpretation' and detects hallucinations by identifying 'unstable or erratic behavior' in an energy landscape, with Section 3 explicitly predicting low δU for hallucinated responses. But Appendix A reports that hallucinated responses have higher values of all three components that make up δU, and since the weights in Eq. 17 are positive, the total variation must also be higher for hallucinated responses. This is an internal inconsistency, not a question of external validity. If the sign issue is real, the stated physics narrative is false even though the empirical score might still separate classes. The paper would need a major revision to replace the hypothesized mechanism or show that the detector's sign convention is the opposite of what the text implies. No code or error bars are provided, which further prevents verifying the sign, but the contradiction can be checked directly from the authors' reported class-conditional means. Given that the central theoretical claim is falsified by the paper's own data, the appropriate verdict is REJECT in the current form, rather than CONDITIONAL acceptance.","tokens_in":17648,"tokens_out":6995,"duration_ms":81584,"concrete_test":"Obtain per-response HalluField scores δU_Q for one model/dataset pair (e.g., LLaMA-2-7B-Chat on TriviaQA) and compute the mean and distribution for hallucinated vs non-hallucinated responses using the exact settings in §C. If mean δU_Q(hallucinated) > mean δU_Q(correct), the Section 3 low-δU hypothesis is refuted by the method's own data; the paper must then either abandon that hypothesis or demonstrate that the reported AUCs use the opposite sign convention. This test is decisive because the class-conditional sign of δU_Q is fixed by Eq. 17 and the positive weights.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3's motivating hypothesis is explicit: 'For a hallucinated response whose internal energy is already high, increasing its temperature has little effect on its internal energy, resulting in a low total variation δU. In contrast, raising the temperature would change a correct, low-energy response into an incorrect, high-energy one, leading to a high δU.' Appendix A then reports the opposite: 'Across different temperatures, models, and datasets, we consistently observe that hallucinated responses yield statistically higher values of all three measures: the base energy variation ΔBQ, the change in potential ΔPQ, and the change in entropy Δ(THQ).' The HalluField score is δU_Q = δF_Q + δ(TH_Q) = ΣΔT [(T0+ΔT)ΔB_Q + ΔP_Q/(T0+ΔT)^2 + Δ(TH_Q)/(T0+ΔT)^2] (Eq. 17). All weights are positive for the temperature sets used in §C, so if hallucinated responses have higher ΔB, ΔP, and Δ(TH), they necessarily have higher δU_Q. Thus the detector's own data imply the mechanism runs opposite to the stated hypothesis: hallucinated responses are the high-variation states, not low-variation states. The paper's 'principled physical interpretation' is therefore internally inconsistent with its empirical findings. This is not a matter of tuning constants or held-out validation; it challenges the central theoretical claim of the paper. Either the hypothesis in §3 must be replaced by the empirical observation (high δU = hallucination) or the detector's sign convention and AUC direction must be reconciled with the stated low-δU prediction.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HalluField, a hallucination-detection method for LLMs that treats a response as a token path with an associated free energy F and entropy H, both derived from token-level logits. A 'total variation' δU = δF + δ(TH) is computed as a weighted sum of temperature-perturbed differences (Eq. 17), and δU (or δU + λ·SE, called HalluFieldSE) is used as a hallucination score. The method is evaluated on four open-domain QA datasets and multiple open-weight models, reporting AUC/accuracy competitive with or better than Semantic Entropy, KLE, and other baselines, and claiming very low per-query processing time. The paper presents the approach as grounded in a variational principle and the first law of thermodynamics, and claims state-of-the-art performance.","tokens_in":18082,"tokens_out":5620,"duration_ms":64926,"significance":"If the method's claims held, a logit-only hallucination detector requiring no auxiliary LLM and negligible processing cost would be practically valuable. The paper does provide a broad experimental comparison across models and datasets and directly uses token-level probabilities, which is a strength. However, the central theoretical mechanism is contradicted by the paper's own Appendix A, and the key weights and hyperparameters are selected from the same data used for the final evaluation. Consequently, the 'principled physical interpretation' is not established, and the reported performance advantages may not be out-of-sample. The contribution, as it stands, is closer to a heuristic temperature-perturbation score than to a field-theoretic detector.","major_comments":[{"comment":"The stated hypothesis and the empirical findings have opposite signs. Section 3 hypothesizes that hallucinated responses have high internal energy, so raising temperature changes their energy little, giving low δU, while correct low-energy responses become high-energy, giving high δU. Appendix A reports the reverse: 'Across different temperatures, models, and datasets, we consistently observe that hallucinated responses yield statistically higher values of all three measures: the base energy variation ΔB_Q, the change in potential ΔP_Q, and the change in entropy Δ(TH_Q).' Since all weights in Eq. (17) are positive for the temperature sets used ({1.0,1.5,2.0} or {1.0,2.0,3.0}), higher ΔB, ΔP, and Δ(TH) imply higher δU. Thus hallucinated responses have higher δU, directly contradicting the proposed mechanism. This is not a tuning or notation issue; it invalidates the 'internal-energy' expl","section":"Section 3, Eq. (17), Appendix A"},{"comment":"The weights w_B = T0 + ΔT, w_P = 1/(T0+ΔT)^2, w_TH = 1/(T0+ΔT)^2 and the mixing constant λ = 2 are chosen after inspecting the same data on which the method is later evaluated. The text explicitly says 'Figure 2 shows the advantage of using weighted sums across variations' and that Figure 2 is based on LLaMa-2-7B-Chat across datasets; the experimental section then evaluates on the same datasets and model family. No held-out validation, cross-validation, or sensitivity analysis is provided for these choices, nor for the temperature sets and number of perturbations. Reported margins over baselines are often only 0.01–0.02 AUC, so without out-of-sample selection of these heuristics the 'state-of-the-art' claim is not credible.","section":"Section 4, Eq. (14), Figure 2, Section 5"},{"comment":"The 'derivation' of δF is not a variational principle. The theoretical parametrized variation in Eq. (30) requires evaluating the same token sequence at T0+ΔT. In practice, Eq. (11) replaces the unobserved F[τ(r0,T0+ΔT)] with an expectation over independently generated paths, and Eq. (12) introduces a new 'change in potential' term with an indicator I(r ≠ r0). Neither replacement follows from Eq. (30), and Eq. (9) already states that the total variation is 'heuristically defined' as a weighted sum. Consequently, δU is a weighted combination of ad hoc statistics, not a discrete implementation of δU = TδH + W. The thermodynamic vocabulary is used as an analogy and supplies no independent constraint on the weights, so the paper's claim of a 'principled physical interpretation' is overstated.","section":"Section B.2, Eq. (30); Section 4, Eqs. (10)–(13)"}],"minor_comments":[{"comment":"Reported AUC differences among methods are often very small (≤0.01–0.02), but no error bars, multiple runs, or significance tests are given. Several rows show KLE or SE matching or exceeding HalluField, so the statement that HalluField 'achieves state-of-the-art ... across models and datasets' is too strong.","section":"Section 5, Tables 2–5"},{"comment":"The runtime of 1×10^-4 seconds for HalluField excludes generation time, yet the algorithm requires generating 50 perturbed responses per temperature. This makes the runtime comparison misleading for end-to-end use.","section":"Section 5, Table 6"},{"comment":"Notation is inconsistent: 'LLaMa' vs 'LLaMA' appear in different places; Eq. (1) states T ≥ 0 but the softmax with T = 0 is undefined; denominators in Eqs. (14)–(17) implicitly require T0 + ΔT > 0, which should be stated.","section":"Throughout"},{"comment":"The claim that hallucinated responses yield 'statistically higher' values is not accompanied by error bars, confidence intervals, or a description of the statistical test used.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper's central issue is not merely incomplete validation: the proposed physical mechanism is internally inconsistent with the paper's own empirical appendix. The detector might still work as a heuristic, but the 'field-theoretic' framing and the first-law analogy would need to be dropped or thoroughly reworked, and the weight choices would need to be validated on held-out data. Given that the reported improvements over baselines are often within 0.01–0.02 AUC, the empirical contribution alone is currently too thin to justify acceptance without substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nQuick take: this paper has a genuinely usable hallucination detector, but its advertised 'principled physical interpretation' is internally contradicted by its own Appendix A, and the key weights are selected after peeking at the test benchmarks. It deserves a serious referee, but the revision needs real work.\n\nWhat's new: HalluField uses temperature-perturbed token-level negative log-likelihood and Shannon entropy to build a hallucination score δU. Unlike semantic entropy or KLE, it needs no auxiliary model. The experiments are broad—nine models, four datasets, following Farquhar et al.'s protocol—and HalluField is competitive, often slightly better or matching SE/KLE, with trivial per-query cost. That is a real result, and the runtime advantage is meaningful.\n\nThe soft spots are not minor. Section 3 hypothesizes that hallucinated responses have high internal energy, so raising temperature changes them little, giving low δU; correct responses go from low to high energy, giving high δU. Appendix A reports the opposite: hallucinated responses have statistically higher ΔB, ΔP, and Δ(TH) across temperatures, models, and datasets. Since all weights in Eq. 17 are positive, hallucinated responses necessarily have higher δU. The appendix even says this 'aligns with our hypothesis,' which is backwards. So the detector's actual empirical sign contradicts the theoretical story. The method can still work as a high-δU-means-hallucination heuristic, but the paper's central physical claim is wrong and must be fixed or removed.\n\nSecond, the weight functions in Eq. 14 and λ=2 are chosen after inspecting Figure 2, which uses the same datasets and model family as the final evaluation. No held-out validation or sensitivity analysis. The differences between HalluField and SE/KLE are often within a few hundredths of AUC; without error bars and a proper validation split, the 'state-of-the-art' claim is not justified. No code is released, which compounds the issue.\n\nThe derivations in the appendix are careful, and the empirical work is reproducible in principle. The authors are clearly capable. This is not a desk-reject; it's a revise-and-resubmit with focused requirements: reconcile the sign of δU with the hypothesis or drop the hypothesis, run a validation split for the weights, report variance, and publish code.\n\nWho should read it: anyone working on hallucination detection or LLM uncertainty who wants a cheap, auxiliary-free baseline. I'd take it for reading group, but I wouldn't cite the theoretical framing until it's fixed.\n\nMy recommendation: send to peer review. The empirical method is useful, and the internal contradiction, while serious, is fixable. It deserves referee time.\n\nBest,","headline":"A genuinely fast, auxiliary-free hallucination detector, but the paper's own appendix contradicts its central theoretical mechanism, and the weights are tuned on the test benchmarks.","tokens_in":18540,"tokens_out":4859,"would_cite":false,"duration_ms":49708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HalluField detects hallucinations by measuring how an LLM's token-level free energy and entropy respond to temperature changes, using the change in internal energy as a hallucination score.","keywords":["hallucination detection","large language models","thermodynamics","variational principles","free energy","semantic entropy","logit-based uncertainty","temperature perturbation"],"falsifier":"Run HalluField with the published weights and λ on a held-out set of models and datasets that were not used to tune the weights (for instance, a different model family and a domain like medical or legal QA). If the AUC gap over semantic entropy on those unseen conditions is within noise or negative, the weight selection is overfit to the four QA benchmarks. A second check: deliberately flatten the logits of a known-correct response so that all tokens have nearly equal probability; the energy variation should still classify it correctly if the thermodynamic hypothesis is the real driver.","tokens_in":17554,"feed_emoji":"⚡","tokens_out":5096,"duration_ms":55769,"temperature":0.7,"pith_summary":"HalluField treats an LLM's answer as a thermodynamic system: each token is assigned a free energy and an entropy from the model's own logits, and the change in internal energy under a temperature shift becomes a hallucination score. The paper's hypothesis is that hallucinated responses already occupy high-energy, unstable states, so heating them barely changes their energy, whereas correct low-energy responses jump to higher energy when heated. This total variation is computed as a weighted sum over several temperature perturbations using a parametrized discrete variational principle, with no auxiliary LLM, fine-tuning, or hidden-state probing. The authors report that HalluField outperforms semantic entropy and kernel language entropy in AUC/accuracy across several open-domain QA datasets and multiple model families, with near-instant runtime. A variant that adds semantic entropy, HalluFieldSE, is the strongest detector overall.","feed_headline":"Hallucinations show up as high-energy token-path instabilities","feed_subtitle":"A logits-only detector that beats semantic entropy on four QA benchmarks needs no auxiliary model.","key_machinery":"The parametrized discrete variation of a functional on token paths, defined as a weighted sum over temperature increments ΔT of the difference of the functional evaluated at T0+ΔT and at T0. The functionals are the free energy F_Q(τ) = −Σ_i log P(τ_i | τ_