{"id":"5149dacf-915a-4ae8-bde5-5396c14b2f1d","arxiv_id":"2506.22486","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-small-model ensemble with sentence splitting, z-score normalization, and harmonic mean detects hallucinations in RAG answers with a reported 10% F1 gain over single-model baselines.","lead":"This paper tests whether several small language models, working together, can check whether a large language model's answer matches the documents it was given. The method splits answers into sentences, asks each small model yes or no questions, and combines the answers to flag hallucinations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out evaluation: thresholds and normalization moments are fit on the same data used to report F1, so the claimed 10% improvement is not shown to be predictive.","rationale":"The reader's weakest assumption identifies the same load-bearing problem: the evaluation protocol fits the decision threshold and normalization statistics on the data used to report F1, so the headline improvement is not a predictive performance estimate. This is not a minor methodological quibble; it directly undermines the paper's central empirical claim. A second concern about calibration of P(token1=yes) is related but secondary, because even a calibrated score would still need a threshold chosen without peeking at the test labels. The dataset is small and single-domain, and no code or data is released, which makes independent verification impossible. The proposed idea of ensembling small models for hallucination detection is plausible and cheap to deploy, so this is not a rejection of the concept; rather, the evidence as presented cannot support the 10% F1 claim. The reader's REJECT verdict remains appropriate.","tokens_in":9659,"tokens_out":4830,"duration_ms":56561,"concrete_test":"Hold out a random 20% of questions (not individual responses, to avoid leakage across the three responses generated from the same question). On the remaining 80%, estimate the normalization moments in Eq. 4 and fit each method's decision threshold to maximize F1; then evaluate all methods on the held-out 20%. Repeat over 5 random splits or use 5-fold cross-validation and report the mean and 95% CI of F1 for proposed, P(yes), ChatGPT, Qwen2, and MiniCPM. If the held-out F1 advantage of the proposed method over P(yes)/ChatGPT is not at least the reported 6.6%/11%, or if the ordering reverses, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section V-D and Conclusion: proposed method is 11% and 6.6% better than ChatGPT and P(yes)) rests on F1 scores computed after selecting thresholds \"yielding the highest F1 scores\" on the evaluation set itself. No train/development/test split is described, and Eq. 4 says the normalization means and variances \"can be computed based on previous responses\" without indicating what was done in the reported runs. If the decision threshold and the normalization statistics are estimated from the same responses that are later scored, every reported F1 is an in-sample optimum. With only \"over 100\" question/context/response sets and three response categories, threshold selection on the evaluation data can easily inflate or even create the reported margin. The paper therefore does not establish that the proposed small-model ensemble would achieve a 10% F1 improvement on new responses; the reported number is an upper bound of what the method can achieve after tuning to the test set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hallucination-detection framework for RAG-generated answers. It splits each answer into sentences, prompts two small language models (Qwen2 and MiniCPM) to output P(token1=Yes) for each sentence given the question and retrieved context, normalizes these probabilities per model using Eq. (4), averages them across models, and aggregates the sentence scores via a harmonic mean (Eqs. (5)-(6)). A response is classified as correct if the final score exceeds a threshold. The framework is evaluated on a proprietary HR-policy dataset of 'over 100' question-context-response triples with correct, partial, and wrong answers, and the paper reports F1/precision/recall numbers, claiming a 10% F1 improvement over baselines and specifically 11% over ChatGPT and 6.6% over P(yes) in the partial-response condition.","tokens_in":9783,"tokens_out":9412,"duration_ms":101793,"significance":"The idea is practically attractive: local small models avoid API costs, sentence-level decomposition addresses partially correct answers, and ensembling two SLMs is a simple, clearly specified procedure. The equations are easy to follow, and the comparison with P(yes) and ChatGPT is a useful sanity check. However, the empirical support as presented is not valid evidence for the claim: thresholds, normalization moments, and the aggregation rule are all selected on the same data used to report F1, no held-out evaluation or uncertainty quantification is given, and the dataset is small, single-domain, and unreleased. These issues are load-bearing because the headline numbers are in-sample optima. If a re-analysis with proper validation preserves the margin, the contribution would be a useful practical tool, but the current manuscript does not establish that.","major_comments":[{"comment":"The reported headline results are in-sample optimized. The text states that the thresholds yielding the highest F1 scores were selected (Fig. 3), but no train/validation/test split is described anywhere in Section V. Because the same data are used both to choose the decision threshold and to compute F1, the reported F1 values (e.g., 0.81 vs 0.73/0.76 in the partial condition) are upper bounds of what the method can achieve after tuning, not estimates of predictive performance. This directly undermines the abstract's 10% improvement claim and the conclusion's 11% and 6.6% claims. The authors should either use a held-out test set with the threshold chosen on a development set, or use nested cross-validation, and should report the threshold values used.","section":"Section V-D"},{"comment":"The same evaluation-fitting problem applies to the normalization moments in Eq. (4) and to the choice of aggregation mean. Eq. (4) says the moments can be computed based on previous responses, but the paper does not state what was actually done; Fig. 5 compares five aggregation options and selects the harmonic mean because it yields the best outcomes. Thus the per-model normalization moments, the decision threshold, and the aggregation function are all selected on the evaluation data. Without an independent or preregistered choice for these components, the comparison against P(yes) and ChatGPT is not a fair out-of-sample comparison.","section":"Section V-C/V-E"},{"comment":"No uncertainty quantification is provided. Section V-A reports over 100 question/context/response sets, but Figs. 3-5 report single F1/precision/recall values without error bars, confidence intervals, or significance tests. With this sample size, the reported gaps (e.g., 0.81 vs 0.76 for F1 in the partial condition) could easily be within sampling noise. The authors should provide bootstrap confidence intervals or paired tests (e.g., McNemar) on the classifications, and should report the exact dataset size and per-class counts.","section":"Section V-A/V-D"},{"comment":"The method assumes that P(token1=yes) from the two SLMs is a meaningful correctness signal, but no calibration analysis is provided. Since the decision threshold is selected on the test data, the reported F1 could be an artifact of arbitrary threshold choice even if the scores carry no calibrated probability information. The authors should report calibration curves or at least give exact score distributions with thresholds marked, and show that the chosen threshold is stable across splits or bootstrap resamples.","section":"Section IV-B/V-C"}],"minor_comments":[{"comment":"The equation uses the symbols µm and σm, but the surrounding text refers to µj and σj; please unify the notation and clarify whether σ is the standard deviation (the equation uses a z-score, so it should be).","section":"Eq. (4)"},{"comment":"The harmonic and geometric means require si,j > 0, but normalized scores in Eq. (4) can be negative or zero; the manuscript states that such values are adjusted but never specifies the adjustment. This matters because the final scores depend on the exact transformation.","section":"Eqs. (6) and (8)"},{"comment":"The abstract describes responses with fully and partially correct sentences, but Section V-A says the labels are not applied at the sentence level; please reconcile this wording.","section":"Section V-A"},{"comment":"ROUGE is cited to reference [21], which is the Guo et al. calibration paper; the citation numbering appears to be off, as [2] already cites Lin's ROUGE paper.","section":"Section II"},{"comment":"The exact SLM prompt template, decoding parameters (temperature, max new tokens), and the SpaCy sentence-splitting settings are not given; please add these details for reproducibility.","section":"Section IV-A/V-C"},{"comment":"The histograms are shown without bin counts or exact axis values; the qualitative interpretation would be stronger with the underlying distributions or tables.","section":"Figs. 6-7"},{"comment":"There are minor typos (e.g., 'vectoried' in Section I, 'differ ent' in Section III-B) and the abstract's 10% improvement is not qualified as relative or absolute; please state the comparison condition and whether the improvement is relative or absolute.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely a short industry paper, and the evaluation is not at the level of a serious journal. The main issue is in-sample threshold selection, which invalidates the current quantitative claims. If the editor is willing to consider a major revision, the authors should be required to provide a proper held-out or cross-validated evaluation with confidence intervals, make the dataset or code available, and temper claims of generalizability. If a re-analysis does not preserve the reported margin, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the proposed method—two small models, sentence splitting, per-model normalization, harmonic mean—is a sensible, cheap-to-run idea, and the paper explains it clearly. But the one number that matters, the 10% F1 improvement, is not trustworthy as reported. The threshold is chosen on the evaluation set (Section V-D says 'thresholds yielding the highest F1 scores'), and Eq. 4 says the normalization moments 'can be computed based on previous responses' without saying what was done. So we don't know if the F1s are in-sample optima. With only about 100 question/context/answer triples, that's a real problem.\n\nWhat's actually new: the specific combination of multiple small models with normalization and harmonic mean over sentence scores. Each ingredient is known—P(True), semantic entropy's sentence splitting, small models—but the ensemble-of-verifiers framing is a modest extension, not a new capability. The paper is honest about its scope: it's a practical verification tool for RAG, not a theory of hallucination. The writing is straightforward and the equations are simple and internally consistent.\n\nSoft spots, in order of severity. First, the evaluation protocol: no train/test split, threshold selection on the evaluation data, and no error bars or significance tests. A 6.6–11% margin could easily shrink or vanish with a proper held-out set or bootstrap intervals. Second, the dataset is small and single-domain (HR handbook, one company). That limits any general claim. Third, the baseline P(yes) is a single SLM on the whole response, so it does not isolate the contribution of the ensemble versus just sentence splitting; the paper attributes the gain to multiple models, but the comparison is confounded. Fourth, no code or data release, so the result can't be checked. The distributions in Section V-F are nice descriptive evidence, but they don't fix the in-sample tuning.\n\nWho gets value: someone building a lightweight hallucination checker for internal RAG applications might find the recipe useful as a starting point, but should not trust the reported numbers. The paper deserves a serious referee—it's coherent, the idea is sensible, and the flaw is fixable with a proper split—but if the evaluation isn't redone, the result should not be published as a 10% improvement.","headline":"A sensible small-model ensemble idea, but the claimed 10% F1 gain is not supported because the evaluation tunes the threshold and normalization on the same data.","tokens_in":10325,"tokens_out":2972,"would_cite":false,"duration_ms":26921,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sentence-level yes-probability scores from two small language models detect hallucinations in retrieval-augmented answers better than a ChatGPT-based judge.","keywords":["hallucination detection","small language models","retrieval-augmented generation","answer verification","sentence splitting","P(yes) probability","model ensemble","F1 score"],"falsifier":"A reader could split the dataset by question, estimate $\\mu_m$, $\\sigma_m$, and the threshold on the first half, and score the second half; if the held-out F1 for detecting correct from partial responses does not stay near 0.81, the core claim of a 10 percent improvement is not supported.","tokens_in":9407,"feed_emoji":"✅","tokens_out":4629,"duration_ms":43665,"temperature":0.7,"pith_summary":"This paper argues that hallucinated answers in retrieval-augmented generation can be caught by small language models rather than by large, closed, or API-only ones. The proposed framework splits a generated answer into sentences, asks two small models whether each sentence is supported by the retrieved context, and turns the probability of a 'yes' first token into a single score via normalization, averaging, and the harmonic mean. On a dataset of HR-policy questions with correct, partially correct, and wrong answers, the framework reports F1 scores about 10 percent higher than the ChatGPT-based baseline and 6.6 percent higher than prompting a single small model on the whole response. The payoff would be that cheap, locally hosted models can serve as a scalable verifier for LLM outputs.","feed_headline":"Two small models catch RAG hallucinations better than ChatGPT","feed_subtitle":"Splitting answers into sentences and averaging yes-probabilities across two small models lifts F1 by up to 11 percent.","key_machinery":"The machinery is a three-stage scoring pipeline. A splitter segments the answer into sentences $r_{i,j}$; each small model $m$ assigns a score $s^{(m)}_{i,j} = P(\\text{token}_1 = \\text{yes} \\mid q_i, c_i, r_{i,j})$, the probability that the first generated token is 'yes' when asked to verify the sentence against the context. Scores are normalized per model as $(s^{(m)}_{i,j} - \\mu_m)/\\sigma_m$ to put different models on a common scale, averaged across the $M$ models, and finally combined over sentences by the harmonic mean (restricted to positive values) to yield the response-level score $s_i$. This score is the object whose distribution separates correct, partial, and wrong responses.","core_discovery":"The paper's central claim is that $P(\\text{token}_1 = \\text{yes} \\mid \\text{question}, \\text{context}, \\text{sentence})$, estimated locally from multiple small models and combined across sentences, is a more reliable hallucination signal than asking one large model to judge the whole answer. For each sentence, the framework normalizes each model's yes-probability by its own mean and standard deviation, averages the normalized scores across models, and takes the harmonic mean over sentences to produce a final score. Thresholding this score separates correct answers from wrong ones almost perfectly and from partially correct ones with the best reported F1 of 0.81, outperforming ChatGPT and single-model P(yes) baselines. The authors take this as evidence that multiple small models can effectively verify answers when ground truth is absent.","pith_inferences":["Beyond the paper: if the normalization moments and decision threshold are estimated on held-out data rather than on the evaluation set, the reported F1 gains are likely to shrink, because the paper does not describe a train/test split and the text only says the best thresholds were the ones yielding the highest F1.","Beyond the paper: the same normalized-yes-probability pipeline could be applied to closed-book fact-checking of summaries, replacing a large judge model with a cheap ensemble.","Beyond the paper: a testable extension is to replace the fixed harmonic-mean aggregator with a learned weighting of sentence scores, which could improve detection of partially correct answers where the current best F1 is 0.81.","Beyond the paper: measuring the calibration of $P(\\text{token}_1 = \\text{yes})$ for each model would reveal whether the ensemble gain comes from better score separation or from averaging away per-model miscalibration."],"forward_implications":["If the framework holds, organizations can run hallucination detection entirely on local small models, avoiding API costs and data-exfiltration concerns.","Sentence splitting is central: the same averaging process fails to separate partial from correct answers when applied to whole responses, so per-sentence checking is what buys the improvement.","Combining multiple small models outperforms either model alone, implying that model diversity, not scale, contributes to verification accuracy.","The harmonic mean over sentences outperforms geometric, arithmetic, min, and max aggregators on partial answers, suggesting that the weakest-sentence penalty is the useful inductive bias.","The approach transfers to any domain where question, answer, and retrieved context are available, not just HR-policy questions."],"supporting_citations":[{"why":"Supplies the yes/no P(True) prompting approach and the ChatGPT-based baseline that the framework extends and compares against.","marker":"[3]"},{"why":"Provides retrieval-augmented generation as the setting in which responses are generated with retrieved context.","marker":"[4]"},{"why":"Supplies the vectorized-database mechanism for retrieving the context that the verifier checks against.","marker":"[5]"},{"why":"Defines small language models (100M to 5B parameters) and gives evidence that they can perform the required tasks.","marker":"[6]"},{"why":"Provides Qwen2, one of the two small models used in the proposed ensemble.","marker":"[35]"},{"why":"Provides MiniCPM, the second small model used in the proposed ensemble.","marker":"[36]"}],"fun_headline_variants":["Small model ensembles detect RAG hallucinations 10% better","Sentence-level yes scores from small LLMs beat ChatGPT","Multiple small models outperform ChatGPT at hallucination detection","Harmonic mean of small-model yes scores catches RAG lies","Splitting answers improves small-model hallucination detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the per-model normalization moments and the decision threshold are available without being fit to the same responses that are later scored; if they are tuned on the evaluation data, the reported F1 values are optimized, not predictive.","fun_headline_variants_meta":{"raw":{"variants":["Small model ensembles detect RAG hallucinations 10% better","Sentence-level yes scores from small LLMs beat ChatGPT","Multiple small models outperform ChatGPT at hallucination detection","Harmonic mean of small-model yes scores catches RAG lies","Splitting answers improves small-model hallucination detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1744,"prompt_tokens":920,"completion_tokens":824,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":746}},"tokens_in":536,"tokens_out":824,"duration_ms":8472,"temperature":1.0,"reasoning_tokens":746,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:57.837486+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could split the dataset by question, estimate $\\mu_m$, $\\sigma_m$, and the threshold on the first half, and score the second half; if the held-out F1 for detecting correct from partial responses does not stay near 0.81, the core claim of a 10 percent improvement is not supported.","supporting_citations":[],"review_version":1}