{"id":"2a2039de-e427-401b-9a37-f863d2f34cdd","arxiv_id":"2505.00065","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ConSens measures context grounding in open-book QA as a sigmoid-transformed log ratio of answer perplexity without context to perplexity with context, reaching ROC AUC 0.88 to 0.93.","lead":"ConSens is a new metric that scores whether an LLM answer is grounded in a supplied context by comparing the model's perplexity with and without that context. It separates grounded from ungrounded answers with ROC AUC between 0.88 and 0.93 and runs on a small 1B-parameter model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unvalidated proxy: using Llama 3.2 1B perplexities to score answers generated by gpt-4o. The reported AUCs may reflect the evaluator's sensitivity to context rather than the generator's reliance.","rationale":"The reader's weakest assumption is the right one. ConSens's formula (Section 2) is a clean likelihood-ratio transform, and the three experiments provide internally consistent evidence: the effect sizes and ROC AUCs are large, and the comparison to Tonic Validate and embedding similarity in Table 2 rules out the trivial explanation that ConSens is just answer-context similarity. Section 3.5's sweep over Llama sizes is real supporting evidence for robustness to the evaluator's capacity. However, none of these checks confronts the gap between the evaluator and the generator. The ground-truth labels are all generation-condition labels; a small evaluator with a strong topical prior could separate them without tracking whether gpt-4o actually used the context. The paper's conclusion goes beyond 'the answer text is more probable under the context according to Llama' to 'the model's answer relies on the context.' That inference requires validating the proxy, which is absent. This is why the paper is plausible but not fully established; a conditional acceptance with a required validation experiment is appropriate. The unrelated Kaggle paragraph and missing code/data are separate issues that do not change the scientific core. No mathematical inconsistency in the metric itself was found.","tokens_in":9987,"tokens_out":6259,"duration_ms":68109,"concrete_test":"Use an open-weights generator (e.g., Llama 3.1 8B Instruct) to produce answers under two conditions: with a conflicting but irrelevant context and without context, using knowledge-conflict items where the model is known to answer from parametric memory (e.g., questions about a changed fact). Compute ConSens for each answer with (a) the generator itself and (b) Llama 3.2 1B. If the evaluator-based ConSens (b) labels answers that ignore the context as 'grounded' (high score) whenever the answer text is lexically related to the context, while the generator's own ratio (a) is near 0 or negative, then the proxy assumption fails. Additionally, compute per-example Spearman correlation between (a) and (b); a low correlation would show the evaluator is measuring its own sensitivity, not the generator's reliance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"This paper claims ConSens measures how strongly a generated answer relies on the provided context, but the metric is always computed with Llama 3.2 1B perplexities, including for answers generated by ChatGPT/gpt-4o (Sections 3.1, 3.2, 3.3). The only support for using an external evaluator is citation [2] plus the evaluator-robustness check in Section 3.5, which varies Llama model size but never compares against the generating model's own probabilities. Since the ground-truth labels in all three experiments are defined by the generation condition (context present/absent, full/partial, correct document included/excluded), any evaluator whose perplexity is reduced by topically matching context will separate the conditions, even if the original generator ignored the context or used it for different reasons. Thus the high ROC AUCs (0.92, 0.93, 0.88) establish that Llama 1B is sensitive to the context manipulation, not that ConSens measures grounding of the answer by the model that produced it. The paper's wording in Section 2 ('the effect of providing context on the probability of output tokens') implicitly refers to the evaluator's probability; bridging this to the generator is exactly the unstated proxy assumption. Unless validated, ConSens may be a measure of textual groundedness of the answer relative to Llama's priors, which is not the same as the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConSens, a metric that quantifies how strongly a generated answer relies on a provided context. ConSens contrasts token-level perplexities of the answer under a context-present condition (PC) and a context-absent condition (PE), taking the log ratio r = log(PE/PC) and mapping it through a sigmoid to [-1, 1]. The authors validate the metric in three experiments: distinguishing context-grounded from ungrounded answers (WikiEval, ROC AUC 0.92), distinguishing full from partial context in a biomedical QA dataset (ROC AUC 0.93), and identifying the most influential document in a three-document RAG setting (ROC AUC 0.88). All ConSens scores are computed with Llama 3.2 1B, including for answers generated by ChatGPT/gpt-4o, and the paper compares against the Tonic Validate answer-consistency metric and embedding-based similarity. The authors argue the metric is lightweight, interpretable, and suitable for real-time evaluation.","tokens_in":10251,"tokens_out":3845,"duration_ms":41395,"significance":"If the central claim holds, ConSens would be a valuable addition to open-book QA evaluation: it is parameter-free in formulation, cheap to compute with a small open model, and avoids the biases and cost of LLM-as-a-judge approaches. The experimental design has strengths: the validation labels in Experiments 2 and 3 are constructed by explicit context manipulation rather than fitted to the metric, the formula contains no parameters fitted to benchmark labels, and the comparison with Tonic Validate and embedding similarity gives useful context. However, the paper's central claim is currently supported only under an unexplicit proxy assumption: that the probability shifts of a small external evaluator (Llama 3.2 1B) reflect the degree to which the original generating model relied on the provided context. The reported AUCs may demonstrate the evaluator's sensitivity to context manipulations rather than the grounding of the answers by ChatGPT/gpt-4o. This concern is load-bearing, because the stated goal is to assess whether model responses are grounded, not whether a particular evaluator is context-sensitive.","major_comments":[{"comment":"The metric is always computed with Llama 3.2 1B perplexities, but the answers being scored were generated by ChatGPT/gpt-4o. Since the ground-truth labels in all three experiments are derived from the generation condition (context present/absent, full/partial, correct document included/excluded), any evaluator whose perplexity is reduced by topically matching context will separate the conditions, regardless of whether the original generator actually relied on the context. The robustness check in Section 3.5 (Table 3) only varies the size of Llama models and never compares against the generating model's own probabilities. To support the central claim, the paper needs either (a) a validation using the generating model's own logits or a faithful surrogate, or (b) a re-scoping of the claims to 'groundedness relative to a reference evaluator,' with additional tests showing that the metric measures more than topical overlap (e.g., counterfactual or paraphrase tests). As written, the high ROC AUCs establish that Llama 1B is sensitive to the context manipulation, not that ConSens measures grounding of the answer by the model that produced it.","section":"Sections 3.1-3.3 and 3.5"},{"comment":"The manually chosen word filter (excluding closed-set words and words appearing in the query, with examples in Table 1) is a free design choice that could materially affect the metric's behavior, yet no ablation is reported. The exact filter definition (the full list of excluded word classes and any stopword list) is not given, and there is no evidence that the results are robust to reasonable variations of this filter. Since the filter is part of the metric as proposed, an ablation study (e.g., with and without the filter, or with different closed-set lists) is needed to establish that the reported AUCs are not an artifact of this particular manual choice.","section":"Section 2, word filter"},{"comment":"The reported ROC AUC values are point estimates with no confidence intervals or statistical tests. Given that the HDI intervals for the underlying score distributions are wide (e.g., the 90% HDI for the Experiment 1 mean difference is [0.2, 2.00]), the claim that ConSens provides a 'reliable signal' is only partially supported. The paper should report bootstrap confidence intervals for the AUCs and ideally a significance test comparing ConSens to the Tonic Validate metric (the differences between 0.92 and 0.92 in Experiment 1, and 0.93 vs 0.75 in Experiment 2, may or may not be meaningful without such intervals).","section":"Tables 2 and 3; Section 3.1-3.3"},{"comment":"The dataset constructed for Experiments 2 and 3 is not released, and the construction procedure is underspecified: the paper does not state how the two 'consecutive sentences' were selected, what prompting was used for gpt-4o to generate questions and answers, how 657 examples survived filtering, or whether the retrieval hits in Experiment 3 were based on topic similarity or exact-match criteria. Since the experiments are central to the validation, this lack of detail and lack of released code/data prevents independent reproduction and limits the paper's contribution as an evaluation metric. The authors should either release the dataset and code or provide a fully specified, reproducible protocol.","section":"Section 3.2 and 3.3; reproducibility"}],"minor_comments":[{"comment":"The paragraph beginning 'Our project is a competition on Kaggle (Predict Future Sales)...' is unrelated to the rest of the paper and appears to be an editorial artifact. It should be removed.","section":"Section 1, after paragraph 2"},{"comment":"The reference for ContextCite provides the arXiv identifier 1904.09675, which is the identifier for BERTScore, not ContextCite. Please correct the reference and verify the author list.","section":"References, [5]"},{"comment":"The formula for P_text as 1/N times the sum of e^{-log p} is nonstandard: perplexity is normally the geometric mean of per-token inverse probabilities (i.e., the exponential of the average negative log-likelihood). The authors should either correct the formula or clarify that they are intentionally averaging per-token perplexities rather than using the standard definition.","section":"Section 2, perplexity formula"},{"comment":"The sentence 'In addition, we proved several important characteristics' overclaims: the paper provides empirical demonstrations, not proofs. Suggest changing 'proved' to 'demonstrated'.","section":"Section 4"},{"comment":"Several sentences contain typos or awkward phrasing, e.g., 'we also checked whether the performance of a ConSens across the three experiments' and 'ConSense's performance' (with inconsistent spelling of ConSens). A thorough proofread and consistent spelling of the metric name are needed.","section":"Sections 3 and 4, English phrasing"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains a clearly extraneous passage about a Kaggle competition ('Predict Future Sales') in the Introduction, which suggests a contamination or copy-paste error in manuscript preparation. The authors should be asked to remove it and to carefully audit the rest of the text for similar artifacts. The citation error for ContextCite is also indicative of a need for careful reference checking. The central scientific concern I raise about the evaluator-proxy assumption is substantive; I would not recommend acceptance until it is addressed either by a validation with the generating model's own probabilities or by a re-scoping of the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper. It proposes ConSens, a metric that scores how much an LLM answer depends on provided context by contrasting perplexity with and without the context, then applying a sigmoid. The formulation is new: they log-ratio the perplexities and filter out closed-class words and words from the question before averaging. The validation is the real contribution. Across three settings — grounded vs ungrounded WikiEval answers, full vs partial biomedical contexts, and RAG with three documents — ROC AUCs are 0.92, 0.93, 0.88, beating or matching the Tonic Validate judge and clearly beating answer-context embedding similarity. They also show the AUC is stable across four Llama evaluator sizes. That is a useful, cheap, local alternative to LLM-as-a-judge.\n\nThe soft spots are real but not fatal. The biggest is the evaluator-proxy assumption: they always compute perplexities with Llama 3.2 1B, even when the answer was generated by gpt-4o. The evidence that the metric tracks the generator's reliance is an external citation plus a robustness check across Llama sizes. That check does not validate the proxy; it only shows different small models behave similarly. Since the labels in all three experiments are defined by the context manipulation, a topically sensitive evaluator would separate conditions regardless of what the original generator actually used. So the high AUCs support ConSens as a measure of textual groundedness relative to the evaluator, not necessarily the generator's internal reliance. The paper's wording in Section 2 claims the latter, so the central claim is plausible but not proven.\n\nSecondary issues: no released code or data, no confidence intervals on the AUCs, and the word filter is hand-picked with no ablation. Also there's an unrelated Kaggle paragraph in the introduction that looks like editorial contamination; it should be deleted. Minor, but a serious revision should clean it up.\n\nWho is this for? People building RAG systems who need a fast, private, real-time faithfulness monitor. The metric is simple enough to implement from any model's logits, and the experiments suggest it works. I'd want the proxy assumption tested by comparing against the generating model's own probabilities before fully trusting it, and code/data released.\n\nRecommendation: send it out. A good referee can separate the useful practical metric from the overclaimed interpretation, and the paper deserves the attention.","headline":"A cheap, interpretable grounding metric with strong ROC AUC across three experiments, but the evaluator-proxy assumption and missing reproducibility keep the central claim from being fully established.","tokens_in":10796,"tokens_out":2408,"would_cite":false,"duration_ms":24280,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces ConSens, a metric that measures how much an LLM's answer to an open-book question relies on the supplied context rather than on the model's own parametric knowledge.","keywords":["ConSens","context grounding","open-book question answering","perplexity","faithfulness","retrieval-augmented generation","LLM evaluation","context attribution"],"falsifier":"Generate answers with the context hidden from the generator but with a topically related filler passage present, then compute ConSens. If scores rise above zero for such filler contexts, the metric is tracking the evaluator's ease with the passage rather than the answer's actual grounding in it.","tokens_in":9749,"feed_emoji":"📊","tokens_out":7119,"duration_ms":66190,"temperature":0.7,"pith_summary":"This paper introduces ConSens, a metric that measures how much an LLM's answer to an open-book question relies on the supplied context rather than on the model's own parametric knowledge. The idea is to compute the perplexity of the answer twice, once with the context present and once with it removed, and to turn the ratio of these perplexities into a score on $[-1,1]$. The authors report that this score separates grounded from ungrounded answers, detects when a critical sentence is removed from the context, and identifies which retrieved document most influenced the answer, with ROC AUC values of 0.92, 0.93, and 0.88. If correct, ConSens provides a cheap, local, interpretable alternative to LLM-as-a-judge evaluation for grounding, useful in retrieval-augmented generation and other knowledge-intensive tasks.","feed_headline":"Perplexity gap reveals whether an LLM answer used the given context","feed_subtitle":"ConSens compares answer perplexity with and without context and matches slower judge models while running on a small model.","key_machinery":"The object that carries the argument is a paired perplexity ratio. Perplexity of the answer is computed as the average token negative log-likelihood under a small instruction-following evaluator model, and the logarithm of the ratio $r = \\log(P_E/P_C)$ is passed through the sigmoid-like transform $\\frac{2}{1+e^{-r}}-1$ to land in $[-1,1]$. Closed-set words and words already present in the question are excluded from the perplexity calculation, so the contrast is not diluted by tokens that are predictable with or without the context. The ratio, rather than either perplexity alone, is what diagnoses grounding; it is also what lets the same procedure identify a critical sentence or the most influential retrieved document by removing segments and observing the drop.","core_discovery":"The paper's central claim is that the strength of context grounding is captured by a single contrast: how much more probable the answer tokens become when the context is provided than when it is empty. Formally, with $P_C$ the answer perplexity given the context and $P_E$ the perplexity with an empty context, the score is $\\mathrm{ConSens} = 2/(1+e^{-r})-1$ where $r=\\log(P_E/P_C)$. Values near 1 mean the context substantially raises answer likelihood; values around and below 0 mean it does not. Across the three validation experiments, the score separates grounded from ungrounded answers (ROC AUC 0.92), full from partial context (0.93), and inclusion from exclusion of the truly relevant retrieved document (0.88), while remaining comparable to or better than a slower LLM-as-a-judge baseline and clearly above answer-context similarity.","pith_inferences":["I would read ConSens as measuring the evaluator's probabilistic sensitivity to the context, not a causal guarantee about the generator; separating those requires controlling the generator's actual dependence on context.","The transform is a scaled log-likelihood ratio: $\\mathrm{ConSens} = \\tanh(r/2)$, so the score can be interpreted as an evidence weight for 'context used' and could be calibrated against error rates.","A natural stress test is paraphrase robustness: if the context contains the answer rewritten rather than verbatim, a drop in ConSens would indicate that the metric partly measures lexical overlap rather than semantic grounding.","Cross-family transfer is untested: using an evaluator from a different small-model family could change the score scale, though the paper's within-family results suggest the ordering is stable."],"forward_implications":["Open-book QA systems can flag answers that ignore the provided context using only a small local model's logits, without paying for or depending on a large judge API.","In retrieval-augmented generation pipelines, the same score can attribute influence to individual retrieved documents, since removing the correct document produces the largest drop in ConSens.","Because the metric uses a ratio rather than absolute perplexity, it avoids known problems of using perplexity as a direct proxy for text quality.","The procedure extends naturally to any task where the question is how much an output depends on a particular input segment, such as summary coverage or prompt-segment importance.","The metric can be computed in a single parallelizable pass, making real-time evaluation feasible on modest hardware."],"supporting_citations":[{"why":"Supplies the paired grounded and ungrounded answer dataset used in Experiment 1.","marker":"[6]"},{"why":"Provides the biomedical abstract collection from which the full/partial context and multi-document retrieval datasets were built.","marker":"[1]"},{"why":"Defines the evaluator model family whose token probabilities are used to compute every perplexity score.","marker":"[16]"},{"why":"Supplies the evidence that small models adhere to provided context, justifying the choice of the evaluator model.","marker":"[2]"},{"why":"Provides the LLM-as-a-judge answer-consistency baseline against which ConSens is compared.","marker":"[23]"},{"why":"Earlier demonstration that token probabilities can support context attribution, which this work extends.","marker":"[5]"},{"why":"Documents pitfalls of absolute perplexity as a text-quality proxy, motivating the ratio-based formulation.","marker":"[7]"}],"fun_headline_variants":["LLM context check: perplexity gap reveals grounding","No judge needed: perplexity contrast scores context use","Perplexity gap: a fast, bias-free context grounding metric","Context reliance measured by perplexity with vs without","One number: how much LLMs lean on provided context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the probability shifts measured by a small, separate evaluator model faithfully reflect how strongly the original answering model relied on the context; if evaluator and generator differ in how they use context, ConSens scores the evaluator, not the answer.","fun_headline_variants_meta":{"raw":{"variants":["LLM context check: perplexity gap reveals grounding","No judge needed: perplexity contrast scores context use","Perplexity gap: a fast, bias-free context grounding metric","Context reliance measured by perplexity with vs without","One number: how much LLMs lean on provided context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1159,"prompt_tokens":922,"completion_tokens":237,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":158}},"tokens_in":538,"tokens_out":237,"duration_ms":3047,"temperature":1.0,"reasoning_tokens":158,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:54:15.697734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate answers with the context hidden from the generator but with a topically related filler passage present, then compute ConSens. If scores rise above zero for such filler contexts, the metric is tracking the evaluator's ease with the passage rather than the answer's actual grounding in it.","supporting_citations":[{"cited_title":"In: Proceedings of the 18th EACL, EACL 2024 - System Demonstr ations, St","cited_arxiv_id":null,"evidence_quote":"Supplies the paired grounded and ungrounded answer dataset used in Experiment 1."},{"cited_title":"Sci Data 10(1), 8 (2023), https://www.nature.com/articles/s41597-022-01920-3 7 CONSENS","cited_arxiv_id":null,"evidence_quote":"Provides the biomedical abstract collection from which the full/partial context and multi-document retrieval datasets were built."},{"cited_title":"tonic.ai","cited_arxiv_id":null,"evidence_quote":"Provides the LLM-as-a-judge answer-consistency baseline against which ConSens is compared."}],"review_version":1}