ConSens: Assessing context grounding in open-book question answering

Adriana Correia; Ivan Vankov; Matyo Ivanov; Victor Botev

REVIEW 4 major objections 5 minor 31 references

ConSens: Assessing context grounding in open-book question answering

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces ConSens, a metric that measures how much an LLM's answer to an open-book question relies on the supplied context rather than on the model's own parametric knowledge.

desk verdict A cheap, interpretable grounding metric with strong ROC AUC across three experiments, but the evaluator-proxy assumption and missing reproducibility keep the central claim from being fully established. read the letter →

arxiv 2505.00065 v1 pith:PEUZORFE submitted 2025-04-30 cs.CL cs.LG

Ivan Vankov , Matyo Ivanov , Adriana Correia , Victor Botev This is my paper · ORCID

classification cs.CLcs.LG

keywords ConSenscontextgroundingopen-bookquestionansweringperplexityfaithfulnessretrieval-augmentedgenerationLLMevaluationattribution

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ConSens, a metric that measures how much an LLM's answer to an open-book question relies on the supplied context rather than on the model's own parametric knowledge. The idea is to compute the perplexity of the answer twice, once with the context present and once with it removed, and to turn the ratio of these perplexities into a score on $[-1,1]$. The authors report that this score separates grounded from ungrounded answers, detects when a critical sentence is removed from the context, and identifies which retrieved document most influenced the answer, with ROC AUC values of 0.92, 0.93, and 0.88. If correct, ConSens provides a cheap, local, interpretable alternative to LLM-as-a-judge evaluation for grounding, useful in retrieval-augmented generation and other knowledge-intensive tasks.

What carries the argument

The object that carries the argument is a paired perplexity ratio. Perplexity of the answer is computed as the average token negative log-likelihood under a small instruction-following evaluator model, and the logarithm of the ratio $r = \log(P_E/P_C)$ is passed through the sigmoid-like transform $\frac{2}{1+e^{-r}}-1$ to land in $[-1,1]$. Closed-set words and words already present in the question are excluded from the perplexity calculation, so the contrast is not diluted by tokens that are predictable with or without the context. The ratio, rather than either perplexity alone, is what diagnoses grounding; it is also what lets the same procedure identify a critical sentence or the most influential retrieved document by removing segments and observing the drop.

What would settle it

Generate answers with the context hidden from the generator but with a topically related filler passage present, then compute ConSens. If scores rise above zero for such filler contexts, the metric is tracking the evaluator's ease with the passage rather than the answer's actual grounding in it.

Watch

Extended reading notes

Core claim

The paper's central claim is that the strength of context grounding is captured by a single contrast: how much more probable the answer tokens become when the context is provided than when it is empty. Formally, with $P_C$ the answer perplexity given the context and $P_E$ the perplexity with an empty context, the score is $\mathrm{ConSens} = 2/(1+e^{-r})-1$ where $r=\log(P_E/P_C)$. Values near 1 mean the context substantially raises answer likelihood; values around and below 0 mean it does not. Across the three validation experiments, the score separates grounded from ungrounded answers (ROC AUC 0.92), full from partial context (0.93), and inclusion from exclusion of the truly relevant retrieved document (0.88), while remaining comparable to or better than a slower LLM-as-a-judge baseline and clearly above answer-context similarity.

Load-bearing premise

The load-bearing premise is that the probability shifts measured by a small, separate evaluator model faithfully reflect how strongly the original answering model relied on the context; if evaluator and generator differ in how they use context, ConSens scores the evaluator, not the answer.

Editorial extensions

If this is right

Open-book QA systems can flag answers that ignore the provided context using only a small local model's logits, without paying for or depending on a large judge API.
In retrieval-augmented generation pipelines, the same score can attribute influence to individual retrieved documents, since removing the correct document produces the largest drop in ConSens.
Because the metric uses a ratio rather than absolute perplexity, it avoids known problems of using perplexity as a direct proxy for text quality.
The procedure extends naturally to any task where the question is how much an output depends on a particular input segment, such as summary coverage or prompt-segment importance.
The metric can be computed in a single parallelizable pass, making real-time evaluation feasible on modest hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

I would read ConSens as measuring the evaluator's probabilistic sensitivity to the context, not a causal guarantee about the generator; separating those requires controlling the generator's actual dependence on context.
The transform is a scaled log-likelihood ratio: $\mathrm{ConSens} = \tanh(r/2)$, so the score can be interpreted as an evidence weight for 'context used' and could be calibrated against error rates.
A natural stress test is paraphrase robustness: if the context contains the answer rewritten rather than verbatim, a drop in ConSens would indicate that the metric partly measures lexical overlap rather than semantic grounding.
Cross-family transfer is untested: using an evaluator from a different small-model family could change the score scale, though the paper's within-family results suggest the ordering is stable.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know about this paper. It proposes ConSens, a metric that scores how much an LLM answer depends on provided context by contrasting perplexity with and without the context, then applying a sigmoid. The formulation is new: they log-ratio the perplexities and filter out closed-class words and words from the question before averaging. The validation is the real contribution. Across three settings — grounded vs ungrounded WikiEval answers, full vs partial biomedical contexts, and RAG with three documents — ROC AUCs are 0.92, 0.93, 0.88, beating or matching the Tonic Validate judge and clearly beating answer-context embedding similarity. They also show the AUC is stable across four Llama evaluator sizes. That is a useful, cheap, local alternative to LLM-as-a-judge.

The soft spots are real but not fatal. The biggest is the evaluator-proxy assumption: they always compute perplexities with Llama 3.2 1B, even when the answer was generated by gpt-4o. The evidence that the metric tracks the generator's reliance is an external citation plus a robustness check across Llama sizes. That check does not validate the proxy; it only shows different small models behave similarly. Since the labels in all three experiments are defined by the context manipulation, a topically sensitive evaluator would separate conditions regardless of what the original generator actually used. So the high AUCs support ConSens as a measure of textual groundedness relative to the evaluator, not necessarily the generator's internal reliance. The paper's wording in Section 2 claims the latter, so the central claim is plausible but not proven.

Secondary issues: no released code or data, no confidence intervals on the AUCs, and the word filter is hand-picked with no ablation. Also there's an unrelated Kaggle paragraph in the introduction that looks like editorial contamination; it should be deleted. Minor, but a serious revision should clean it up.

Who is this for? People building RAG systems who need a fast, private, real-time faithfulness monitor. The metric is simple enough to implement from any model's logits, and the experiments suggest it works. I'd want the proxy assumption tested by comparing against the generating model's own probabilities before fully trusting it, and code/data released.

Recommendation: send it out. A good referee can separate the useful practical metric from the overclaimed interpretation, and the paper deserves the attention.

Referee Report

4 major / 5 minor

Summary. The paper introduces ConSens, a metric that quantifies how strongly a generated answer relies on a provided context. ConSens contrasts token-level perplexities of the answer under a context-present condition (PC) and a context-absent condition (PE), taking the log ratio r = log(PE/PC) and mapping it through a sigmoid to [-1, 1]. The authors validate the metric in three experiments: distinguishing context-grounded from ungrounded answers (WikiEval, ROC AUC 0.92), distinguishing full from partial context in a biomedical QA dataset (ROC AUC 0.93), and identifying the most influential document in a three-document RAG setting (ROC AUC 0.88). All ConSens scores are computed with Llama 3.2 1B, including for answers generated by ChatGPT/gpt-4o, and the paper compares against the Tonic Validate answer-consistency metric and embedding-based similarity. The authors argue the metric is lightweight, interpretable, and suitable for real-time evaluation.

Significance. If the central claim holds, ConSens would be a valuable addition to open-book QA evaluation: it is parameter-free in formulation, cheap to compute with a small open model, and avoids the biases and cost of LLM-as-a-judge approaches. The experimental design has strengths: the validation labels in Experiments 2 and 3 are constructed by explicit context manipulation rather than fitted to the metric, the formula contains no parameters fitted to benchmark labels, and the comparison with Tonic Validate and embedding similarity gives useful context. However, the paper's central claim is currently supported only under an unexplicit proxy assumption: that the probability shifts of a small external evaluator (Llama 3.2 1B) reflect the degree to which the original generating model relied on the provided context. The reported AUCs may demonstrate the evaluator's sensitivity to context manipulations rather than the grounding of the answers by ChatGPT/gpt-4o. This concern is load-bearing, because the stated goal is to assess whether model responses are grounded, not whether a particular evaluator is context-sensitive.

major comments (4)

[Sections 3.1-3.3 and 3.5] The metric is always computed with Llama 3.2 1B perplexities, but the answers being scored were generated by ChatGPT/gpt-4o. Since the ground-truth labels in all three experiments are derived from the generation condition (context present/absent, full/partial, correct document included/excluded), any evaluator whose perplexity is reduced by topically matching context will separate the conditions, regardless of whether the original generator actually relied on the context. The robustness check in Section 3.5 (Table 3) only varies the size of Llama models and never compares against the generating model's own probabilities. To support the central claim, the paper needs either (a) a validation using the generating model's own logits or a faithful surrogate, or (b) a re-scoping of the claims to 'groundedness relative to a reference evaluator,' with additional tests showing that the metric measures more than topical overlap (e.g., counterfactual or paraphrase tests). As written, the high ROC AUCs establish that Llama 1B is sensitive to the context manipulation, not that ConSens measures grounding of the answer by the model that produced it.
[Section 2, word filter] The manually chosen word filter (excluding closed-set words and words appearing in the query, with examples in Table 1) is a free design choice that could materially affect the metric's behavior, yet no ablation is reported. The exact filter definition (the full list of excluded word classes and any stopword list) is not given, and there is no evidence that the results are robust to reasonable variations of this filter. Since the filter is part of the metric as proposed, an ablation study (e.g., with and without the filter, or with different closed-set lists) is needed to establish that the reported AUCs are not an artifact of this particular manual choice.
[Tables 2 and 3; Section 3.1-3.3] The reported ROC AUC values are point estimates with no confidence intervals or statistical tests. Given that the HDI intervals for the underlying score distributions are wide (e.g., the 90% HDI for the Experiment 1 mean difference is [0.2, 2.00]), the claim that ConSens provides a 'reliable signal' is only partially supported. The paper should report bootstrap confidence intervals for the AUCs and ideally a significance test comparing ConSens to the Tonic Validate metric (the differences between 0.92 and 0.92 in Experiment 1, and 0.93 vs 0.75 in Experiment 2, may or may not be meaningful without such intervals).
[Section 3.2 and 3.3; reproducibility] The dataset constructed for Experiments 2 and 3 is not released, and the construction procedure is underspecified: the paper does not state how the two 'consecutive sentences' were selected, what prompting was used for gpt-4o to generate questions and answers, how 657 examples survived filtering, or whether the retrieval hits in Experiment 3 were based on topic similarity or exact-match criteria. Since the experiments are central to the validation, this lack of detail and lack of released code/data prevents independent reproduction and limits the paper's contribution as an evaluation metric. The authors should either release the dataset and code or provide a fully specified, reproducible protocol.

minor comments (5)

[Section 1, after paragraph 2] The paragraph beginning 'Our project is a competition on Kaggle (Predict Future Sales)...' is unrelated to the rest of the paper and appears to be an editorial artifact. It should be removed.
[References, [5]] The reference for ContextCite provides the arXiv identifier 1904.09675, which is the identifier for BERTScore, not ContextCite. Please correct the reference and verify the author list.
[Section 2, perplexity formula] The formula for P_text as 1/N times the sum of e^{-log p} is nonstandard: perplexity is normally the geometric mean of per-token inverse probabilities (i.e., the exponential of the average negative log-likelihood). The authors should either correct the formula or clarify that they are intentionally averaging per-token perplexities rather than using the standard definition.
[Section 4] The sentence 'In addition, we proved several important characteristics' overclaims: the paper provides empirical demonstrations, not proofs. Suggest changing 'proved' to 'demonstrated'.
[Sections 3 and 4, English phrasing] Several sentences contain typos or awkward phrasing, e.g., 'we also checked whether the performance of a ConSens across the three experiments' and 'ConSense's performance' (with inconsistent spelling of ConSens). A thorough proofread and consistent spelling of the metric name are needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ConSens is a parameter-free perplexity ratio validated against independently constructed context-manipulation labels.

full rationale

The paper's central quantity is defined by the closed-form equation ConSens = 2/(1+e^{-r}) - 1 with r = log(PE/PC), and no parameter in this formula is fitted to the benchmark labels. The validation labels in Experiment 1 come from the external WikiEval dataset's generation condition (grounded vs ungrounded); in Experiments 2 and 3 the labels are constructed by manipulating the context (full vs partial, inclusion vs exclusion of the source abstract), independently of the metric's score. The word filter and the choice of Llama 3.2 1B as evaluator are fixed design decisions, not fitted to maximize ROC AUC, and the evaluator-robustness check in Section 3.5 varies only the evaluator size. The one self-citation, WISDM [3], appears in related work as an example of semantic similarity metrics and plays no role in the derivation or validation. The use of an external evaluator to score answers generated by gpt-4o is an empirical validity assumption about evaluator sensitivity to context rather than a circular reduction: there is no equation or fitted value that makes the reported AUCs follow from the definition of ConSens. The unrelated Kaggle passage in Section 1 is a copy-paste artifact and does not affect the derivation chain. Accordingly, no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The metric itself has no fitted numeric parameters, but it depends on two hand-chosen components, the word filter and the evaluator model, and on three domain assumptions about label validity and the evaluator proxy. No invented entities are introduced.

free parameters (2)

word exclusion filter = hand-chosen rule (closed-set words and query-overlap words removed)
In Section 2, the paper excludes closed-set words and words that appear in the user query from the perplexity computation. This is a discretionary design choice with no ablation or formal justification, and it directly affects the score.
evaluator model selection = Llama 3.2 1B, with robustness checks on Llama 3.2 3B, Llama 3.1 8B, and Llama 3.1 70B
The paper chooses a small model for perplexity calculation because small models are claimed to adhere to context. Robustness across Llama sizes is shown, but the choice is still hand-picked rather than derived from the problem.

assumptions (3)

domain assumption Token probabilities of an external evaluator model reflect the generating model's context grounding.
Used to compute all ConSens scores; the evaluator is Llama 3.2 1B while many answers come from gpt-4o. The paper gives a plausibility argument in Section 3 but no direct validation against the generating model.
domain assumption The gpt-4o-generated labels in Experiments 2 and 3 are correct ground truth.
Questions are generated from two consecutive sentences, answers from the full abstract, and the two removed sentences are assumed to be the critical grounding. This labeling is not independently verified.
ad hoc to paper Excluding closed-set and query-overlap words improves the metric's signal.
Stated in Section 2 with an intuitive rationale; no theoretical or empirical proof that the filtering increases validity rather than just shifting scores.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ConSens: Assessing context grounding in open-book question answering." pith.science (2026). https://pith.science/paper/PEUZORFE

@misc{pith2026250500065,
  author       = {Pith},
  title        = {Pith review of: ConSens: Assessing context grounding in open-book question answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PEUZORFE}},
  note         = {Machine review of arXiv:2505.00065}
}

read the original abstract

Large Language Models (LLMs) have demonstrated considerable success in open-book question answering (QA), where the task requires generating answers grounded in a provided external context. A critical challenge in open-book QA is to ensure that model responses are based on the provided context rather than its parametric knowledge, which can be outdated, incomplete, or incorrect. Existing evaluation methods, primarily based on the LLM-as-a-judge approach, face significant limitations, including biases, scalability issues, and dependence on costly external systems. To address these challenges, we propose a novel metric that contrasts the perplexity of the model response under two conditions: when the context is provided and when it is not. The resulting score quantifies the extent to which the model's answer relies on the provided context. The validity of this metric is demonstrated through a series of experiments that show its effectiveness in identifying whether a given answer is grounded in the provided context. Unlike existing approaches, this metric is computationally efficient, interpretable, and adaptable to various use cases, offering a scalable and practical solution to assess context utilization in open-book QA systems.

Figures

Figures reproduced from arXiv: 2505.00065 by the authors.

**Figure 2.** Experiment 2 results. Values on the x axis represen [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Experiment 3 results. The x axis represents the min [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 11 canonical work pages

[1]

Sci Data 10(1), 8 (2023), https://www.nature.com/articles/s41597-022-01920-3 7 CONSENS

Attal, K., Ondov, B., Demner-Fushman, D.: A dataset for p lain language adaptation of biomedical abstracts. Sci Data 10(1), 8 (2023), https://www.nature.com/articles/s41597-022-01920-3 7 CONSENS

work page 2023
[2]

, Mei, L., Fang, J., Li, Z., Wei, F., Deng, W ., Sun, F., Zhang, Q., Liu, S.: Context-dpo: Aligning language models for context-faithfulness

Bi, B., Huang, S., Wang, Y ., Y ang, T., Zhang, Z., Huang, H. , Mei, L., Fang, J., Li, Z., Wei, F., Deng, W ., Sun, F., Zhang, Q., Liu, S.: Context-dpo: Aligning language models for context-faithfulness. CoRR (2024), https://doi.org/10.48550/arXiv.2412.15280

arXiv doi:10.48550/arxiv.2412.15280 2024
[3]

In: Proceed- ings of the 6th International Workshop on Mining Scientiﬁc P ublications

Botev, V ., Marinov, K., Schäfer, F.: Word importance-ba sed similarity of documents metric (WISDM): Fast and scalable document similarity metric for analysis o f scientiﬁc documents. In: Proceed- ings of the 6th International Workshop on Mining Scientiﬁc P ublications. pp. 17–23. ACM (2017), https://dl.acm.org/doi/10.1145/3127526.3127530

work page arXiv 2017
[4]

In: Proceedings of the 2022 EMNLP , EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022

Chen, H.T., Zhang, M.J.Q., Choi, E.: Rich knowledge sour ces bring complex knowledge con- ﬂicts: Recalibrating models to reﬂect conﬂicting evidence . In: Proceedings of the 2022 EMNLP , EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pp. 2292–2307. ACL (2022), https://doi.org/10.18653/v1/2022.emnlp-main.146

doi:10.18653/v1/2022.emnlp-main.146 2022
[6]

In: Proceedings of the 18th EACL, EACL 2024 - System Demonstr ations, St

ES, S., James, J., Anke, L.E., Schockaert, S.: Ragas: Aut omated evaluation of retrieval augmented generation. In: Proceedings of the 18th EACL, EACL 2024 - System Demonstr ations, St. Julians, Malta, March 17-22, 2024. pp. 150–158. ACL (2024), https://aclanthology.org/2024.eacl-demo.16

work page 2024
[7]

Fang, L., Wang, Y ., Liu, Z., Zhang, C., Jegelka, S., Gao, J ., Ding, B., Wang, Y .: What is wrong with perplexity for long-context language modeling? CoRR (2024), https://doi.org/10.48550/arXiv.2410.23771

arXiv doi:10.48550/arxiv.2410.23771 2024
[8]

In: Proceedings of the 2024 NAACL (V ol- ume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16 -21, 2024

Fu, J., Ng, S., Jiang, Z., Liu, P .: Gptscore: Evaluate as y ou desire. In: Proceedings of the 2024 NAACL (V ol- ume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16 -21, 2024. pp. 6556–6576. ACL (2024), https://doi.org/10.18653/v1/2024.naacl-long.365

doi:10.18653/v1/2024.naacl-long.365 2024
[9]

Gao, M., Hu, X., Ruan, J., Pu, X., Wan, X.: LLM-based NLG ev aluation: Current status and challenges (2024), http://arxiv.org/abs/2402.01383

arXiv 2024

Show all 31 references

[10]

CoRR (2024), https://doi.org/10.48550/arXiv.2410.15821

Hawkins, W ., Mittelstadt, B.D., Russell, C.: The effec t of ﬁne-tuning on language model toxicity. CoRR (2024), https://doi.org/10.48550/arXiv.2410.15821

2024 doi arXiv
[11]

Jacovi, A., Wang, A., Alberti, C., Tao, C., Lipovetz, J. , Olszewska, K., Haas, L., Liu, M., Keating, N., Blo- niarz, A., Sarouﬁm, C., Fry, C., Marcus, D., Kukliansky, D., Tomar, G.S., Swirhun, J., Xing, J., Wang, L., Gurumurthy, M., Aaron, M., Ambar, M., Fellinger, R., Wang, ...

2025 arXiv
[12]

In: Proceedings of the 2024 EMNLP (EMNLP)

Kim, S., Suk, J., Longpre, S., Lin, B.Y ., Shin, J., Welle ck, S., Neubig, G., Lee, M., Lee, K., Seo, M.: Prometheus 2: An open source language model specialized in evaluating o ther language models. In: Proceedings of the 2024 EMNLP (EMNLP). pp. 4334–4353. Miami, Florida, USA...

2024 doi
[13]

In: Advances in Neural Information Processing Sy stems 33: NeurIPS 2020 (2020), https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html

Lewis, P .S.H., Perez, E., Piktus, A., Petroni, F., Karp ukhin, V ., Goyal, N., Küttler, H., Lewis, M., Yih, W ., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augm ented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Sy stems 3...

2020
[14]

In: Text Summarization Branches Out

Lin, C.Y .: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81. ACL (2004), https://aclanthology.org/W04-1013/

2004
[15]

CoRR (2023), https://doi.org/10.48550/arXiv.2308.08747

Luo, Y ., Y ang, Z., Meng, F., Li, Y ., Zhou, J., Zhang, Y .: An empirical study of catastrophic forgetting in large lan- guage models during continual ﬁne-tuning. CoRR (2023), https://doi.org/10.48550/arXiv.2308.08747

2023 doi arXiv
[16]

CoRR (2024), https://doi.org/10.48550/arXiv.2407.21783

Meta, L.T.A..: The llama 3 herd of models. CoRR (2024), https://doi.org/10.48550/arXiv.2407.21783

2024 doi arXiv
[17]

OpenAI: Gpt-4 technical report (2024), https://arxiv.org/abs/2303.08774

2024 arXiv
[18]

In: Findings of the Associa tion for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024

Pham, Q., Ngo, H., Luu, A.T., Nguyen, D.Q.: Who’s who: La rge language models meet knowledge conﬂicts in practice. In: Findings of the Associa tion for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024. pp. 1 0142–10151. ACL (2024), https://aclan...

2024
[19]

Rashkin, H., Nikolaev, V ., Lamm, M., Aroyo, L., Collins , M., Das, D., Petrov, S., Tomar, G.S., Turc, I., Reitter, D.: Measuring attribution in natural language generation m odels. Comput. Linguistics 49(4), 777–840 (2023), https://doi.org/10.1162/coli_a_00486

2023 doi
[20]

Roberts, A., Raffel, C., Shazeer, N.: How much knowledg e can you pack into the parameters of a language model? In: Proceedings of the 2020 EMNLP , EMNLP 2020, Online , November 16-20, 2020. pp. 5418–5426. ACL (2020), https://doi.org/10.18653/v1/2020.emnlp-main.437 8 CONSENS

2020 doi
[21]

In: Proc eedings of the 2024 NAACL (V olume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2 024

Saad-Falcon, J., Khattab, O., Potts, C., Zaharia, M.: A RES: an automated evaluation frame- work for retrieval-augmented generation systems. In: Proc eedings of the 2024 NAACL (V olume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2 024. pp. 338–354. ACL (2024)...

2024 doi
[22]

In: The Twelfth ICLR, ICLR 2024, Vienna, Austria, May 7-11, 2 024

Sclar, M., Choi, Y ., Tsvetkov, Y ., Suhr, A.: Quantifyin g language models’ sensitivity to spu- rious features in prompt design or: How I learned to start wor rying about prompt formatting. In: The Twelfth ICLR, ICLR 2024, Vienna, Austria, May 7-11, 2 024. OpenReview.net (202...

2024
[23]

tonic.ai

Tonic.ai: Tonic Validate guide | Tonic Validate — docs. tonic.ai. https://docs.tonic.ai/validate (2023), [Accessed 28-03-2025]

2023
[24]

CoRR (2022), https://doi.org/10.48550/arXiv.2210.05892

Wang, Y ., Deng, J., Sun, A., Meng, X.: Perplexity from PL M is unreliable for evaluating text quality. CoRR (2022), https://doi.org/10.48550/arXiv.2210.05892

2022 doi arXiv
[25]

Wu, K., Wu, E., Zou, J.: ClashEval: Quantifying the tug- of-war between an LLM’s internal prior and external evidence (2024), https://arxiv.org/abs/2404.10198

2024 arXiv
[26]

In: The Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austr ia, May 7-11, 2024

Xie, J., Zhang, K., Chen, J., Lou, R., Su, Y .: Adaptive ch ameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conﬂicts . In: The Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austr ia, May 7-11, 2024. O...

2024
[27]

In: Proceedings of the 2024 EMNLP , EMNLP 2024, Miami, FL, USA, November 12-16, 2024

Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y ., Xu, W .: Knowledge conﬂicts for llms: A survey. In: Proceedings of the 2024 EMNLP , EMNLP 2024, Miami, FL, USA, November 12-16, 2024. pp. 8541–8565. ACL (2024), https://aclanthology.org/2024.emnlp-main.486

2024
[28]

CoRR (2024), https://doi.org/10.48550/arXiv.2404.09077

Y ang, Z., Zhu, Z.: Curiousllm: Elevating multi-docume nt QA with reasoning-infused knowledge graph prompt- ing. CoRR (2024), https://doi.org/10.48550/arXiv.2404.09077

2024 doi arXiv
[29]

CoRR (2024), https://doi.org/10.48550/arXiv.2410.02736

Y e, J., Wang, Y ., Huang, Y ., Chen, D., Zhang, Q., Moniz, N ., Gao, T., Geyer, W ., Huang, C., Chen, P ., Chawla, N.V ., Zhang, X.: Justice or prejudice? quantify ing biases in llm-as-a-judge. CoRR (2024), https://doi.org/10.48550/arXiv.2410.02736

2024 doi arXiv
[30]

In: Advances in Neural Information Processing Sys tems 34: NeurIPS 2021, NeurIPS 2021, December 6-14, 2021, virtual

Y uan, W ., Neubig, G., Liu, P .: Bartscore: Evaluating ge nerated text as text gen- eration. In: Advances in Neural Information Processing Sys tems 34: NeurIPS 2021, NeurIPS 2021, December 6-14, 2021, virtual. pp. 27263 –27277 (2021), https://proceedings.neurips.cc/paper/2021...

2021
[31]

CoRR (2024), https://doi.org/10.48550/arXiv.2403.10131

Zhang, T., Patil, S.G., Jain, N., Shen, S., Zaharia, M., Stoica, I., Gonzalez, J.E.: RAFT: adapting language model to domain speciﬁc RAG. CoRR (2024), https://doi.org/10.48550/arXiv.2403.10131

2024 doi arXiv
[32]

Zhang, T., Kishore, V ., Wu, F., Weinberger, K.Q., Artzi , Y .: BERTScore: Evaluating text generation with BERT (2020), http://arxiv.org/abs/1904.09675 9

2020 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Sci Data 10(1), 8 (2023), https://www.nature.com/articles/s41597-022-01920-3 7 CONSENS

Attal, K., Ondov, B., Demner-Fushman, D.: A dataset for p lain language adaptation of biomedical abstracts. Sci Data 10(1), 8 (2023), https://www.nature.com/articles/s41597-022-01920-3 7 CONSENS

work page 2023

[2] [2]

, Mei, L., Fang, J., Li, Z., Wei, F., Deng, W ., Sun, F., Zhang, Q., Liu, S.: Context-dpo: Aligning language models for context-faithfulness

Bi, B., Huang, S., Wang, Y ., Y ang, T., Zhang, Z., Huang, H. , Mei, L., Fang, J., Li, Z., Wei, F., Deng, W ., Sun, F., Zhang, Q., Liu, S.: Context-dpo: Aligning language models for context-faithfulness. CoRR (2024), https://doi.org/10.48550/arXiv.2412.15280

arXiv doi:10.48550/arxiv.2412.15280 2024

[3] [3]

In: Proceed- ings of the 6th International Workshop on Mining Scientiﬁc P ublications

Botev, V ., Marinov, K., Schäfer, F.: Word importance-ba sed similarity of documents metric (WISDM): Fast and scalable document similarity metric for analysis o f scientiﬁc documents. In: Proceed- ings of the 6th International Workshop on Mining Scientiﬁc P ublications. pp. 17–23. ACM (2017), https://dl.acm.org/doi/10.1145/3127526.3127530

work page arXiv 2017

[4] [4]

In: Proceedings of the 2022 EMNLP , EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022

Chen, H.T., Zhang, M.J.Q., Choi, E.: Rich knowledge sour ces bring complex knowledge con- ﬂicts: Recalibrating models to reﬂect conﬂicting evidence . In: Proceedings of the 2022 EMNLP , EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pp. 2292–2307. ACL (2022), https://doi.org/10.18653/v1/2022.emnlp-main.146

doi:10.18653/v1/2022.emnlp-main.146 2022

[5] [6]

In: Proceedings of the 18th EACL, EACL 2024 - System Demonstr ations, St

ES, S., James, J., Anke, L.E., Schockaert, S.: Ragas: Aut omated evaluation of retrieval augmented generation. In: Proceedings of the 18th EACL, EACL 2024 - System Demonstr ations, St. Julians, Malta, March 17-22, 2024. pp. 150–158. ACL (2024), https://aclanthology.org/2024.eacl-demo.16

work page 2024

[6] [7]

Fang, L., Wang, Y ., Liu, Z., Zhang, C., Jegelka, S., Gao, J ., Ding, B., Wang, Y .: What is wrong with perplexity for long-context language modeling? CoRR (2024), https://doi.org/10.48550/arXiv.2410.23771

arXiv doi:10.48550/arxiv.2410.23771 2024

[7] [8]

In: Proceedings of the 2024 NAACL (V ol- ume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16 -21, 2024

Fu, J., Ng, S., Jiang, Z., Liu, P .: Gptscore: Evaluate as y ou desire. In: Proceedings of the 2024 NAACL (V ol- ume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16 -21, 2024. pp. 6556–6576. ACL (2024), https://doi.org/10.18653/v1/2024.naacl-long.365

doi:10.18653/v1/2024.naacl-long.365 2024

[8] [9]

Gao, M., Hu, X., Ruan, J., Pu, X., Wan, X.: LLM-based NLG ev aluation: Current status and challenges (2024), http://arxiv.org/abs/2402.01383

arXiv 2024