Pith. sign in

REVIEW 4 major objections 5 minor 29 references

PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single perplexity gap between a question-answer pair and the answer alone ranks LLM response quality without ground truth.

desk verdict The absolute-value perplexity difference conflates good and bad answers, and the paper's own correlations are too weak to support its 'correlates well' claim. read the letter →

arxiv 2411.15320 v1 pith:2ZP6NFLW submitted 2024-11-22 cs.CL cs.AI

Gerald Friedland , Xin Huang , Yueying Cui , Vishaal Kapoor , Ashish Khetan , Sanjiv Das This is my paper · ORCID
classification cs.CLcs.AI
keywords perplexityLLMevaluationunsupervisedqualitymetricmodelrankingquestionansweringinformationtheoryprompt-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PPLqa is a proposed answer-quality score for generative language models that needs no ground truth, no human annotations, and no evaluation prompt: it is simply the absolute difference between the perplexity of a question-answer pair and the perplexity of the answer alone, with a lower value taken to mean a better response. The paper argues that this one number captures both the writing quality of the response (through perplexity) and its relevance and consistency with the question (through the surprise of seeing the question and answer together). If the claim holds, anyone choosing among LLMs for a task can rank candidate models by feeding their responses through the metric, bypassing the costly creation of expert-written reference answers. The paper reports that across four subject domains and on the MT-Bench human-judgment set, PPLqa matches or beats prompt-based metrics such as GPTScore and G-EVAL, and correlates with both human rankings and rankings from an LLM judge. Because the score is computed from token probabilities alone, it is language-independent and could apply to any trained language model.

What carries the argument

The load-bearing object is the perplexity differential of Equation (2), computed with a separate, performant evaluator LLM that assigns token probabilities. Perplexity is the exponentiated average negative log-probability of a token sequence, interpretable as the number of binary choices needed to encode the text. The paper's argument is that $|\mathrm{PPL}(X(qa)) - \mathrm{PPL}(X(a))|$ measures the surprise of a response given a question: a coherent, on-topic response adds little perplexity when the question is prepended, while an unrelated or hallucinated response makes the joint sequence much less predictable. The absolute-value differencing is the mechanism that removes length effects and isolates the context-dependent component of surprise, so the same fixed formula can rank responses without any task-specific prompt.

What would settle it

Take a set of question-answer triples where the answer is factually correct, subtly wrong, or off-topic, with the three answers matched for length and style, and check whether PPLqa strictly orders them from best to worst; any failure to separate the correct from the subtly wrong answer would refute the claim that lower PPLqa means better quality.

Watch

Extended reading notes

Core claim

The central discovery is that the quantity $\mathrm{PPLqa} = |\mathrm{PPL}(X(qa)) - \mathrm{PPL}(X(a))|$, where $X(qa)$ is the concatenation of the prompt and response and $X(a)$ is the response alone, orders LLM responses by quality: a lower value of PPLqa indicates a better response. The reasoning is that perplexity of the response measures coherence and fluency, while the gap between the joint and marginal perplexities measures how surprising, and therefore how relevant and consistent, the response is given the question. Explicitly, an unrelated answer raises $\mathrm{PPL}(X(qa))$, and differencing also normalizes the well-known tendency for perplexity to fall as text gets longer. On the authors' experiments, PPLqa performed in the same range as GPTScore and G-EVAL on binary and four-way comparisons across macroeconomics, astronomy, AI, and electronics questions, and it outperformed both on MT-Bench when used without prompts.

Load-bearing premise

The load-bearing premise is that the gap in perplexity between a question-and-answer pair and the answer alone tracks how good the answer is, rather than how long, stylized, or familiar to the scoring model it is.

Editorial extensions

If this is right

  • A user comparing LLMs for a task can rank them with only a list of questions and the models' answers, no expert-written references.
  • Long-form question answering is where PPLqa shows its strongest agreement with human judgments; short binary or multiple-choice responses are outside its intended scope.
  • No evaluation prompt needs to be designed; in the MT-Bench experiments, removing prompts improved PPLqa's accuracy while prompt-based competitors required them.
  • The metric is white-box and fully transparent in computation, allowing inspection of the softmax activations that produce the perplexity values.
  • Because no language-specific processing is used, the same formula applies to any language a trained evaluator model can score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely untested consequence is that PPLqa inherits the biases of the evaluator LLM; swapping the scoring model could change rankings, so a multi-evaluator average would be a natural extension.
  • The differencing operation suggests a normalized variant, such as the ratio $\mathrm{PPL}(X(qa))/\mathrm{PPL}(X(a))$, which might give more stable comparisons across domains or languages where absolute perplexity scales differ.
  • A controlled test that holds response length and topic fixed while varying only factual correctness would separate genuine quality tracking from length or style effects; the paper does not report such a test.
  • The same formula could be applied to retrieval-augmented generation by replacing the question with the retrieved context plus question, providing a prompt-free way to score groundedness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PPLqa, an unsupervised quality metric for generative LLM responses, defined as the absolute difference between the perplexity of the concatenated question-answer (PPL(X(qa))) and the perplexity of the answer alone (PPL(X(a))). The authors claim that a lower PPLqa value implies better response quality, that the metric subsumes coherence, fluency, relevance, and consistency, and that it performs comparably to GPTScore and G-EVAL on the authors' four-domain dataset and better than those baselines on MT-Bench. Experiments compare PPLqa, GPTScore, and G-EVAL against human and Claude-based rankings using F1, accuracy, MCC, and Kendall's tau.

Significance. If validated, a prompt-free, unsupervised, language-independent quality metric would be practically valuable for model selection and would reduce reliance on ground-truth annotation or LLM-as-a-judge prompting. The paper's strengths include a simple functional form, a fixed evaluator model (Mistral 7B Instruct V0.2), reproducible experimental setup, and direct comparison with two popular baselines. The metric is falsifiable and the paper ships enough detail to recompute it. However, the central claim that 'a lower value of PPLqa implies better response quality' is not theoretically derived, and the empirical correlations are weak and sometimes negative, so the significance is only conditional on a substantial revision that addresses the definitional and evidential gaps.

major comments (4)
  1. [Section 3, Eq. (2)] The absolute value in PPLqa = |PPL(X(qa)) - PPL(X(a))| breaks the claimed monotonicity between the metric and answer quality. A correct, highly entailed answer will make PPL(X(qa)) much lower than PPL(X(a)) because the question makes answer tokens predictable, yielding a large absolute difference that the paper would label as low quality. Conversely, a generic or loosely related answer with similar perplexities in both conditions yields a small difference and is labeled high quality. The paper's own intuition in Section 3 ('If the response is unrelated to the question, then PPL(X(qa)) will be high') describes only one of the two regimes; the opposite regime is equally consistent with Eq. (2). Please provide a derivation or controlled failure-case analysis showing that the problematic regime does not dominate, or replace the absolute value with a signed difference whose direction is justified by the intended quality ordering.
  2. [Section 3, paragraph on length normalization] The claim that 'differencing normalizes across responses that are of different lengths' is not supported by the paper's own appendix. Appendix Table 5 shows that PPL(X(qa)) is negatively correlated with response length for all four listed models across most domains, and PPL(X(a)) is expected to behave similarly. Because the relationship between perplexity and length is nonlinear and model-dependent, simple subtraction of two length-dependent quantities does not cancel the length effect. The paper should report the correlation of PPLqa itself with response length and, if the correlation is substantial, either adjust the metric or explicitly limit the claim to responses of comparable length. As written, the metric likely rewards longer responses regardless of quality.
  3. [Section 4, Table 4 and abstract] The overall Kendall's tau of PPLqa against Claude rankings is 0.137 (p=0.01), which is lower than GPTScore's 0.197, and G-EVAL's value is negative (-0.074). A p-value of 0.01 only indicates that the correlation is distinguishable from zero; it does not indicate a strong or 'well' correlation as claimed in the abstract. The per-domain tau values are significant only for Macroeconomics (0.234, p=0.02) and are close to zero for AI (0.049), Astronomy (0.126, p=0.22), and Electronics (0.026, p=0.80). The authors should report confidence intervals or effect sizes, and temper the abstract's 'correlates well' claim. The claim that PPLqa 'performs as well as other related metrics' is contradicted by Table 4, where GPTScore achieves the highest overall correlation.
  4. [Section 4, Experiment I and Tables 1-2] Several reported MCC values are near zero or negative, which undermines the claim that PPLqa aligns with human or Claude rankings. For example, in Table 1, PPLqa (w/o prompt) has overall MCC 0.14, and MCC values of 0.01 in both AI and Electronics; in Table 2, PPLqa (w/prompt) has negative MCC in all four domains. In Table 3, PPLqa with prompts on MT-Bench yields MCC = -0.274. The paper should reconcile these weak results with the strong wording in the abstract, and should also clarify the orientation of the two F1 values in Tables 1-3 (label 0 vs. label 1) so the reader can interpret the binary classification results.
minor comments (5)
  1. [Section 3, after Eq. (2)] The phrase 'subsumes, but is not explicitly based on, coherence and fluency ... and relevance and consistency' is vague; please specify which terms in Eq. (2) are intended to capture each property, or state explicitly that these properties are emergent rather than designed.
  2. [Section 4, Table 4] The layout of Table 4 is confusing: each domain has two rows, the first giving tau and the second giving the p-value, but the column headers do not distinguish them. Use a sub-table or parenthetical notation such as '0.137 (p=0.01)' to make the numbers self-explanatory.
  3. [Section 4, Experiment II] The exact prompts used for 'PPLqa (w/prompt)' are not shown; the prompt set from GPTScore is mentioned but an example of only the coherence prompt is given. Please provide the full prompt set in an appendix or a supplementary file.
  4. [Section 5, Limitations] The reference to Friedland (2024) for the 'information-theoretic interpretation' is not sufficient for a self-contained argument; a brief derivation or a concise explanation of why the perplexity differential is information-theoretically meaningful would strengthen the paper.
  5. [Section 6, Conclusions] The statement that the metric is a 'white-box' metric because 'one may investigate each neuron's activation in the softmax layer' is overstated; perplexity is a scalar aggregate over token probabilities and does not expose per-neuron activations. Please revise the wording.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; PPLqa is defined independently of the rankings it is tested against and is evaluated against external human and LLM labels.

full rationale

The central metric is defined in Equation (2) as PPLqa = |PPL(X(qa)) - PPL(X(a))| with no parameters fitted to human judgments, Claude rankings, or MT-Bench labels, so no fitted input is being renamed as a prediction. The claim that a lower PPLqa implies better response quality is an assumption about the direction of the metric, but it is not derived from the evaluation data; the experiments test this assumption against external ground truth (human labels, Claude 3, MT-Bench). The only self-citation is the passing remark in Section 5 that 'The information-theoretic interpretation of such models is explained in Friedland (2024)', which is not used to justify Equation (2) or any experimental conclusion, so it is not load-bearing. Potential concerns about the absolute value in Equation (2) not being monotone in relevance, or about length confounds documented in Appendix Table 5, are correctness and validity risks rather than circularity: they do not show that the metric's output reduces to its own inputs by construction. The paper also explicitly acknowledges in Section 6 that theoretical analysis is left for future work, which further confirms that no derivation is being smuggled in through self-citation. The evaluation is self-contained against external benchmarks, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numerical parameters are fitted in the PPLqa formula; the only input choices are the evaluator LLM and the concatenation order, which are treated as domain assumptions. The central claim rests on five listed axioms, most of which are asserted without formal support.

assumptions (5)
  • domain assumption The absolute value of the perplexity difference |PPL(qa) - PPL(a)| quantifies response quality, namely coherence, fluency, relevance, and consistency.
    Section 3, after Eq. (2), states 'This metric embodies two attributes' and 'If the response is unrelated to the question, then PPL(X(qa)) will be high.' No derivation or counterexample analysis is given.
  • domain assumption Lower PPLqa implies better response quality.
    Section 3 states 'A lower value of PPLqa implies better response quality.' This assumption is used throughout the experiments.
  • domain assumption Taking the difference between the two perplexities normalizes across responses of different lengths.
    Section 3 says 'a useful byproduct of differencing... normalizes across responses that are of different lengths.' Appendix Table 5 only documents length correlation for PPL(X(qa)), not for PPLqa.
  • domain assumption Mistral 7B Instruct V0.2 provides a reliable evaluator probability model for all tested domains.
    Section 4 states 'the underlying model used in PPLqa... across all experiments is Mistral 7B Instruct V0.2.' No sensitivity analysis or justification is provided.
  • standard math Aggregating Kendall's tau across questions is a valid comparison metric for evaluator agreement.
    Section 4, 'Rank Correlations', Eqs. (3)-(4), uses standard rank correlation, though statistical independence across questions is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models." pith.science (2026). https://pith.science/paper/2ZP6NFLW

@misc{pith2026241115320,
  author       = {Pith},
  title        = {Pith review of: PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZP6NFLW}},
  note         = {Machine review of arXiv:2411.15320}
}
read the original abstract

We propose PPLqa, an easy to compute, language independent, information-theoretic metric to measure the quality of responses of generative Large Language Models (LLMs) in an unsupervised way, without requiring ground truth annotations or human supervision. The method and metric enables users to rank generative language models for quality of responses, so as to make a selection of the best model for a given task. Our single metric assesses LLMs with an approach that subsumes, but is not explicitly based on, coherence and fluency (quality of writing) and relevance and consistency (appropriateness of response) to the query. PPLqa performs as well as other related metrics, and works better with long-form Q\&A. Thus, PPLqa enables bypassing the lengthy annotation process required for ground truth evaluations, and it also correlates well with human and LLM rankings.

Figures

Figures reproduced from arXiv: 2411.15320 by the authors.

Figure 1
Figure 1. Sample questions generated by a LLM for four topics: AI, astronomy, electronics, and macroeconomics [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 13 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bai, Y.; Ying, J.; Cao, Y.; Lv, X.; He, Y.; Wang, X.; Yu, J.; Zeng, K.; Xiao, Y.; Lyu, H.; Zhang, J.; Li, J.; and Hou, L. 2023. Benchmarking Foundation Models with Language - Model -as-an- Examiner . ArXiv:2306.04181 [cs]

    arXiv 2023

  4. [4]

    Banerjee, S.; and Lavie, A. 2005. METEOR : An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments . In Goldstein, J.; Lavie, A.; Lin, C.-Y.; and Voss, C., eds., Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , 65--72. Ann Arbor, Michigan: Association...

    work page 2005

  5. [5]

    J.; and Jurman, G

    Chicco, D.; Warrens, M. J.; and Jurman, G. 2021. The Matthews correlation coefficient (MCC) is more informative than Cohen’s Kappa and Brier score in binary classification assessment. Ieee Access, 9: 78368--78381

    work page 2021

  6. [6]

    Es, S.; James, J.; Espinosa Anke, L.; and Schockaert, S. 2024. RAGA s: Automated Evaluation of Retrieval Augmented Generation. In Aletras, N.; and De Clercq, O., eds., Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150--158. St. Julians, Malta: Association for Computation...

    work page 2024

  7. [7]

    Fomicheva, M.; Sun, S.; Yankovskaya, L.; Blain, F.; Guzmán, F.; Fishel, M.; Aletras, N.; Chaudhary, V.; and Specia, L. 2020. Unsupervised Quality Estimation for Neural Machine Translation . Transactions of the Association for Computational Linguistics, 8: 539--555

    work page 2020

  8. [8]

    Friedland, G. 2024. Information- Driven Machine Learning : Data Science as an Engineering Discipline . Cham: Springer International Publishing. ISBN 978-3-031-39476-8 978-3-031-39477-5

    work page 2024

Show all 29 references
  1. [9]

    Friel, R.; and Sanyal, A. 2023. ChainPoll : A High Efficacy Method for LLM Hallucination Detection

    2023

  2. [10]

    Fu, J.; Ng, S.-K.; Jiang, Z.; and Liu, P. 2023. GPTScore : Evaluate as You Desire . ArXiv:2302.04166 [cs]

    2023 arXiv

  3. [11]

    Gao, L.; Tow, J.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; McDonell, K.; Muennighoff, N.; Phang, J.; Reynolds, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; and Zou, A. 2021. A framework for few-shot language model evaluation

    2021

  4. [12]

    A.; and Zettlemoyer, L

    Gonen, H.; Iyer, S.; Blevins, T.; Smith, N. A.; and Zettlemoyer, L. 2022. Demystifying Prompts in Language Models via Perplexity Estimation . ArXiv:2212.04037 [cs]

    2022 arXiv

  5. [13]

    Gu, Y.; Tafjord, O.; Kuehl, B.; Haddad, D.; Dodge, J.; and Hajishirzi, H. 2024. OLMES : A Standard for Language Model Evaluations . ArXiv:2406.08446 [cs]

    2024 arXiv

  6. [14]

    Liang, P.; and et al. 2022. Holistic Evaluation of Language Models . ArXiv:2211.09110 [cs]

    2022 arXiv

  7. [15]

    Lin, C.-Y. 2004. ROUGE : A Package for Automatic Evaluation of Summaries . In Text Summarization Branches Out , 74--81. Barcelona, Spain: Association for Computational Linguistics

    2004

  8. [16]

    Lin, Y.-T.; and Chen, Y.-N. 2023. LLM - Eval : Unified Multi - Dimensional Automatic Evaluation for Open - Domain Conversations with Large Language Models . ArXiv:2305.13711 [cs]

    2023 arXiv

  9. [17]

    Liu, Y.; Iter, D.; Xu, Y.; Wang, S.; Xu, R.; and Zhu, C. 2023. G- Eval : NLG Evaluation using Gpt -4 with Better Human Alignment . In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2511--2522....

    2023

  10. [18]

    OpenAI. 2023. GPT -4 Technical Report . ArXiv:2303.08774 [cs]

    2023 arXiv

  11. [19]

    Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. BLEU : a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ACL '02, 311--318. USA: Association for Computational Linguistics

    2002

  12. [20]

    Powers, D. M. 2020. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061

    2020 arXiv

  13. [21]

    Srivastava, A.; and et al. 2022. Beyond the Imitation Game : Quantifying and extrapolating the capabilities of language models. ArXiv:2206.04615 [cs, stat]

    2022 arXiv

  14. [22]

    L.; and Parikh, D

    Vedantam, R.; Zitnick, C. L.; and Parikh, D. 2015. CIDEr : Consensus -based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) , 4566--4575. ISSN: 1063-6919

    2015

  15. [23]

    Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019. SuperGLUE : a stickier benchmark for general-purpose language understanding systems. In Proceedings of the 33rd International Conference on Neural Information Processin...

    2019

  16. [24]

    Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018. GLUE : A Multi - Task Benchmark and Analysis Platform for Natural Language Understanding . In Linzen, T.; Chrupała, G.; and Alishahi, A., eds., Proceedings of the 2018 EMNLP Workshop BlackboxNLP : Analy...

    2018

  17. [25]

    Wang, C.; Cheng, S.; Guo, Q.; Yue, Y.; Ding, B.; Xu, Z.; Wang, Y.; Hu, X.; Zhang, Z.; and Zhang, Y. 2024. Evaluating open- QA evaluation. In Proceedings of the 37th International Conference on Neural Information Processing Systems , NIPS '23, 77013--77042. Red Hook, NY, USA: C...

    2024

  18. [26]

    F.; Alon, U.; Neubig, G.; and Hellendoorn, V

    Xu, F. F.; Alon, U.; Neubig, G.; and Hellendoorn, V. J. 2022. A Systematic Evaluation of Large Language Models of Code . ArXiv:2202.13169 [cs]

    2022 arXiv

  19. [27]

    B.; Lee, S.; and Batra, D

    Yadav, D.; Jain, R.; Agrawal, H.; Chattopadhyay, P.; Singh, T.; Jain, A.; Singh, S. B.; Lee, S.; and Batra, D. 2019. EvalAI : Towards Better Evaluation Systems for AI Agents . ArXiv:1902.03570 [cs]

    2019 arXiv

  20. [28]

    Q.; and Artzi, Y

    Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2020. BERTScore : Evaluating Text Generation with BERT . ArXiv:1904.09675 [cs]

    2020 arXiv

  21. [29]

    P.; Zhang, H.; Gonzalez, J

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM -as-a-judge with MT - Bench and Chatbot Arena . ArXiv:2306.05685 [cs]

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.