Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's claim is that a language model can be evaluated without any external question bank by measuring how well it answers questions it wrote from its own explanation of a topic, and that this self-quiz accuracy predicts MMLU-Pro…

desk verdict Useful combination, but the ranking claim is overstated: the reported correlation is category-level, and the formal proof is circular. read the letter →

arxiv 2501.11721 v2 pith:EDBWG6AS submitted 2025-01-20 cs.CL cs.LG

Saeid Asgari Taghanaki , Joao Monteiro This is my paper · ORCID
classification cs.CLcs.LG
keywords self-evaluationlargelanguagemodelsexplanationgenerationquestionansweringMMLU-Proanswerconsistencycomprehensiongapbenchmark-freeevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Explain-Query-Test (EQT), a self-evaluation loop in which a language model explains a topic, creates multiple-choice questions from its own explanation, and later answers those questions without seeing the explanation. The paper's central claim is that accuracy on this self-generated quiz is predictive of performance on an established external benchmark, MMLU-Pro, reporting a Pearson correlation of $r = 0.361$ ($p = 0.003$) across category–model pairs. If that holds, EQT would let developers rank LLMs using only a list of topic names, with no external question bank. The paper also claims that the gap between how fluently models explain and how well they answer questions about their own explanations exposes limits in their internal knowledge representation and reasoning.

What carries the argument

The load-bearing object is the formalised self-contained feedback loop, written as $E = g_\theta(K)$, $Q = h_\theta(E)$, $A = f_\theta(Q)$, with full-loop accuracy $\text{Acc}_{\text{loop}} = \frac{\text{Correct answers on self-generated questions}}{\text{Total questions}}$. The paper's argument is that when $f_\theta(h_\theta(g_\theta(K))) \approx K$, the loop isolates the model's internal knowledge representation $K$ and reasoning function $f_\theta$, so intermediate flaws in the explanation $E$ or questions $Q$ do not break the measurement. The Answer Consistency Score (ACS) is the supporting metric for stability under paraphrase: $\text{ACS} = 1 - \frac{|U(A)|-1}{|A|}$, where $A$ is the set of option letters chosen across a question and its paraphrases and $U(A)$ is the set of unique letters among them.

What would settle it

Run EQT with the test step fed questions whose explanations were deliberately corrupted or swapped across unrelated topics; if accuracy stays high, the loop is scoring memorized topic knowledge, not comprehension of the generated explanation. A second decisive check is to apply EQT to a new family of models and see whether the $r = 0.361$ correlation with MMLU-Pro reproduces; a large divergence in model rankings would falsify the proxy claim.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the full loop Explain → Query → Test yields a usable self-consistency signal: a model that can articulate a topic and then answer its own derived questions at high accuracy is exhibiting internal knowledge coherence, while a low full-loop accuracy flags misalignment among explanation, question generation, and reasoning. The evidence is a moderate but statistically significant correlation between EQT accuracy and MMLU-Pro accuracy ($r = 0.361$, $p = 0.003$), along with ranking heatmaps that show partial agreement between EQT-based and MMLU-Pro-based model orderings. The paper further finds that models with high MMLU-Pro accuracy drop more under EQT in some categories, and that answer-consistency scores across paraphrases are high for frontier models, with Sonnet-3.5 and GPT-4o leading.

Load-bearing premise

The entire inference rests on the assumption that all three steps draw on one stable internal knowledge store (the paper's $K$), so that answering questions the model wrote from its own explanation reflects comprehension rather than topic memorization, prompt familiarity, or repetition of just-written text.

Editorial extensions

If this is right

  • EQT can rank LLMs without an external test set: only lists of topics are required.
  • EQT accuracy predicts MMLU-Pro accuracy moderately well, so low EQT scores flag models whose explanation fluency exceeds their tested comprehension.
  • The explanation–comprehension gap quantifies a limitation of current LLMs in internal knowledge representation and reasoning.
  • The Answer Consistency Score provides a benchmark-free consistency measure: high stability across paraphrased questions is achievable by frontier models, while O1-preview lags despite being designed for reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if the correlation is reproduced on other benchmarks and newer model families, EQT could become a cheap continuous-monitoring signal during model development.
  • Beyond the paper, the loop could be made diagnostic by feeding deliberately flawed or shuffled explanations and watching whether EQT accuracy changes, which would separate memorized topic knowledge from comprehension of the generated text.
  • Beyond the paper, the explanation–comprehension gap suggests a training objective that rewards full-loop accuracy, so models are tuned for consistency between generation and downstream use rather than next-token likelihood alone.
  • Beyond the paper, the modest $r = 0.361$ leaves open whether question-generation quality caps the correlation; using questions written by a stronger model while keeping the answering model fixed would separate generator effects from reasoner effects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Explain-Query-Test (EQT), a self-evaluation pipeline in which an LLM (i) generates an explanation from a topic, (ii) generates multiple-choice questions and correct answers from that explanation, (iii) paraphrases each question, and then (iv) answers the original and paraphrased questions. The authors define full-loop accuracy (Accloop) and an Answer Consistency Score (ACS), report correlations between EQT accuracy and MMLU-Pro accuracy across category–model pairs, and claim that EQT can rank LLMs without external test sets, using only lists of topics. They also report that models with higher MMLU-Pro accuracy often show larger drops under EQT, which they interpret as a gap between surface-level accuracy and deeper comprehension.

Significance. If properly validated, EQT would be a valuable low-cost proxy for benchmark performance and a diagnostic for self-consistency, reducing dependence on extensive evaluation datasets. The paper contributes a simple, reusable pipeline and releases code (https://github.com/asgsaeid/EQT), which is a concrete asset for reproducibility. The ACS metric is clearly defined and easy to apply. However, the current evidence does not support the model-ranking claim, and the theoretical justification in Section 3.2 contains a circular step. The central idea is worth developing, but the load-bearing claims need stronger empirical and conceptual support.

major comments (4)
  1. [§4.2, Figure 3a] The headline claim that EQT can rank models is not supported by the reported evidence. The correlation r=0.361 is computed across 70 category–model pairs, pooling 14 categories per model; these points are not independent, and no clustering or model-level analysis is provided. To substantiate ranking, the authors need to show that model-level mean EQT accuracy orders the five models consistently with MMLU-Pro, e.g., via a model-level correlation, a rank table, or a paired test. With only five models, a single divergent model can flip the conclusion.
  2. [§3.2.3] The 'proof' that high Accloop demonstrates effective reasoning is circular. The framework defines correctness as agreement with the model's self-generated answers and assumes E=gθ(K), Q=hθ(E), A=fθ(Q) share a latent K; the proof then simply restates that high Accloop implies fθ(hθ(gθ(K)))≈K. This establishes self-consistency only under the assumption that K exists and is equally accessible at each step, but K is never operationalized or tested. The authors should either remove the proof and present the claim as a hypothesis, or provide an independent test (e.g., ablating access to the explanation across the loop's steps).
  3. [§3.1, Appendix A] The ground-truth labels for Accloop are generated by the same model being evaluated: the question-generation prompt in Appendix A asks the model to output 'Correct Answers: ...' alongside each question. High accuracy on these self-labeled questions may reflect the model's ability to recognize its own earlier outputs rather than comprehension of the concept. The correlation with MMLU-Pro provides some external grounding, but the paper should report, for example, human or external verification of the generated questions and correct answers, or an analysis of how often model answers match its own labels by chance.
  4. [Abstract, §4.2] The strength of the reported correlation is overstated. The abstract says 'correlates strongly,' while the full text (Section 4.2) says 'moderatly' and Figure 3a reports r=0.361, which corresponds to r²≈0.13. This effect size leaves 87% of variance unexplained and is insufficient to support the claim of predictive ranking without further evidence (see first major comment). Please add confidence intervals and a benchmark comparison of correlation strength.
minor comments (7)
  1. [§4.2] The text refers to 'Figure ?? provides a spider plot,' but the figure is missing from the manuscript.
  2. [Figure 1, §4.3] The caption of Figure 1 says 'Comparison of EQT Accuracy across corresponding MMLU-Pro Categories,' but the figure is first referenced in Section 4.3 for stability scores; the caption and the text should be aligned.
  3. [Equation (1)] The multiset A is denoted both as a list and with |A| = n+1; the variable A is overloaded with the answer symbol, which may confuse readers.
  4. [Algorithm 1] The line 'Results ← {}{Stores results for each concept}' contains a stray '{' and should be formatted consistently.
  5. [§4.2] There is a grammatical error: 'Figure 3a show a statistically significant correlation' should be 'Figure 3a shows.'
  6. [Throughout] The dataset name is inconsistently rendered as 'MMLU-P RO' and 'MMLU-Pro'; please standardize.
  7. [§4] The paper states '20 concepts' per category and '20 questions per concept' but does not clarify whether the k paraphrases are included in the 20 questions or additional; please state this explicitly.

Circularity Check

1 steps flagged · score 6.0 of 10

Section 3.2.3's proof that high Accloop measures internal reasoning is definitional: the metric is scored against self-generated correct answers and then equated with fidelity to latent K; the MMLU-Pro correlation is external evidence and is not itself circular.

  1. self definitional [Section 3.2.2 and Section 3.2.3 (Full-Loop Accuracy and Significance of High/Low Accloop)]
    "Accloop = Correct Answers Based on Self-Generated Q / Total Questions. ... If Accloop is high, it validates that EQT faithfully captures the model's internal reasoning and consistency. ... The EQT framework ensures that explanations (E), questions (Q), and answers (A) are all derived from the model's internal knowledge representation (K). Formally: E = gθ(K), Q = hθ(E), A = fθ(Q)."

    By definition, Accloop counts how often the model re-selects the 'Correct Answers' that the model itself wrote when generating Q from its own E; no independent ground truth is involved. The proof then identifies a high value of this self-agreement with fθ(hθ(gθ(K))) ≈ K, i.e., robustness of an unobserved internal representation K. That identification is exactly the claim under test: it assumes self-authored correct answers are faithful expressions of K. A model that writes labels and then reproduces them can achieve high Accloop regardless of whether those labels correspond to K, so the conclusion that the loop measures reasoning or comprehension is true by construction, not by proof.

full rationale

The empirical core of the paper—EQT accuracy correlating with MMLU-Pro accuracy (r = 0.361, p = 0.003; Figure 3a)—is externally grounded and not circular: EQT questions are generated from model-authored explanations and are not fit to MMLU-Pro answers. The self-citations to Monteiro et al. (2024a,b) are contextual related-work references and carry no load-bearing weight. The circularity is localized to the formal interpretation: Section 3.2.3 'proves' that high Accloop validates the framework by defining Accloop as agreement with self-generated correct answers and then asserting that this agreement equals fidelity to latent internal knowledge K. That is a definitional equivalence, not an empirical derivation, and the paper's conclusion that the explanation/answer gap exposes 'fundamental limitations in the internal knowledge representation and reasoning' inherits this tautology. Separately, Section 4.2 contains a dangling 'Figure ??' reference, and the 'rank models' headline is not backed by a model-level correlation test; these are validity/completeness defects rather than circularity and are not counted in the score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The paper's central inference depends on several unverified premises: a stable latent K, the equivalence of self-generated accuracy with comprehension, the proxy validity relative to MMLU-Pro, and the absence of answer leakage. The free parameters are experimental design choices that are not varied or justified. No new physical or mathematical entity is introduced beyond the theoretical construct K.

free parameters (3)
  • number of concepts per MMLU-Pro category = 20
    Chosen by hand in Section 4; not varied or justified, and the reported correlation could depend on topic selection.
  • paraphrases per question (k) = 3
    Chosen in Section 4; the Answer Consistency Score and stability results depend on this choice.
  • questions per concept (q) = 5 (implied from 20 total questions per concept with k=3)
    Not stated explicitly; affects sample size and the difficulty of the EQT accuracy estimate.
assumptions (4)
  • domain assumption A stable internal knowledge representation K is shared across explanation, question generation, and answering (E=gθ(K), Q=hθ(E), A=fθ(Q)).
    Invoked in Section 3.2.1 and used by the proof in Section 3.2.3; no independent evidence is given for the existence or stability of K.
  • ad hoc to paper High accuracy on self-generated questions implies robust internal reasoning and self-consistency.
    This is the conclusion of Section 3.2.3, asserted as a proof but actually assumed: correctness is defined within the loop, so the implication is built into the definitions.
  • domain assumption EQT accuracy is a valid proxy for MMLU-Pro performance and can rank models.
    Section 4.2 interprets r=0.361 as predictive power; this is the central inference of the paper, not a proven theorem.
  • domain assumption Self-contained generated questions prevent the model from using its own explanation or topic memory during answering.
    The method relies on this to claim the answering step measures comprehension; the prompt in Appendix A asks for self-contained questions but does not verify that leakage is absent.
invented entities (1)
  • Internal knowledge representation K
    purpose: Latent variable used in Section 3.2 to define the EQT loop and to justify that high full-loop accuracy measures reasoning.
    K is not operationalized or measured; the proof in Section 3.2.3 assumes its existence and stability rather than demonstrating it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy." pith.science (2026). https://pith.science/paper/EDBWG6AS

@misc{pith2026250111721,
  author       = {Pith},
  title        = {Pith review of: Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EDBWG6AS}},
  note         = {Machine review of arXiv:2501.11721}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable proficiency in generating detailed and coherent explanations of complex concepts. However, the extent to which these models truly comprehend the concepts they articulate remains unclear. To assess the level of comprehension of a model relative to the content it generates, we implemented a self-evaluation pipeline where models: (i) given a topic generate an excerpt with information about the topic, (ii) given an excerpt generate question-answer pairs, and finally (iii) given a question generate an answer. We refer to this self-evaluation approach as Explain-Query-Test (EQT). Interestingly, the accuracy on generated questions resulting from running the EQT pipeline correlates strongly with the model performance as verified by typical benchmarks such as MMLU-Pro. In other words, EQT's performance is predictive of MMLU-Pro's, and EQT can be used to rank models without the need for any external source of evaluation data other than lists of topics of interest. Moreover, our results reveal a disparity between the models' ability to produce detailed explanations and their performance on questions related to those explanations. This gap highlights fundamental limitations in the internal knowledge representation and reasoning abilities of current LLMs. We release the code at https://github.com/asgsaeid/EQT.

Figures

Figures reproduced from arXiv: 2501.11721 by the authors.

Figure 1
Figure 1. Comparison of EQT Accuracy across corresponding MMLU-Pro Categories. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Accuracy (%) comparison across MMLU-PRO categories for various language models. Each model is representedf by two bars: the first (solid) represents the original MMLU-PRO accuracy, and the second (hatched) indicates the adjusted accuracy due to the application of EQT since new questions are added. diverse yet semantically equivalent prompts. The derived questions are compared to the models’ known performance on the … view at source ↗
Figure 3
Figure 3. Analysis of MMLU-PRO and EQT results. 4.2 RESULTS [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Making Sense of the Unsensible: Reflection, Survey, and Challenges for XAI in Large Language Models Toward Human-Centered AI

    cs.CY 2025-05 conditional novelty 4.0 of 10

    A survey and position paper proposing faithfulness, truthfulness, plausibility, and contrastivity as four core dimensions of LLM explainability and arguing for governance-ready, audience-sensitive XAI.

Reference graph

Works this paper leans on

24 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introducing gemini: Google's multimodal ai model

    Google AI. Introducing gemini: Google's multimodal ai model. https://ai.googleblog.com/2023/05/introducing-gemini-googles-multimodal.html, 2023

    work page 2023

  2. [2]

    Introducing claude: Anthropic's ai assistant

    Anthropic. Introducing claude: Anthropic's ai assistant. https://www.anthropic.com/index/claude, 2023

    work page 2023

  3. [3]

    Explainability in ai: A survey

    Trisha Bansal and Prashant Verma. Explainability in ai: A survey. In Proceedings of the 2022 International Joint Conference on Artificial Intelligence (IJCAI), 2022

    work page 2022

  4. [4]

    On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 610--623, 2021

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 610--623, 2021

    work page 2021

  5. [5]

    On the opportunities and risks of foundation models

    Rishi Bommasani et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  6. [6]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 0 1877--1901, 2020

    work page 1901

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT, 2019

    work page 2019

  8. [8]

    Evaluating para-consistency in large language models: A novel benchmark

    Qi Li and Wei Zhang. Evaluating para-consistency in large language models: A novel benchmark. arXiv preprint arXiv:2305.05678, 2023

    arXiv 2023

Show all 24 references
  1. [9]

    Self-evaluation in large language models: Bridging the gap between generation and understanding

    Eli Lightman and Yao Xu. Self-evaluation in large language models: Bridging the gap between generation and understanding. arXiv preprint arXiv:2304.01234, 2023

    2023 arXiv

  2. [10]

    Self-critique training: Improving large language models through iterative feedback

    Aman Madaan, Yujie Wu, and Lianmin Gao. Self-critique training: Improving large language models through iterative feedback. arXiv preprint arXiv:2306.04500, 2023

    2023 arXiv

  3. [11]

    Xc-cache: Cross-attending to cached context for efficient llm inference

    Joao Monteiro, Etienne Marcotte, Pierre-Andre Noel, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, and Perouz Taslakian. Xc-cache: Cross-attending to cached context for efficient llm inference. arXiv preprint arXiv:2404.15420, 2024 a

    2024 arXiv

  4. [12]

    Repliqa: A question-answering dataset for benchmarking llms on unseen reference content

    Joao Monteiro, Pierre-Andre Noel, Etienne Marcotte, Sai Rajeswar, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, and Perouz Taslakian. Repliqa: A question-answering dataset for benchmarking llms on unseen reference content. arXiv preprint arXiv:2406.1...

    2024 arXiv

  5. [13]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. https://openai.com/research/gpt-4, 2023

    2023

  6. [14]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 2383--2392, 2016

    2016

  7. [15]

    A statistical analysis of llms' self-evaluation using proverbs

    Ryosuke Sonoda and Ramya Srinivasan. A statistical analysis of llms' self-evaluation using proverbs. arXiv preprint arXiv:2410.16640, 2024

    2024 arXiv

  8. [16]

    Llama: Open and efficient foundation language models

    Hugo Touvron et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  9. [17]

    Robustness of language models to syntactic variations: An empirical study

    Xinyi Wang, Haoran Liu, and Jared Johnson. Robustness of language models to syntactic variations: An empirical study. arXiv preprint arXiv:2302.06789, 2023

    2023 arXiv

  10. [18]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024

    2024 arXiv

  11. [19]

    Teach me to explain: A review of machine learning interpretability through explanations

    Sarah Wiegreffe and Ana Marasovic. Teach me to explain: A review of machine learning interpretability through explanations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, pp.\ 2055--2071, 2021

    2021

  12. [20]

    Language models can evaluate themselves via probability discrepancy

    Tingyu Xia, Bowen Yu, Yuan Wu, Yi Chang, and Chang Zhou. Language models can evaluate themselves via probability discrepancy. arXiv preprint arXiv:2405.10516, 2024

    2024 arXiv

  13. [21]

    Evaluating paraphrase sensitivity in large language models

    Jingyuan Zhao, Fei Wang, and Yichen Li. Evaluating paraphrase sensitivity in large language models. In Proceedings of the 2023 Annual Meeting of the Association for Computational Linguistics (ACL), 2023

    2023

  14. [22]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  15. [23]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  16. [24]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.