{"id":"ee465fd1-59ce-41da-8046-4ad1c2efbd93","arxiv_id":"2501.11721","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A model's accuracy on questions it generates about its own explanations correlates with MMLU-Pro only at r = 0.361, and the paper claims this can serve as a test-set-free ranking proxy.","lead":"This paper proposes Explain-Query-Test (EQT), a self-evaluation method in which a language model explains a topic, writes multiple-choice questions about its own explanation, and then answers those questions. The authors report a modest correlation between EQT accuracy and MMLU-Pro scores and argue EQT could rank models without a labeled test set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that EQT can rank models is unsupported because the reported correlation is across category–model pairs, not across models, and no model-level analysis is provided.","rationale":"I focused on the empirical support for the central claim of model ranking rather than the theoretical framing around K. The reader's weakest_assumption concerns the latent internal knowledge representation and the circular proof in Section 3.2.3, which speaks to whether the loop measures comprehension. That is an important conceptual issue, but the headline claim is specifically about predictive power and ranking. Even if the loop does not perfectly isolate comprehension, EQT might still empirically track benchmark performance. Conversely, even if the loop measures comprehension perfectly, the current statistics do not demonstrate model-level ranking. The reported r=0.361 is modest, computed over non-independent category–model pairs, and no model-level ranking is shown. A correlation across 70 points does not establish an ordering across 5 models. The heatmap in Figure 3b is informal and shows only partial agreement. Therefore, the most load-bearing concern is the gap between the evidence and the ranking claim. I agree with the reader's overall CONDITIONAL verdict: the idea is plausible and potentially useful, but the central claim requires model-level statistics and the interpretation of EQT's accuracy needs scrutiny. The suggested test (model-level Spearman, permutation test, mixed-effects model) would directly resolve whether the ranking claim holds. I also note an internal inconsistency: the title-page abstract says 'correlates strongly' while the full-text abstract says 'correlates moderatly' (typo for 'moderately'); the reported r=0.361 supports the moderate wording. This discrepancy further weakens the abstract's strong wording. My disagreement with the reader's weakest_assumption is not a rejection of it; I simply think the empirical statistical gap is more directly load-bearing for the stated central claim.","tokens_in":9154,"tokens_out":4468,"duration_ms":46075,"concrete_test":"Compute the mean EQT accuracy and mean MMLU-Pro accuracy for each of the five models, and test the model-level association. Use Spearman's rank correlation across the five models and a permutation test (shuffle model labels) to obtain an exact p-value. Also fit a linear mixed-effects model with random intercepts for model and category to obtain a cluster-corrected p-value for the category–model association. If the model-level rank correlation is not significant or the adjusted p-value exceeds 0.05, the central claim that EQT ranks models without external data is not supported by the reported evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that EQT can rank LLMs without external evaluation data. The supporting evidence is a Pearson correlation of r=0.361 (p=0.003) computed over 70 category–model pairs (14 categories × 5 models) in Figure 3a. This analysis conflates two sources of variation: differences between models and differences between categories within a model. The 14 data points for a given model are not independent, and the p-value likely understates the uncertainty because no clustering correction is applied. More importantly, ranking models is a model-level claim: one must show that, on average, EQT accuracy orders the five models in the same way as MMLU-Pro accuracy. The paper does not report a model-level correlation or a model-level ranking table; Figure 3b shows only per-category ranking matches, which are partial (GPT-4o has the fewest matches). With only five models, a single model's divergent ranking can flip the aggregate ordering. Thus the headline assertion that EQT 'can be used to rank models' is not directly supported by any statistical test at the model level.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Explain-Query-Test (EQT), a self-evaluation pipeline in which an LLM (i) generates an explanation from a topic, (ii) generates multiple-choice questions and correct answers from that explanation, (iii) paraphrases each question, and then (iv) answers the original and paraphrased questions. The authors define full-loop accuracy (Accloop) and an Answer Consistency Score (ACS), report correlations between EQT accuracy and MMLU-Pro accuracy across category–model pairs, and claim that EQT can rank LLMs without external test sets, using only lists of topics. They also report that models with higher MMLU-Pro accuracy often show larger drops under EQT, which they interpret as a gap between surface-level accuracy and deeper comprehension.","tokens_in":9309,"tokens_out":4433,"duration_ms":42990,"significance":"If properly validated, EQT would be a valuable low-cost proxy for benchmark performance and a diagnostic for self-consistency, reducing dependence on extensive evaluation datasets. The paper contributes a simple, reusable pipeline and releases code (https://github.com/asgsaeid/EQT), which is a concrete asset for reproducibility. The ACS metric is clearly defined and easy to apply. However, the current evidence does not support the model-ranking claim, and the theoretical justification in Section 3.2 contains a circular step. The central idea is worth developing, but the load-bearing claims need stronger empirical and conceptual support.","major_comments":[{"comment":"The headline claim that EQT can rank models is not supported by the reported evidence. The correlation r=0.361 is computed across 70 category–model pairs, pooling 14 categories per model; these points are not independent, and no clustering or model-level analysis is provided. To substantiate ranking, the authors need to show that model-level mean EQT accuracy orders the five models consistently with MMLU-Pro, e.g., via a model-level correlation, a rank table, or a paired test. With only five models, a single divergent model can flip the conclusion.","section":"§4.2, Figure 3a"},{"comment":"The 'proof' that high Accloop demonstrates effective reasoning is circular. The framework defines correctness as agreement with the model's self-generated answers and assumes E=gθ(K), Q=hθ(E), A=fθ(Q) share a latent K; the proof then simply restates that high Accloop implies fθ(hθ(gθ(K)))≈K. This establishes self-consistency only under the assumption that K exists and is equally accessible at each step, but K is never operationalized or tested. The authors should either remove the proof and present the claim as a hypothesis, or provide an independent test (e.g., ablating access to the explanation across the loop's steps).","section":"§3.2.3"},{"comment":"The ground-truth labels for Accloop are generated by the same model being evaluated: the question-generation prompt in Appendix A asks the model to output 'Correct Answers: ...' alongside each question. High accuracy on these self-labeled questions may reflect the model's ability to recognize its own earlier outputs rather than comprehension of the concept. The correlation with MMLU-Pro provides some external grounding, but the paper should report, for example, human or external verification of the generated questions and correct answers, or an analysis of how often model answers match its own labels by chance.","section":"§3.1, Appendix A"},{"comment":"The strength of the reported correlation is overstated. The abstract says 'correlates strongly,' while the full text (Section 4.2) says 'moderatly' and Figure 3a reports r=0.361, which corresponds to r²≈0.13. This effect size leaves 87% of variance unexplained and is insufficient to support the claim of predictive ranking without further evidence (see first major comment). Please add confidence intervals and a benchmark comparison of correlation strength.","section":"Abstract, §4.2"}],"minor_comments":[{"comment":"The text refers to 'Figure ?? provides a spider plot,' but the figure is missing from the manuscript.","section":"§4.2"},{"comment":"The caption of Figure 1 says 'Comparison of EQT Accuracy across corresponding MMLU-Pro Categories,' but the figure is first referenced in Section 4.3 for stability scores; the caption and the text should be aligned.","section":"Figure 1, §4.3"},{"comment":"The multiset A is denoted both as a list and with |A| = n+1; the variable A is overloaded with the answer symbol, which may confuse readers.","section":"Equation (1)"},{"comment":"The line 'Results ← {}{Stores results for each concept}' contains a stray '{' and should be formatted consistently.","section":"Algorithm 1"},{"comment":"There is a grammatical error: 'Figure 3a show a statistically significant correlation' should be 'Figure 3a shows.'","section":"§4.2"},{"comment":"The dataset name is inconsistently rendered as 'MMLU-P RO' and 'MMLU-Pro'; please standardize.","section":"Throughout"},{"comment":"The paper states '20 concepts' per category and '20 questions per concept' but does not clarify whether the k paraphrases are included in the 20 questions or additional; please state this explicitly.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style submission and the abstract's claims considerably exceed the evidence presented. The core idea is interesting and the code release is a plus, but the analysis needs a proper model-level statistical test and a re-framing of the theoretical justification. The circularity in Section 3.2.3 and the self-generated labels in the pipeline are correctness risks that should be addressed before the claims can be taken at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is the three-stage loop—explain, generate QA from the explanation, then answer without seeing the explanation—used as a benchmark-free proxy. The components exist separately (self-evaluation, generated QA, paraphrase consistency), but the specific combination is not in the cited literature, and the code is released. That is a legitimate contribution worth a workshop slot.\n\nWhat the paper does well: it is clearly written, the pipeline is easy to reproduce, and the ACS metric is simple and sensible. The observation that performance drops on self-generated questions relative to MMLU-Pro, and that the gap varies by category, is interesting and worth reporting.\n\nThe soft spot is the central claim. The abstract and text say EQT 'can be used to rank models,' and 'strongly correlates' in one place (the abstract in the arXiv version says 'moderately,' which is at least honest). The supporting evidence is r = 0.361 over 70 category–model pairs. That is a weak-to-moderate correlation, and the statistical treatment ignores clustering by model. With 14 points per model, the effective sample size for model-level ranking is five, not 70. No model-level correlation or ranking table is shown; Figure 3b is only per-category, and GPT-4o has the fewest matches, which cuts against the claim. So the headline assertion is not directly supported. The stress-test note lands.\n\nAlso, Section 3.2.3 'proves' that high Accloop demonstrates reasoning by assuming K is stable and that fθ(hθ(gθ(K))) ≈ K whenever accuracy is high. That is circular. The authors do hedge by saying 'partially flawed' intermediate artifacts, but the proof does not actually derive anything; it restates the definition. This should be flagged but is not fatal—the empirical correlation with MMLU-Pro is external grounding, and the method is the contribution, not the proof.\n\nThe citation pattern is fine. Self-citations to Monteiro et al. are relevant and not inflated. The related work covers the main prior art.\n\nBottom line: this is a salvageable workshop paper. With proper model-level analysis (e.g., a correlation across the five models, or at least a clear per-model ranking table), a held-out topic split, and softened language about ranking, the method would be a useful evaluation tool for practitioners. The current version overclaims, but the core idea is sound enough to merit review.\n\nRecommendation: send to peer review. The paper deserves a serious referee; expect major revision on the statistics and the proof.","headline":"Useful combination, but the ranking claim is overstated: the reported correlation is category-level, and the formal proof is circular.","tokens_in":9887,"tokens_out":639,"would_cite":false,"duration_ms":8622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's claim is that a language model can be evaluated without any external question bank by measuring how well it answers questions it wrote from its own explanation of a topic, and that this self-quiz accuracy predicts MMLU-Pro…","keywords":["self-evaluation","large language models","explanation generation","question answering","MMLU-Pro","answer consistency","comprehension gap","benchmark-free evaluation"],"falsifier":"Run EQT with the test step fed questions whose explanations were deliberately corrupted or swapped across unrelated topics; if accuracy stays high, the loop is scoring memorized topic knowledge, not comprehension of the generated explanation. A second decisive check is to apply EQT to a new family of models and see whether the $r = 0.361$ correlation with MMLU-Pro reproduces; a large divergence in model rankings would falsify the proxy claim.","tokens_in":8876,"feed_emoji":"🧠","tokens_out":6362,"duration_ms":57060,"temperature":0.7,"pith_summary":"This paper introduces Explain-Query-Test (EQT), a self-evaluation loop in which a language model explains a topic, creates multiple-choice questions from its own explanation, and later answers those questions without seeing the explanation. The paper's central claim is that accuracy on this self-generated quiz is predictive of performance on an established external benchmark, MMLU-Pro, reporting a Pearson correlation of $r = 0.361$ ($p = 0.003$) across category–model pairs. If that holds, EQT would let developers rank LLMs using only a list of topic names, with no external question bank. The paper also claims that the gap between how fluently models explain and how well they answer questions about their own explanations exposes limits in their internal knowledge representation and reasoning.","feed_headline":"Self-made quiz scores track MMLU-Pro rankings","feed_subtitle":"A model explains a topic, writes its own questions, then answers them — no external test set required beyond a topic list.","key_machinery":"The load-bearing object is the formalised self-contained feedback loop, written as $E = g_\\theta(K)$, $Q = h_\\theta(E)$, $A = f_\\theta(Q)$, with full-loop accuracy $\\text{Acc}_{\\text{loop}} = \\frac{\\text{Correct answers on self-generated questions}}{\\text{Total questions}}$. The paper's argument is that when $f_\\theta(h_\\theta(g_\\theta(K))) \\approx K$, the loop isolates the model's internal knowledge representation $K$ and reasoning function $f_\\theta$, so intermediate flaws in the explanation $E$ or questions $Q$ do not break the measurement. The Answer Consistency Score (ACS) is the supporting metric for stability under paraphrase: $\\text{ACS} = 1 - \\frac{|U(A)|-1}{|A|}$, where $A$ is the set of option letters chosen across a question and its paraphrases and $U(A)$ is the set of unique letters among them.","core_discovery":"On its own terms, the paper's discovery is that the full loop Explain → Query → Test yields a usable self-consistency signal: a model that can articulate a topic and then answer its own derived questions at high accuracy is exhibiting internal knowledge coherence, while a low full-loop accuracy flags misalignment among explanation, question generation, and reasoning. The evidence is a moderate but statistically significant correlation between EQT accuracy and MMLU-Pro accuracy ($r = 0.361$, $p = 0.003$), along with ranking heatmaps that show partial agreement between EQT-based and MMLU-Pro-based model orderings. The paper further finds that models with high MMLU-Pro accuracy drop more under EQT in some categories, and that answer-consistency scores across paraphrases are high for frontier models, with Sonnet-3.5 and GPT-4o leading.","pith_inferences":["Beyond the paper, if the correlation is reproduced on other benchmarks and newer model families, EQT could become a cheap continuous-monitoring signal during model development.","Beyond the paper, the loop could be made diagnostic by feeding deliberately flawed or shuffled explanations and watching whether EQT accuracy changes, which would separate memorized topic knowledge from comprehension of the generated text.","Beyond the paper, the explanation–comprehension gap suggests a training objective that rewards full-loop accuracy, so models are tuned for consistency between generation and downstream use rather than next-token likelihood alone.","Beyond the paper, the modest $r = 0.361$ leaves open whether question-generation quality caps the correlation; using questions written by a stronger model while keeping the answering model fixed would separate generator effects from reasoner effects."],"forward_implications":["EQT can rank LLMs without an external test set: only lists of topics are required.","EQT accuracy predicts MMLU-Pro accuracy moderately well, so low EQT scores flag models whose explanation fluency exceeds their tested comprehension.","The explanation–comprehension gap quantifies a limitation of current LLMs in internal knowledge representation and reasoning.","The Answer Consistency Score provides a benchmark-free consistency measure: high stability across paraphrased questions is achievable by frontier models, while O1-preview lags despite being designed for reasoning."],"supporting_citations":[{"why":"Supplies MMLU-Pro, the benchmark whose category structure and topic list drive the EQT experiments and whose accuracy is the external correlate.","marker":"Wang et al., 2024"},{"why":"Documents paraphrase sensitivity in LLMs, the phenomenon the ACS stability metric is designed to measure.","marker":"Zhao et al., 2023"},{"why":"Establishes consistency across paraphrased inputs as a robustness criterion that underlies the paper's stability analysis.","marker":"Wang et al., 2023"},{"why":"A self-evaluation framework that EQT extends by removing the need for external test data.","marker":"Lightman & Xu, 2023"},{"why":"Question answering over unseen reference content, relevant to testing whether models can answer questions about content they generated.","marker":"Monteiro et al., 2024b"}],"fun_headline_variants":["LLM self-quizzes predict benchmark rankings without external data","Explain, query, test: a self-check for LLM comprehension","Models that ace their own exams also ace MMLU-Pro","LLM self-tests reveal comprehension gaps","Self-evaluation pipeline predicts LLM benchmark scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire inference rests on the assumption that all three steps draw on one stable internal knowledge store (the paper's $K$), so that answering questions the model wrote from its own explanation reflects comprehension rather than topic memorization, prompt familiarity, or repetition of just-written text.","fun_headline_variants_meta":{"raw":{"variants":["LLM self-quizzes predict benchmark rankings without external data","Explain, query, test: a self-check for LLM comprehension","Models that ace their own exams also ace MMLU-Pro","LLM self-tests reveal comprehension gaps","Self-evaluation pipeline predicts LLM benchmark scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3098,"prompt_tokens":955,"completion_tokens":2143,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2064}},"tokens_in":571,"tokens_out":2143,"duration_ms":16688,"temperature":1.0,"reasoning_tokens":2064,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:56:21.006842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EQT with the test step fed questions whose explanations were deliberately corrupted or swapped across unrelated topics; if accuracy stays high, the loop is scoring memorized topic knowledge, not comprehension of the generated explanation. A second decisive check is to apply EQT to a new family of models and see whether the $r = 0.361$ correlation with MMLU-Pro reproduces; a large divergence in model rankings would falsify the proxy claim.","supporting_citations":[{"cited_title":"Evaluating paraphrase sensitivity in large language models","cited_arxiv_id":null,"evidence_quote":"Documents paraphrase sensitivity in LLMs, the phenomenon the ACS stability metric is designed to measure."},{"cited_title":"Prediction of solar wind speed by applying convolutional neural network to potential field source surface (PFSS) magnetograms","cited_arxiv_id":"2304.01234","evidence_quote":"A self-evaluation framework that EQT extends by removing the need for external test data."}],"review_version":1}