{"id":"3148c53c-0519-43f8-bf07-1efb05d08c6c","arxiv_id":"2411.14465","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Across four LLMs on 823 physics questions, accuracy and response entropy form a bell-shaped curve, and confidently wrong answers cluster in reasoning-heavy categories.","lead":"The paper tests how often four large language models answer physics multiple-choice questions correctly and how consistent they are when asked the same question twenty times. It finds that a model can be very sure and still be wrong, especially on questions that require multi-step reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Category asymmetry claim rests on unquantified 2D histograms: error rate and entropy share the same 20-sample counts, so the bell shape is a mathematical envelope (Appendix E); no conditional accuracy-vs-entropy curves or error bars support the S/M vs D/F difference.","rationale":"The reader identified the shared-counts metric coupling and the unquantified scatter as the main weakness. My stress-test builds on that but centers on the category-level asymmetry claim, which is the most novel and load-bearing part of the abstract. The mathematical envelope in Appendix E is derived correctly and transparently, which is a point in the paper's favor; however, transparency about the envelope does not convert visual density into statistical evidence. The 2D histograms alone cannot distinguish a genuine category-dependent certainty–accuracy trade-off from differences in category size or base accuracy. The proposed permutation test and conditional error-rate curves would settle whether the S/M vs D/F asymmetry is real or an artifact of the plotting and category composition. Until that check is run, a conditional verdict is appropriate, so I leave the reader's verdict unchanged.","tokens_in":8632,"tokens_out":4975,"duration_ms":47937,"concrete_test":"For each model, stratify questions by entropy bin (e.g., H=0, 0<H≤0.2, ...) and compute the empirical error rate per bin with 95% bootstrap confidence intervals over questions. Then test the null hypothesis that the low-entropy (H≤0.2) error rate in S∪M equals that in D∪F, using a permutation test that resamples questions within each category to account for category size. If the difference is not significant at the 0.05 level, the claimed intensification of the accuracy–uncertainty asymmetry in reasoning categories is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the accuracy–uncertainty asymmetry 'intensifies' for reasoning categories S and M—is read off 2D histograms (Figures 2–4) without any statistical test, error bar, or conditional accuracy curve. This matters because error rate and entropy are computed from the same N=20 responses per question, so every point is confined to a deterministic family of curves derived in Appendix E. The bell-shaped envelope is therefore a mathematical consequence of the metrics, not an empirical observation; the only empirical content is the density of points within the envelope. The paper does not quantify that density, and the categories differ sharply in size (D:153, F:138, C:238, S:223, M:71) and in marginal accuracy, so a raw count of high-certainty errors in S/M may reflect lower accuracy or larger categories rather than a distinct certainty–accuracy relation. Without a conditional analysis of error rate given entropy, the asymmetry claim is not supported by the presented evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates four LLMs (Llama3.1-8B-Instruct, Mixtral-8x7B-Instruct-v0.1, Mistral-7B-Instruct-v0.3, GPT-3.5-turbo) on the mlphys101 physics multiple-choice dataset, which contains 823 questions in five categories (D, F, C, S, M). For each question, the authors sample N=20 responses at temperature 0.7, compute the response entropy and the error rate, and display 2D histograms of error rate versus entropy overall and per category. The headline claims are that the error-rate/entropy relationship forms a broad bell-shaped distribution, that most models are accurate when certain but not universally so, and that the accuracy–uncertainty asymmetry intensifies as questions demand more logical reasoning (categories S and M). The paper also makes a methodological point, derived in Appendix E, that the bell-shaped envelope corresponds to a family of theoretical curves parameterized by the number of distinct incorrect responses.","tokens_in":8799,"tokens_out":3616,"duration_ms":35059,"significance":"If the empirical claims were properly supported, the paper would offer a useful contribution to evaluating LLM trustworthiness in a scientific domain: the mlphys101 dataset is a novel resource, and the paper ships code and data, which is commendable. However, the central descriptive claims are weakened by a mathematical coupling between the two plotted quantities and by the absence of any statistical quantification of the category differences. The dataset and the general framework have value, but the analysis needs substantial strengthening before the reported conclusions can be accepted.","major_comments":[{"comment":"The abstract and Section 3 present the 'broad horizontal bell-shaped distribution' as an empirical discovery, but Appendix E shows that the envelope is a mathematical identity: for any question where the model produces exactly two distinct responses, one correct and one incorrect, entropy is exactly the binary entropy of the error rate (Eq. 1). The 2D histograms must therefore lie within this deterministic family of curves, so the bell shape carries no empirical information. The only empirical content is the density of points within the envelope. Please reframe the claims to focus on the scatter and its category dependence, and quantify deviations from the theoretical curves rather than presenting the envelope itself as a finding.","section":"Section 3 and Appendix E, Eqs. (1)–(4)"},{"comment":"The claim that the accuracy–uncertainty asymmetry 'intensifies' for single-step (S) and multi-step (M) reasoning questions is read off 2D histograms without any statistical test, error bar, or conditional analysis. Because the categories differ substantially in size (D:153, F:138, C:238, S:223, M:71) and in marginal accuracy, a visually higher density of high-certainty error points in S/M could simply reflect lower overall accuracy or larger category counts. The paper should report, for example, the conditional error rate given entropy intervals per category with confidence intervals, or at least provide counts of high-certainty errors and a formal test (e.g., Fisher's exact test or a logistic regression with category as a predictor) to support the asymmetry claim.","section":"Section 3, Figure 3 (Appendix C)"},{"comment":"The reliability of the scatter plots rests on the stability of entropy and accuracy estimates from N=20 responses per question, but the paper gives no error bars on either quantity and does not report how many responses were excluded as malformed (Section 2.2 says such replies were 'replaced with None and excluded'). Since entropy and error rate are computed from the same 20 counts, sampling noise can generate apparent scatter even for a fixed underlying distribution. Please report the number of excluded responses per model, the effective number of responses used per question, and bootstrap or analytical confidence intervals for the plotted quantities.","section":"Sections 2.2 and 2.3"}],"minor_comments":[{"comment":"Typo: 'infracture' should be 'infrastructure'.","section":"Section 2.2"},{"comment":"'We can hypothesis at this point' should be 'We can hypothesize at this point'.","section":"Section 3"},{"comment":"Several references are incomplete: reference [8] has no year and reference [9] has a missing year. Please check all entries for completeness.","section":"References"},{"comment":"The few-shot prompt example is hard to read because the code block contains irregular spacing. Format it as a clean code block for clarity.","section":"Appendix B"},{"comment":"The dataset is described as 'university-level' in Section 2.1 but as 'high-school grade physics questions' in the abstract; clarify which level applies and keep the description consistent.","section":"Section 2.1 and Abstract"},{"comment":"The claim that this is 'the first publication which focuses on the trustworthiness of LLM answers in physics reasoning' is strong; consider softening the novelty claim or providing a more systematic comparison with prior works on uncertainty estimation in QA.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style empirical study with a useful dataset and available code. The main issues are fixable in scope: the authors should reanalyze their released data to quantify the scatter and category differences with proper statistics, and reframe the bell-shaped curve from a discovery to a sanity check of the metric. If the category asymmetry claim survives a conditional analysis, the paper could make a modest but solid contribution. If the authors cannot provide the requested quantification, the central claims should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this is a small empirical study, not a methodological advance. The authors sample four LLMs 20 times per question on a physics MCQ dataset, compute entropy over the sampled letters, and plot error rate against entropy. The bell-shaped envelope they report is not a discovery—they derive it in Appendix E from the fact that both quantities come from the same counts. What is left empirical is the density of points inside that envelope, and that density is read off 2D histograms without error bars or tests.\n\nGive credit where due. The paper is transparent about the mathematical constraint, more than most would be. The category analysis is a genuinely new data point: no one else has mapped accuracy-certainty across definition, fact, conceptual, single-step, and multi-step physics questions for these models. The finding that confidently wrong answers cluster in the reasoning categories is plausible and worth knowing for anyone building physics tutors. The N=20 repeated sampling at a fixed temperature is a standard but legitimate protocol.\n\nSoft spots. First, the central asymmetry claim—that the asymmetry intensifies as questions demand more logical reasoning—rests on visual inspection of Figures 2–4. Category sizes differ a lot (M has 71 questions, D 153) and marginal accuracy differs, so the raw scatter could reflect difficulty rather than a distinct certainty-accuracy relation. They should compute error rate conditional on entropy bands, per category, with confidence intervals. Second, there is no report of how many responses were excluded as malformed, even though they say Mistral sometimes returned two letters. Third, the code and data links are placeholders, which blocks verification. The \"first publication\" claim is overbroad; semantic entropy and accuracy-certainty analyses exist in the literature. What is new is the physics domain and the category breakdown, not the method.\n\nThe paper is honest, short, and descriptive. It would benefit from a proper statistical pass and full data release. As it stands, the headline is a mathematical shape, and the empirical claim needs missing support.\n\nWho gets value: people working on trustworthiness of LLMs in education and anyone using MCQ benchmarks. It deserves a referee—it would survive with major revision—but I would not cite it in its current form.\n\nRecommendation: send it to review, conditional on the authors supplying the code and data and adding basic statistics.","headline":"Descriptive, honest workshop paper whose headline bell curve is derived in its own appendix; the real empirical content is the unquantified scatter, which needs statistical support before the category claim holds.","tokens_in":9322,"tokens_out":2235,"would_cite":false,"duration_ms":23477,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Error rate and response entropy are coupled on physics multiple-choice questions, but the coupling fails exactly where reasoning is required.","keywords":["large language models","uncertainty estimation","hallucination","physics multiple choice","entropy","accuracy-certainty trade-off","reasoning vs knowledge retrieval","few-shot prompting"],"falsifier":"A reader could resample each question with N=200 at the same temperature and rebuild the error-rate-versus-entropy histograms; if the high-certainty, high-error points in the S and M categories disappear or shift substantially, the reported asymmetry is an artifact of the 20-sample estimate. Alternatively, permuting the correct-answer labels while keeping response counts fixed would reproduce the same envelope, confirming that curve shape alone cannot identify hallucination.","tokens_in":8411,"feed_emoji":"⚛️","tokens_out":5022,"duration_ms":44474,"temperature":0.7,"pith_summary":"This paper tries to establish how reliably a large language model's own response consistency signals correctness on physics questions. Sampling each of 823 multiple-choice questions 20 times from four models, the authors measure answer variability as entropy and compare it with error rate. They find that low uncertainty usually accompanies correct answers, but not always: a noticeable set of questions receive wrong answers with high certainty, and this mismatch grows as questions move from recalling definitions and facts to single- and multi-step reasoning. The overall error-rate-versus-entropy scatter takes a broad horizontal bell-shaped form, which the paper shows in an appendix to be largely a mathematical consequence of how the two quantities are constructed from the same response counts.","feed_headline":"Confident LLM physics answers are often wrong in reasoning tasks","feed_subtitle":"Sampling 20 responses per question shows high-certainty errors cluster in multi-step problems, not in fact recall.","key_machinery":"The carrying object is the pairwise plot of error rate (1 minus accuracy) against response entropy $H(Y|x,h)=-\\sum_i p(y_i|x,h)\\ln p(y_i|x,h)$, computed from a question's 20 repeated answers, together with the parameterized family of theoretical curves derived in Appendix E. For a question with only two distinct answers, one correct and one incorrect, entropy is forced to $-(1-e)\\ln(1-e)-e\\ln e$ where $e$ is the error rate; with three, four, or five distinct answers the same relation holds as a family of curves with one or more free probabilities. This identity explains the bell-shaped envelope of the histograms and shifts the empirical content to the density of questions across the envelope.","core_discovery":"The central claim is that accuracy and uncertainty are coupled for LLMs on physics multiple-choice questions, but the coupling is not uniform: most models provide accurate replies when they are certain, but this is by no means a general behavior, and the asymmetry between accuracy and uncertainty intensifies as questions demand more logical reasoning while staying comparatively sharp for knowledge retrieval. Concretely, single-step (S) and multi-step (M) reasoning questions show more high-certainty wrong answers and more high-diversity scattered responses across models, whereas definition (D) and fact (F) categories concentrate accurate, low-entropy replies. The authors present this through two-dimensional histograms of error rate versus entropy for four models and five question categories, interpreting the high-certainty, high-error region as a signature of hallucination.","pith_inferences":["Beyond the paper: if the 20-sample entropy proves unstable, the reported scatter could change with sample size; re-running with N=100 or N=500 would test whether high-certainty errors are robust or a small-sample artifact.","Beyond the paper: the same entropy-versus-error-rate identity applies to any discrete classifier with repeated sampling, so the method transfers to other domains and could be used as a cheap hallucination screen for non-physics MCQ benchmarks.","Beyond the paper: the paper's category ordering D < F < C < S < M suggests a testable gradient: if prompting style shifts (e.g., chain-of-thought) reduce the asymmetry in S and M, the reported gap is partly a prompting effect rather than an intrinsic reasoning deficit."],"forward_implications":["In knowledge-retrieval categories (D and F), low entropy is a fairly reliable indicator of a correct answer for the tested models.","In reasoning categories (S and M), confidence-like signals from sampling degrade; users should not read low entropy as correctness in multi-step physics problems.","The bell-shaped error-rate-versus-entropy envelope is a mathematical consequence of the measurement, so comparisons between models must be made through scatter density and category-wise patterns, not through the curve's gross shape.","Larger models in this set (Mixtral) produce lower diversity, while GPT-3.5-turbo shows higher diversity, implying consistency differences track model choice and prompting."],"supporting_citations":[{"why":"Supplies the mlphys101 dataset of 823 physics multiple-choice questions with five answer categories that all experiments and entropy computations run on.","marker":"[12]"},{"why":"Motivates the hallucination problem and provides the semantic-entropy approach to detecting inconsistent LLM outputs that the paper adapts to letter answers.","marker":"[5]"},{"why":"Guides the manual construction of the three-shot prompts used to elicit letter-only answers from all models.","marker":"[15]"},{"why":"Supports the paper's interpretation that failures in single- and multi-step reasoning questions match known reasoning breakdowns in large language models.","marker":"[16]"},{"why":"Provides the prompt-dependence result the paper cites to explain GPT-3.5-turbo's high response diversity and to bound the prompting limitation.","marker":"[18]"},{"why":"Grounds the choice of multiple-choice question format as a standard and efficient way to assess LLM abilities.","marker":"[10]"}],"fun_headline_variants":["Confident LLM physics answers often wrong in reasoning","High certainty LLM responses mask reasoning errors","Reasoning questions reveal LLM confidence-accuracy split","LLM certainty doesn't guarantee physics accuracy","Physics reasoning tasks trip confident LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that asking each question 20 times at temperature 0.7 gives a stable estimate of a model's response distribution, and that the bell-shaped curve's shape is an empirical finding rather than a consequence of how error rate and entropy are computed from the same 20 answers.","fun_headline_variants_meta":{"raw":{"variants":["Confident LLM physics answers often wrong in reasoning","High certainty LLM responses mask reasoning errors","Reasoning questions reveal LLM confidence-accuracy split","LLM certainty doesn't guarantee physics accuracy","Physics reasoning tasks trip confident LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2650,"prompt_tokens":870,"completion_tokens":1780,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":1711}},"tokens_in":486,"tokens_out":1780,"duration_ms":12202,"temperature":1.0,"reasoning_tokens":1711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:21:25.497305+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could resample each question with N=200 at the same temperature and rebuild the error-rate-versus-entropy histograms; if the high-certainty, high-error points in the S and M categories disappear or shift substantially, the reported asymmetry is an artifact of the 20-sample estimate. Alternatively, permuting the correct-answer labels while keeping response counts fixed would reproduce the same envelope, confirming that curve shape alone cannot identify hallucination.","supporting_citations":[{"cited_title":"Buczek, C","cited_arxiv_id":null,"evidence_quote":"Supplies the mlphys101 dataset of 823 physics multiple-choice questions with five answer categories that all experiments and entropy computations run on."},{"cited_title":"Prompting","cited_arxiv_id":null,"evidence_quote":"Guides the manual construction of the three-shot prompts used to elicit letter-only answers from all models."}],"review_version":1}