{"id":"57f040fc-18ca-4df3-8623-c41086cc4005","arxiv_id":"2504.15604","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Comparing GPT-2 and Llama-2 on belief-tracking story prompts, the paper finds that added context and higher temperature reduce the probability of each model's own most likely next token.","lead":"This paper tests how two language models, GPT-2 and Llama-2, predict the next words in short stories about what characters know, after extra sentences are added to the stories and the model temperature is changed. It reports that added context and higher temperature lower the probability of the model's top choice, but it does not check whether that choice is actually the right answer.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ground-truth scoring: 'accuracy' is top-token confidence, so the central claims are unsupported.","rationale":"The reader's weakest assumption is exactly the linchpin of the paper's central claim. The reported 'accuracy' is the average probability assigned to the model's own argmax token, not the frequency with which that token matches an expected answer. The appendix plots show that both models assign probability 1.0 to different tokens at temperature 0.01, and many of those tokens are subword fragments or semantically wrong locations; that is direct visual evidence that top-token probability is not accuracy. The tokenizer mismatch between GPT-2 (e.g., 'leather, brief, case') and Llama-2 (e.g., 'le, ather, brief') further complicates any cross-model comparison of token-level probabilities, though the absence of ground truth is the more fundamental problem. The paper does include reproducible code and a large set of plots, and the confidence trends with temperature and infills may be real, but those trends cannot support the accuracy and Theory of Mind superiority claims as written. A repair would require scoring against ExploreToM gold answers and rephrasing the claims as confidence or calibration trends. I therefore agree with the reader's rejection of the current version.","tokens_in":97644,"tokens_out":2984,"duration_ms":31958,"concrete_test":"Re-run the evaluation with ground-truth scoring: for each (story, question order, infill level, temperature), take the model's top-1 and top-5 tokens for each of the three predicted steps, decode them into strings, and compare them with the ExploreToM gold answer for that question (or with the canonical continuation of the story). Report accuracy as the fraction of exact or prefix matches, separately from average top-token probability. Then re-plot accuracy against infill level and GPT-2 vs Llama-2. If the downward trend and Llama-2 advantage do not survive ground-truth scoring, the central claims fail; if they do survive, the paper should report both metrics separately.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claims, stated in the Conclusion, are that increasing infill density 'reduces prediction accuracy for both models' and that Llama-2 'outperforms GPT-2 in higher-order reasoning tasks, particularly in second-order inference questions.' These claims rest on equating the model's own highest-probability token with the correct answer. Section 5.1 defines the 'correct token (CT)' as 'the token with the highest probability in the list,' Section 5.2 averages CT probabilities, and Section 6 then relabels those averages as 'prediction probability' and 'accuracy.' Nowhere is any prediction compared with the ExploreToM gold answer or with the actual next token of a canonical continuation. The reported effects are therefore confidence effects: at temperature 0.01 the CT probability is often 1.0 for both models, but that does not indicate the token is semantically correct. The displayed CTs themselves make this clear—for the second-order prompt of Story 1, GPT-2's CT is 'leather, brief, case' at infill 0 but 'hotel, lobby, .' at infill 1, both with probability 1.0 at temperature 0.01. Without an external reference, the headline 'accuracy' statements do not follow from the measurements.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper constructs a dataset from 10 ExploreToM stories by programmatically inserting 0, 1, 4, 16, or 64 GPT-4-generated infill sentences, then prompts GPT-2 and Llama-2-7b-chat-hf to predict the next three tokens for zero-, first-, and second-order theory-of-mind questions under several temperature settings. The authors define the 'correct token' as the model's own highest-probability token, average the probabilities of these top tokens, and report that increasing infill density reduces prediction 'accuracy' for both models, that higher temperatures reduce 'accuracy', and that Llama-2 'outperforms' GPT-2, particularly on second-order questions. The central results are presented as claims about prediction accuracy and reasoning performance, but the evaluation never compares any prediction to an external ground-truth continuation or to the ExploreToM gold answers.","tokens_in":97841,"tokens_out":3273,"duration_ms":30833,"significance":"If the claims were valid, the study would offer useful evidence on how context length, temperature, and reasoning complexity affect next-token prediction in theory-of-mind tasks. The experimental matrix is broad (two models, five infill levels, five temperatures, three question orders, ten stories), and the authors provide a code repository and a reproducible prompting protocol, which are strengths. However, the validity of every headline claim depends on an external standard of correctness, and that standard is absent. The reported quantities are self-confidence scores, not accuracies, so the paper's main conclusions do not follow from the measurements. The work is better framed as an exploratory study of model confidence and distributional spread, but as written the accuracy-based conclusions are unsupported.","major_comments":[{"comment":"The central evaluation is self-referential. Section 5.1 states that 'The token with the highest probability in the list is referred to as the correct token (CT)', and Section 5.2 averages the probabilities of these CTs. Section 6 then relabels these averages as 'prediction probability' and 'accuracy', and Section 7 concludes that 'increasing infill density reduces prediction accuracy for both models'. Because CT is defined as the model's own argmax, the reported 'accuracy' is a measure of self-confidence, not correctness. No comparison is made to the actual next token of a canonical continuation or to any ExploreToM gold answer. The displayed tokens make the problem concrete: for the second-order prompt of Story 1 (Figure 1), GPT-2's top-token sequence at temperature 0.01 is 'leather, brief, case' at infill 0 but 'hotel, lobby, .' at infill 1, both with average probability 1.00; both cannot be correct, yet both are scored as fully correct by the paper's metric.","section":"Section 5.1, 5.2, 6, 7"},{"comment":"The temperature effects on 'accuracy' are also not measured. The observation that higher temperatures lower average top-token probability is a near-definitional consequence of the softmax temperature sharpening the distribution, not evidence about whether the predicted token is semantically correct. The claim in Section 6 that 'higher temperatures also led to greater response diversity ... this also increased the likelihood of incorrect or less coherent outputs' is not supported by any ground-truth scoring or human evaluation. To support such a claim, the authors would need to sample completions at each temperature and compare them against expected answers, or at least report top-1 agreement with the original story continuation.","section":"Section 6, 7"},{"comment":"The cross-model comparison is not meaningful as reported. The conclusion that 'Llama-2 consistently outperforms GPT-2 in accuracy' and 'outperforms GPT-2 in higher-order reasoning tasks' rests on comparing average top-token probabilities across models whose tokenizers and top-token strings differ completely. For example, in Figure 2 Llama-2's top tokens are often subword fragments such as 'le, ather, brief', while GPT-2's are full words in Figure 1; there is no common output space over which the comparison is defined. Additionally, no statistical tests, confidence intervals, or repeated trials are reported, and some conditions are skipped for GPT-2 because of its 1024-token limit, as acknowledged in Section 6. With 10 stories and unmatched conditions, the comparative accuracy claims are not established.","section":"Section 6, 7, Figures 1-2"},{"comment":"A ground-truth evaluation was available but not used. The prompts constructed in Section 5 continue actual story sentences (e.g., 'Liam moved the silver letter opener to the'), and the ExploreToM dataset used in Section 4 contains narrative facts and question answers that could serve as a reference. The paper's own research questions in Section 5.3 ask 'does the probability of correct token prediction decrease?', but no operational definition of 'correct' beyond the model's highest-probability token is ever given. This is a load-bearing gap: without an external reference standard, the paper cannot distinguish between a model becoming less confident and a model becoming less accurate.","section":"Section 4.2, 5"}],"minor_comments":[{"comment":"The formula 'Average Probability = P P N' is unclear; please define P and N explicitly and clarify whether the average is taken over the three predicted tokens within a single completion, over stories, or over both.","section":"Section 5.2"},{"comment":"The abstract and Section 5 list temperature settings as 0.01, 0.5, 1.0, and 2.0, but Section 6.2 and the appendix also report results at 1.5; this inconsistency should be reconciled.","section":"Section 5, 6.2, Appendix"},{"comment":"Several figures contain tokens rendered as '#', '!', '$', '□', and '<unk>'; these special tokens and truncated subwords should be explained in the caption or in Section 5 so readers can interpret them.","section":"Figures 1-45"},{"comment":"The tolerance of ±3 sentences for higher infill conditions is mentioned but not quantified in the analysis; please describe how the actual infill counts vary and whether they were controlled for in the comparison.","section":"Section 4"},{"comment":"The conclusion states that 'GPT-2 remains a faster alternative' but no runtime or computational efficiency measurements appear anywhere in the paper.","section":"Section 7"}],"recommendation":"reject","confidential_remarks":"The core problem is not a presentation issue but the definition of the evaluation metric: 'accuracy' is the model's own top-token probability, so the central claims about accuracy, temperature effects, and cross-model superiority cannot be supported. I see no indication of deliberate misrepresentation; rather, the paper appears to be an early-stage exploratory report that would need a substantially redesigned evaluation (ground-truth scoring, sampling, and statistical analysis) before it could support its conclusions. I would not encourage a major revision unless the authors are willing to re-run the experiments with an external correctness criterion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper measures the model's own top-token probability and then reports that as accuracy. Section 5.1 defines the 'correct token (CT)' as the highest-probability token, Section 5.2 averages CT probabilities, and Sections 6–7 call the result 'prediction accuracy' and 'accuracy.' There is no comparison to ExploreToM ground-truth answers anywhere. The central claims—that infill density reduces accuracy and that Llama-2 outperforms GPT-2 on second-order questions—are therefore unsupported. The temperature trend is real but near-definitional: softmax temperature flattens the distribution, so the argmax's probability falls. That is confidence, not correctness. The stress-test note holds up; the displayed CTs make it concrete, e.g., the same second-order prompt yields 'leather, brief, case' at infill 0 and 'hotel, lobby, .' at infill 1, both at probability 1.0 under temperature 0.01.\n\nCredit where it is due: the infill manipulation is a genuinely new evaluation axis. Generating 1/4/16/64-sentence variants of the same stories, running two model families over temperature and question order, and shipping the code is real work. The qualitative observation that added context shifts the argmax token is worth reporting as a context-sensitivity or instability result. The plots are legible and the appendix is thorough.\n\nSoft spots beyond the central flaw: the model comparison is not apples-to-apples because GPT-2's 1024-token limit skips many long-context questions, and the paper just notes this. There are no repeated samples or error bars, so all reported differences could be noise. Temperature reporting is inconsistent—the abstract lists four settings, while Section 6.2 uses five including 1.5. The ±3-sentence tolerance on infill counts is a free parameter that is never analyzed. Despite the Theory of Mind framing, the prompts are cloze completions with no validation that the expected continuation is unique.\n\nWho this is for: someone studying next-token confidence or context-sensitivity in small models could salvage the measurement by rescoring against gold tokens and rephrasing the claims as confidence trends. As written, it should not be cited for ToM accuracy.\n\nMy recommendation: I would not desk-reject this if I were the editor—a serious referee could demand ground-truth scoring, variance reporting, and consistent temperature handling, and the infill axis is worth examining. But my own verdict is reject with major revision. It is also a good reading-group example of circular evaluation.","headline":"A small confidence-measurement study whose headline claims about Theory of Mind accuracy do not survive contact with its own definition of 'correct token.'","tokens_in":98400,"tokens_out":2724,"would_cite":false,"duration_ms":29065,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding more context sentences lowers next-token prediction confidence in Theory of Mind prompts, and that Llama-2-7b handles the added complexity better than GPT-2, especially on second-order belief questions.","keywords":["next-token prediction","Theory of Mind","GPT-2","LLaMA-2","infill context","temperature","belief reasoning","token probability"],"falsifier":"Take the same prompts and compare each top token against the actual completion string from the ExploreToM dataset; if the top token is often not the expected answer while its probability stays high, the reported 'prediction accuracy' collapses to confidence rather than correctness.","tokens_in":97429,"feed_emoji":"🧠","tokens_out":1819,"duration_ms":19074,"temperature":0.7,"pith_summary":"The paper tests whether two language models, GPT-2 and Llama-2-7b-chat-hf, can track characters' beliefs and false beliefs by predicting the next token in short narratives. It builds story variants with 0, 1, 4, 16, or 64 extra sentences inserted by GPT-4, then measures the probability the models assign to their top token across three reasoning levels: factual state tracking, first-order belief reasoning, and second-order recursive belief reasoning. The central claim is that more context reduces top-token probability for both models, but Llama-2 degrades less and achieves higher top-token probability than GPT-2, particularly on second-order questions. A sympathetic reader would care because next-token probability is offered as a window into whether language models implicitly represent who knows what in a conversation.","feed_headline":"More context lowers next-token confidence in ToM prompts","feed_subtitle":"Llama-2 holds up better than GPT-2 as infill sentences grow, and both models struggle most on second-order belief questions.","key_machinery":"The measuring instrument is top-token probability: for each prompt the model produces a ranked token list, the highest-probability token is labeled the correct token (CT), and the paper averages CT probabilities over three predicted tokens as its accuracy metric. This machinery lets the authors study context, temperature, and reasoning level without labeling ground truth, but it also means the reported 'accuracy' is really the model's self-assigned confidence.","core_discovery":"On the ten ExploreToM stories and their infilled variants, increasing infill density reduces the average probability of the model's most likely next token, and Llama-2-7b-chat-hf is more resilient to this degradation than GPT-2. Second-order questions, which require reasoning about one character's belief about another character's belief, produce the largest probability drops and the greatest response diversity in both models. Higher temperature settings flatten the token distribution and spread probability across alternatives, with GPT-2 showing more variability and Llama-2 retaining more consistent top-token confidence.","pith_inferences":["The paper's 'prediction accuracy' is better read as 'model confidence', since the top token is never checked against the ExploreToM ground-truth answer; an extension would compare CT tokens to the expected completion string.","The infill sentences are generated by GPT-4 and may not match the style or factual structure of the original stories, so the observed decline could partly reflect distribution shift rather than reasoning failure.","A testable follow-up is to insert matched distractors that are semantically relevant versus irrelevant to the belief question, which would separate genuine context-tracking from superficial priming effects."],"forward_implications":["Longer narrative context, at least when it is programmatically inserted, tends to lower a model's confidence in its next-token choice on Theory of Mind-style prompts.","Larger models with longer context windows, such as Llama-2-7b-chat, can handle more injected context without as much confidence loss as GPT-2.","Second-order belief reasoning is harder for both models than zero-order state tracking, as measured by top-token probability and response diversity.","Higher sampling temperatures reduce top-token probability for both models and increase the spread of alternative tokens, so temperature is a real control knob for prediction confidence.","The 1024-token input limit of GPT-2 caused some 64-infill questions to be skipped, so cross-model comparisons at the highest context level are incomplete."],"supporting_citations":[{"why":"Supplies the ten stories and the zero-order, first-order, and second-order question taxonomy that define the experiment.","marker":"[1]"},{"why":"Identifies the Llama-2-7b-chat-hf model used and its context window.","marker":"[5]"},{"why":"Identifies the GPT-2 model used and its 1024-token context limit.","marker":"[6]"},{"why":"Provides the comparative ToM-evaluation background against which next-token probing is positioned.","marker":"[2]"},{"why":"Prior work on second-order belief reasoning that motivates the question-complexity axis.","marker":"[4]"}],"fun_headline_variants":["Llama-2 beats GPT-2 on ToM next-token as context grows","ToM context hurts next-token confidence, Llama-2 resists","More infill sentences weaken LLM Theory of Mind predictions","Second-order ToM questions trip up next-token prediction most"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole accuracy story rests on assuming the model's own highest-probability token is the correct answer, so no ground-truth comparison is needed.","fun_headline_variants_meta":{"raw":{"variants":["Llama-2 beats GPT-2 on ToM next-token as context grows","ToM context hurts next-token confidence, Llama-2 resists","More infill sentences weaken LLM Theory of Mind predictions","Second-order ToM questions trip up next-token prediction most"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2456,"prompt_tokens":1001,"completion_tokens":1455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":1377}},"tokens_in":617,"tokens_out":1455,"duration_ms":9707,"temperature":1.0,"reasoning_tokens":1377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:22:09.551687+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same prompts and compare each top token against the actual completion string from the ExploreToM dataset; if the top token is often not the expected answer while its probability stays high, the reported 'prediction accuracy' collapses to confidence rather than correctness.","supporting_citations":[{"cited_title":"Meta’s Llama-2 Model: meta-llama/Llama-2-7b-chat-hf","cited_arxiv_id":null,"evidence_quote":"Identifies the Llama-2-7b-chat-hf model used and its context window."},{"cited_title":"OpenAI’s GPT-2 Model: GPT-2 on Hugging Face","cited_arxiv_id":null,"evidence_quote":"Identifies the GPT-2 model used and its 1024-token context limit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the comparative ToM-evaluation background against which next-token probing is positioned."}],"review_version":1}