Pith. sign in

REVIEW 5 major objections 5 minor 18 references

What am I missing here?: Evaluating Large Language Models for Masked Sentence Prediction

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Commercial LLMs are poor at masked sentence prediction in low-structured domains: they generate plausible, locally fluent substitutes rather than faithful reconstructions of the original text.

desk verdict Useful diagnostic with a real fidelity–cohesion split, but the 'poor at MSP' claim needs baselines and a temperature control before it is convincing. read the letter →

arxiv 2508.07702 v1 pith:GBVOXAXO submitted 2025-08-11 cs.CL

Charlie Wyatt , Aditya Joshi , Flora Salim This is my paper · ORCID
classification cs.CL
keywords maskedsentencepredictionnexttokenlargelanguagemodelstextinfillingfidelitycohesiondiscoursecoherenceLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether LLMs trained on next-token prediction can reconstruct a sentence randomly removed from a longer text. Across stories, recipes, and Wikipedia, it evaluates GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash, finding that fidelity to the original sentence is modest at best and weakest in open-ended narratives and encyclopedic prose. In structured procedural text, models do better at recovering the original but human readers more often notice when the generated sentence is substituted. The authors argue this exposes a gap between local fluency and global coherence, one that matters for any application that requires faithful text reconstruction.

What carries the argument

Masked Sentence Prediction (MSP), the task of infilling a single removed sentence, serves as the probe. The paper isolates two capabilities: fidelity, measured automatically via BLEURT, SBERT, ROUGE-1, and BLEU similarity to the original sentence, and cohesion, measured by a blind human preference test between original and generated sentences. Mask position and mask density are varied to control how much context the model can use, making MSP a tunable diagnostic for discourse-level understanding.

What would settle it

Re-run the study with an open-weight model whose training data provably exclude the three corpora and with multiple independent annotators scoring a 100-sample subset per domain. If the open model reconstructs masked sentences with high fidelity (e.g., BLEURT > 0.8) in low-structure domains, or if annotators reliably prefer the original over the generation in stories and Wikipedia, the paper's central claim would be undercut; if the pattern persists, it is confirmed.

Watch

Extended reading notes

Core claim

The paper's central discovery is that commercial LLMs, despite strong benchmark performance, are poor at masked sentence prediction: across all three domains, BLEURT scores rarely exceed 0.55, and in ROCStories and Wikipedia the models tend to return coherent but semantically divergent substitutes rather than the original sentence. The authors attribute this to next-token prediction's incentive for local fluency without explicit pressure for document-level coherence. They also identify a fidelity–cohesion trade-off: in Recipe1M, where procedure constrains the possible completions, fidelity is higher but human preference for the original over the generation is also stronger, meaning violation

Load-bearing premise

The conclusions rest on the assumption that the automatic similarity measures (BLEURT, SBERT, ROUGE-1, BLEU) faithfully capture fidelity and that a single blind annotator's preferences faithfully capture cohesion; if either measurement is noisy or biased, the reported gap between fluency and reconstruction could be overstated.

Editorial extensions

If this is right

  • Any system that relies on LLM infilling to restore damaged or missing text—historical documents, corrupted records, edited legal or journalistic copy—should treat the output as a plausible guess, not a faithful reconstruction.
  • Fluency-only benchmarks will overstate LLM discourse skills; sentence-level reconstruction tests add a distinct signal that should be tracked separately.
  • In structured domains such as procedures, generated substitutions are more detectable, which could aid automated detection of altered or fabricated text.
  • The results point toward training objectives that explicitly model sentence relations or global coherence, rather than relying on next-token prediction alone.
  • Among the tested models, Claude 3.5 Sonnet shows the most consistent fidelity with lower variance, making it the safer default when reconstruction quality matters and variation is costly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The high equal-preference rates in stories and Wikipedia may indicate that many acceptable sentences exist for a given context; if so, fidelity to the original is a conservative yardstick, and a fairer test would measure whether generated sentences are judged as valid continuations independently of the original.
  • The poor performance on first and last masked sentences suggests models use nearby local cues rather than a global document representation; a testable prediction is that adding a distant sentence to the prompt will improve reconstruction less than adding an adjacent one.
  • Because the evaluation corpora almost certainly appeared in pretraining, the low fidelity scores are surprising: memorization would have produced higher similarity. Replicating with an open-weight model trained on data provably excluding the test corpora would clarify whether the deficit is fundamental or an artifact of evaluation.
  • The single-blind-annotator and uncalibrated automatic metrics mean the numerical boundaries should be read as indicative; a multi-annotator replication with inter-annotator agreement could turn the directional finding into a quantitative benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces Masked Sentence Prediction (MSP) as a diagnostic task: a full sentence is removed from a document and a model is asked to infill it. The authors evaluate three commercial LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash) on ROCStories, Recipe1M, and Wikipedia, measuring fidelity to the original sentence with BLEURT, SBERT, ROUGE-1, and BLEU, and cohesion with a single blind human preference test. The central claim is that LLMs are 'poor at predicting masked sentences in low-structured domains' such as narratives and expository text, and that this reflects a limitation of next-token prediction for global, sentence-level reconstruction. The paper also reports that human annotators often rate the generated sentence as equally preferable to the original, and observes higher fidelity in the more structured Recipe1M domain.

Significance. If the central claim holds, the paper addresses an important and timely question: whether next-token pretraining yields models that can reconstruct and reason over document-level structure, not only locally fluent continuations. The task is simple and reproducible, the use of three domains and multiple metrics is a reasonable first step, and the qualitative examples in Appendix B are instructive. The paper also honestly acknowledges several limitations: data contamination, the single-annotator human evaluation, and the small sample size. However, the manuscript as written does not yet provide sufficient evidence for its headline conclusion. The absence of human or chance baselines, the use of temperature-1 sampling, the lack of statistical grounding, and a likely data error in Table 3 all prevent the reported numbers from supporting the strong 'poor at MSP' framing. With appropriate baselines, a fixed decoding protocol, and a more rigorous human evaluation, this could become a useful empirical contribution.

major comments (5)
  1. [§4.1, Table 1] The central claim—that LLMs are 'poor at predicting masked sentences'—is not interpretable without a baseline. BLEURT scores around 0.31–0.53 and ROUGE-1 scores around 0.19–0.36 are absolute similarities to one surface form; they do not have a natural scale. A human baseline performing the same MSP task, or a chance/trivial baseline (e.g., repeating a random sentence from the dataset), is needed to establish that these values are actually low. The authors' own Table 2 shows that in ROCStories and Wikipedia humans prefer the generated sentence as often as the original (60%+ 'Equal Preference'), which undercuts the 'poor' framing. Please add a human reconstruction baseline or a chance-level baseline, and report confidence intervals or significance tests for the model/domain comparisons.
  2. [§3.2] The paper states that all models were queried with 'default decoding settings (e.g., temperature = 1.0)'. At temperature 1.0, the model samples from the full next-token distribution, so even a model that assigns high probability to the original surface form will frequently generate a different, equally plausible sentence. The reported low fidelity may therefore reflect the decoding strategy rather than a representational or capability limitation. This is a load-bearing confound for the central claim. Please either run the MSP task with greedy/zero-temperature decoding, or, if sampling is used deliberately, report multiple samples and the distribution of fidelity scores, and show that the conclusion is robust across decoding settings. Also report all decoding parameters for reproducibility.
  3. [§4.2 and Limitations] The cohesion results are based on a single annotator who is also an author, with n=50 per cell. This is acknowledged in the Limitations section, and the acknowledgment is appreciated, but it remains a serious threat to the paper's secondary claim of an inverse relationship between fidelity and cohesion. The 'Equal Preference' rates are used to argue that LLMs generate plausible substitutes in open-ended domains; that argument needs multiple independent annotators, inter-annotator agreement statistics, and ideally a pre-registered protocol. As written, the human-preference data cannot be treated as reliable evidence.
  4. [Table 3, §4.1.2] In Table 3, the GPT-4o ROCStories rows for 'First' and 'Last' are identical across all four metrics (BLEURT 0.3754, SBERT 0.4305, ROUGE-1 0.1807, BLEU 0.0160). This is almost certainly a data-entry or aggregation error, and it directly affects the mask-position analysis in Figure 3 and §4.1.2, which claims that models perform best on middle masks and worst on final masks. The duplicate row must be corrected, and the position analysis re-run. Additionally, the methodology says masking is applied to 'first, last, or a middle sentence', but Table 3 only reports 'Random', 'First', and 'Last' for ROCStories and Recipe1M, and says Wikipedia used 'only random middle sentences'. Please clarify how 'middle' positions were computed and why they are not reported as a separate condition.
  5. [§4.1.2, Figure 3] The claim that 'models perform best when the masked sentence appears in the middle of the text' is not directly supported by the table: for ROCStories and Recipe1M, the table reports only Random, First, and Last. 'Random' is an average over all positions, so it does not isolate the middle position. If Figure 3 is based on additional position-specific data (e.g., 0.1–0.9 quantile bins in Table 5), that data should be explicitly connected to the figure. The current exposition leaves the reader unable to verify the position effect from the reported numbers.
minor comments (5)
  1. [Appendix A] The text refers to 'Tables ?? and ??'—the cross-references are not resolved. Please fix all table references.
  2. [Throughout] Several formatting issues: 'ROCS TORIES', 'RECIPE 1M', 'W IKIPEDIA' appear with spurious spacing; the dataset names should be consistent (e.g., ROCStories, Recipe1M, Wikipedia).
  3. [Figures] Figures 2–4 present distributions and trends without any confidence intervals, error bars, or statistical tests. Given the small effect sizes (e.g., BLEURT differences of ~0.01–0.03 between models), the figures would be more informative with error bars or bootstrapped CIs.
  4. [§3.2] The models are accessed via API, but no specific version identifiers, API request dates, or max token settings are reported. This is important for reproducibility, especially for commercial models that may change over time.
  5. [Appendix A] Table 5 uses quantile bins (0.0–0.9) for masked position, but the mapping between these bins and sentence-level positions in a document is not explained. Please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical evaluation with standard metrics and no derived predictions.

full rationale

The paper contains no derivation chain that reduces to its own inputs. Its central claim—that commercial LLMs are poor at masked sentence prediction in low-structured domains—is an empirical conclusion drawn from observed fidelity scores (BLEURT, SBERT, ROUGE-1, BLEU) measured on model outputs. No parameters are fitted to a subset of the data and then used to predict a closely related quantity; the metrics are standard, applied post hoc, and not used to construct the task. The only human evaluation uses a single blind annotator who is an author (Limitations), which is a methodological limitation regarding objectivity, but it does not make the reasoning circular: the preference judgments are independent observations, not derived from the automatic metrics. The duplicated numeric values for GPT-4o ROCStories First/Last mask positions in Table 3 suggest a data transcription or processing error, but an error of accuracy is not an instance of circularity. There are no self-citations that carry load-bearing assumptions, and no uniqueness theorem or ansatz is imported from prior work by the same authors. The paper is self-contained as an empirical study; its conclusions may be debated on grounds of task underdetermination, decoding temperature, or insufficient baselines, but these are validity threats, not circularity. Therefore, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is an empirical evaluation study with no derivations, free parameters, or invented constructs. The key load-bearing assumptions are about metric validity, annotator reliability, decoding stability, and sampling representativeness.

assumptions (4)
  • domain assumption BLEURT and SBERT cosine similarity are valid automatic measures of sentence fidelity for the MSP task.
    The paper relies on these metrics without validating them against human fidelity judgments or a baseline; a metric bias could change the magnitude of the reported 'poor' performance.
  • domain assumption A single blind annotator, who is an author, provides representative human cohesion judgments.
    Only one annotator scored 50 items per model and dataset; acknowledged in the Limitations section as limiting generalizability and objectivity.
  • domain assumption Default API decoding settings (temperature 1.0) are appropriate for evaluating generation behavior.
    Temperature 1.0 introduces sampling variance; no multiple runs or seeds are reported, so part of the observed variance could be decoding noise.
  • domain assumption The 400 randomly sampled documents per dataset are representative of each domain's sentence structure.
    Random sampling is used but no stratification or power analysis is described, and the authors note larger experiments would improve significance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What am I missing here?: Evaluating Large Language Models for Masked Sentence Prediction." pith.science (2026). https://pith.science/paper/GBVOXAXO

@misc{pith2026250807702,
  author       = {Pith},
  title        = {Pith review of: What am I missing here?: Evaluating Large Language Models for Masked Sentence Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GBVOXAXO}},
  note         = {Machine review of arXiv:2508.07702}
}
read the original abstract

Transformer-based models primarily rely on Next Token Prediction (NTP), which predicts the next token in a sequence based on the preceding context. However, NTP's focus on single-token prediction often limits a model's ability to plan ahead or maintain long-range coherence, raising questions about how well LLMs can predict longer contexts, such as full sentences within structured documents. While NTP encourages local fluency, it provides no explicit incentive to ensure global coherence across sentence boundaries-an essential skill for reconstructive or discursive tasks. To investigate this, we evaluate three commercial LLMs (GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash) on Masked Sentence Prediction (MSP) - the task of infilling a randomly removed sentence - from three domains: ROCStories (narrative), Recipe1M (procedural), and Wikipedia (expository). We assess both fidelity (similarity to the original sentence) and cohesiveness (fit within the surrounding context). Our key finding reveals that commercial LLMs, despite their superlative performance in other tasks, are poor at predicting masked sentences in low-structured domains, highlighting a gap in current model capabilities.

Figures

Figures reproduced from arXiv: 2508.07702 by the authors.

Figure 1
Figure 1. Our experimental pipeline to evaluate masked sentence prediction. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of BLEURT scores by model and domain. ing the original sentence may exist verbatim in the model’s training data. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. BLEURT by number of masked sentences [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: MSP performance (BLEURT) by the number of sentences in the RECIPE1M dataset. A Detailed Experimental Results This section provides detailed numerical results supplementing the analysis in the main paper [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 3 canonical work pages

  1. [1]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet model card addendum . Accessed: 2025-05-19

    work page 2024

  2. [2]

    Gregor Bachmann and Vaishnavh Nagarajan. 2024. https://arxiv.org/abs/2403.06963 The pitfalls of next-token prediction . Preprint, arXiv:2403.06963

    arXiv 2024

  3. [3]

    Google DeepMind. 2024. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/ Introducing gemini 2.0: our new ai model for the agentic era . Accessed: 2025-05-19

    work page 2024

  4. [4]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakan...

    arXiv 2024

  5. [5]

    Matthew Honnibal and Ines Montani. 2017. spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing. To appear

    2017

  6. [6]

    Yichen Huang, Yizhe Zhang, Oussama Elachqar, and Yu Cheng. 2020. https://doi.org/10.18653/v1/2020.acl-main.226 INSET : Sentence infilling with IN ter- SE ntential transformer . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2502--2515, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.226 2020

  7. [7]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. https://arxiv.org/abs/1910.13461 Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension . Preprint, arXiv:1910.13461

    arXiv 2019

  8. [8]

    Dayiheng Liu, Jie Fu, Pengfei Liu, and Jiancheng Lv. 2019. https://doi.org/10.18653/v1/p19-1406 Tigs: An inference algorithm for text infilling with gradient search . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, page 4146–4156. Association for Computational Linguistics

    doi:10.18653/v1/p19-1406 2019

Show all 18 references
  1. [9]

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. https://arxiv.org/abs/2402.17753 Evaluating very long-term conversational memory of llm agents . Preprint, arXiv:2402.17753

    2024 arXiv

  2. [10]

    Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba. 2019. https://arxiv.org/abs/1810.06553 Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images . Preprint, arXiv:1810.06553

    2019 arXiv

  3. [11]

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696

    2016 arXiv

  4. [12]

    OpenAI, Aaron Hurst, and et al. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    2024 arXiv

  5. [13]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . Preprint, arXiv:1910.10683

    2023 arXiv

  6. [14]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . Preprint, arXiv:1908.10084

    2019 arXiv

  7. [15]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  8. [16]

    Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang, and Nan Duan. 2023. https://arxiv.org/abs/2302.01626 Modeling sequential sentence relation to improve cross-lingual dense retrieval . Preprint, arXiv:2302.01626

    2023 arXiv

  9. [17]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  10. [18]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.