{"id":"e3a70c63-732f-4e36-b002-9daad5f96de6","arxiv_id":"2502.09307","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Humans and large language models show similar comprehension failures on garden-path sentences, with stronger models correlating more closely with human performance across three tasks.","lead":"This paper compares how humans and a large suite of language models handle garden-path sentences, where the first verb looks like it takes an object. It finds that both humans and many LLMs make the same kind of comprehension error, and larger models resemble humans more closely.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The binary scoring rule ('No' correct, 'Yes' wrong) for questions like 'Did the man hunt the deer?' is the load-bearing assumption under every accuracy and correlation result in the paper.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing point: the entire error-pattern analysis depends on the forced-choice coding of 'No' as correct for questions about the non-entailed object reading. I agree this is the most foundational assumption, rather than the statistical reporting issues or the paraphrase filtering, because if this coding is invalid the human 'error' rates themselves dissolve, and with them the comparison to LLMs. I do not claim the coding is wrong: it follows standard psycholinguistic practice and is supported by prior work such as Christianson et al. (2001) and Patson et al. (2009), which the paper cites. But the paper does not independently validate the coding for its own human sample, and the paper explicitly notes that the accurate answer is 'Not necessarily,' which is not identical to 'No.' A three-choice follow-up is a concrete, low-cost way to settle whether the binary 'No' responses are driven by true rejection of the object reading or by a more general 'not determined' response strategy. The reader's conditional accept is appropriate: the central claim is plausible and well supported by converging tasks, but this foundational assumption deserves explicit verification before the human-LLM similarity conclusions are treated as firmly established. My read does not move the reader's verdict; it reinforces the need for the conditional element.","tokens_in":16012,"tokens_out":21819,"duration_ms":217475,"concrete_test":"Run the identical single-trial experiment on the same 228 sentences with a three-option comprehension question (Yes / No / Cannot tell) on a fresh sample of native speakers, in addition to the original binary version. Map 'Cannot tell' responses to 'No' as the paper's binary scoring effectively does, and compare response distributions across GP versus non-GP and plausible versus implausible conditions. If the proportion of 'Cannot tell' responses is substantial (for example, above 20%) and is systematically higher in GP or plausible conditions, then binary 'No' accuracy is not a clean measure of correct parse; re-estimate all human condition means and the human-LLM correlations from the three-option data to determine whether the three-factor pattern and the model-size trend survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All human and LLM accuracy numbers, condition means, and the Kendall/Spearman correlations in Section 5 are computed against the coding that 'Yes' is always wrong and 'No' is always right for GP questions such as 'Did the man hunt the deer?' given 'While the man hunted the deer ran into the woods.' The paper itself concedes in Section 2 that the logically accurate answer is 'Not necessarily': the sentence does not entail that the man hunted the deer, but it also does not entail that he did not. In a forced Yes/No format, 'No' is therefore an approximation that conflates recognizing non-entailment with asserting falsity, and it scores as fully wrong a respondent who accepts the plausible but non-entailed object reading. Every comparison between humans and LLMs inherits this coding. This is the standard convention in the garden-path literature the paper extends, so it is not an internal inconsistency, but it is the least externally secured link in the argument: if a substantial fraction of 'No' responses reflect uncertainty avoidance rather than successful reanalysis, or if 'Yes' responses reflect a pragmatically licensed inference rather than a lingering misparse, the three-factor pattern and the claimed human-LLM similarity would need reinterpretation. The paraphrasing and text-to-image results for LLMs provide converging evidence, but no human paraphrase or three-choice data are collected here to validate the binary coding for the human side.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares humans and a broad suite of LLMs on object/subject garden-path sentences using the same comprehension-question task. It manipulates three factors—GP vs. non-GP syntax, plausible vs. implausible direct-object readings, and optionally transitive vs. reflexive/unaccusative verbs—and measures accuracy for humans (Prolific, speeded word-by-word presentation) and token probabilities for LLMs. It also reports paraphrase and text-to-image tasks for LLMs. The main empirical claims are that all three hypothesized factors affect humans and many LLMs, that stronger models correlate better with human item difficulty, and that the two auxiliary tasks reproduce the same error pattern.","tokens_in":16221,"tokens_out":9973,"duration_ms":99208,"significance":"If the pattern holds, this is a useful contribution to the emerging comparison between human and LLM sentence processing: it uses a same-task design, covers multiple model families and checkpoints, and triangulates with paraphrasing and image generation. The human experiment is carefully controlled (single-trial design, native speakers, GLMM analyses), and the full sentence materials are provided in Appendix B. The main limitations are the absence of confidence intervals for the rank correlations, the unvalidated binary scoring rule for the comprehension questions (acknowledged in Section 2), and the treatment of timeouts as incorrect. Still, the central observation—garden-path difficulty is not unique to humans—is credible and should be of interest to both psycholinguists and NLP researchers.","major_comments":[{"comment":"The claim that 'all models show a high Spearman rank correlation with human data' is contradicted by the condition-level numbers in Table 4. For example, Olmo-7B-Tokens-8B has accuracies [0.655, 0.665, 0.654, 0.663, 0.649, 0.657] across the six conditions; against the human condition means this yields Spearman rho of about 0.09, not a high correlation, and Olmo-7B-Tokens-111B is essentially flat across conditions, making its rank correlation near zero or undefined. The paper should report the per-model Spearman values (with a note on tied ranks) and restrict the claim to the models that actually show high correlation, or provide a statistical summary with confidence intervals. This is load-bearing because the 'stronger models are more similar to humans' narrative depends on the reliability of these correlations.","section":"Section 5, Figure 5 and Table 4"},{"comment":"The paper states that for questions like 'Did the man hunt the deer?' the logically accurate answer is 'Not necessarily', but then scores 'yes' as wrong and 'no' as right. All human and LLM accuracy numbers, condition means, and rank correlations inherit this coding. This is the standard convention in the garden-path literature, so it is not an internal inconsistency, but it is a validity limitation: some 'no' responses may reflect uncertainty avoidance rather than successful reanalysis, and some 'yes' responses may reflect a pragmatically licensed inference rather than a lingering misparse. The authors should report a sensitivity analysis (e.g., excluding timeouts, or re-scoring with a permissive rule) and should also state whether any human data were collected on a three-option or paraphrase version to validate the forced binary coding.","section":"Section 2, scoring of comprehension questions"},{"comment":"The procedure marks a response as incorrect if it is not given within five seconds after the question. Because garden-path conditions are likely to slow response times, this rule can inflate the human GP deficit purely through a speed-accuracy trade-off. The paper does not report timeout rates or model them statistically. Please report the proportion of timed-out trials by condition and re-run the main GLMMs excluding or modeling timeouts, to show that the human effects are not an artifact of the response deadline.","section":"Section 3.1, Procedure"}],"minor_comments":[{"comment":"There are typos: 'paraphraing' should be 'paraphrasing', and 'Gloabal' should be 'Global'.","section":"Section 6 heading and Figure 4 caption"},{"comment":"The plausibility manipulation was selected using GPT-4 ratings, and GPT-family models are then evaluated on those same items. This does not by itself invalidate the results because the effect also appears in humans and non-GPT models, but the paper should discuss the possible circularity and ideally show that the plausibility effect survives when the items were selected by human norms (or by a non-GPT model).","section":"Section 3.1, plausibility pretesting"},{"comment":"Spearman correlations based on six condition means are coarse and unstable; the paper should report the actual correlation values, their uncertainty, and the handling of tied ranks, especially for near-flat models like the early OLMo checkpoints.","section":"Section 5, correlation methodology"},{"comment":"The figure caption does not explicitly define the order of the six condition bars; please add a legend or a sentence such as 'conditions are GP/non-GP for plausible, implausible, and reflexive verb types'.","section":"Figure 1"},{"comment":"The sentence 'The man hunted the child. seem to be too out of distribution for our LLMs to generate' contains a grammatical error and should be rephrased; also, the format metric description says 'consistes' instead of 'consists'.","section":"Section 6.1"},{"comment":"The paper does not state whether the experimental materials, human response data, and model output probabilities will be released; providing these would strengthen reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about GPT-4 being nearly opposite to humans does not survive inspection of Table 4: GPT-4's condition-level accuracies are very close in ranking to the human means (Spearman about 0.94), so that specific concern should not be a basis for rejection. The more serious issues are the overclaim in Section 5 that all models show high Spearman correlation, which is false for some OLMo checkpoints, and the need for a sensitivity analysis around the binary scoring rule and the timeout-as-incorrect convention. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things to know. This is the most systematic human-LLM comparison of garden-path comprehension I've seen, with one human experiment, many models, and three explicit hypotheses. The human data are solid and support all three factors, though the syntactic effect in the plausible condition is only marginal (p = .065), not significant. And the paper's claim that 'all models show a high Spearman rank correlation with human data' is not backed by its own Table 4: GPT-4's condition-level accuracies look almost inversely ranked relative to humans, so that sentence is false as written.\n\nWhat's genuinely new is the breadth and the same-task design. Prior work (Arehalli et al. 2022, Irwin et al. 2023, Li et al. 2024) already showed LLMs make garden-path-like errors, but this paper tests three contributing factors in carefully controlled sentence sets, uses the same comprehension question for humans and models, and validates on paraphrasing and image generation. The paraphrase and image tasks provide converging evidence, and the item-level AUC between question accuracy and paraphrase correctness is a thoughtful addition. The human experiment is well run: 10 participants per item, single-trial presentation, standard word-by-word timing.\n\nThe soft spots are statistical rather than conceptual. The Spearman overclaim needs fixing; I'd like item-level data or at least a corrected summary. The p = .065 effect should be reported as a trend. The paraphrase experiment filters out two models that failed the format checks; that's reasonable but it is a post-hoc filter and should be flagged. The binary scoring rule — 'Yes' always wrong, 'No' always right for the GP question — is standard in the garden-path literature, and the paper honestly notes that the logically accurate answer is 'Not necessarily.' That is a real limitation: it conflates rejecting an entailment with asserting its negation, and no human paraphrase data are collected to verify that 'No' responses reflect successful reanalysis. But the converging LLM paraphrase and image results make me less worried that the whole pattern is an artifact. The circularity concern about using GPT-4 for plausibility pretesting is minor, because the plausibility effect appears in humans and in non-GPT models.\n\nBottom line: this paper is for psycholinguists interested in LLM evaluation as a window into human processing, and for NLP researchers who want a cheap behavioral benchmark for garden-path effects. It deserves a serious referee. I'd recommend conditional accept: request corrected correlation reporting, item-level analyses, and a more transparent discussion of the scoring rule and the paraphrase filter.","headline":"A systematic, well-run comparison of garden-path comprehension in humans and many LLMs with honest human data, but the paper overclaims by-condition Spearman correlations and should report a marginal syntactic effect as non-significant.","tokens_in":16818,"tokens_out":3526,"would_cite":true,"duration_ms":34640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Both humans and LLMs misread garden-path sentences for the same three reasons.","keywords":["garden-path sentences","language models","human sentence processing","comprehension","psycholinguistics","plausibility","transitivity","paraphrasing"],"falsifier":"Run the same comprehension questions with a three-way forced choice (Yes / No / Not necessarily) on the same garden-path and non-garden-path items. If participants select 'Yes' or 'No' at similar rates across structures, or if accuracy computed with 'Yes' counted as correct matches non-GP accuracy, the reported garden-path deficit is an artifact of the binary scoring rule.","tokens_in":15766,"feed_emoji":"🧠","tokens_out":10853,"duration_ms":88597,"temperature":0.7,"pith_summary":"This paper asks whether large language models fail on the same sentences that reliably trip up human readers. It tests humans and a wide range of LLMs on the same comprehension questions about garden-path sentences—sentences like \"While the man hunted the deer ran into the woods,\" where the first clause tempts a misparse. The paper claims that three factors—the garden-path syntax itself, the semantic plausibility of the noun as a direct object, and whether the verb is transitive—drive errors in both humans and models, and that stronger models resemble humans more closely. If true, it means LLM errors on this construction are not arbitrary; they follow the same psycholinguistic pressures that shape human misreading, and LLMs could serve as a test bed for theories of human sentence processing. The claim is validated further by paraphrasing and text-to-image tasks, which show the same error pattern.","feed_headline":"Garden-path sentences fool LLMs exactly where they fool humans","feed_subtitle":"Syntax, plausibility, and verb type drive the same misreadings in people and language models; larger models align with humans more.","key_machinery":"The central object is the object/subject garden-path sentence, a temporarily ambiguous construction of the form \"While [embedded verb] [NP] [main verb] ...\" whose first parse (NP as object of embedded verb) must be revised. The argument works by manipulating three features—clause order (garden-path vs. non-garden-path), plausibility of the NP as a direct object, and verb type (optionally transitive vs. reflexive/unaccusative)—and measuring, on identical items, human accuracy and the probability LLMs assign to the correct answer token. The comparison metric is the rank correlation between human item accuracy and LLM answer probability, with Spearman correlation across the six experimental conditions as a secondary check.","core_discovery":"The paper's central discovery is that object/subject garden-path sentences—where a noun phrase is temporarily attachable as the object of an embedded verb—produce the same pattern of comprehension errors in humans and in large language models. Using a single-trial comprehension task with questions like \"Did the man hunt the deer?\", the authors show that accuracy falls when the sentence requires syntactic reanalysis, when the noun is a plausible object for the verb, and when the verb is optionally transitive rather than reflexive or unaccusative. For LLMs, the same manipulations move answer probabilities in the same direction, and the rank correlation of item difficulty with human accuracy rises with model scale (the strongest model reaches 78% accuracy and the highest correlations). The authors take this as evidence that LLMs and humans share underlying sensitivity to the same syntactic and semantic pressures, and they corroborate the finding with a paraphrasing task and image generation, where the same misinterpretations appear.","pith_inferences":["An unresolved question is whether the reported 'misinterpretation' is partly a task artifact: the paper scores 'yes' as wrong for 'Did the man hunt the deer?' even though the sentence permits that reading. Reanalyzing the same data with a permissive scoring rule—counting 'yes' as acceptable—would show how much of the human-model error pattern survives.","The paper groups all optionally transitive verbs together despite finding only a weak correlation (≤0.19) between transitivity bias and accuracy; a finer-grained analysis separating high- and low-bias verbs might reveal that verb-specific statistics, not a categorical distinction, drive the effect.","If LLM errors mirror human errors because both reflect shallow distributional plausibility, then making the implausible condition more extreme (e.g., 'the rhino ran into the woods' after 'hunted') should push both humans and models toward ceiling; a graded plausibility curve could separate the syntactic and semantic contributions."],"forward_implications":["If the claim holds, LLM performance on comprehension questions can serve as a behavioral proxy for human garden-path processing, letting psycholinguists run large-scale, cheap replications of reading studies.","Model scaling (size and pretraining tokens) should continue to increase alignment with human error patterns, so the correlation becomes a usable benchmark for whether a new model is becoming more 'human-like' in this specific sense.","The three-factor account—syntax, plausibility, verb transitivity—should predict error rates on new garden-path items, not just the sets tested here; a new set of sentences varying these factors should reproduce the same ordering.","Since the effect appears in comprehension, paraphrasing, and image generation, the misinterpretation is not an artifact of the question format; any model that understands a GP sentence correctly should also avoid the error in open-ended generation."],"supporting_citations":[{"why":"Supplies the original demonstration that comprehension questions after garden-path sentences show lingering misinterpretation, and 24 of the 45 sentence sets come from this study.","marker":"Christianson et al. (2001)"},{"why":"Supplies the human paraphrasing evidence that the paper extends to LLMs, motivating the paraphrasing task.","marker":"Patson et al. (2009)"},{"why":"Provides the baseline finding that LLM-based metrics underestimate human garden-path processing difficulty, which the paper's direct comparison revisits.","marker":"Arehalli et al. (2022)"},{"why":"Prior observation that four LLMs make garden-path parsing errors similar to humans, which the paper extends to a larger model suite and three factors.","marker":"Li et al. (2024)"},{"why":"Justifies the single-trial design used to prevent fatigue and learning effects in the human experiment.","marker":"Christianson et al. (2022)"},{"why":"Foundational eye-movement evidence for the garden-path effect, grounding the claim that reanalysis is difficult.","marker":"Ferreira and Henderson (1990)"},{"why":"Foundational evidence on verb bias and plausibility in temporarily ambiguous sentences, grounding the plausibility and verb-type hypotheses.","marker":"Garnsey et al. (1997)"}],"fun_headline_variants":["LLMs and humans fall for the same garden-path traps","Garden-path sentences trip up LLMs just like humans","AI models mirror human misreads on garden-path sentences","When sentences mislead, LLMs and humans agree","Syntactic complexity fools LLMs as much as humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire error pattern rests on the labeling rule that a 'yes' answer to 'Did the man hunt the deer?' is wrong for the sentence 'While the man hunted the deer ran into the woods,' because the authors treat the sentence as not entailing that the man hunted the deer; if a permissive reading that allows 'hunt the deer' were treated as correct, the reported human and model accuracy gaps would shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["LLMs and humans fall for the same garden-path traps","Garden-path sentences trip up LLMs just like humans","AI models mirror human misreads on garden-path sentences","When sentences mislead, LLMs and humans agree","Syntactic complexity fools LLMs as much as humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1485,"prompt_tokens":882,"completion_tokens":603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":524}},"tokens_in":498,"tokens_out":603,"duration_ms":5918,"temperature":1.0,"reasoning_tokens":524,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:57:19.912442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same comprehension questions with a three-way forced choice (Yes / No / Not necessarily) on the same garden-path and non-garden-path items. If participants select 'Yes' or 'No' at similar rates across structures, or if accuracy computed with 'Yes' counted as correct matches non-GP accuracy, the reported garden-path deficit is an artifact of the binary scoring rule.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the original demonstration that comprehension questions after garden-path sentences show lingering misinterpretation, and 24 of the 45 sentence sets come from this study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the human paraphrasing evidence that the paper extends to LLMs, motivating the paraphrasing task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the single-trial design used to prevent fatigue and learning effects in the human experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Foundational eye-movement evidence for the garden-path effect, grounding the claim that reanalysis is difficult."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Foundational evidence on verb bias and plausibility in temporarily ambiguous sentences, grounding the plausibility and verb-type hypotheses."}],"review_version":1}