{"id":"4f5ff625-bb88-41e5-9f10-ec99a01c0921","arxiv_id":"2501.12980","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Most tested LLMs fail to reproduce human implicit causality biases in coreference, coherence, and referring-expression form, even when they show partial coreference effects.","lead":"Researchers tested whether German-language LLMs show the same discourse biases as humans when continuing sentences with verbs like 'fascinate' and 'admire'. They found that almost no tested model matched human behavior on coreference, explanation, or referring-expression biases, and they propose the test set as a benchmark for evaluating LLM discourse skills.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unvalidated constrained-beam emulation of the forced-reference task (Section 8.1) is the weakest load-bearing step: it restricts the response set to three forms, omits der/die demonstratives, and constrains only the first token's gender, so the null form-bias result may be an artifact.","rationale":"I read the paper as a careful behavioral benchmark with a mostly negative result. The central claim has three pillars; the form-bias pillar (Experiments 3 and 4) is the most fragile because it is the only one that abandons free generation in favor of a constrained decoding emulation of the human forced-reference task. The authors explicitly say the forced-reference procedure was emulated (Section 8.1) and provide no validation that the emulated continuations behave like human productions. The emulation has two concrete defects: (1) the allowed first-token set excludes the der/die demonstrative class that is part of the human annotation categories, and (2) constraining only the first token's gender does not guarantee coreference to the intended argument, and no reference-validity filter was applied. Either defect could change the measured proportions of pronouns versus names and thereby erase or create a form bias. A targeted validation study of constrained outputs would settle this. The coreference and coherence results are also subject to annotation and exclusion caveats, but they rely on more standard generation and the authors provide some manual checks; the form-bias result is the least supported. Because the concern is testable and not demonstrably fatal, the reader's CONDITIONAL verdict is appropriate; I would not change it.","tokens_in":28091,"tokens_out":10581,"duration_ms":128659,"concrete_test":"Have two annotators judge a sample of 200 constrained continuations per condition from each model in Experiments 3 and 4 for (a) whether the first referring expression uniquely refers to the intended subject/object and (b) whether the continuation is natural. If more than 10% of continuations fail either criterion, the emulation is not comparable to the human forced-reference task and the null form-bias result is uninterpretable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central negative claim that all models lack human-like referring-expression form bias (Experiments 3 and 4) rests entirely on the forced-reference emulation described in Section 8.1. In the human gold standard (Bott and Solstad, 2023), participants were instructed to continue about a particular referent and could choose any form; annotation coded personal pronouns, demonstratives of both dieser- and der/die-type, and repeated names. In the LLM emulation, the decoder only allowed first tokens that are personal pronouns, the prompt names, or dieser/diese; der/die demonstratives were excluded. Moreover, constraining only the first token's gender does not guarantee that the model establishes reference to the intended argument: the token can be gender-congruent without being anaphoric to the target (e.g., sie as plural), or the rest of the clause can switch reference. Because no acceptability or reference-validity filter was applied to LLM continuations, the measured form proportions may reflect the constraint set and decoding artifacts rather than the model's form preferences under genuine forced reference. If the emulation distorts the relative availability of names versus pronouns, the null form-bias result, and hence the claim that no model reproduces the full trio of human biases, is not yet secure. The authors acknowledge that the forced-reference procedure was only emulated (Section 8.1) and provide no validation that emulated continuations are comparable to human productions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper probes ten mid-size, open-weight, non-instruction-tuned LLMs (German GPT-2, German BLOOM 350M/1.5B/6.4B, mGPT, and XGLM 0.564B/1.7B/2.9B/4.5B/7.5B) for three Implicit Causality (IC) discourse biases that are well-established in human psycholinguistic experiments: next-mention coreference bias (Experiment 1), explanation vs. other coherence relations (Experiment 2), and anaphoric form choice under forced reference (Experiments 3 and 4). Human gold-standard data come from the authors' earlier studies (Solstad and Bott 2022 for coreference and coherence; Bott and Solstad 2023 for referring forms). The central empirical findings are negative: none of the models shows the full human trio of biases. Specifically, only German BLOOM 6.4B exhibits a significant human-like I-Caus coreference verb-class effect; no model shows an explanation bias above 50%; and no model shows the human-like modulation of referring-expression form by bias congruency, although most models reproduce the general human preference for simpler forms for subject reference. The paper proposes this triad as a discourse benchmark, arguing that apparent coreference sensitivity in LLMs can exist without human-like causal discourse understanding.","tokens_in":28413,"tokens_out":5490,"duration_ms":61302,"significance":"If the findings survive the methodological concerns raised below, the paper makes a useful contribution: it is one of the few studies to evaluate IC in LLMs beyond the coreference bias, and it provides a concrete, multi-level benchmark that later work can apply to larger or instruction-tuned models. The authors also make several methodologically commendable choices: they use the same automatic annotation pipeline for human and LLM data where possible, report mixed-effects regression and bootstrap analyses, and run multiple models from the same family to probe size effects. The three-level model (relation -> referent -> form) is a clear and plausible way to structure the benchmark. However, the validity of the benchmark, and in particular the claim that no LLM exhibits the referring-expression form bias, currently rests on an unvalidated constrained-decoding emulation and on moderate-quality automatic annotation with substantial data exclusion in Experiment 2. These are not fatal to the coreference or coherence results per se, but they are load-bearing for the strongest negative claims.","major_comments":[{"comment":"","section":null}],"minor_comments":[{"comment":"There is a missing Section 3: the text jumps from Section 2 to a numbered but empty '3.' and then to '4. Language Model Setup'. This should be fixed.","section":"Sections 2-4"},{"comment":"The papers contains numerous typos and small errors, e.g., 'constitues', 'perfomance', 'Or focus', 'futher', 'seperate', 'allmost', 'proported' (for 'purported'), and 'dependency' (for 'dependencies'). A careful copyedit is needed.","section":"Sections 6-9"},{"comment":"The sentence 'yielding κ = .85 and κ = .996 for coreference labeling in experiments 1 and 2, respectively' is confusing: Experiment 2 is about coherence relations, not coreference. Please clarify which annotations were compared in each experiment.","section":"Section 6.2"},{"comment":"The full statistical results for Experiments 3 and 4 are not reported. Only selected models are plotted, and no table gives the regression coefficients, standard errors, and p-values for all ten models. Since the central claim is that all models fail to show the form bias, full results should be provided in a table or supplementary material.","section":"Sections 8 and 9"},{"comment":"The exact prompts and the complete list of allowed token sets for the constrained decoding are not fully specified. The paper should list the exact set of allowed first tokens for each condition (which names, which pronoun forms, and which demonstrative forms) so that the experiments are reproducible.","section":"Section 8.1"},{"comment":"The OSF repository is described as 'not yet published'. For a benchmark paper, data, code, and model outputs should be made available with the manuscript, or at least a stable anonymized repository should be provided for review.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of a computational linguistics journal and addresses a timely question with a clean negative-result design. The stress-test concern about the constrained-decoding emulation in Section 8.1 is, in my reading, a legitimate validity threat to the Experiments 3/4 conclusions; it is not merely a hypothetical issue because the constraint set omits a category that appears in the human gold standard and no reference-validity filter is applied. I would advise the editor that the paper can become publishable after the authors either validate the emulation or substantially soften the claims about the form-bias null result, and after they provide the missing statistical details and data availability. There is also a missing Section 3 in the submitted text that suggests the manuscript is not fully assembled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: this is the first benchmark that systematically tests all three implicit causality biases — coreference, coherence, referring-expression form — across a range of open-weight LLMs, and the headline negative result is mostly credible. The part I'd stake less on is the form-bias null, because it rests on an unvalidated constrained-decoding emulation.\n\nWhat's genuinely new: previous LLM work on implicit causality looked almost exclusively at coreference. This paper adds coherence and referring-form, tests I-Caus and I-Cons, uses human gold-standard data from the authors' earlier psycholinguistic studies, and runs the same annotation pipeline over human and model outputs. That is real work. The finding that only German BLOOM 6.4B shows a significant human-like I-Caus coreference bias, while no model shows an explanation bias and form biases are absent, is a useful challenge to the idea that coreference sensitivity implies discourse understanding. The statistical treatment — mixed-effects regressions plus bootstrap confidence intervals — is careful and appropriate.\n\nSoft spots, in order of importance. First, Experiments 3 and 4. The forced-reference task was emulated by constraining beam search to personal pronouns, names, and dieser/diese, with only the first token's gender constrained. Der/die demonstratives were excluded, and no filter checked whether the continuation actually referred to the intended argument. A gender-congruent sie can be plural or switch reference. Without validation against the human productions, the null form-bias result might be an artifact of the constraint set rather than a real model property. The authors acknowledge the emulation, but they don't validate it. Second, Experiment 2's automatic discourse annotation has kappa=0.68 and excludes 39.9–77.1% of LLM data. That's a lot, though the negative result is consistent across models and the authors show higher agreement after removing main clauses. Third, the promised OSF repository isn't live, so the data and scripts can't be checked yet.\n\nNone of this sinks the paper. The coherence and coreference results stand independently, and the main conclusion — mid-size open LLMs don't reproduce the human trio — is plausible. But the form-bias claim should be tempered until the emulation is validated or replaced.\n\nWho this is for: anyone building discourse benchmarks or comparing LLM linguistic behavior with human psycholinguistic data. It deserves a serious referee. I'd send it to review, with the expectation that data are released and the forced-reference emulation is either validated or the claim is narrowed.","headline":"First multi-level implicit causality benchmark for LLMs; the negative story is mostly believable, but the referring-form null depends on an unvalidated constrained decoding trick.","tokens_in":28945,"tokens_out":2863,"would_cite":true,"duration_ms":30851,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that none of ten mid-size open-weight LLMs reproduces the full trio of human implicit-causality biases — coreference, coherence, and referring-expression form — and that the coreference biases seen in some larger models…","keywords":["implicit causality","discourse bias","coreference","coherence relations","referring expressions","LLM benchmarking","psycholinguistics","German"],"falsifier":"A reader could run Experiment 3 again with free generation followed by a post-hoc filter that keeps only continuations whose first referring expression is a gender-congruent personal pronoun, name, or demonstrative. If a human-like form bias (more pronouns for bias-congruent objects than for bias-incongruent objects) emerges in that filtered set, the paper's null result for referring-expression form would be shown to depend on the constrained decoding rather than on the models' discourse competence.","tokens_in":27871,"feed_emoji":"🧠","tokens_out":9589,"duration_ms":83889,"temperature":0.7,"pith_summary":"This paper asks whether large language models share the discourse biases humans show for implicit-causality verbs such as fascinate and admire. It compares ten mid-size open-weight models with human gold-standard data on three linked biases: which argument the continuation refers to (coreference), whether the continuation explains or follows the event (coherence), and whether the referring expression is a simple pronoun or a more complex form (form). The central result is negative: no model reproduces the full trio. Only the largest monolingual model, German BLOOM 6.4B, shows a statistically significant human-like coreference bias for because-continuations, the explanation bias is absent in every model, and no model shows the form bias. The authors conclude that apparent coreference sensitivity can emerge in LLMs without human-like discourse understanding, and they present the four-experiment setup as a benchmark for future models.","feed_headline":"Ten LLMs tested: none match human causal discourse biases","feed_subtitle":"Coreference quirks appear in some models, but explanation and referring-form biases are missing in all.","key_machinery":"The mechanism under test is a three-level cascade of discourse expectations associated with implicit-causality verbs: level 1 predicts a coherence relation (explanation vs. consequence), level 2 predicts the next-mention referent given that relation, and level 3 predicts the morpho-syntactic form of the referring expression given the referent. The benchmark operationalizes the cascade as four experiments: a coreference continuation task with weil and sodass prompts (Experiment 1), a coherence-relation annotation task after comma prompts (Experiment 2), and two forced-reference-style tasks measuring referring-expression form after weil and sodass (Experiments 3 and 4). LLM continuations are generated with diverse beam search, and the forced-reference tasks are emulated by constraining the first generated token to a gender-congruent personal pronoun, name, or demonstrative; human gold-standard data from the same prompt designs are re-annotated with the same automatic pipeline for comparison.","core_discovery":"The paper's central claim is that, across ten models from the GPT-2, mGPT, XGLM, and German BLOOM families, none exhibits human-like causal discourse when coherence and referring-expression form are considered together with coreference. In Experiment 1, only German BLOOM 6.4B shows a statistically significant human-like I-Caus coreference bias after weil 'because'; nearly all models show an I-Cons bias after sodass 'and so', but weaker than the human bias and shifted toward the object argument. In Experiment 2, no model produces explanations at a rate above 50%, and temporal relations dominate. In Experiments 3 and 4, every model prefers pronouns for subject coreference, but none shows the human pattern in which bias-congruent objects receive more pronouns than bias-incongruent objects. The paper concludes that a model can display coreference-like preferences without the coherence and form biases that form an integrated discourse-expectation system in humans.","pith_inferences":["We infer that the absence of explanation bias may partly reflect corpus statistics, since causal relations are often implicit in text while temporal relations are explicitly marked; prompting with full stops instead of commas, or probing with implicit-discourse annotations, would test this.","The constrained-decoding null result for form bias could be checked with a free-generation control: generate without constraints, then keep only continuations whose first referring expression is a pronoun, name, or demonstrative, and re-run the analysis.","The models' tendency toward female-argument coreference, opposite to the male bias often reported for LLMs, suggests that implicit-causality prompts engage gender dynamics differently from stereotype-heavy tasks; this deserves direct study.","Because I-Cons performance is stronger than I-Caus performance, the benchmark offers an empirical route to separating formal-linguistic competence from functional-causal competence in models, a distinction the paper raises in its discussion."],"forward_implications":["Coreference-only tests of LLMs overestimate their discourse competence: a model can pass the coreference-bias test while failing on coherence and referring-expression form.","The coherence experiment provides a sharp behavioral probe: no tested mid-size model produced explanations above 50 percent, whereas humans produced around 80 percent after comma prompts, so this single measure separates human from model behavior cleanly.","The I-Cons bias after 'and so' is more robust and less noisy than the I-Caus bias after 'because' across models, suggesting that these models handle consequentiality better than explanation-oriented causality.","Applying the same four-experiment benchmark to larger, instruction-tuned, or closed models would show whether the missing biases emerge with scale or alignment training, and whether ChatGPT-like strong coreference behavior is accompanied by the coherence and form biases.","The hierarchical ordering of the three biases in humans predicts that models that lack the coherence and coreference biases will also lack the form bias, and the observed pattern is consistent with that ordering."],"supporting_citations":[{"why":"Supplies the human gold-standard coreference and coherence data that the LLM outputs are compared against.","marker":"Solstad and Bott, 2022"},{"why":"Provides the human forced-reference production data that define the expected referring-expression form bias in Experiments 3 and 4.","marker":"Bott and Solstad, 2023"},{"why":"Establishes the explanation-default coherence bias for implicit-causality verbs that Experiment 2 tests.","marker":"Kehler et al., 2008"},{"why":"Earlier LLM study showing weak or inverse implicit-causality coreference bias; supplies the baseline for Experiment 1.","marker":"Upadhye et al., 2020"},{"why":"Earlier LLM study finding a general object bias in coreference; the paper's I-Cons results are discussed against this.","marker":"Kementchedjhieva et al., 2021"},{"why":"Showed an IC coreference effect in GPT-2 XL while arguing it is surface-level; motivates the paper's separation of bias types.","marker":"Davis and van Schijndel, 2020"},{"why":"German GPT-2/BERT IC study showing weak biases and strong name/gender effects; motivates the German prompt design and gender-order analysis.","marker":"Zarrieß et al., 2022"},{"why":"Reported strong human-like IC coreference bias for ChatGPT; used as the qualitative contrast case for the paper's negative results.","marker":"Cai et al., 2023"},{"why":"Validates the diverse beam search decoding procedure used for generating LLM continuations.","marker":"Sieker et al., 2023"}],"fun_headline_variants":["Ten LLMs fail to match human causal discourse biases","Only one LLM shows human-like coreference bias, none others","LLMs miss explanation and referring-form biases that humans show","Benchmark reveals LLMs lack integrated discourse expectations","Ten models tested, one matches coreference, none match coherence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that forcing the first word after the prompt to be a gender-matching pronoun, name, or demonstrative produces the same kind of continuation a human would write when asked to refer to a particular person; if this forcing changes how naturally pronouns versus names are used, the finding that models show no form bias could be a measurement artifact rather than a real deficit.","fun_headline_variants_meta":{"raw":{"variants":["Ten LLMs fail to match human causal discourse biases","Only one LLM shows human-like coreference bias, none others","LLMs miss explanation and referring-form biases that humans show","Benchmark reveals LLMs lack integrated discourse expectations","Ten models tested, one matches coreference, none match coherence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1750,"prompt_tokens":949,"completion_tokens":801,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":720}},"tokens_in":565,"tokens_out":801,"duration_ms":8956,"temperature":1.0,"reasoning_tokens":720,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:33:40.347753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could run Experiment 3 again with free generation followed by a post-hoc filter that keeps only continuations whose first referring expression is a gender-congruent personal pronoun, name, or demonstrative. If a human-like form bias (more pronouns for bias-congruent objects than for bias-incongruent objects) emerges in that filtered set, the paper's null result for referring-expression form would be shown to depend on the constrained decoding rather than on the models' discourse competence.","supporting_citations":[{"cited_title":"This isn’t the bias you’re looking for: Implicit causality, names and gender in german language models","cited_arxiv_id":null,"evidence_quote":"Supplies the human gold-standard coreference and coherence data that the LLM outputs are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier LLM study showing weak or inverse implicit-causality coreference bias; supplies the baseline for Experiment 1."}],"review_version":1}