{"id":"8d2b2e90-ed43-4a8c-8f9f-5376f9850bc2","arxiv_id":"2505.02590","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A deterministic sentence comprehension model becomes uncertainty-aware when its last layer is treated with an ensemble Kalman filter, making it less overconfident on reversal anomaly sentences.","lead":"This paper applies a Bayesian version of an ensemble Kalman filter to a neural network model of sentence comprehension, letting the model show uncertainty on impossible role reversals such as 'the pizza eats the woman.' Instead of one confident answer, the model keeps a spread of possibilities, and the authors argue this is closer to how human readers handle ambiguous sentences.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported uncertainty on reversal anomalies is an artifact of a data-dependent prior: πprior is centered at θ_MLE trained on the same data, and Appendix D shows the effect disappears for Pprior=0.01I, so the Bayesian claim is circular.","rationale":"The reader's weakest_assumption identified two issues: the missing human benchmark and the Gaussian prior centered at MLE. I agree with both, but I consider the prior-centered-at-MLE problem the more load-bearing one because it attacks the internal validity of the computational result, not just its external interpretation. If the posterior is not a legitimate posterior, then the reported 'uncertainty' is not a property of the Bayesian method but an artifact of the chosen prior. Appendix D already demonstrates the effect's sensitivity to Pprior, which strengthens this concern. The human-approximation issue is also real but could be addressed by adding EEG/behavioral comparisons; the prior issue would require a fundamental re-analysis. I therefore keep the reader's CONDITIONAL verdict: the paper is not acceptable as is, but the concern is testable and could be resolved by re-running with an independent prior. If the effect disappears, the verdict would need to move to REJECT.","tokens_in":13649,"tokens_out":8662,"duration_ms":109894,"concrete_test":"Re-run the Bayesian SG analysis with an independent prior that does not use the training labels, e.g., mprior = 0 and Pprior = I (or with hyperparameters selected on a held-out split), while keeping the same MLE-trained feature map, dropout rate, step size, and 10 runs. If the reversal-anomaly pattern of Figure 5 — decreased activation for semantically plausible and increased activation for syntactically indicated roles — does not reproduce, the headline uncertainty effect is an artifact of centering the prior on the MLE. Also rerun Pprior = 0.01I as a control; Appendix D suggests the effect will already vanish there.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3, the authors set the Gaussian prior to mprior = θ_MLE and Pprior = I, where θ_MLE is obtained by training the full SG model on the same dataset D (Section 2.4). This makes πprior in Eq. (1) dependent on the very labels used in the likelihood, so the resulting 'posterior' is not a Bayesian posterior under any pre-data belief. The uncertainty estimates and the reversal-anomaly activation shifts (Figures 4-5) therefore do not measure epistemic uncertainty in a principled sense; they measure the combined effect of the likelihood and an ad hoc prior centered on the MLE. Appendix D confirms the load-bearing role of the prior scale: with Pprior = 0.01I the paired differences between Bayesian and ADAM activations become insignificant (p > 0.05), whereas Pprior = I and Pprior = 5I produce highly significant differences. This is not a minor calibration issue: the central empirical claim — that Bayesian inference 'increases uncertainty' on RA sentences — is controlled by an arbitrary hyperparameter, not by the data or by a valid posterior. The last-layer feature map is also frozen at the MLE solution, so the model cannot revisit earlier-layer uncertainty. Consequently, the paper's stated improvement in uncertainty representation is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Bayesian extension of the Sentence Gestalt (SG) model of sentence comprehension, using an ensemble Kalman filter (EnKF) based dropout deterministic sampler to perform last-layer Bayesian inference. The SG model is first trained by maximum likelihood estimation (MLE), the feature map is frozen, and a Gaussian prior is placed on the output-layer parameters; predictions are obtained by averaging sigmoid outputs over the final particle ensemble (Eq. 5). The authors compare this Bayesian SG model with the MLE-trained model on semantically congruent and reversal-anomaly (RA) sentences, reporting that the Bayesian model lowers activations for semantically plausible but syntactically incorrect role assignments and raises activations for syntactically indicated roles. This behavior is interpreted as greater uncertainty under conflicting cues and as being closer to human comprehension. The evidence consists of model-level and item-level t-tests on simulated test sentences; no human behavioral or neurophysiological data are collected.","tokens_in":13947,"tokens_out":4551,"duration_ms":60544,"significance":"If the central claim were established, the paper would make a useful methodological contribution by showing that a relatively cheap last-layer Bayesian correction can change the qualitative behavior of a cognitive model under cue conflict, and the deterministic EnKF sampler is of independent interest. The paper is transparent about its hyperparameters, provides pseudocode, and includes a prior-sensitivity appendix, which are strengths. However, the central empirical claim is not yet supported: the prior is centered at the MLE estimated on the same training data, the reversal-anomaly effect largely disappears under a smaller prior covariance (Appendix D), and no human data directly test the claim that the Bayesian model is closer to human processing. These issues are load-bearing for the paper's stated conclusions rather than cosmetic.","major_comments":[{"comment":"The prior is data-dependent in a way that undermines the Bayesian interpretation. The manuscript states that a Gaussian prior with mean mprior = θMLE and covariance Pprior = I is used, where θMLE is obtained by training the full SG model on the same dataset D (Section 2.4). Because the prior mean is a function of the same labels that appear in the likelihood, Eq. (1) does not define a posterior under any pre-data belief. Consequently the ensemble spread and the predictive distribution in Eq. (5) do not measure epistemic uncertainty in the standard Bayesian sense; they reflect a mixture of the likelihood and an ad hoc, data-dependent regularization. This is load-bearing because the central claim of the paper is about uncertainty representation. The authors should either use a prior that is genuinely independent of D (e.g., a zero-mean prior with a justified covariance) or explicitly reframe the method as a regularized ensemble procedure and avoid the language of Bayesian posterior uncertainty.","section":"Section 3, Eq. (1)"},{"comment":"The main qualitative result is controlled by an arbitrary hyperparameter. The paired t-tests in Table D.8 show that with Pprior = 0.01I, activations from the dropout deterministic sampler are not significantly different from the ADAM-trained model (p > 0.05), whereas with Pprior = I and Pprior = 5I they are highly significant. This means the reported 'increased uncertainty' on reversal anomaly sentences is not a robust consequence of the data or of Bayesian inference; it is a consequence of choosing a sufficiently large prior covariance. Because this effect is exactly what the paper uses to support its main conclusion, the authors need to provide a principled justification for the prior scale or demonstrate that the qualitative pattern holds across a range of defensible priors. As it stands, Appendix D documents that the central empirical claim is prior-driven rather than data-driven.","section":"Appendix D, Table D.8"},{"comment":"The claim that the Bayesian SG model is 'closer to human cognitive processing' is not tested against any human data. The paper reports only model-model differences on synthetic test sentences; the mapping from output activations to human uncertainty is qualitative and is borrowed from cited behavioral and ERP studies (Ferreira 2003; Kuperberg et al. 2003). No behavioral accuracy, reaction time, or ERP benchmark is used to validate the proposed model. Even if the Bayesian model shows reduced confidence on RA sentences, this by itself does not establish that it approximates human comprehension. The authors should either add a direct comparison with human behavioral or ERP data, or explicitly limit the conclusion to 'increased activation uncertainty' in the model and remove or substantially weaken the human-approximation claim.","section":"Section 4, Discussion"},{"comment":"Part of the reported 'greater uncertainty' is built in by the ensemble averaging formula. Equation (5) averages sigmoid outputs over the particle ensemble, so any ensemble with positive covariance will smooth extreme activations relative to a single point estimate. Since the ensemble spread is governed by the prior covariance, the difference between the Bayesian and ADAM activations in Figures 4 and 5 is at least partly inherited from the prior rather than learned from the data. This is related to the prior-sensitivity problem above, but it deserves its own statement because it means the qualitative pattern is not evidence of a specific cognitive mechanism unless the prior is justified independently.","section":"Eq. (5), Section 3.1"}],"minor_comments":[{"comment":"There appears to be an inconsistency between the t-test results and the figure: Table 3 reports that ADAM activations for reversed words do not differ significantly from zero, yet Figure 5 shows high mean activations for the semantically plausible roles. Please clarify what quantity is being tested, correct any mismatch between the table labels and the figure legends, and report the mean and standard deviation for the tested units.","section":"Table 3 and Figure 5"},{"comment":"There are several typos: 'extention' in the abstract, 'determinsitic' in Section 2.1.1, and 'the the' in Section 3.1. These should be corrected before publication.","section":"Abstract and Section 2.1.1"},{"comment":"The figure captions refer to standard deviations, but the text reports t-tests over 10 runs and item-level analyses over 8 items. Please specify whether the error bars represent variability across models, across items, or across both, and state which analysis is displayed.","section":"Figures 4 and 5"},{"comment":"The significance threshold is used inconsistently: the main text uses p < 0.05, while Table D.8 uses p < 0.001. Please use a consistent alpha level and state it explicitly.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is a computational modeling study, and for a journal of mathematical psychology the absence of any human data is a significant scope concern. The prior-sensitivity issue documented in Appendix D is the most serious technical problem; it should be addressed before publication. The deterministic EnKF sampler is an interesting methodological contribution, but the manuscript currently overclaims the human relevance of the results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a modest but legitimate computational demonstration: the authors take an existing ensemble Kalman sampler for Bayesian logistic regression and apply it to last-layer inference in the Sentence Gestalt model, showing that the resulting predictions are less overconfident on reversal anomalies like \"During dinner, pizza eats woman.\" That application is new, and the numerical work is carefully done: model- and item-level t-tests, a clear story about how the Bayesian version shifts from pure semantic cue weighting toward a blend of syntactic and semantic cues, and a sensitivity analysis. The authors are also transparent about the main weakness: the prior is a Gaussian centered at the MLE from the same training set, with an identity covariance that is essentially hand-picked. Appendix D shows the headline effect disappears when the prior covariance is scaled down to 0.01I. That is not a minor calibration detail; it means the \"uncertainty\" reported is substantially a property of the chosen prior, not of the data or the model alone. In other words, the posterior is not a Bayesian posterior under any pre-data belief, so the epistemic-uncertainty interpretation is shaky.\n\nThe paper also claims the model is \"closer to human processing\" without any human behavioral or ERP data. That claim rests entirely on a qualitative analogy to Ferreira (2003) and Kuperberg et al. (2003). The model-model difference is real under the wide prior, but the mapping to human uncertainty is not established.\n\nWhat the paper does well: it gives a reproducible algorithmic description (though not code or data), the statistical analysis is appropriate for the small sample, and the authors acknowledge the prior sensitivity in the appendix. The sampler itself is theoretically grounded in previous work, so the mathematical core is sound.\n\nWho this is for: computational psycholinguists who want a practical way to add uncertainty to small connectionist models, and method developers interested in a novel application of EnKF-based samplers. It deserves a serious referee: the combination is new and the execution is honest, but the claims need to be reined in and the prior issue needs to be addressed — either by using a true pre-data prior or by explicitly reframing the result as sensitivity to the prior rather than Bayesian evidence. I would not cite this in my own work in the near term, but I would bring it to a reading group focused on Bayesian cognitive modeling.","headline":"A new application of an EnKF sampler to a small sentence-processing model yields a real but prior-sensitive change in predictions; the Bayesian and human-uncertainty claims need more support.","tokens_in":14448,"tokens_out":3302,"would_cite":false,"duration_ms":41777,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","65C35","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A Bayesian last layer makes a sentence model hedge when syntax and meaning conflict.","keywords":["Bayesian inference","ensemble Kalman filter","uncertainty quantification","sentence comprehension","Sentence Gestalt model","reversal anomaly","last-layer inference","interacting particle systems"],"falsifier":"Run the same eight reversal-anomaly sentences with human participants and measure a behavioral or neural index of uncertainty, such as error rate, reaction time, or the late positive brain response associated with reanalysis. If the Bayesian model's activation shift appears on items where humans show no uncertainty, or fails to appear where humans do show uncertainty, the claimed correspondence between model activations and human processing would be refuted. A computational check would be to vary the prior covariance over a wide range; the paper's own appendix shows a very small prior covariance removes the effect.","tokens_in":13451,"feed_emoji":"🧠","tokens_out":10663,"duration_ms":123405,"temperature":0.7,"pith_summary":"This paper asks whether a neural network model of sentence comprehension can represent uncertainty instead of committing to a single reading. The authors take the Sentence Gestalt model, train it with maximum likelihood, and then apply an ensemble-Kalman-filter-based Bayesian sampler to its output layer, comparing the two versions on reversal anomalies—sentences in which syntactic role assignment and semantic plausibility point in opposite directions. They find that the standard model confidently keeps the semantically plausible but syntactically wrong reading, while the Bayesian model lowers activation for that reading and raises activation for the syntactically indicated roles. They argue that this graded, cue-sensitive response is closer to the uncertainty humans show on such sentences, which behavioral studies link to errors and reanalysis. If the finding holds, last-layer Bayesian inference offers a practical route to uncertainty estimates in existing comprehension models without retraining their core.","feed_headline":"Bayesian filter lets a sentence model hedge on reversed roles","feed_subtitle":"A neural comprehension model stays uncertain when syntax and meaning conflict, mirroring human readers.","key_machinery":"The central mechanism is the dropout deterministic sampler, an interacting-particle-system ordinary differential equation that evolves an ensemble of output-layer weight vectors from a Gaussian prior toward the posterior in the limit of large simulation time. It is an extension of the ensemble Kalman filter adapted to logistic-regression-style losses, and it is applied only to the last layer of the Sentence Gestalt model after the earlier layers have been fixed by maximum-likelihood training. Dropout randomly zeroes entries of the ensemble to prevent the particles from collapsing into the low-dimensional subspace spanned by their initial values, and a Trotter-splitting time-stepping scheme keeps the numerical cost low. The final ensemble's averaged sigmoid predictions replace the single point estimate in the predictive distribution.","core_discovery":"On the paper's own terms, the discovery is that replacing the point estimate of the Sentence Gestalt model's output layer with a posterior over weights changes how the model handles reversal anomalies. For semantically congruent sentences, the Bayesian model's output activations are comparable to those of the maximum-likelihood model. For reversal anomalies, the MLE model continues to activate the semantically plausible but syntactically wrong role assignments with high confidence, whereas the Bayesian model's mean activations for those assignments drop and its activations for the syntactically indicated roles rise. Statistical tests in the paper show that the Bayesian activations for reversed words differ significantly from zero and from the MLE model's activations, while the MLE model's do not; the authors interpret this as the model representing graded uncertainty when semantic and syntactic cues conflict, and take it as closer to human processing.","pith_inferences":["A direct item-level experiment, not run in the paper, would compare the Bayesian model's activation differences with human reaction times, error rates, or ERP amplitudes on the same sentences; that would test whether the borrowed human-uncertainty mapping actually holds.","Because the prior is centered at the maximum-likelihood solution and the paper's Appendix D shows the effect vanishes for a very small prior covariance, some of the reported uncertainty may be a prior-width artifact rather than purely data-driven posterior behavior.","Only the output layer is treated Bayesian here; uncertainty in the learned feature representation itself is not quantified, so genuinely ambiguous semantic representations might require a deeper Bayesian treatment.","The same EnKF-based sampler could in principle be applied to output layers of other layered cognitive or language models as a cheap uncertainty signal, though the frozen-feature assumption would need to be checked in each setting."],"forward_implications":["A last-layer Bayesian treatment can be added to an already maximum-likelihood-trained sentence model, so uncertainty estimates do not require retraining the feature layers.","On reversal anomalies, the model no longer commits fully to the semantically plausible reading; both the syntactic and the semantic candidate roles remain represented with graded activation.","The same activation shift does not appear for irrelevant role assignments, indicating that the increased uncertainty tracks the actual semantic-syntactic conflict rather than general noise.","Predictive probabilities, not just point estimates, become available for any probe question, offering a route to calibrated confidence in sentence-processing models."],"supporting_citations":[{"why":"Supplies the dropout deterministic sampler, the interacting-particle-system method used to perform Bayesian inference in the output layer.","marker":"Bhandari et al. (2024)"},{"why":"Provides the behavioral evidence that human readers misassign thematic roles on noncanonical sentences, the uncertainty phenomenon the model aims to approximate.","marker":"Ferreira (2003)"},{"why":"Supplies the ERP semantic-illusion findings that link conflicting syntactic and semantic cues to human processing signatures.","marker":"Kuperberg et al. (2003)"},{"why":"Defines the original Sentence Gestalt model architecture that the paper extends with Bayesian inference.","marker":"St. John and McClelland (1990)"},{"why":"Provides the modern Sentence Gestalt training setup and the N400-related modeling context that the experiments build on.","marker":"Rabovsky et al. (2018)"},{"why":"Underlies the affine-invariant ensemble transform and homotopy ideas from which the deterministic sampler is derived.","marker":"Pidstrigach and Reich (2022)"},{"why":"Introduces dropout, which the sampler uses to keep small ensembles from collapsing into a low-dimensional subspace.","marker":"Srivastava et al. (2014)"}],"fun_headline_variants":["Bayesian filter helps sentence model embrace uncertainty","Ensemble Kalman filter brings human-like doubt to language AI","Sentence model hedges on reversal anomalies via Bayesian update","Uncertainty in comprehension: Bayesian approach outperforms MLE","Neural sentence model learns to be uncertain with Bayes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that changes in the Bayesian model's output activations correspond to the uncertainty humans experience on reversal anomalies; that mapping is borrowed from earlier behavioral and ERP studies and is not directly tested with human data in this paper, and the posterior itself is defined relative to a Gaussian prior centered at the maximum-likelihood solution.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian filter helps sentence model embrace uncertainty","Ensemble Kalman filter brings human-like doubt to language AI","Sentence model hedges on reversal anomalies via Bayesian update","Uncertainty in comprehension: Bayesian approach outperforms MLE","Neural sentence model learns to be uncertain with Bayes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1426,"prompt_tokens":853,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":495}},"tokens_in":469,"tokens_out":573,"duration_ms":7584,"temperature":1.0,"reasoning_tokens":495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:48:20.132529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eight reversal-anomaly sentences with human participants and measure a behavioral or neural index of uncertainty, such as error rate, reaction time, or the late positive brain response associated with reanalysis. If the Bayesian model's activation shift appears on items where humans show no uncertainty, or fails to appear where humans do show uncertainty, the claimed correspondence between model activations and human processing would be refuted. A computational check would be to vary the prior covariance over a wide range; the paper's own appendix shows a very small prior covariance removes the effect.","supporting_citations":[{"cited_title":", author Pidstrigach, J","cited_arxiv_id":null,"evidence_quote":"Supplies the dropout deterministic sampler, the interacting-particle-system method used to perform Bayesian inference in the output layer."},{"cited_title":", author Sitnikova, T","cited_arxiv_id":null,"evidence_quote":"Supplies the ERP semantic-illusion findings that link conflicting syntactic and semantic cues to human processing signatures."},{"cited_title":", author Hansen, S","cited_arxiv_id":null,"evidence_quote":"Provides the modern Sentence Gestalt training setup and the N400-related modeling context that the experiments build on."},{"cited_title":", author Reich, S","cited_arxiv_id":null,"evidence_quote":"Underlies the affine-invariant ensemble transform and homotopy ideas from which the deterministic sampler is derived."},{"cited_title":", author Hinton, G","cited_arxiv_id":null,"evidence_quote":"Introduces dropout, which the sampler uses to keep small ensembles from collapsing into a low-dimensional subspace."}],"review_version":1}