{"id":"57bac697-42d4-4762-bd15-249088b8d57b","arxiv_id":"1909.00303","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Sentences that take humans longer to read also show larger disagreement between layers of pretrained language encoders.","lead":"This paper applies Representational Similarity Analysis, a brain-imaging technique, to compare how BERT and ELMo encode sentences with how hard humans find those sentences to read. It reports that difficult sentences produce larger disagreement between encoder layers, offering a new way to inspect language models with eye-tracking data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim may be a length artifact: no length controls for the layer-disagreement measure, while fixations are length-normalized; a partial-correlation check is needed.","rationale":"The reader's weakest_assumption identifies exactly the condition that must hold for the central claim: the layer-disagreement metric must not be a proxy for sentence length. I agree. The paper's own normalization in Section 3 removes length from the fixation response but never from the disagreement regressor, and sentence length can plausibly influence mean-pooled representations (averaging over more tokens changes the geometry of the sentence vector in the RDM) and hence row-wise Kendall taus. This is the most load-bearing concern because it attacks the correlation that the entire Discussion interprets as mutual information between human processing and representation divergence; a spurious correlation from length would leave the RSA demonstration intact but the central claim unsupported. I did not choose the layer-pair selection issue as the primary concern, because Figure 2 shows the negative correlation appears broadly across BERT layer pairs, so selective reporting in Table 1 is secondary. The proposed partial-correlation check is inexpensive, uses data already in the paper (Dundee sentence lengths are implicit in the corpus; log frequency and Yngve are already computed), and would settle the issue: a robust partial rho supports the claim, and a collapse would require reinterpreting the finding as a length or complexity artifact. This keeps the verdict CONDITIONAL, so no change to the reader's recommendation.","tokens_in":7877,"tokens_out":7563,"duration_ms":76898,"concrete_test":"Recompute the analysis for the strongest layer pair (B11-B12) and, if feasible, for all 276 BERT layer pairs, adding sentence length (number of words), average log frequency, and Yngve score as covariates. Compute partial Spearman correlations between VCorr and Vtotfix (and Vfirstpass) controlling for these variables, or fit a regression with fixation as outcome and layer disagreement plus covariates as predictors, using bootstrap confidence intervals over sentences. If the -0.88 coefficient collapses to near zero or loses significance after controlling for length alone, the central claim fails; if it remains substantially negative (e.g., below -0.3) with length controlled, the length-confounding concern is ruled out and the correspondence is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that per-sentence layer disagreement (1 minus the Kendall row correlation between two layer RDMs) tracks processing difficulty rather than low-level surface properties. In Section 3, fixation vectors are obtained by summing word measurements and dividing by sentence length, so length is removed from the dependent variable. But the disagreement vectors VCorr Li-Lj receive no analogous length adjustment: sentence representations are mean-pooled, and mean-pooled vectors from longer sentences can have different variance, isotropy, and neighbor structure, which can directly affect row-wise Kendall taus. If sentence length is correlated both with average fixation duration and with row-wise correlation, the headline negative Spearman correlations (e.g., -0.88 for B11-B12) could be spurious. The paper also reports no partial correlations controlling for its own linguistic predictors: Yngve score correlates 0.66 with total fixation, and layer disagreement correlates with Yngve (Figure 2, right), so the 'correspondence' may be mediated by syntactic complexity, word frequency, or length rather than being a distinct property of encoder representations. Because the central claim in the Discussion ('sentences which are difficult for humans... have more divergent representations') is stated without these controls, the association is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Representational Similarity Analysis (RSA) as a method for comparing sentence encoder representations across layers and across models, and applies it to BERT and ELMo. The authors construct RDM-based per-sentence layer-disagreement vectors, defined as one minus the row-wise Kendall correlation between two layers' RDMs, and correlate these vectors with Dundee eye-tracking fixation durations and three linguistic predictors (log frequency, number of senses, and Yngve score). They report strong negative Spearman correlations between layer disagreement and total fixation duration (e.g., -0.88 for BERT layers 11 and 12), and interpret this as evidence that sentences difficult for humans to process have more divergent encoder representations both within and across models.","tokens_in":8065,"tokens_out":2733,"duration_ms":27847,"significance":"If the central claim holds, the paper offers a conceptually clean, parameter-free way to relate neural language model representations to human processing difficulty, with clear advantages over probing classifiers: it avoids training auxiliary models, makes representational geometry directly comparable, and produces falsifiable predictions about layer pairs. The analysis is self-contained against an external eye-tracking benchmark, no parameters are fitted to the fixation data, and the correlations are reported as statistically significant. The main contributions are methodological (RSA for NLP interpretability) and empirical (a new correspondence between layer disagreement and processing difficulty), and the findings would be of interest to the interpretability and psycholinguistics communities.","major_comments":[{"comment":"The paper does not control for sentence length in the layer-disagreement measure. The fixation vectors are explicitly length-normalized ('dividing by its length'), but the disagreement vectors VCorr Li-Lj receive no analogous adjustment: sentence representations are mean-pooled, and mean-pooled vectors from longer sentences can differ in variance, isotropy, and neighborhood structure, which can directly affect row-wise Kendall taus. Because sentence length is plausibly correlated both with fixation duration and with row-wise correlation, the headline negative correlations (e.g., -0.88 for B11-B12) could be partly or wholly artifactual. The authors should report partial Spearman correlations controlling for sentence length, or show that the correlation persists within length-matched bins.","section":"Section 3"},{"comment":"The middle section of Table 1 reports only the 'top three BERT layer pairs Li and Lj which maximize the correlation scores,' selected from the 276 possible BERT layer pairs. This selection procedure inflates the apparent strength of the relationship and makes the reported -0.88 an extreme-order statistic rather than a pre-specified effect. The authors should report the full distribution of correlations across all layer pairs, or, if selection is used, apply a proper multiple-comparison correction and describe the selection rule before reporting p-values.","section":"Table 1"},{"comment":"The central claim is not tested against simple baselines. No comparison is made to bag-of-words representations, random or untrained encoders, or representations derived from non-linguistic features. If such baselines show similar negative correlations with fixation duration, then the observed effect would reflect a general property of high-dimensional sentence representations rather than the trained encoders' processing of difficult sentences. Adding these baseline conditions is necessary to support the claim that the correspondence is specific to the language-model representations.","section":"Section 3 and Section 4"},{"comment":"The reported correlations between layer disagreement and fixation duration are not accompanied by partial correlations controlling for the paper's own linguistic predictors. Yngve score correlates 0.66 with total fixation duration, and Figure 2 shows that layer disagreement also correlates with Yngve. Because the linguistic features are known processing-difficulty predictors, the authors should report partial Spearman correlations between layer disagreement and fixation duration after controlling for Yngve, log frequency, and word sense, or a mediation analysis. Without this, the 'tripartite affinity' described in the Discussion does not establish that layer disagreement carries information about processing difficulty beyond syntactic complexity and other low-level confounds.","section":"Section 4 and Figure 2"}],"minor_comments":[{"comment":"The correlation distance formula contains a typo in the norm notation ('‖(ni− ¯ni‖2' should be '‖ni− ¯ni‖2'), and the figure caption spells 'similarity' as 'similairty'. These should be corrected.","section":"Section 2 and Figure 1"},{"comment":"The name 'Malahanobis distance' is misspelled; it should be 'Mahalanobis distance'.","section":"Section 2"},{"comment":"The notation VCorr Li−Lj mixes a superscript and subscript in a way that is hard to parse; a clearer notation, such as V_Corr(Li,Lj), would improve readability.","section":"Section 3"},{"comment":"The asterisk convention is ambiguous: the text says 'All correlations significant with p<0.0001 after Bonferroni correction unless marked with *,' but it would be clearer to state explicitly that the starred entries are not significant and to report their p-values.","section":"Table 1"},{"comment":"The word 'complimentary' in the Conclusion should be 'complementary' to match the intended meaning.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things to know. First, the specific empirical result here – per-sentence layer disagreement in BERT and ELMo negatively correlates with Dundee total fixation duration, down to -0.88 for B11-B12 – is genuinely new, and it is presented with a refreshingly simple method: no probes, no fitted parameters. Second, the paper is honest about its lineage. It cites Abnar et al., Chrupała, and others, and does not oversell RSA itself as novel. Third, the central claim in the Discussion, that sentences difficult for humans have more divergent representations, is not yet supported because the analysis lacks control variables.\n\nThe method itself is sound. The authors build RDMs per layer, compute row-wise Kendall correlations for each sentence, and correlate those with fixation times. That is a legitimate higher-order RSA move. The ANOVA on layer groups helps show the pattern is not random. For a subfield that often reaches for diagnostic classifiers, this is a reasonable and transparent alternative.\n\nThe soft spots are real but addressable. Fixation vectors are normalized by sentence length, but the layer-disagreement vectors are not. Mean-pooled representations from longer sentences can have different variance and neighbor structure, which can affect row-wise Kendall taus. No partial correlation is reported for length, or for the paper's own Yngve score, which correlates 0.66 with total fixation. Since Yngve also correlates with layer disagreement (Figure 2, right), the headline association could be mediated by syntactic complexity or length rather than being a distinct property of encoder representations. The Table 1 selection of the top three layer pairs makes the strongest numbers look better than the full picture, though the full heatmap in Figure 2 partly mitigates that. There are also no baselines; bag-of-words, random representations, or shuffled labels would make the specificity of the BERT/ELMo result much clearer.\n\nNone of this is fatal. The correlation is large and visible across nearly all layer pairs, so it is unlikely to be pure noise. But the title and conclusion claim a 'previously unknown correspondence' between encoders and human processing difficulty, and that correspondence is not yet disentangled from simple confounds. A careful revision that adds partial correlations, baselines, and a length control would make the claim convincing. I would send this to a serious referee; it is a solid empirical paper that deserves a fair review rather than a desk rejection. I would not cite it in its current form, but I would cite a controlled version.","headline":"A useful RSA-based interpretability demonstration with a striking correlation, but the central claim needs length and baseline controls before it lands.","tokens_in":8608,"tokens_out":3500,"would_cite":false,"duration_ms":34664,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hard-to-process sentences show more divergent representations inside language encoders, the paper argues.","keywords":["representational similarity analysis","RSA","sentence encoders","eye-tracking","BERT","ELMo","layer disagreement","syntactic complexity"],"falsifier":"Recompute the Spearman correlations between layer disagreement and total fixation duration after splitting the Dundee sentences into narrow sentence-length bins (or after partialling out sentence length). If the negative correlations largely disappear or reverse within bins, the claim that processing difficulty specifically drives layer disagreement would be refuted; if they survive, the length-confounding worry is addressed.","tokens_in":7675,"feed_emoji":"👁️","tokens_out":5508,"duration_ms":44351,"temperature":0.7,"pith_summary":"This paper argues that the sentences human readers find hardest to process are also the sentences on which pretrained language encoders' internal representations diverge most, both across layers of a single model and between different models. To show this, it adapts Representational Similarity Analysis (RSA), a neuroscience technique that compares representational geometries rather than decoding accuracy, to sentences from the Dundee eye-tracking corpus read by BERT and ELMo. The central result is a strong negative correlation between a sentence's gaze fixation duration and how much two layers disagree about that sentence's similarity to other sentences, e.g. -0.88 for BERT layers 11 and 12. The finding matters because it offers a probe-free way to relate neural model representations to human processing behavior.","feed_headline":"Human reading difficulty tracks language model layer disagreement","feed_subtitle":"An RSA-based method links eye-tracking reading times to divergence inside BERT and ELMo without probes.","key_machinery":"The load-bearing object is the representational dissimilarity matrix (RDM), a matrix whose entry for a pair of sentences is the correlation distance between their mean-pooled layer representations. For each pair of layers, the paper builds one RDM per layer and then computes a row-wise Kendall's tau between corresponding rows, yielding a per-sentence vector of layer correlations; subtracting from 1 gives per-sentence 'layer disagreement.' These vectors are then correlated with eye-tracking fixation durations and linguistic features using Spearman's rho, a 'third-order' step that connects geometry to behavior.","core_discovery":"The paper's central discovery is a previously unknown correspondence: sentences with high human processing difficulty, measured by eye-tracking fixation durations, tend to have higher layer disagreement in BERT and ELMo, and also higher disagreement between the two encoders. Layer disagreement for a sentence is defined as 1 minus the row-wise Kendall correlation between two layers' representational dissimilarity matrices, restricted to that sentence's row. Spearman correlations with total fixation duration reach -0.88 for adjacent BERT layers 11-12 and -0.87 for BERT layers 12-13, with similar but weaker negative correlations between ELMo and BERT layers (e.g. -0.66 for E2-B23). The same pattern appears for first-pass reading times. The authors interpret this as evidence that difficult sentences produce degraded or unstable encoder representations, and they report that mid-layer disagreements correlate most with both fixation time and Yngve syntactic complexity, which they read as modest support for the view that middle layers encode syntax.","pith_inferences":["If the correlation is causal rather than merely correlational, layer disagreement could serve as a cheap proxy for annotating reading difficulty in languages and domains where eye-tracking data do not exist; this is an extension the paper does not test.","Because the fixation vectors are normalized by sentence length but no length-matched control is reported, the strongest test of the claim is to recompute the correlations within length bins; the paper's central interpretation would be weakened if the effect disappears.","The method could be turned around: instead of predicting human difficulty from model divergence, one could use fixation data to select sentences that expose representational failure, and then use those sentences as evaluation sets for encoders."],"forward_implications":["RSA can serve as an interpretability tool that compares model representations with human behavioral data without training diagnostic classifiers, avoiding overfitting and ceiling effects.","Layer disagreement, not just individual layer quality, is a meaningful summary statistic of encoder behavior on a sentence.","The negative correlations predict that similar effects should appear for other pretrained encoders and other eye-tracking corpora, and that middle layers are the most syntax-sensitive.","Cross-encoder divergence between ELMo and BERT can act as a second, independent signal of processing difficulty."],"supporting_citations":[{"why":"Introduces RSA and the RDM framework that the paper's higher-order comparisons build on.","marker":"Kriegeskorte et al. (2008)"},{"why":"Recommends Kendall's tau for RDM comparison and provides the toolbox the analysis follows.","marker":"Nili et al. (2014)"},{"why":"Supplies the Dundee eye-tracking corpus, the source of all fixation-duration data.","marker":"Kennedy et al. (2003)"},{"why":"Provides BERT-large, one of the two encoders whose layers are compared.","marker":"Devlin et al. (2018)"},{"why":"Provides ELMo and the mean-pooling sentence representation method.","marker":"Peters et al. (2018)"},{"why":"Defines the syntactic-complexity score used as a linguistic comparison feature.","marker":"Yngve (1960)"},{"why":"Establishes the link between word frequency, ambiguity, and fixation times that motivates the hypothesis.","marker":"Rayner and Duffy (1986)"}],"fun_headline_variants":["Reading strain syncs with AI layer clashes in BERT and ELMo","Why hard sentences trip up both readers and language models","Eye-tracked reading time links to encoder layer discord","No probes? RSA reveals BERT and ELMo reflect human effort","Layer disagreement in AI mirrors human eye-tracking difficulty"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a sentence's layer-disagreement score reflects genuine representation degradation rather than being driven by sentence length or low-level lexical properties, and the paper reports no length-control analysis to rule that out.","fun_headline_variants_meta":{"raw":{"variants":["Reading strain syncs with AI layer clashes in BERT and ELMo","Why hard sentences trip up both readers and language models","Eye-tracked reading time links to encoder layer discord","No probes? RSA reveals BERT and ELMo reflect human effort","Layer disagreement in AI mirrors human eye-tracking difficulty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":2916,"prompt_tokens":843,"completion_tokens":2073,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":1990}},"tokens_in":459,"tokens_out":2073,"duration_ms":15324,"temperature":1.0,"reasoning_tokens":1990,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:55:06.106966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Spearman correlations between layer disagreement and total fixation duration after splitting the Dundee sentences into narrow sentence-length bins (or after partialling out sentence length). If the negative correlations largely disappear or reverse within bins, the claim that processing difficulty specifically drives layer disagreement would be refuted; if they survive, the length-confounding worry is addressed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces RSA and the RDM framework that the paper's higher-order comparisons build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Dundee eye-tracking corpus, the source of all fixation-duration data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the syntactic-complexity score used as a linguistic comparison feature."},{"cited_title":"and Duffy, S","cited_arxiv_id":null,"evidence_quote":"Establishes the link between word frequency, ambiguity, and fixation times that motivates the hypothesis."}],"review_version":1}