{"id":"134fee7b-82c3-468e-8986-ad23e52d9e9b","arxiv_id":"2505.17071","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Deep-layer embeddings of short literary excerpts carry enough information to identify their source book and author, with same-author works more confused, indicating style is encoded in the prompt representation.","lead":"Short excerpts from classic novels become separable in a language model's internal representations once the prompt has passed through several transformer layers, so a simple classifier can often identify the source book. The authors read this as evidence that models encode writing style, not just topic, which could give literary scholars a new tool for attribution.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evidence does not isolate style from topic: same-author confusion and shuffled-text separability can be explained by content-word frequencies, and the chunk-level 70/30 split leaks autocorrelation.","rationale":"The reader identified the same load-bearing concern I find: the paper's evidence does not control for topic or content confounds, and the chunk-level split risks autocorrelation leakage. My attack adds a concrete mechanism: the shuffling result at B = 1 shows near-perfect separability with syntax destroyed, implying that preserved open-class vocabulary alone is sufficient for the classifier, which is exactly the confound the paper admits in Section 3.1. The same-author confusion pattern cannot rule this out because same-author books in the corpus share historical period, register, and topical vocabulary. The cross-lingual transfer is a genuine check, but translation preserves style and content together, so it does not separate them. The proposed held-out-novel test would directly determine whether the classifier generalizes across works by the same author, which is the minimal condition for claiming an author-level stylistic signature rather than a book-level content signature. Since this is the same concern the reader already used to justify a conditional verdict, my stress-test does not change the verdict; it reinforces it with a specific, feasible experimental test.","tokens_in":9508,"tokens_out":2158,"duration_ms":23328,"concrete_test":"Retrain the multiclass probe of Fig. 3 under a held-out-novel split: train on one novel per author (e.g., HM1, JA1, MT1, NH1, VW1) and test on the second novel by the same author (HM2, JA2/JA3, MT2, NH2, VW2). If same-author test accuracy is at or near chance while the chunk-level split remains high, the separability signal is within-novel topic or content, not author style. As a complementary check, mask all open-class tokens (nouns, verbs, adjectives) to placeholders, preserving function words and syntax; if separability collapses, the style interpretation is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that embeddings encode 'style' rests on showing that the separability signal is stylistic rather than topical. Section 3.1 explicitly concedes the alternative: 'It could be that they contain enough factual information (names, subjects, etc.) to reveal their provenance.' The same-author confusion pattern is suggestive but not controlling: same-author pairs such as Melville's Moby Dick and Pierre, or Twain's Life on the Mississippi and Roughing It, share period, register, and broad topic domains, and Austen's novels share Regency social vocabulary and settings. The shuffling experiment in Table 1 makes the confound sharper: at B = 1, where word order and syntax are destroyed, probe accuracy remains 97%, showing that bag-of-words lexical content alone is sufficient. Since shuffling preserves open-class content words, this result does not distinguish style from topic. The 70/30 split is performed at chunk level, and chunks are consecutive N-token windows from a single tokenized novel, so neighboring chunks share proper nouns, sentence fragments, and discourse context; this autocorrelation can inflate reported accuracies. The cross-lingual transfer in Table 2 does not resolve the issue because translation preserves both style and topic. The paper provides no held-out-novel or held-out-author evaluation and no content-word ablation, so the style interpretation remains underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether the final-token embeddings produced by transformer language models encode information about the literary source of a prompt. Using chunks of 8–128 tokens drawn consecutively from 19th- and early-20th-century novels, the authors train linear SVM and MLP probes on rightmost-token activations of Llama-3.2-1B and three other base models. They report high binary and multiclass classification accuracy, increased classifier confusion between books by the same author, a low-dimensional PCA subspace that preserves separability, preserved separability under token shuffling, and cross-lingual transfer between French novels and their English translations. The paper concludes that LLM embeddings encode a stylistic signature of the source oeuvre. The central interpretive claim is that the separable signal is style rather than topic or other factual content; the manuscript itself acknowledges this alternative reading in Section 3.1.","tokens_in":9727,"tokens_out":2384,"duration_ms":26198,"significance":"If the stylistic interpretation were established, the paper would be a valuable contribution to LLM interpretability and to authorship attribution, since it would show that intangible prompt-level properties are linearly readable in deep embeddings of short excerpts. The empirical work has real strengths: the use of multiple open models in Appendix D, the explicit attempt to separate syntax from lexical content through shuffling in Section 3.3, and the exploratory cross-lingual transfer experiment in Section 3.4. The paper also promises public code, which is welcome. However, the load-bearing inference from 'classifier separability of novels' to 'embeddings encode style' is not yet supported, because topic, register, and local discourse context are not controlled in the experiments that are offered as evidence. The result as stated is therefore an interesting phenomenon whose interpretation remains underdetermined.","major_comments":[{"comment":"The same-author confusion pattern is the principal evidence for the stylistic interpretation, but it does not control for topic, register, or period vocabulary. Same-author pairs such as Melville's Moby Dick and Pierre, Twain's Life on the Mississippi and Roughing It, and the three Austen novels share narrative settings, era-specific social vocabulary, and broad thematic domains, so higher confusion between same-author books is exactly what a content-based classifier would produce. The paper needs a control that removes or ablates content words, or an evaluation on held-out novels or held-out authors, before the increased same-author confusion can be attributed to style.","section":"Section 3.1, Fig. 3"},{"comment":"The 70/30 train/test split is applied at the level of individual chunks, and chunks are consecutive non-overlapping windows of N tokens from a single tokenized novel. Neighboring chunks therefore share proper nouns, sentence fragments, discourse context, and local topic, so the test set is autocorrelated with the training set. This leakage can inflate classification accuracy substantially. The authors should either split at the novel or author level, or interleave/downsample chunks to break autocorrelation, and should report the effect of such a split on the accuracies in Figs. 2–3 and Tables 1–2.","section":"Section 2 and Appendix B.3"},{"comment":"The shuffling experiment shows that at B=1, where word order and syntax are destroyed, the linear probe still achieves 97% accuracy on the GE versus VW pair. Because shuffling preserves the multiset of open-class content words, this result demonstrates that bag-of-words lexical content alone is sufficient for separability; it does not distinguish style from topic. The statement in Section 5 that 'the main signal might be about lexical content rather than syntax' is too weak, given that lexical content is itself confounded with topic and factual references. A function-word-only or content-word-ablated analysis is needed to make the stylistic claim.","section":"Section 3.3, Table 1"},{"comment":"The cross-lingual transfer experiment does not resolve the style-versus-topic confound because translation preserves both stylistic choices and topical content. The observation that a French-trained probe transfers to English translations is consistent with the embeddings encoding topic, named entities, or register, as well as with the embeddings encoding style. The paper should state this limitation explicitly and, if possible, test transfer on a pair of languages or registers where topical overlap is controlled.","section":"Section 3.4, Table 2"}],"minor_comments":[{"comment":"The conclusion says 'short (102 tokens) literary excerpts' but the methods and figures use N up to 128; please align the token count or correct the typo.","section":"Conclusion"},{"comment":"The caption says 'Wolf (VW1, VW2)' but the author name is Woolf; please correct the spelling.","section":"Fig. 5 caption"},{"comment":"The two rows of accuracies are unlabeled; please indicate which row corresponds to the reference French-trained probe and which to the English-transferred probe, and report the standard errors or confidence intervals.","section":"Table 2"},{"comment":"The classifier description does not specify the number of random seeds, the validation procedure for hyperparameters, or whether the 70/30 split was stratified; these details are needed for reproducibility given the small number of novels.","section":"Appendix B.3"},{"comment":"The label 'GEVW' is unclear; please indicate whether this refers to the combined GE/VW ensemble or to each ensemble separately, and state the embedding layer used for the intrinsic-dimension estimate.","section":"Fig. 4B"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about the main confound in Section 3.1, but the conclusion nevertheless overstates the stylistic interpretation. The missing controls—content-word ablation, held-out novel or author splitting, and variance reporting—are all within the manuscript's experimental scope and should be added before the claims can be accepted. I do not see an unresolvable flaw; the issue is that the central interpretative claim needs additional experimental support. I would also ask the editor to encourage the authors to release the code and data as promised, since the current manuscript states that code 'will be made available' without a repository link."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical probing paper, not a breakthrough. The headline claim—that deep prompt embeddings carry a linearly readable signature of the source oeuvre—is plausible and mostly supported by their measurements, but the \"style\" interpretation is underdetermined. The paper's own shuffling result (97% accuracy at B=1) shows bag-of-words lexical content is sufficient, so topic and named-entity leakage remain live confounds.\n\nWhat's actually new: the accuracy-vs-layer-depth-vs-context-length surfaces across four 1B-class base models; the PCA finding that about 16 top principal components carry the separable signal; the same-author confusion pattern; and the cross-lingual transfer experiment. These are concrete, reproducible-ish measurements (no code shipped, but the method is straightforward). The paper also honestly admits in Sec 3.1 that factual information could explain the provenance signal. That's good scholarly hygiene.\n\nWhere it's soft: the central confound is not controlled. The 70/30 chunk-level split over consecutive windows from a single novel risks autocorrelation leakage (adjacent chunks share names, settings, discourse context). A held-out-novel evaluation or a function-word-only baseline would have settled a lot. The shuffling experiment is informative but actually strengthens the topic-leakage worry: at B=1, syntax is gone, yet accuracy stays at 97%, so the classifier is mainly using content-word frequencies. Same-author confusion is suggestive, but same-period/same-register authors also share topic distributions (e.g., Melville's sea novels vs. Twain's travelogues share domain vocabulary). No error bars anywhere; with a 70/30 split and random seeds, that's a fixable gap, not a fatal one.\n\nThe cross-lingual transfer is the most convincing piece: a probe trained on French Flaubert vs Hugo transfers to the English translations. That suggests something structural rather than surface names, though translation preserves topic as well as style, so it doesn't fully break the tie.\n\nBottom line: the paper deserves a serious referee, and the gaps are addressable with a moderate revision. For a reading group, it's a useful case study in why binary classifier accuracy alone can't establish a \"style\" claim. I'd recommend conditional accept with a request for topic-controlled baselines, held-out-novel evaluation, and variance reporting.","headline":"A careful probe study showing style signals in LLM prompt embeddings, but the style-vs-topic confound is not controlled; the specific N/L surfaces and cross-lingual transfer are the real novelties.","tokens_in":10318,"tokens_out":2018,"would_cite":false,"duration_ms":18963,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Short literary excerpts carry a stylistic signature readable from a single deep LLM embedding.","keywords":["prompt embeddings","literary style","authorship attribution","transformer internal representations","last-token embedding","style geometry","principal components","cross-lingual transfer"],"falsifier":"Train the same probe on two books by the same author after replacing all proper nouns and content nouns with placeholders while keeping syntax and function words intact; if accuracy drops to chance, the separability was carried by factual content, not style. A second check is to split the chunk ensembles so that entire contiguous blocks of a novel go to either training or test rather than being interleaved; if accuracy collapses under that split, the result depends on autocorrelation between neighboring chunks rather than on a stable style signature.","tokens_in":9272,"feed_emoji":"🖋️","tokens_out":9207,"duration_ms":71632,"temperature":0.7,"pith_summary":"This paper sets out to show that when a large language model reads a short passage of prose, the internal vector of the passage's final token has absorbed information about who wrote it, not just what it means. Using chunks of 8 to 128 tokens from 19th- and early 20th-century novels, the authors probe the rightmost embedding after each transformer layer and find that a linear classifier can separate two authors with over 90% accuracy, while a multiclass probe distinguishes among thirteen books with roughly 75% accuracy. The same-author books are much harder to tell apart than different-author books, which the paper takes as evidence that the encoded signal is authorial style rather than topic. If this is right, authorship becomes readable from a single embedding of about a hundred tokens, without any task-specific fine-tuning.","feed_headline":"One LLM embedding can name the author from 102 tokens","feed_subtitle":"A single last-token vector carries a stylistic signature that survives shuffling and translation.","key_machinery":"The central object is the rightmost-token embedding of a chunk of $N$ tokens after transformer layer $L$, written $\\vec{x}_N^{(L)}$; because the attention mask is causal, this single vector has accumulated information from every preceding token and serves as the prompt's summary at that depth. The authors collect one such vector per chunk for each novel, forming ensembles, and test whether the ensembles are separable using linear SVM probes for binary comparisons and MLP probes for multiclass comparisons. Two observations carry the argument: separability increases with both context length $N$ and layer depth $L$, and classifier confusion is systematically higher between books by the same author than between books by different authors. PCA truncation, block shuffling, and French-to-English transfer are used as controls to locate the signal in a low-dimensional subspace and to test whether syntax or surface language is responsible.","core_discovery":"The paper's central claim is that deep transformer embeddings encode a stylistic signature of the prompt. After enough transformer layers (about $L \\ge 10$) and with enough context (about $N \\ge 64$), the rightmost token's activation in Llama-3.2-1B contains linearly readable information about the literary source of a short excerpt. This signature occupies a low-dimensional subspace of the largest principal components (around 16 directions), persists when token order is scrambled at block sizes 1, 4, and 32, and transfers across languages: a probe trained on distinguishing French novelists keeps most of its accuracy on the English translations of the same books. The higher confusion between different novels by the same author than between novels by different authors is the paper's core evidence that what is encoded is style rather than factual content.","pith_inferences":["A direct test the paper does not run: compare these deep-embedding probes with a function-word-frequency baseline on the same chunks; if the baseline matches the probe accuracy, the transformer may simply be re-encoding count statistics rather than a new kind of style representation.","The cross-language result hints that style occupies a language-neutral region of the embedding space; one could search for a direction whose subtraction removes authorship information while preserving meaning.","Because different shuffling scales are perfectly separable from one another, syntax is still represented somewhere in the embeddings; ablating the style subspace could show whether this authorship signal is causally used in next-token generation or is a by-product."],"forward_implications":["Authorship attribution can work from a single deep embedding of a roughly 100-token excerpt, without fine-tuning a model for the task.","The stylistic signal is carried mainly by lexical content rather than word order, because shuffling tokens in blocks of 1, 4, or 32 leaves probe accuracy high.","The relevant representation is low-dimensional, concentrated in about 16 principal-component directions of the embedding ensemble.","The style signature survives translation between French and English, suggesting it is not tied to the surface language of the text.","The information is built up gradually by the attention mechanism: shallow layers and short contexts do not show separability, while deeper layers and longer contexts do."],"supporting_citations":[{"why":"Supplies the Llama-3.2-1B model whose activations are probed throughout the study.","marker":"MetaAI, 2024"},{"why":"Establishes that prompt embedding ensembles stretch along a few principal directions, motivating the PCA analysis.","marker":"Sarfati et al., 2025"},{"why":"Introduces the block-shuffling perturbation and intrinsic-dimension observations used to test syntax sensitivity.","marker":"Viswanathan et al., 2025"},{"why":"Provides the TwoNN method used to estimate the intrinsic dimension of the embedding ensembles.","marker":"Valeriani et al., 2023"},{"why":"Supports the linear representation hypothesis that abstract attributes can lie along linear directions in embedding space.","marker":"Park et al., 2024"},{"why":"Shows that embeddings can linearly encode factual attributes such as truth, the baseline for encoding non-stylistic information.","marker":"Marks and Tegmark, 2024"},{"why":"Found lexical stylistic features such as formality along linear axes in embedding space.","marker":"Lyu et al., 2023"},{"why":"Shows multilingual embedding spaces separate language-specific style from language-neutral content, supporting the cross-language result.","marker":"Chang et al., 2022"},{"why":"Demonstrates that transformer models can attribute authorship from raw text, the task this paper probes with embeddings.","marker":"Hicke and Mimno, 2023"}],"fun_headline_variants":["One vector bets the author from a short excerpt","LLM hidden state knows the novelist from 64 tokens","Style lives in a low-dim corner of the prompt embedding","Authorship emerges in deep transformer embeddings","Shuffled and translated, the stylistic signature remains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the separability signal being style rather than factual content or shared register; the paper itself allows that names, subjects, and era vocabulary could reveal the source, and because chunks are consecutive windows from one novel, nearby chunks resemble one another in ways that can inflate test accuracy.","fun_headline_variants_meta":{"raw":{"variants":["One vector bets the author from a short excerpt","LLM hidden state knows the novelist from 64 tokens","Style lives in a low-dim corner of the prompt embedding","Authorship emerges in deep transformer embeddings","Shuffled and translated, the stylistic signature remains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000467,"raw_usage":{"total_tokens":2280,"prompt_tokens":845,"completion_tokens":1435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":1361}},"tokens_in":461,"tokens_out":1435,"duration_ms":10472,"temperature":1.0,"reasoning_tokens":1361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:16:06.043571+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same probe on two books by the same author after replacing all proper nouns and content nouns with placeholders while keeping syntax and function words intact; if accuracy drops to chance, the separability was carried by factual content, not style. A second check is to split the chunk ensembles so that entire contiguous blocks of a novel go to either training or test rather than being interleaved; if accuracy collapses under that split, the result depends on autocorrelation between neighboring chunks rather than on a stable style signature.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Llama-3.2-1B model whose activations are probed throughout the study."}],"review_version":1}