{"id":"fc294e50-6e6b-4cfb-bff1-df9d56e5968c","arxiv_id":"2501.10072","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A small empirical study reports that bigram POS features with a CNN give higher author classification accuracy (59%) than unigram POS features with a feedforward network (44%), but the comparison is confounded and not statistically validated.","lead":"Researchers trained simple neural networks to identify literary authors from part-of-speech counts and bigram counts in 193 German-language books, finding higher accuracy with bigram features (59% vs 44%). The result is suggestive, but the two models differ in architecture and the study lacks statistical validation and public code or data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bigram-superiority claim is unsupported: feature type and network architecture are confounded, and the test set is likely too small for the 0.59 vs 0.44 gap to be reliable.","rationale":"The reader's weakest assumption identifies the same load-bearing problem: the unigram and bigram experiments differ in both feature representation and model architecture, so the reported improvement cannot be attributed to the features. I agree with that assessment, and I would sharpen it in two ways. First, the architecture confound alone invalidates the causal claim, because CNNs and fully connected networks differ in capacity, regularization, and inductive bias; the bigram model's higher training accuracy (0.81 vs 0.61) and test accuracy (0.59 vs 0.44) could come entirely from the CNN. Second, the test-set size issue is severe: the paper describes a stratified split of 193 books, but the classification task concerns only 8 authors with at least 5 works each, so the test fold is likely only a handful of books; with such a small test set, the difference between 0.59 and 0.44 may be a difference of one or two books. The absence of confidence intervals, bootstrap estimates, or a significance test makes it impossible to rule this out. The proposed 2x2 experiment directly isolates feature representation from architecture and would settle whether bigram features actually add information. The MDS plots, while visually suggestive, are not a quantitative substitute. Because the strongest claim is not established by the current experimental design, the reader's REJECT verdict is appropriate; no change to that verdict is needed.","tokens_in":7372,"tokens_out":5706,"duration_ms":58333,"concrete_test":"Run a 2x2 controlled comparison on the same stratified split: (a) FNN on 11-dimensional POS vectors; (b) FNN on flattened 121-dimensional bigram matrices; (c) CNN on 11x11 bigram matrices; (d) CNN on an 11x11 unigram-derived input, for example the unigram vector expanded or broadcast to 11x11. Use multiple random seeds and report test accuracy with 95% bootstrap confidence intervals at the book level, together with the test-set size and per-class counts. If bigram inputs beat unigram inputs within both architectures and the confidence intervals do not overlap, the confound is resolved; otherwise the paper's headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that bigram frequency matrices are significantly more effective than unigram POS-tag vectors rests entirely on comparing a CNN trained on 11x11 bigram matrices (test accuracy 0.59, Results: Author Classification Using Bigram Matrices) with a fully connected network trained on 11-dimensional POS vectors (test accuracy 0.44, Results: Author Classification Using POS-Tag Vectors). This is not a controlled comparison: the feature representation and the network architecture change simultaneously, so the 0.15 accuracy gap cannot be attributed to sequential word-class information. The paper never runs the same architecture on both representations, nor any simple baseline such as logistic regression or SVM, and it reports no confidence intervals, per-class test counts, or significance test. The test set is also almost certainly very small: the 80/10/10 stratified split is described for the corpus of 193 books, but the classification task uses only 8 authors with at least 5 works each; if the relevant subset contains roughly 40 to 80 books, the test fold contains only about 4 to 8 books total, so the reported accuracies correspond to a handful of classification decisions. The qualitative MDS cluster separation in Figures 2 through 6 does not quantify classification accuracy and cannot substitute for a controlled evaluation. The central claim therefore lacks the experimental support needed to distinguish a genuine stylistic signal from architecture choice and sampling noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes distinguishing literary authors using part-of-speech (POS) tag unigram frequency vectors and POS bigram frequency matrices, trained with a fully connected network and a convolutional neural network, respectively. The reported test accuracy is 0.44 for the POS-tag/fully connected condition and 0.59 for the bigram/CNN condition on a subset of 8 authors with at least 5 works from a corpus of 193 (apparently German) books, with additional multi-dimensional scaling (MDS) visualizations of cluster structure. The central claim is that sequential word-class patterns (bigrams) are more distinctive of authorial style than unigram POS tag frequencies, and that this is demonstrated by the higher test accuracy of the bigram/CNN model.","tokens_in":7637,"tokens_out":5133,"duration_ms":49831,"significance":"If the comparison were valid, this would be a modest but potentially useful contribution to authorship attribution and computational stylistics, since it would show that POS bigram frequency matrices carry author-discriminative information beyond unigram POS frequencies. The paper has a clear task and makes its network architectures explicit in Tables I and II, and the MDS figures provide a qualitative look at the data. However, the paper does not provide code or data, and the central empirical comparison is not controlled, so the significance is conditional on the authors' ability to substantiate the claimed feature-based improvement with a valid experimental design.","major_comments":[{"comment":"The central comparison of test accuracies 0.44 versus 0.59 is confounded because the POS-tag condition uses a fully connected network (Table I) while the bigram condition uses a CNN (Table II), so the feature representation and the network architecture change simultaneously. The claim that bigram features \"significantly improve performance\" (Abstract, Discussion) cannot be distinguished from an architecture effect on this evidence. The authors must run the same architecture on both feature types, or at least include a standard baseline such as logistic regression or an SVM on both representations, to support the stated conclusion.","section":"Results: Author Classification Using POS-Tag Vectors and Author Classification Using Bigram Matrices (Figures 5 and 6…"},{"comment":"The paper does not report the test set size, the number of works per author in the 8-author subset, per-class counts, confidence intervals, or significance tests. Given the 80/10/10 stratified split described for the full corpus and the restriction to authors with at least five works, the test fold likely contains only a handful of books (possibly as few as 4–8), making the point estimates 0.44 and 0.59 statistically unreliable. The authors should report exact sample sizes for the subset and the split, and provide a measure of uncertainty such as bootstrap confidence intervals or exact binomial intervals.","section":"Data Set and Results: Author Classification sections"},{"comment":"The MDS visualizations are used as evidence that bigram features produce \"more pronounced separations\" and \"tighter clusters,\" but these are qualitative projections of input features or hidden-layer outputs, not a quantitative measure of classification performance. Cluster compactness in an MDS plot does not translate directly into test accuracy and can be affected by projection parameters and stress; the paper should either quantify cluster separability with a standard index or base its feature-comparison claim solely on the classification experiments.","section":"Results: Multi-Dimensional Scaling of Linguistic Patterns and Clustering Analysis of Frequent Authors (Figures 2–4)"},{"comment":"The statement that MDS is \"parameter-free and all mutual distances of the points are preserved\" is inaccurate: MDS requires choosing between metric and nonmetric variants, defining an initialization, and selecting the number of output dimensions, and it minimizes a stress function rather than preserving all pairwise distances exactly. This sentence should be corrected, as it overstates the properties of the method.","section":"Methods: Multi-dimensional Scaling (MDS)"}],"minor_comments":[{"comment":"Table II contains several typos: \"Convolutio 2D\" should be \"Convolution 2D,\" \"Faltten\" should be \"Flatten,\" \"Droput\" should be \"Dropout,\" and the table title \"BIGRAM -VECTORS\" is inconsistent with the input being a matrix; please correct these.","section":"Table II"},{"comment":"The figure captions contain typographical errors, including \"anlysis\" in Figure 2 and \"An simple fully-connected network\" in Figure 5; these should be corrected to \"analysis\" and \"A simple fully-connected network.\"","section":"Figure captions"},{"comment":"The language of the literary texts is not stated explicitly, although the spaCy German model is used; if the works by Poe, Verne, and Zweig are German translations, this should be stated because it affects the interpretation of authorial style and the generalization of the results.","section":"Data Set"},{"comment":"The classification task uses only \"eight authors with at least five works,\" but the selection criteria, the identities of these authors, and their individual work counts are not reported; please provide these details to enable reproducibility.","section":"Data Set"},{"comment":"The word \"significantly\" in the Abstract and Discussion is not supported by any statistical test; the text should say the reported test accuracy is higher in the bigram/CNN experiment, not that the improvement is statistically significant.","section":"Abstract and Discussion"},{"comment":"The data and code availability statements say materials will be made available \"upon request\" but provide no repository, license, or contact mechanism; a persistent public repository would substantially improve reproducibility.","section":"Additional Information"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is methodologically thin, and the central feature-comparison claim is invalid as reported because architecture and features are confounded and the test set is almost certainly too small for reliable accuracy estimates. The paper also leans heavily on self-citations in the MDS motivation (e.g., refs [16]–[18] concerning sleep EEG), which are not directly relevant to the linguistic question. A major revision with new controlled experiments and proper statistical reporting could make the claim defensible, but the current version is not acceptable for publication. The fit for a cs.CL venue is questionable given the very short experimental report format."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: what is new here is only the specific result on this particular German corpus; what the paper wants to conclude is not supported by its own experimental design.\n\nThe study uses standard methods: POS tagging with spaCy, bigram counts for consecutive word classes, fully connected and convolutional networks, and MDS for visualization. The positive side is that the writeup is clear, the preprocessing is described, and the network architecture tables are concrete. The MDS plots give a qualitative sense that authors do cluster on the basis of POS and bigram counts, which is a reasonable sanity check.\n\nThe soft spot is central. The paper claims bigrams are 'significantly more effective' than unigrams because the CNN on 11x11 bigram matrices reaches 0.59 test accuracy while the FNN on 11-dimensional POS vectors reaches 0.44. But the two setups differ in feature type and network architecture at the same time. No experiment holds the architecture fixed across feature representations, no baseline classifier is added, and no confidence intervals or significance tests appear. The test set is also likely tiny: the 80/10/10 split is done on all 193 books, but classification only uses the eight authors with at least five works. If that subset is 40-80 books, the test fold has 4-8 books, so the reported accuracies are based on a handful of decisions. The MDS cluster separation in Figures 2-6 is qualitative and cannot substitute for a quantitative comparison.\n\nThe paper is not circular and I don't see any fabricated data. The references are generally on point, though they lean heavily on the authors' own prior MDS work in unrelated domains. Code and data are only 'available upon request', which is a reproducibility negative.\n\nWho should read this? Someone looking for a quick illustrative example of a common design trap in applied ML. Not someone building an attribution system. My recommendation: as submitted, this should not go to peer review. If the authors return with matched architectures, simple baselines, and uncertainty quantification on a properly sized test set, the result could become a minor but valid empirical data point. Until then, treat the headline claim as unverified.","headline":"A small, clearly written stylometry study whose main bigram-vs-unigram claim is undermined by a confounded comparison and a likely tiny test set.","tokens_in":8139,"tokens_out":3415,"would_cite":false,"duration_ms":34678,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sequential word-class patterns, not just their frequencies, carry an author's stylistic signature, as shown by bigram matrices classifying eight authors at 59% test accuracy versus 44% for unigram counts.","keywords":["author classification","part-of-speech tagging","bigram frequency matrix","convolutional neural network","authorial style","multi-dimensional scaling","computational stylometry","German literary corpus"],"falsifier":"Re-run the classification with the same CNN on unigram POS vectors and with the same fully connected network on flattened bigram matrices; if the bigram advantage disappears or reverses, the paper's claim that sequential patterns are more distinctive is not supported. Alternatively, compute a confidence interval for the 0.59 versus 0.44 difference on the actual test split; a large overlap would settle the comparison.","tokens_in":7180,"feed_emoji":"📚","tokens_out":3654,"duration_ms":35333,"temperature":0.7,"pith_summary":"This paper asks whether the order of grammatical categories—the sequence of word classes in a sentence—carries an author's stylistic fingerprint. Working with 193 German literary works, it compares two feature representations: unigram frequency vectors of part-of-speech tags and 11x11 bigram frequency matrices of consecutive word-class pairs. Fed to a neural network, bigram matrices classify the eight most frequent authors with 59 percent test accuracy, while unigram vectors reach 44 percent, both far above chance at 12.5 percent. The paper argues the gap shows sequential patterns in word classes are more distinctive of authorial style than simple frequency counts.","feed_headline":"Bigram word-class sequences identify authors at 59 percent accuracy","feed_subtitle":"In 193 German literary works, sequential POS bigrams beat unigram counts (44 percent), with chance at 12.5 percent.","key_machinery":"The central object is the bigram frequency matrix: an 11x11 table whose entry (i,j) counts how often POS tag i is immediately followed by POS tag j in a text, normalized by word count. The paper feeds these matrices into a two-dimensional convolutional network with max pooling, treating each matrix as a single-channel image, and compares this with a fully connected network fed the 11-dimensional unigram POS vector. The CNN is what lets the model exploit local spatial structure in the bigram matrix, and the comparison is meant to isolate the value of sequential word-class information.","core_discovery":"The authors claim that sequential word class distributions—bigrams of part-of-speech tags—capture author-specific stylistic traits more strongly than isolated POS tag frequencies. Using a convolutional network on 11x11 bigram matrices, they reach 0.81 training and 0.59 test accuracy for eight authors, compared with a fully connected network on unigram POS vectors that reaches 0.61 and 0.44. MDS projections show tighter clusters for bigram features, consistent with better separability. The central discovery, as the paper presents it, is that the added contextual relationship between consecutive word classes carries a substantial share of the stylistic signal.","pith_inferences":["A cleaner test would feed unigram POS vectors into the same CNN architecture used for bigrams; if the accuracy gap persists, the features themselves cause the improvement, making the paper's conclusion stronger.","The finding suggests that authors may be distinguished by their grammatical rhythm—the predictability of word-class transitions—which could be combined with lexical features such as function-word frequencies in a multi-stream model.","Because the bigram matrix is only 11x11, the method is cheap and could scale to larger authorship corpora or to other languages by retraining the POS tagger.","The 59 percent ceiling hints that word-class bigrams alone are not enough for high-accuracy attribution in larger author sets; combining them with longer n-grams or syntactic features could be a natural next step."],"forward_implications":["Bigram word-class matrices can serve as a usable feature for author attribution in German literary texts, reaching 59 percent test accuracy for eight authors with 193 books.","Sequential word-class patterns appear more stable within an author than simple POS frequencies, as shown by tighter MDS clusters and higher classification accuracy.","The approach is interpretable, since POS tags and bigram counts can be inspected in histograms without relying on learned internal representations.","Success on unseen books indicates that some consistent, author-specific grammatical sequencing persists across different works of the same author.","Both accuracies exceed chance by a wide margin, so both unigram and bigram word-class distributions carry a real stylistic signal."],"supporting_citations":[{"why":"Positions authorship attribution as the application domain the study contributes to.","marker":"[1]"},{"why":"Supplies the spaCy German model used to tokenize and POS-tag the literary corpus.","marker":"[7]"},{"why":"Documents the spaCy NLP toolchain that produces the POS tags and bigram input features.","marker":"[8]"},{"why":"Provides the Keras deep learning framework in which both neural network architectures were implemented.","marker":"[6]"},{"why":"Establishes word-class distribution analysis on linguistic corpora, the direct precursor to the unigram and bigram features used here.","marker":"[19]"}],"fun_headline_variants":["Sequential POS bigrams beat unigrams for author identification","59% accuracy: word-class bigrams reveal author style","Beyond word counts: bigram patterns identify authors at 59%","In 193 German works, POS bigrams classify authors better than counts","From 44% to 59%: sequential bigrams sharpen author attribution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy gain is attributed to the features, but the two models differ in architecture as well as features, and the paper reports no test-set size or confidence intervals, so the cause of the gain is not isolated.","fun_headline_variants_meta":{"raw":{"variants":["Sequential POS bigrams beat unigrams for author identification","59% accuracy: word-class bigrams reveal author style","Beyond word counts: bigram patterns identify authors at 59%","In 193 German works, POS bigrams classify authors better than counts","From 44% to 59%: sequential bigrams sharpen author attribution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001113,"raw_usage":{"total_tokens":4575,"prompt_tokens":823,"completion_tokens":3752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":3662}},"tokens_in":439,"tokens_out":3752,"duration_ms":26993,"temperature":1.0,"reasoning_tokens":3662,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:22:56.715002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the classification with the same CNN on unigram POS vectors and with the same fully connected network on flattened bigram matrices; if the bigram advantage disappears or reverses, the paper's claim that sequential patterns are more distinctive is not supported. Alternatively, compute a confidence interval for the 0.59 versus 0.44 difference on the actual test split; a large overlap would settle the comparison.","supporting_citations":[{"cited_title":"A survey of modern authorship attribution methods,","cited_arxiv_id":null,"evidence_quote":"Positions authorship attribution as the application domain the study contributes to."},{"cited_title":"Srinivasa-Desikan, Natural Language Processing and Computational Linguistics: A practical guide to text analysis with Python, Gensim, spaCy, and Keras","cited_arxiv_id":null,"evidence_quote":"Supplies the spaCy German model used to tokenize and POS-tag the literary corpus."},{"cited_title":"spacy-industrial-strength natural language processing in python,","cited_arxiv_id":null,"evidence_quote":"Documents the spaCy NLP toolchain that produces the POS tags and bigram input features."},{"cited_title":"Chollet, Deep Learning mit Python und Keras: Das Praxis-Handbuch vom Entwickler der Keras-Bibliothek","cited_arxiv_id":null,"evidence_quote":"Provides the Keras deep learning framework in which both neural network architectures were implemented."},{"cited_title":"Analysis of continuous neuronal activity evoked by natural speech with computational corpus linguistics methods,","cited_arxiv_id":null,"evidence_quote":"Establishes word-class distribution analysis on linguistic corpora, the direct precursor to the unigram and bigram features used here."}],"review_version":1}