REVIEW 3 major objections 4 minor 12 references
Words of War: Exploring the Presidential Rhetorical Arsenal with Deep Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read By labeling each presidential speech by whether the US entered a major war within a year, this paper claims that gated recurrent networks and transformer models classify pre-war rhetoric with test AUC-ROC above 0.9.
desk verdict The reported 0.98 AUC is an artifact of SMOTE applied before the train/test split; the paper is a well-written pilot study whose central result does not survive its own evaluation protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The paper's central object is the War label, a binary variable equal to 1 for speeches delivered within one year before US entry into a major war and 0 otherwise, with start dates taken from the authors' historical source. The mechanism carrying the argument is supervised text classification: BERT, a pre-trained transformer language model, tokenizes and vectorizes cleaned transcripts, and each neural model is trained with a binary loss to separate the two classes. The LSTM-with-attention model adds a learned weighting over positions in the speech, and the paper also applies local and global explanation methods to attribute predictions to input features. Together these pieces convert a historical claim about pre-war rhetoric into a measurable classification signal, then open the model enough to say which parts of a speech drive the signal.
What would settle it
The most direct falsifier is a label-shuffle placebo test: randomly reassign which speeches are labeled War=1, retrain the LSTM-with-attention model under the same pipeline, and check whether AUC-ROC collapses from 0.982 toward 0.5. A weaker version would keep the real labels but use fake war dates drawn from the same marginal distribution; if the score stays high, the model is learning label timing rather than pre-war wording.
Extended reading notes
Core claim
The central discovery the paper argues for is that pre-war rhetoric is learnable from text alone. Using a corpus of 964 US presidential speeches, cleaning the transcripts, and balancing the heavily imbalanced labels with SMOTE, the authors train four architectures and find that the LSTM with an attention layer reaches an AUC-ROC of 0.982, the LSTM-only RNN reaches an F1-score of 0.929, and fine-tuned BERT also exceeds 0.9 AUC-ROC. In the paper's own reading, this accuracy is not merely a classification exercise: it indicates that neural networks can capture rhetorical patterns that precede US entry into major wars, and the interpretability analyses (attention weights, LIME, SHAP) begin to show what those patterns look like. The authors are careful to call this a hint of predictive power for future involvement rather than a proven forecasting method.
Load-bearing premise
The load-bearing premise is that 'within one year of a war start date' cleanly identifies pre-war rhetoric; if the dates or the window leak in other timing signals, the classifier could be separating label eras rather than war language.
Editorial extensions
If this is right
- If the central claim is right, a model trained on presidential transcripts can flag the year before a major US war at near-perfect separation (AUC-ROC 0.982), meaning the pre-war register is present in the text itself.
- The attention-weight distributions separate the two classes, so attention is doing substantive work rather than reweighting noise, which makes the LSTM-with-attention result more than an accuracy number.
- Because LIME and SHAP point to different key features, the paper's interpretive layer implies that local and global explanations capture different aspects of the pre-war signal, and both are needed to describe it.
- The authors' conclusion extends one step further: the same classifiers hint at predictive power for determining involvement in future wars, provided future rhetoric resembles the historical pattern.
Reading between the lines
- A natural next test the paper does not run is a placebo: shuffle the war dates and retrain; if AUC stays near 0.982, the models are reading the calendar, not the rhetoric.
- Varying the window from one year to six months or two years would show whether the signal is a gradual ramp-up before war or an artifact of the cutoff.
- A temporal train/test split, training on conflicts before 1950 and testing on later ones, would reveal whether the learned pattern generalizes across eras or reflects one president's style.
- Applying the classifier to crises that never escalated would show whether it detects specifically war-bound rhetoric or a broader crisis register.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper trains several deep learning models (MLP, LSTM, LSTM with attention, and fine-tuned BERT) on a corpus of US presidential speeches to classify whether a speech was delivered within one year of US entry into a major war. After preprocessing and class balancing with SMOTE and random undersampling, the authors report strong test performance, with the LSTM with attention achieving an AUC-ROC of 0.982 and the LSTM achieving an F1-score of 0.929. They also perform interpretability analyses using attention weights, LIME, and SHAP, and conclude that their models show predictive potential for identifying the rhetoric preceding US involvement in future wars.
Significance. The interdisciplinary goal of linking presidential rhetoric to war onset is interesting, and the explicit attempt to make deep learning predictions interpretable via attention, LIME, and SHAP is commendable. The paper also demonstrates careful data cleaning, such as handling the missing Jefferson transcript and removing numeric artifacts. However, the central empirical claim is not supported because the evaluation protocol contains a data leakage error: the class-balancing transformations are applied to the full dataset before splitting into training, validation, and test sets. This invalidates the reported AUC-ROC and F1 values. The absence of error bars, multiple seeds, temporal splits, and baselines further weakens the evidence. If the evaluation were corrected, the paper could make a modest contribution, but as it stands, the headline results cannot be trusted.
major comments (3)
- [Methods (data preprocessing and resampling)] The SMOTE and random undersampling pipeline is applied to the full dataset before the train/validation/test split, which causes data leakage. The text states that the transformations are combined 'into a single pipeline' and that the predictor features are stacked before this pipeline is applied; it then says the models train on 80% of the data with the remaining 20% split for validation and test. Nowhere is it stated that resampling is confined to the training set. Consequently, synthetic minority examples generated from neighborhoods in the training portion can appear in the test set, making test examples non-independent of training data. This leakage can inflate AUC-ROC and F1 substantially and explains the unusually high values (e.g., 0.982 for LSTM with Attention and 0.929 for LSTM). The claim that the models 'can accurately classify' pre-war rhetoric is unsupported until the resampling is performed inside a proper cross-validation or training-only pipeline.
- [Experiments and Results (model evaluation)] All reported test metrics come from a single run on a single data split, with no error bars, no multiple random seeds, and no variation in the train/test assignment. With only 81 positive examples, the test set is small and the point estimates are highly variable. Moreover, there are no baseline comparisons (e.g., majority-class classifier, logistic regression, or TF-IDF with a simple classifier), so the reader cannot judge whether the complex neural architectures are necessary or whether the task is trivially separable by surface features. Adding these elements is essential for the central claim to be evaluated.
- [Conclusions and label definition] The conclusion that the results 'hint at their predictive power in determining involvement in future wars' is an overreach because the evaluation uses a random split rather than a temporal split. A model trained on speeches from across two centuries and tested on a random held-out subset can achieve high accuracy by exploiting period-specific stylistic correlations, not by learning a generalizable pre-war signal. A temporal split (e.g., training on speeches before 1950 and testing on those after) would be needed to support any claim about future prediction. Relatedly, the definition of the War label as 'within one year of the president's speech' is ambiguous about direction; if it includes speeches after a war has started, the model is partly detecting contemporaneous war rhetoric rather than preparatory rhetoric. This ambiguity should be clarified and, ideally, the label direction fixed as 'within one year before'.
minor comments (4)
- [Throughout] Several references to 'the plot above' and 'the chart above' indicate figures that are not present in the manuscript; these figures need to be included or the text should be revised.
- [References] Reference [6] spells the first author's name as 'Riberio' but the correct spelling is 'Ribeiro'.
- [Future Research] The phrase 'find-grained interpretation' contains a typo; it should be 'fine-grained interpretation'.
- [Methods (BERT preprocessing)] The paper does not describe how the BERT model handles speeches longer than the 512-token maximum sequence length; the text mentions converting speeches to 'fixed-length vectors' but does not specify truncation, chunking, or other long-text handling, leaving this part of the methodology incomplete.
Circularity Check
No circularity: the classification pipeline is self-contained and the reported scores are empirical measurements, not identities.
full rationale
The paper's derivation chain is a standard supervised text-classification pipeline. The War label is assigned from Congressional Research Service war start dates using a one-year window; the features are BERT tokenizations of cleaned speech transcripts; and the models are fit with binary cross-entropy and evaluated on a held-out split. No fitted parameter is defined in terms of the reported AUC-ROC or F1-score, no result is imported from a same-author uniqueness theorem, and no target quantity is renamed as an output. The reported test values, such as 'the LSTM with Attention has the highest value at 0.982' and the 'RNN with LSTM slightly outperforms the others in terms of F1-Score, achieving 0.929,' are empirical measurements from the test set, not consequences of how the model or labels were constructed. The concluding claim that the models 'hint at their predictive power in determining involvement in future wars' is an extrapolation beyond the random-split evaluation, but extrapolation is not circular reasoning. The Methods section's SMOTE-before-split protocol is a possible data-leakage and validity flaw that could inflate test performance, but that is a correctness concern: the test labels are still external to the fitted model's objective, so the result does not reduce to its own input by construction. The paper also contains no load-bearing self-citations; its references are to external data sources, external methods papers, and external background work. Therefore no circular step is exhibited and the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (3)
- SMOTE and undersampling balance ratio =
unspecified, presumably near 50/50
- BERT maximum sequence length =
unspecified (likely 512 by default)
- Model hyperparameters (learning rates, LSTM units, L2, dropout) =
various, e.g., LSTM units 128, BERT learning rate 3e-7
assumptions (4)
- domain assumption The Miller Center transcripts accurately represent the delivered speeches.
- domain assumption The Congressional Research Service war start dates and the one-year window define a meaningful positive class.
- domain assumption SMOTE-generated synthetic speeches are valid representatives of the minority class.
- domain assumption BERT embeddings capture the relevant semantic signal for classifying historical rhetoric.
Cite this review
Pith. "Pith review of Words of War: Exploring the Presidential Rhetorical Arsenal with Deep Learning." pith.science (2026). https://pith.science/paper/R3VRIG6U
@misc{pith2026241208868,
author = {Pith},
title = {Pith review of: Words of War: Exploring the Presidential Rhetorical Arsenal with Deep Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/R3VRIG6U}},
note = {Machine review of arXiv:2412.08868}
}
read the original abstract
In political discourse and geopolitical analysis, national leaders words hold profound significance, often serving as harbingers of pivotal historical moments. From impassioned rallying cries to calls for caution, presidential speeches preceding major conflicts encapsulate the multifaceted dynamics of decision-making at the apex of governance. This project aims to use deep learning techniques to decode the subtle nuances and underlying patterns of US presidential rhetoric that may signal US involvement in major wars. While accurate classification is desirable, we seek to take a step further and identify discriminative features between the two classes (i.e. interpretable learning). Through an interdisciplinary fusion of machine learning and historical inquiry, we aspire to unearth insights into the predictive capacity of neural networks in discerning the preparatory rhetoric of US presidents preceding war. Indeed, as the venerable Prussian General and military theorist Carl von Clausewitz admonishes, War is not merely an act of policy but a true political instrument, a continuation of political intercourse carried on with other means (Clausewitz, 1832).
Reference graph
Works this paper leans on
-
[1]
von Clausewitz, C. (1997). On War (J. J. Graham, Trans.). Wordsworth Editions
work page 1997
-
[2]
Lilleberg, J. (2020). United States presidential speeches. Kaggle. https://www.kaggle.com/datasets/littleotter/united-states-presidential- speeches. Data scraped from The Miller Center at the University of Virginia, https://millercenter.org/the-presidency/presidential-speeches
work page 2020
-
[3]
Miller, US Congressional Re- search Service
Barbara Salazar Torreon and Carly A. Miller, US Congressional Re- search Service. (2024). U.S. Periods of War and Dates of Recent Conflicts, available at https://sgp.fas.org/crs/natsec/RS21405.pdf
work page 2024
-
[4]
SMOTE: Synthetic Minority Over-sampling Technique,
Nitesh V . Chawla et al., “SMOTE: Synthetic Minority Over-sampling Technique,” Journal of Artificial Intelligence 16 (2002) pp. 321-357
work page 2002
-
[5]
Jacob Devlin et al. (2019). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1
work page 2019
-
[6]
Riberio et al., Why Should I Trust You?: Explaining the Predictions of Any Classifier, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016), pp. 1135- 1144
work page 2016
-
[7]
Scott M. Lundberg and Su Lee. (2017). A Unified Approach to Inter- preting Model Predictions. In I. Guyon & U. V . Luxburg & S. Bengio & H. Wallach & R. Fergus & S. Vishwanathan & R. Garnett (Eds.), Advances in Neural Information Processing Systems (pp. 4765-4774). Curran Associates, Inc
work page 2017
-
[8]
Zican Dong et al. (2022). A Survey on Long Text Modeling with Transformers. ArXiv 2302.14502v1. See also Park et al. (2022). Ef- ficient Classification of Long Documents Using Transformers. ArXiv 2203.11258v1
arXiv 2022
Show all 12 references
-
[9]
Aman Jaiswal and Evangelos Milios. (2023). Breaking the Token Bar- rier: Chunking and Convolution for Efficient Longer Text Classification with BERT. ArXiv 2310.2055av1
2023
-
[10]
Michal Brzozowski. (2023). Fine-tuning BERT model for arbitrarily long texts Part 1. MIM AI. See also Michal Brzozowski. (2023). Fine-tuning BERT model for arbitrarily long texts, Part 2. MIM AI. For technical documentation, see Michal Brzozowski and Marek Wachnicki. (2023). W...
2023
-
[11]
Xianming Li et al. (2023). Recurrent Attention Networks for Long-text Modeling. Findings of the Association for Computational Linguistics (ACL), pp. 3006-3019
2023
-
[12]
Dat Hong et al. (2023). ProtoryNet - Interpretable Text Classification Via Prototype Trajectories. Journal of Machine Learning Research 24, pp. 1-39
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.