{"id":"6a12647c-1e2a-425e-afbe-4bc3ced26a75","arxiv_id":"2412.08868","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Deep learning classifiers appear to distinguish pre-war presidential speeches from others, but the reported high performance is inflated by resampling the full dataset before splitting into train and test sets.","lead":"This preprint applies several deep learning models, including fine-tuned BERT and LSTMs, to US presidential speeches to classify whether each speech was given within one year of US entry into a major war, reporting high test accuracy. A generalist might read it to see whether neural networks can detect pre-war rhetoric from text, but the evaluation has a critical data leakage problem.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SMOTE is applied before the train/validation/test split, so synthetic minority examples generated from training data can appear in the test set; the reported AUC-ROC above 0.9 likely reflects leakage rather than generalization.","rationale":"The paper's central claim is empirical: that neural networks can accurately classify pre-war presidential rhetoric, with test AUC-ROC above 0.9. The strongest evidence for that claim is the test-set performance. The Methods section describes SMOTE and random undersampling as a single pipeline applied before the data are split into training, validation, and test sets. That ordering means synthetic minority examples generated from training data can be present in the test set, violating the independence needed for a valid generalization estimate. The reported metrics are therefore not trustworthy as evidence of classification ability. This is the most load-bearing concern because the conclusion, the abstract, and the headline results all rest on those test numbers. If the evaluation is corrected and the high performance persists, the central claim would be much better supported; but as presented, the empirical contribution fails. The reader's rationale identifies the same leakage issue, though the reader's stated 'weakest assumption' focuses on the one-year war window and war-start dates, which is a related but secondary concern. No ad hominem is intended; the issue is purely methodological and testable by re-running the pipeline correctly.","tokens_in":5367,"tokens_out":3065,"duration_ms":35223,"concrete_test":"Re-run all four models with a time-ordered split: train on speeches before a cutoff date, validate on a later contiguous block, and test on the most recent block; apply SMOTE and random undersampling only to the training block, never to validation or test. Also recompute with a random stratified split but with SMOTE applied inside each training fold. Report test AUC-ROC and F1-score with 95% confidence intervals over at least five seeds. If the LSTM-with-Attention AUC drops materially below 0.9, or approaches the majority-class baseline, leakage is confirmed.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing step is the evaluation protocol. The Methods section states that SMOTE is used to balance the classes and is combined with random undersampling 'into a single pipeline,' and only afterward do the models train on 80% of the data with validation and test each taking half of the remaining 20%. As written, the resampling pipeline runs on the full dataset before the split, so the test set contains synthetic minority-class vectors generated from training-set neighborhoods. Test examples are therefore not independent of training data; a model can score high by recognizing interpolated neighbors rather than by learning to generalize war rhetoric. This could explain the unusually high test AUC-ROC (0.982 for LSTM with Attention) and F1-score (0.929 for LSTM) obtained from a single run with no error bars. The claim that gated RNNs and BERT 'can accurately classify' pre-war rhetoric is unsupported until this leakage is removed. A secondary concern is that the War label is defined as entry into a major war 'within one year' of the speech, with no explicit directionality and no temporal split; the model may be detecting label timing or stylistic autocorrelation around wars. But the leakage alone invalidates the headline result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains several deep learning models (MLP, LSTM, LSTM with attention, and fine-tuned BERT) on a corpus of US presidential speeches to classify whether a speech was delivered within one year of US entry into a major war. After preprocessing and class balancing with SMOTE and random undersampling, the authors report strong test performance, with the LSTM with attention achieving an AUC-ROC of 0.982 and the LSTM achieving an F1-score of 0.929. They also perform interpretability analyses using attention weights, LIME, and SHAP, and conclude that their models show predictive potential for identifying the rhetoric preceding US involvement in future wars.","tokens_in":5590,"tokens_out":3225,"duration_ms":34629,"significance":"The interdisciplinary goal of linking presidential rhetoric to war onset is interesting, and the explicit attempt to make deep learning predictions interpretable via attention, LIME, and SHAP is commendable. The paper also demonstrates careful data cleaning, such as handling the missing Jefferson transcript and removing numeric artifacts. However, the central empirical claim is not supported because the evaluation protocol contains a data leakage error: the class-balancing transformations are applied to the full dataset before splitting into training, validation, and test sets. This invalidates the reported AUC-ROC and F1 values. The absence of error bars, multiple seeds, temporal splits, and baselines further weakens the evidence. If the evaluation were corrected, the paper could make a modest contribution, but as it stands, the headline results cannot be trusted.","major_comments":[{"comment":"The SMOTE and random undersampling pipeline is applied to the full dataset before the train/validation/test split, which causes data leakage. The text states that the transformations are combined 'into a single pipeline' and that the predictor features are stacked before this pipeline is applied; it then says the models train on 80% of the data with the remaining 20% split for validation and test. Nowhere is it stated that resampling is confined to the training set. Consequently, synthetic minority examples generated from neighborhoods in the training portion can appear in the test set, making test examples non-independent of training data. This leakage can inflate AUC-ROC and F1 substantially and explains the unusually high values (e.g., 0.982 for LSTM with Attention and 0.929 for LSTM). The claim that the models 'can accurately classify' pre-war rhetoric is unsupported until the resampling is performed inside a proper cross-validation or training-only pipeline.","section":"Methods (data preprocessing and resampling)"},{"comment":"All reported test metrics come from a single run on a single data split, with no error bars, no multiple random seeds, and no variation in the train/test assignment. With only 81 positive examples, the test set is small and the point estimates are highly variable. Moreover, there are no baseline comparisons (e.g., majority-class classifier, logistic regression, or TF-IDF with a simple classifier), so the reader cannot judge whether the complex neural architectures are necessary or whether the task is trivially separable by surface features. Adding these elements is essential for the central claim to be evaluated.","section":"Experiments and Results (model evaluation)"},{"comment":"The conclusion that the results 'hint at their predictive power in determining involvement in future wars' is an overreach because the evaluation uses a random split rather than a temporal split. A model trained on speeches from across two centuries and tested on a random held-out subset can achieve high accuracy by exploiting period-specific stylistic correlations, not by learning a generalizable pre-war signal. A temporal split (e.g., training on speeches before 1950 and testing on those after) would be needed to support any claim about future prediction. Relatedly, the definition of the War label as 'within one year of the president's speech' is ambiguous about direction; if it includes speeches after a war has started, the model is partly detecting contemporaneous war rhetoric rather than preparatory rhetoric. This ambiguity should be clarified and, ideally, the label direction fixed as 'within one year before'.","section":"Conclusions and label definition"}],"minor_comments":[{"comment":"Several references to 'the plot above' and 'the chart above' indicate figures that are not present in the manuscript; these figures need to be included or the text should be revised.","section":"Throughout"},{"comment":"Reference [6] spells the first author's name as 'Riberio' but the correct spelling is 'Ribeiro'.","section":"References"},{"comment":"The phrase 'find-grained interpretation' contains a typo; it should be 'fine-grained interpretation'.","section":"Future Research"},{"comment":"The paper does not describe how the BERT model handles speeches longer than the 512-token maximum sequence length; the text mentions converting speeches to 'fixed-length vectors' but does not specify truncation, chunking, or other long-text handling, leaving this part of the methodology incomplete.","section":"Methods (BERT preprocessing)"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a course-style project report rather than a fully developed research article. The empirical claims are, at present, invalid because of the data leakage in the resampling pipeline. The good news is that the leakage is fixable by re-running the experiments with SMOTE applied within cross-validation folds, and by adding error bars, baselines, and a temporal-split analysis. However, even with these corrections, the contribution may be modest given the small dataset and the lack of a clear theoretical framework. The journal should weigh whether the paper's interdisciplinary framing and interpretability analyses are sufficient to warrant the additional revision cycles."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2412.08868. The paper applies standard deep learning to classify US presidential speeches as within one year before a major war. The dataset is new (Miller Center via Kaggle) and the writing is clear, but the headline numbers (AUC 0.982, F1 0.929) are not trustworthy. The methods say the authors combined SMOTE and random undersampling into a single pipeline and then, after that, split the data 80/10/10. That means synthetic minority examples generated from training neighborhoods are in the test set. The classifier is partly recognizing its own interpolated neighbors, so the reported generalization is largely leakage.\n\nWhat is genuinely new: the specific dataset and label construction, and the attempt to add interpretability via attention weights, LIME, and SHAP. The authors also deserve credit for a clean, readable narrative and for acknowledging the class imbalance problem. But novelty is modest: BERT and LSTMs are standard, and the paper does not compare to simpler baselines (e.g., logistic regression on bag-of-words) or to prior work on political rhetoric and conflict.\n\nThe soft spots are proportionate. The leakage alone invalidates the central claim, and it is a fixable error: resample only the training portion after the split, or use cross-validation with resampling inside each fold. The paper also lacks error bars, uses a single run, and does not do a temporal split, which matters because the positive labels are tied to specific historical periods; the model may be picking up era-specific style rather than war-specific rhetoric. The conclusion that the models 'hint at predictive power in determining involvement in future wars' is an overreach; classification accuracy on a balanced, leaked test set says nothing about forecasting.\n\nIf the evaluation were corrected, this could be a reasonable pilot study for a workshop, but in current form the empirical contribution is unsupported. The interpretability analysis is too shallow to salvage it: attention weight distribution overlap and a single LIME example are not enough to claim insight into rhetoric.\n\nWho is this for? A reader interested in a cautionary example of SMOTE leakage or a student project template might find value. Not for someone who wants reliable evidence about pre-war rhetoric.\n\nMy recommendation: desk reject in current form, but tell the authors the core question is worth pursuing with a proper evaluation. If they fix the split and add baselines and error bars, a resubmission could be worth a look.","headline":"The reported 0.98 AUC is an artifact of SMOTE applied before the train/test split; the paper is a well-written pilot study whose central result does not survive its own evaluation protocol.","tokens_in":6126,"tokens_out":3965,"would_cite":false,"duration_ms":41508,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By labeling each presidential speech by whether the US entered a major war within a year, this paper claims that gated recurrent networks and transformer models classify pre-war rhetoric with test AUC-ROC above 0.9.","keywords":["presidential rhetoric","deep learning","text classification","war prediction","LSTM with attention","BERT","model interpretability","SMOTE"],"falsifier":"The most direct falsifier is a label-shuffle placebo test: randomly reassign which speeches are labeled War=1, retrain the LSTM-with-attention model under the same pipeline, and check whether AUC-ROC collapses from 0.982 toward 0.5. A weaker version would keep the real labels but use fake war dates drawn from the same marginal distribution; if the score stays high, the model is learning label timing rather than pre-war wording.","tokens_in":5152,"feed_emoji":"⚔️","tokens_out":9546,"duration_ms":92003,"temperature":0.7,"pith_summary":"This paper asks whether the words US presidents choose before the country enters a major war are measurably different from ordinary presidential speech. It turns that question into a binary text-classification problem: each speech is labeled 1 if the US entered a major war within one year of the address and 0 otherwise, using war-start dates from the paper's historical source. On 964 cleaned transcripts, several neural architectures are trained, and the authors report that the gated recurrent and transformer models all reach test AUC-ROC above 0.9, with LSTM with attention at 0.982 and the RNN with LSTM at an F1-score of 0.929. They then use attention weights, local explanations, and global feature attributions to identify which parts of a speech push the classifier toward the pre-war class. If the result holds, it suggests that war-preparatory rhetoric is a learnable textual signal, not just a retrospective label.","feed_headline":"Neural nets spot pre-war presidential rhetoric at 0.98 AUC","feed_subtitle":"LSTM-with-attention hits 0.982 AUC; pre-war speech is learnable.","key_machinery":"The paper's central object is the War label, a binary variable equal to 1 for speeches delivered within one year before US entry into a major war and 0 otherwise, with start dates taken from the authors' historical source. The mechanism carrying the argument is supervised text classification: BERT, a pre-trained transformer language model, tokenizes and vectorizes cleaned transcripts, and each neural model is trained with a binary loss to separate the two classes. The LSTM-with-attention model adds a learned weighting over positions in the speech, and the paper also applies local and global explanation methods to attribute predictions to input features. Together these pieces convert a historical claim about pre-war rhetoric into a measurable classification signal, then open the model enough to say which parts of a speech drive the signal.","core_discovery":"The central discovery the paper argues for is that pre-war rhetoric is learnable from text alone. Using a corpus of 964 US presidential speeches, cleaning the transcripts, and balancing the heavily imbalanced labels with SMOTE, the authors train four architectures and find that the LSTM with an attention layer reaches an AUC-ROC of 0.982, the LSTM-only RNN reaches an F1-score of 0.929, and fine-tuned BERT also exceeds 0.9 AUC-ROC. In the paper's own reading, this accuracy is not merely a classification exercise: it indicates that neural networks can capture rhetorical patterns that precede US entry into major wars, and the interpretability analyses (attention weights, LIME, SHAP) begin to show what those patterns look like. The authors are careful to call this a hint of predictive power for future involvement rather than a proven forecasting method.","pith_inferences":["A natural next test the paper does not run is a placebo: shuffle the war dates and retrain; if AUC stays near 0.982, the models are reading the calendar, not the rhetoric.","Varying the window from one year to six months or two years would show whether the signal is a gradual ramp-up before war or an artifact of the cutoff.","A temporal train/test split, training on conflicts before 1950 and testing on later ones, would reveal whether the learned pattern generalizes across eras or reflects one president's style.","Applying the classifier to crises that never escalated would show whether it detects specifically war-bound rhetoric or a broader crisis register."],"forward_implications":["If the central claim is right, a model trained on presidential transcripts can flag the year before a major US war at near-perfect separation (AUC-ROC 0.982), meaning the pre-war register is present in the text itself.","The attention-weight distributions separate the two classes, so attention is doing substantive work rather than reweighting noise, which makes the LSTM-with-attention result more than an accuracy number.","Because LIME and SHAP point to different key features, the paper's interpretive layer implies that local and global explanations capture different aspects of the pre-war signal, and both are needed to describe it.","The authors' conclusion extends one step further: the same classifiers hint at predictive power for determining involvement in future wars, provided future rhetoric resembles the historical pattern."],"supporting_citations":[{"why":"Supplies the corpus of United States presidential speeches that every model trains and evaluates on.","marker":"Lilleberg, 2020"},{"why":"Provides the major-war start dates from which the paper constructs the War label's one-year window.","marker":"Torreon and Miller, 2024"},{"why":"Introduces SMOTE, the oversampling method used to balance the heavily imbalanced War=0 and War=1 classes.","marker":"Chawla et al., 2002"},{"why":"Provides BERT, the pre-trained transformer whose tokenization and vectorization turn each speech into model input.","marker":"Devlin et al., 2019"},{"why":"Supplies LIME, the local explanation method used to identify features behind a single prediction.","marker":"Ribeiro et al., 2016"},{"why":"Supplies SHAP, the global feature-attribution method used to rank features by influence on war predictions.","marker":"Lundberg and Lee, 2017"}],"fun_headline_variants":["Deep learning decodes war rhetoric in presidential speeches","AI learns to spot pre-war language in White House speeches","LSTM attention model uncovers pre-war rhetorical patterns","Neural nets recognize pre-war speech patterns at 0.98 AUC","Interpretable deep learning reveals pre-war rhetoric cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 'within one year of a war start date' cleanly identifies pre-war rhetoric; if the dates or the window leak in other timing signals, the classifier could be separating label eras rather than war language.","fun_headline_variants_meta":{"raw":{"variants":["Deep learning decodes war rhetoric in presidential speeches","AI learns to spot pre-war language in White House speeches","LSTM attention model uncovers pre-war rhetorical patterns","Neural nets recognize pre-war speech patterns at 0.98 AUC","Interpretable deep learning reveals pre-war rhetoric cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3211,"prompt_tokens":895,"completion_tokens":2316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2237}},"tokens_in":511,"tokens_out":2316,"duration_ms":17306,"temperature":1.0,"reasoning_tokens":2237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:28:06.479252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The most direct falsifier is a label-shuffle placebo test: randomly reassign which speeches are labeled War=1, retrain the LSTM-with-attention model under the same pipeline, and check whether AUC-ROC collapses from 0.982 toward 0.5. A weaker version would keep the real labels but use fake war dates drawn from the same marginal distribution; if the score stays high, the model is learning label timing rather than pre-war wording.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the corpus of United States presidential speeches that every model trains and evaluates on."},{"cited_title":"Miller, US Congressional Re- search Service","cited_arxiv_id":null,"evidence_quote":"Provides the major-war start dates from which the paper constructs the War label's one-year window."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BERT, the pre-trained transformer whose tokenization and vectorization turn each speech into model input."}],"review_version":1}