{"id":"a5fa4b4f-46f4-428b-835e-f0111c8f492a","arxiv_id":"2507.13913","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors compile large multi-dataset benchmarks for political leaning and politicalness classification, show that single-dataset models fail out-of-distribution, and release new models with improved cross-domain F1 scores.","lead":"This paper gathers 12 existing datasets for political leaning and builds a new politicalness dataset from 18 sources, then benchmarks transformer classifiers on in-distribution and out-of-distribution text. The authors train their own POLITICS- and DeBERTa-based classifiers and report improved generalization across diverse text types, plus a practical takeaway: most existing political text classifiers degrade sharply on unfamiliar domains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leave-one-out hyperparameters were tuned on one target fold (§3.5.2), so the reported OOD gains and Table 9 SOTA are not a clean generalization estimate.","rationale":"The reader's weakest assumption about politicalness labels is real but secondary for the central SOTA/OOD claim: the politicalness labels do not directly enter the political leaning models' training, and Table 9's comparisons would be affected only if label noise propagated through the CommonCrawl filter. The more load-bearing issue is that the leave-one-out protocol is used to validate the exact training setup that later produces the 'new state-of-the-art' model, and that protocol leaks one target domain into hyperparameter selection. Tuning dropout, learning rate, warmup, batch size, and weight decay on the Article bias prediction fold means the 'left-out' number for that row is not independent, and using those hyperparameters on all folds makes the reported average optimistic in an unquantified way. The paper is transparent about many limitations, and this is not an accusation of dishonesty; it is a standard nested-validation concern. The proposed check is feasible with one additional Optuna run plus one held-out evaluation and would directly test whether the headline result survives a clean protocol. If it survives, the current CONDITIONAL verdict can be upgraded; if not, the generalization and state-of-the-art claims need to be scaled back.","tokens_in":23465,"tokens_out":7977,"duration_ms":95819,"concrete_test":"Hold out Article bias prediction and run the hyperparameter search a second time using validation splits drawn only from the nine training datasets, with no access to the left-out target. If the model's F1 on the Article bias prediction test set and the average leave-one-out F1 across the other folds are materially lower (e.g., more than 2 F1 points) than the values in Table 7, the reported OOD improvement is partly an artifact of tuning on a target domain. In addition, evaluate the final Table 9 model on a genuinely unseen corpus (e.g., 2025 political news or an annotated political text set not among the 10), and compare it with Political bias prediction AllSides DeBERTa; if the final model does not beat the existing model there, the 'state-of-the-art' generalization claim should be restricted to the collected datasets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5.2 states that, for both the Optuna search (POLITICS) and the manual tuning (DeBERTa V3 large), 'we have chosen the Article bias prediction to be the left-out dataset and aimed for the best performance when evaluating the model on it.' Those hyperparameters are then frozen for every other leave-one-out fold and for the final all-data models in Table 9. This makes the reported 'left-out' scores a partially in-sample estimate of generalization: one of the ten target domains (Article bias prediction) was used as the validation set for model-selection decisions, so the mean left-out F1 in Table 7 is not a clean estimate for an arbitrary unseen domain. Moreover, Table 9 is not a cross-domain result: the final models were trained on all ten datasets (except the two Webis sets) and evaluated on held-out 15% slices of the same datasets, so the 'state-of-the-art across all evaluated datasets' is an in-distribution benchmark. The paper candidly admits in §6 that the new models have not been extensively tested on truly novel inputs, but the abstract and conclusion wording ('enhanced generalization capabilities', 'new state-of-the-art') is stronger than what the experiments establish. The central claim therefore rests on a validation protocol that has not cleanly separated target-domain selection from model selection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compiles and unifies twelve political-leaning datasets and eighteen politicalness datasets (sixteen of which are relabeled by topic inspection or implicit assumptions), evaluates eight existing leaning models and three existing politicalness models, and conducts leave-one-in and leave-one-out benchmarks using BERT, RoBERTa, DeBERTa V3, and POLITICS as base models. The main contributions claimed are: (1) a demonstration that single-dataset models degrade substantially out-of-distribution, (2) a new politicalness dataset covering diverse text types, and (3) two newly trained models (POLITICS-based and DeBERTa V3 large) that reportedly establish a new state of the art for political leaning classification after training on the combined data with leave-one-out-optimized hyperparameters.","tokens_in":23716,"tokens_out":3684,"duration_ms":41861,"significance":"If the claims held in full, the paper would be a valuable empirical contribution: it provides a broad benchmark, a new politicalness dataset, released code and models, and a systematic measurement of cross-dataset transfer that aligns with prior work. The finding that all tested models, including the new ones, perform consistently worse on unseen datasets is an honest and useful result. The strengths are the scope of the benchmark, the dataset-intersection analysis, and the public release of resources. However, the headline claims of 'enhanced generalization capabilities' and 'new state-of-the-art' are only partially supported, because the validation protocol and the evaluation setup for the final models do not cleanly separate in-sample target selection from model selection, and because Table 9 is an in-distribution evaluation rather than a cross-domain test. The significance therefore depends on whether these claims are appropriately qualified.","major_comments":[{"comment":"The hyperparameter optimization for both POLITICS (Optuna search) and DeBERTa V3 large (manual tuning) explicitly used Article bias prediction as the left-out dataset for validation, and the resulting hyperparameters were then frozen for every other leave-one-out fold and for the final all-data models. Consequently, the Article bias prediction row in Table 7 is not an out-of-distribution estimate, and the reported average left-out F1 mixes one in-sample target with nine out-of-sample targets. This compromises the leave-one-out benchmark as a clean estimate of generalization to an arbitrary unseen domain and weakens the abstract's claim of 'enhanced generalization capabilities.' Please either conduct a nested validation in which hyperparameters are selected without using the target fold, or explicitly report the results with this caveat and soften the generalization claim accordingly.","section":"§3.5.2, Table 7"},{"comment":"Table 9 evaluates the final models on held-out 15% slices of the same datasets that were used for training (with only the two Webis datasets excluded), so it is an in-distribution benchmark, not a cross-domain test. The sentence 'They establish the new state-of-the-art performance across all the evaluated datasets' is defensible only as an in-distribution claim, and the conclusion's 'our newly trained models based on POLITICS and DeBERTa large establish a new state-of-the-art performance for political leaning classification' overstates the evidence, particularly in light of the paper's own §6 admission that extensive testing on truly unseen data was not carried out. The claims in the abstract, conclusion, and §4.5 should be reworded to specify that the state-of-the-art result is on the collected benchmark datasets and that cross-domain generalization remains an open question.","section":"§4.5, Table 9, and §8 Conclusion"},{"comment":"For sixteen of the eighteen politicalness datasets, the binary political/non-political labels were created by the authors rather than obtained from explicit annotations. In particular, datasets marked 'implicit' (Yelp review full, IMDB, Recipes, Dialogsum) are assigned entirely to the non-political class without per-text inspection, and 'topics' datasets are labeled by inspecting topic category names rather than the text content. If a nontrivial fraction of these texts concern political subjects, the politicalness benchmark in Table 8 and the politicalness filter models are trained and evaluated on systematically noisy labels, and the near-ceiling F1 scores may be inflated. The paper should provide evidence of label quality, such as a manual validation sample or an inter-annotator agreement study, and should discuss the potential impact of this label noise on the reported politicalness results.","section":"§3.1.2, Table 2"}],"minor_comments":[{"comment":"The 'center multiplier' used to balance the center class is described only verbally ('we have added a multiplier... We adjust it to even out the class distribution'). The exact multiplier value or the balancing algorithm should be reported to make the training setup reproducible.","section":"§3.5.2"},{"comment":"The table mixes models that support the center class with models that do not, and although the text notes that models with different class counts should be compared separately, the 'overall average' column is still likely to be misinterpreted. A footnote restating the random-baseline accuracy for 2-class vs. 3-class problems would improve clarity.","section":"Table 4"},{"comment":"The choice of Article bias prediction as the left-out validation dataset is justified after the fact, but the sentence explaining the reasons currently appears after the description of both hyperparameter searches. Moving this justification to the first mention would make the protocol easier to follow.","section":"§3.5.2"},{"comment":"The text refers to the 'Levenstein distance'; the standard spelling is 'Levenshtein distance.'","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical benchmark study, and the authors are transparent about limitations in §6. The main issues are not fatal errors in the experiments but a mismatch between the strength of the claims in the abstract and conclusion and what the evaluation protocol actually supports. If the authors add a nested validation or clearly re-scope the claims, the paper could be appropriate for publication. I would also encourage the editor to consider whether the journal's scope fits a paper whose primary contribution is a benchmark and dataset release rather than a new method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on political text classification. The paper's real contribution is the resource: 12 leaning datasets unified, a new 18-source politicalness dataset, and a reproducible benchmark with code, models, and full result tables. The empirical core—single-dataset models lose at least 7 F1 points out-of-distribution—is solid, well-tabulated, and consistent with earlier work like Cohen & Ruths. That negative finding is the durable part.\n\nThe soft spots are in the positive claims. Section 3.5.2 says hyperparameters for both optimized models were chosen by tuning on Article bias prediction as the left-out validation set. That makes one of the ten leave-one-out folds—and the average derived from it—partially in-sample. Table 9 is not a cross-domain test at all: those models were trained on all datasets and evaluated on held-out slices of the same datasets, so calling that 'state-of-the-art across all evaluated datasets' is true only in-distribution. The abstract's 'enhanced generalization capabilities' oversells what a single target-domain validation choice can establish. The authors admit in Section 6 that they haven't tested on truly novel inputs; that admission should be in the abstract.\n\nThe politicalness label construction is the second soft spot. For 16 of 18 datasets, labels are inferred from genre or topic inspection—Yelp, IMDB, Recipes, Dialogsum are marked entirely non-political without checking individual texts for political content. That's defensible for a first pass, but it means the politicalness benchmark and any filter trained on it inherit that assumption. The authors do inspect and discard ambiguous topics, so it's not careless, just noisy.\n\nNo error bars anywhere, which matters for a benchmark paper, but the tables are large and the main pattern is clear.\n\nWho's this for? Anyone building or evaluating political leaning/politicalness classifiers. It's a useful resource paper, not a methodological breakthrough. A serious referee should engage with it; the SOTA claim needs rewriting and the validation caveat needs to be front-facing, but the data and negative result merit publication.","headline":"A genuinely useful benchmark resource with an overstated SOTA claim; the validation protocol leaks one target domain into model selection.","tokens_in":24298,"tokens_out":2359,"would_cite":true,"duration_ms":26112,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that political leaning classifiers trained on a single text domain consistently fail on out-of-distribution data, and that models fine-tuned on a combined 12-dataset corpus with leave-one-out-tuned hyperparameters set a…","keywords":["political leaning classification","politicalness classification","out-of-distribution generalization","leave-one-out benchmarking","transformer fine-tuning","multi-dataset training","media bias classification","text classification"],"falsifier":"Run the two final models on a freshly collected, independently labeled corpus of political and non-political texts created after the paper's datasets, for example news and social posts from 2026; if their F1 falls to the level of single-dataset models, roughly 40 to 50 on unseen data, rather than staying near the reported 87 for political leaning, the leave-one-out benchmark overstates real-world generalization. In parallel, hand-label a random sample of texts from datasets the paper marks entirely non-political, such as IMDB, Yelp, or Recipes; a material share of political reviews would falsify the politicalness labels.","tokens_in":23261,"feed_emoji":"🗳️","tokens_out":10278,"duration_ms":95954,"temperature":0.7,"pith_summary":"Political leaning classifiers are usually trained and tested on a single dataset, and this paper argues that the practice is the root of their failure on new text: across every model tested, F1 drops consistently and markedly on datasets that were not in the training mix. To confront that, the authors assembled 12 public political-leaning datasets, created a new binary politicalness (political vs. non-political) dataset by extending 18 existing corpora, and benchmarked both existing and newly trained transformer models under leave-one-in and leave-one-out protocols. The leave-one-out runs confirm that single-dataset models collapse on unfamiliar text, while models fine-tuned on the combined corpus with hyperparameters tuned against a held-out dataset perform better on every evaluated dataset and set what the authors call a new state of the art. They also find politicalness classification to be substantially easier than leaning classification, with the zero-shot natural-language-inference classifier Political DEBATE reaching about 90% F1 on a balanced aggregate. The practical point is a recipe: diverse combined training data plus out-of-distribution-aware tuning produces text classifiers that are more likely to work on the messy, future text a real application would encounter.","feed_headline":"12-dataset training beats single-domain political classifiers","feed_subtitle":"Leave-one-out tests show one-domain models collapse on new text; the new models set state-of-the-art F1.","key_machinery":"The central mechanism is the leave-one-out benchmark turned into a training signal: fine-tune the model on samples from every dataset except one, validate and select checkpoints purely on examples from the left-out dataset, and repeat with each dataset left out. The configuration that scores best on held-out domains, found by leaving out Article bias prediction and then confirmed across all folds, is applied to the full corpus to produce the final models. The second essential piece is the compiled corpus itself: twelve political-leaning datasets unified to left/center/right, with a multiplier to rebalance the underrepresented center examples, and 28 source datasets merged into a single politicalness benchmark, with the Webis datasets excluded from training folds because their heavy intersection with Article bias prediction would cause label spillover.","core_discovery":"The paper's central claim is that existing political leaning classifiers are siloed solutions that perform poorly on out-of-distribution texts, and that this gap can be substantially closed by the combination of a diverse training corpus and a training protocol that optimizes for out-of-distribution performance. The authors demonstrate the failure mode first: every existing model and every benchmark model fine-tuned on one dataset scores consistently lower on datasets it never trained on, with the drop at least 7 F1 points and often much larger. They then show the remedy: fine-tuning POLITICS and DeBERTa V3 large on the full 12-dataset corpus, with hyperparameters selected by the leave-one-out benchmark, yields models that score the highest F1 on every evaluated dataset (Table 9), averaging 87.2 for DeBERTa V3 large. The same protocol applied to the 28-source politicalness corpus shows that the binary political/non-political task is much easier, with base transformers already above 90% F1 on a balanced aggregate, and with Political DEBATE being the strongest existing filter. On the paper's terms, this establishes a new state of the art for political leaning classification and the first large-scale benchmark and models for politicalness.","pith_inferences":["Taken as a recipe, the leave-one-out tuning protocol is a form of domain-generalization training, and the same procedure could plausibly improve other text classifiers that suffer from distribution shift, such as sentiment or stance detection, whenever several heterogeneous datasets are available.","The politicalness benchmark's measured difficulty depends on the genre-wide labeling assumption; auditing samples from datasets like IMDB, Yelp, and Recipes would show how much headroom remains once genuinely political reviews are accounted for.","The consistently lower out-of-distribution scores on article-level three-class datasets such as Qbias suggest that genuinely neutral or center text, not just domain shift, remains the hard core of the task.","The authors' own caveat that generalization can only truly be shown on future text implies a direct extension: re-running the released checkpoints when a new political dataset appears, and comparing their F1 against the reported 87.2 baseline."],"forward_implications":["If the claim is right, any future political leaning classifier should be evaluated on at least one held-out domain, since single-dataset F1 substantially overstates real-world performance.","A single model trained on the combined corpus can be deployed across news, social media, and other text types without per-domain fine-tuning, which is the practical use case the paper points to for aggregator and media-audit tools.","Politicalness filtering is accurate enough at around 90% F1 on balanced data to serve as a practical sieve in front of a leaning classifier, so non-political inputs no longer have undefined behavior.","The continued-pretraining advantage of POLITICS over newer general-purpose architectures suggests that domain-specific pretraining on political text is a more direct route to transferable political classifiers than scaling the base model alone."],"supporting_citations":[{"why":"Supplies the Article bias prediction dataset and the evidence that article-level labels differ from outlet-level labels only 3.11% of the time, justifying reliance on its labels.","marker":"(Baly et al., 2020)"},{"why":"The POLITICS continued-pretraining model that consistently outperforms newer architectures in the benchmarks and serves as the base of the authors' final political leaning model.","marker":"(Liu et al., 2022)"},{"why":"The Political DEBATE NLI classifier that the authors find is the best existing politicalness filter and use to clean the CommonCrawl dataset.","marker":"(Burnham et al., 2024)"},{"why":"Prior evidence that political orientation classifiers drop from above 90% to about 65% on out-of-distribution Twitter users, supporting the paper's transfer hypothesis.","marker":"(Cohen and Ruths, 2021)"},{"why":"Prior finding of poor cross-domain generalizability across congressional records, media, and wikis, which the paper cites as aligned with its leave-one-out results.","marker":"(Yan et al., 2017)"},{"why":"Webis bias flipper 18 is part of the leaning corpus; its large intersection with Article bias prediction motivates the exclusion rules in the leave-one-out benchmark.","marker":"(Chen et al., 2018)"},{"why":"Webis news bias 20 likewise intersects heavily with Article bias prediction, so it is excluded from training folds to avoid label spillover.","marker":"(Chen et al., 2020)"}],"fun_headline_variants":["One-domain classifiers collapse; multi-dataset training beats them","Political AI fails on new text; 12-dataset fix gets state-of-the-art","Training on 12 datasets beats siloed political classifiers by 7+ F1","Out-of-distribution political text? 12-dataset training is the fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The state-of-the-art claim rests on the assumption that leave-one-out scores on existing datasets faithfully predict performance on genuinely new text, and the politicalness benchmark further assumes that entire genres such as reviews, recipes, and dialogues are universally non-political.","fun_headline_variants_meta":{"raw":{"variants":["One-domain classifiers collapse; multi-dataset training beats them","Political AI fails on new text; 12-dataset fix gets state-of-the-art","Training on 12 datasets beats siloed political classifiers by 7+ F1","Out-of-distribution political text? 12-dataset training is the fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001503,"raw_usage":{"total_tokens":5994,"prompt_tokens":876,"completion_tokens":5118,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":5036}},"tokens_in":492,"tokens_out":5118,"duration_ms":41997,"temperature":1.0,"reasoning_tokens":5036,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:12:50.112657+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the two final models on a freshly collected, independently labeled corpus of political and non-political texts created after the paper's datasets, for example news and social posts from 2026; if their F1 falls to the level of single-dataset models, roughly 40 to 50 on unseen data, rather than staying near the reported 87 for political leaning, the leave-one-out benchmark overstates real-world generalization. In parallel, hand-label a random sample of texts from datasets the paper marks entirely non-political, such as IMDB, Yelp, or Recipes; a material share of political reviews would falsify the politicalness labels.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior evidence that political orientation classifiers drop from above 90% to about 65% on out-of-distribution Twitter users, supporting the paper's transfer hypothesis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior finding of poor cross-domain generalizability across congressional records, media, and wikis, which the paper cites as aligned with its leave-one-out results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Webis bias flipper 18 is part of the leaning corpus; its large intersection with Article bias prediction motivates the exclusion rules in the leave-one-out benchmark."}],"review_version":1}