{"id":"23b3cfee-a818-44be-9032-de3743914800","arxiv_id":"2505.12535","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-country framework predicts individual parliamentary votes with up to 85% accuracy and bill outcomes with up to 84% accuracy, but the evaluation does not include trivial baselines.","lead":"This paper describes a machine-learning framework that predicts how individual lawmakers will vote and whether bills will pass, using parliamentary data from five countries. The reported accuracy looks high, but the paper lacks comparisons against simple baselines, so the actual predictive value is unclear.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No baseline comparison is reported, and US bill-level accuracy (82.781%) exactly equals always predicting the majority class, so the headline accuracy is not yet evidence of predictive skill.","rationale":"The reader's weakest_assumption concerns temporal leakage in the 'Opinion on Subject' feature. That is a serious issue, but the published table already shows a more direct failure: US bill-level accuracy is exactly the majority-class rate under the authors' own numbers. The paper claims time-series splitting in Section 3.3, but no control or baseline is provided, and Section 5 reports accuracy without comparing to trivial predictors. Section 6 acknowledges data-accuracy assumptions, yet never addresses baseline-relative performance. The internal dataset inconsistencies and the future-dated UK cutoff further reduce confidence in the evaluation pipeline. The load-bearing concern is therefore the absence of evidence that VPF improves on naive prediction, with leakage as a secondary aggravator. The proposed baseline recomputation is a single concrete check that would settle this concern: if the deltas over baselines are negligible or negative in any country, the central accuracy claim fails. This supports the reader's REJECT verdict, though the emphasis shifts from leakage alone to the baseline comparison issue.","tokens_in":18322,"tokens_out":3612,"duration_ms":40497,"concrete_test":"Recompute Table 4 and Table S5 on the identical train/test splits with trivial baselines: (i) always predict the majority class per country; (ii) predict each member's majority vote within their party (or coalition), estimated on training data only. Report accuracy/F1 deltas and 95% confidence intervals. If US bill-level accuracy equals the majority-class baseline (250/302), or if any country's vote-level accuracy does not exceed both baselines by a significant margin, the claim of predictive skill is not established. Separately, rerun the 'Opinion on Subject' feature using only protocol references dated before each vote; if accuracy drops materially, temporal leakage is confirmed.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that VPF predicts votes and bills with 78–85% accuracy—requires that the reported numbers beat trivial prediction rules. This is not demonstrated. Table S6 lets us check the US bill-level result directly: 250 of 302 test bills actually passed, so always predicting 'pass' yields 250/302 = 82.781% accuracy, exactly the value reported in Table 4. Thus, for the largest, most carefully curated dataset, the model provides zero measured signal over the majority-class rule at bill level. The same risk applies to vote-level accuracy: with strong party discipline, predicting each member's party-line majority (or simply the majority class) can plausibly reach 80%+, and no such baseline, confidence interval, or significance test appears anywhere. Section 3.3 describes only time-series splitting; it never compares against a majority-class or party-line predictor. Section 6 lists assumptions about data accuracy and majority voting but does not address baseline-relative performance. Additional red flags compound the concern: Table 1 reports 2,850,296 US voting records while Table 3 uses 1,048,600 (and the UK numbers differ: 273,948 vs 867,523); Section 4.4 sets the UK training cutoff at 'before August 2026,' after the 18 May 2025 submission; and no code, data, or error bars are provided. Any single discrepancy might be explainable, but together they mean the published accuracy figures are not yet evidence that VPF outperforms naive prediction. The load-bearing assumption—that the evaluation establishes genuine predictive skill—is therefore unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Voting Prediction Framework (VPF), a generic pipeline that collects parliamentary data from five countries (Canada, Israel, Tunisia, the UK, and the US), enriches it with features such as political affiliation, importance rank, protocol-based opinion signals, and bill embeddings, and trains machine-learning classifiers (with XGBoost performing best) to predict individual legislator votes and overall bill outcomes. The authors report vote-level accuracies between 78% and 85% and bill-level accuracies between 82% and 84%, and they use SHAP values to interpret feature importance and discuss anomalous false predictions. The central claim is that VPF provides accurate, cross-country voting prediction and can help prioritize legislation.","tokens_in":18564,"tokens_out":5122,"duration_ms":51628,"significance":"A working multi-country framework for forecasting parliamentary votes would be genuinely valuable for transparency, political analysis, and legislative decision-making, and the paper's attempt to unify heterogeneous parliamentary data into a common schema is a useful direction. The scale of the collected data and the inclusion of temporal splits are also positive features. However, the significance is conditional: the evaluation does not demonstrate that the reported accuracies exceed trivial baselines, and for the US bill-level result the reported accuracy exactly equals the majority-class rule. Potential temporal leakage in a key feature and unexplained dataset inconsistencies further weaken the empirical contribution. If the authors can add baselines, fix the leakage and data issues, and show robust gains over naive rules, the framework could be a meaningful contribution.","major_comments":[{"comment":"The US bill-level accuracy of 82.781% is exactly equal to always predicting the majority class, since 250 of the 302 test bills actually passed (250/302 = 0.82781). The manuscript provides no comparison against majority-class, party-line, or past-vote baselines anywhere in the evaluation, so the headline claim that VPF 'achieves up to 84% accuracy in predicting overall bill outcomes' is not evidence of predictive skill for the largest, most carefully curated dataset. The authors must add trivial baselines and statistical significance tests for all countries before the claimed predictive value can be assessed.","section":"Section 5, Tables 4 and S6"},{"comment":"The 'Opinion on Subject' feature counts references to a bill's subject in meeting protocols without specifying whether those protocols predate the vote. Because the evaluation uses chronological splits, any post-vote protocol content in the test set would leak future information into the model, making the reported test accuracy not a genuine forecast. The manuscript must either document that all protocol references are time-constrained to occur before the vote date or the reported numbers must be treated as corrupted by leakage.","section":"Section 3.2, 'Opinion on Subject' feature"},{"comment":"The dataset sizes are inconsistent across the paper: Table 1 reports 273,948 UK voting records and 2,850,296 US records, while Table 3 reports 867,523 UK records and 1,048,600 US records, with no explanation of the reconciliation. Additionally, Section 4.4 sets the UK training cutoff at 'before August 2026,' which is after the manuscript's 18 May 2025 submission date and is impossible; this also contradicts the 77%-23% training-test split reported in Table 3. These issues prevent reproducibility and undermine confidence in the reported evaluation pipeline.","section":"Tables 1 and 3; Section 4.4"}],"minor_comments":[{"comment":"The abstract and introduction use 'precision' when reporting accuracy figures (e.g., 'up to 85% precision in predicting individual votes'); precision and accuracy are distinct metrics and should be labeled correctly.","section":"Abstract and Introduction"},{"comment":"The caption reads 'class 0 = Yes; class 0 = No; class 2 = Abstention,' which should likely be 'class 0 = Yes; class 1 = No; class 2 = Abstention.'","section":"Figure 4 caption"},{"comment":"The text says 'more than 2.8 million protocols over 19 years from the US parliament,' but Table 1 (and the surrounding context) refers to voting records, not protocols; this appears to be a typo.","section":"Section 5 opening"},{"comment":"The UK data description states 'over 850,000 voting records' while Table 1 reports 273,948 records; the discrepancy should be resolved or explicitly explained.","section":"Section 4.4"},{"comment":"The sentence 'Random Forest perform well but outperform XGBoost' appears to say the opposite of the reported results; it should read 'but do not outperform XGBoost.'","section":"Section 6"},{"comment":"The paper repeatedly states the framework 'will be open source' but provides no code, data, or repository link; this claim is not verifiable from the manuscript and should be substantiated.","section":"Section 7 and Availability"}],"recommendation":"reject","confidential_remarks":"The manuscript has several signs of being an unreviewed draft: the impossible UK cutoff date, the unexplained dataset-size differences, and the exact coincidence of the US bill-level accuracy with the majority-class baseline. The latter is particularly worrying because it suggests the model may be providing no signal on the largest dataset, and without baselines we cannot tell whether the other countries' results are meaningfully better than naive rules. If the authors can address these issues with a rigorous re-evaluation, the paper might become salvageable, but as submitted the central empirical claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zahi et al. have put together a multi-country voting-prediction pipeline that is genuinely useful on the data-collection side: five parliaments, three languages, over 5 million voting records, a unified schema, and a stated commitment to open-source. That part is real and worth building on. The paper is weaker as a demonstration of predictive skill.\n\nThe headline numbers (79–85% vote-level, 82–84% bill-level) are presented without any comparison to trivial rules. This is not a minor omission. For the US, Table S6 shows 250 of 302 test bills actually passed; always predicting 'pass' gives 82.78%, which is exactly Table 4's US bill accuracy. The model adds zero measured signal for US bill outcomes. The other four countries do beat the majority class based on the numbers in Table S6 (e.g., Canada majority 'reject' is 79.6%, model 82.2%), so the framework may have real skill—but the paper never makes that case because no baseline is reported. Vote-level results have the same problem: with strong party discipline, a party-line or majority-class model could plausibly reach 80%, and we don't get that comparison.\n\nThere are also fixable but worrying inconsistencies: US records are 2.85M in Table 1 and 1.05M in Table 3; UK records are 274K in Table 1 and 868K in Table 3. Section 4.4 sets the UK training cutoff at 'before August 2026' on a paper submitted in May 2025. The 'Opinion on Subject' feature counts references to a bill's subject in meeting protocols without stating whether those protocols are all prior to the vote; if not, that is future leakage. The abstract says 'precision' but the tables report accuracy. No code, data, or error bars are provided yet.\n\nWhat the paper does well: the unified schema and collection effort are a legitimate extension, the SHAP-based discussion of country differences is thoughtful, and the anomaly case studies are a useful qualitative check. The authors also list their assumptions and limitations honestly.\n\nMy recommendation: send it to review, but the authors need to add majority-class and party-line baselines, fix the dataset numbers, clarify the cutoff date and the protocol temporal ordering, and report confidence intervals. With those changes, the framework could be a solid contribution for political science and legislative databases. As it stands, the central claim is unproven, but it is a fixable problem.","headline":"Useful multi-country voting dataset, but the predictive accuracy claims are unproven because no trivial baseline is reported; for US bills the model exactly matches always predicting 'pass'.","tokens_in":19118,"tokens_out":2697,"would_cite":false,"duration_ms":26698,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A data-driven framework trained on more than five million parliamentary roll-call records predicts individual votes with up to 85 percent accuracy and bill outcomes with up to 84 percent accuracy across five countries.","keywords":["parliamentary voting prediction","roll-call votes","open government data","machine learning","gradient boosting","bill outcome prediction","cross-country comparison","SHAP values"],"falsifier":"Check the Israeli dataset's protocol references used for the Opinion on Subject feature and recompute the 85.2 percent accuracy with only references dated strictly before each vote; if accuracy drops noticeably, the time-series split is leaking post-vote information. A simpler first look is to verify whether any protocol date in the feature table is later than the corresponding vote date.","tokens_in":18060,"feed_emoji":"🗳️","tokens_out":7377,"duration_ms":70531,"temperature":0.7,"pith_summary":"The Voting Prediction Framework (VPF) is a data-driven pipeline that predicts how individual parliament members will vote on bills and whether those bills will pass. It is built from three components: collecting parliamentary data from official sources, parsing and enriching it with features such as coalition affiliation, seniority rank, and bill content, and applying machine learning models to produce predictions. The paper evaluates VPF on more than five million voting records from Canada, Israel, Tunisia, the United Kingdom, and the United States, reporting vote-level accuracies of roughly 79 to 85 percent and bill-level accuracies of 82 to 84 percent. If these numbers hold, VPF would give legislators, staff, and the public a working way to anticipate which bills pass and to spot members whose votes break from party expectations.","feed_headline":"Parliament votes predicted at up to 85% across five parliaments","feed_subtitle":"Trained on 5 million records from five parliaments, it forecasts bill outcomes with up to 84% accuracy.","key_machinery":"The machinery is the VPF pipeline: web crawlers and APIs feed raw parliamentary data into a unified schema; parsers enrich it with features; and a multi-class classifier turns those features into per-member vote predictions, with a gradient-boosted tree model performing best in every country tested. Three engineered features carry much of the predictive load: political affiliation, indicating whether the member's party is part of the governing coalition; importance rank, a numeric seniority score derived from the member's parliamentary position; and Opinion on Subject, the count of references to a bill's subject in that member's meeting protocols. Bill text enters through pre-trained embeddings, and SHAP values are used to attribute each prediction to these features.","core_discovery":"The paper's central claim is that a single, largely generic framework can forecast individual parliamentarians' votes and full-bill outcomes from open parliamentary data alone. Using time-series splits, the best model reaches 79.8 percent accuracy in Canada, 85.2 percent in Israel, 78.3 percent in Tunisia, 80.3 percent in the United Kingdom, and 80.8 percent in the United States; aggregating vote predictions to bills gives 82 to 84 percent accuracy across those countries. VPF rests on a small set of features, including coalition affiliation, parliamentary position rank, counts of how often a legislator has referenced a bill's subject in meeting protocols, and embeddings of bill text, suggesting that most of the signal comes from party alignment and bill content rather than from hand-built country-specific rules.","pith_inferences":["Beyond the paper's own claims, a strict temporal audit would be the cleanest test: recompute accuracy using only protocol references dated before the vote, because the current feature definition does not specify that the references must precede the vote.","A second extension the authors do not develop is calibration: reporting predicted probabilities for each bill would let users set a threshold for 'likely to pass' and test whether the 82 to 84 percent bill accuracy reflects well-separated probabilities or just a good ranking.","A third extension is transfer learning across parliaments: training the feature pipeline on one country and testing on another would show whether coalition-affiliation and bill-embedding features capture general legislative behavior or only country-specific party discipline."],"forward_implications":["Legislative staff could use VPF to screen bills before floor votes, prioritizing those predicted to pass and reworking or dropping those predicted to fail.","Watchdog groups could use VPF's false-negative cases to flag members voting against their party line, such as the Canadian carbon-tax example discussed in the paper.","The unified schema makes cross-country comparison feasible, so a researcher could compare coalition discipline or issue-driven voting across the five parliaments with the same pipeline.","If the open-source release works as stated, anyone can adapt the parsers and features to a new parliament instead of building collection tooling from scratch.","Feature-importance results suggest voting drivers differ by country, with party and coalition shape in Canada and Israel and individual voting history mattering more in the United Kingdom and the United States."],"supporting_citations":[{"why":"Supplies the text-based roll-call prediction approach that motivates the bill-embedding feature.","marker":"[1]"},{"why":"Provides the generic parliamentary data-collection and parsing framework that VPF extends to voting prediction.","marker":"[23]"},{"why":"Supplies the Canadian roll-call analysis and the party-loyalty interpretation used to explain Canada's feature importances.","marker":"[36]"},{"why":"Offers a prior roll-call prediction system that VPF generalizes from committee gatekeeping to full votes and bill outcomes.","marker":"[37]"},{"why":"Provides a recent benchmark dataset for bill-status prediction that frames VPF's task and comparison.","marker":"[38]"},{"why":"Supplies the gradient-boosted tree algorithm that achieves the highest reported accuracies in every country.","marker":"[53]"},{"why":"Provides the SHAP method used to attribute feature contributions to individual predictions.","marker":"[58]"},{"why":"Supplies the Hebrew language model used for bill embeddings in the Israeli dataset.","marker":"[59]"}],"fun_headline_variants":["Predicting how MPs vote with 85% precision across 5 parliaments","Open-source framework forecasts parliamentary votes at 85%","VPF: 5M votes, 5 countries, 85% vote prediction","From 5M records to 85% vote forecasts: VPF framework","Parliament vote prediction hits 85% with generic features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the split between training and test periods is genuinely out-of-sample; if the feature that counts how often a bill's subject appears in meeting protocols includes meetings held after the vote, the model could be peeking at the future and the reported accuracy would not be a true forecast.","fun_headline_variants_meta":{"raw":{"variants":["Predicting how MPs vote with 85% precision across 5 parliaments","Open-source framework forecasts parliamentary votes at 85%","VPF: 5M votes, 5 countries, 85% vote prediction","From 5M records to 85% vote forecasts: VPF framework","Parliament vote prediction hits 85% with generic features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3407,"prompt_tokens":1011,"completion_tokens":2396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2311}},"tokens_in":627,"tokens_out":2396,"duration_ms":15628,"temperature":1.0,"reasoning_tokens":2311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:31:11.237176+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the Israeli dataset's protocol references used for the Opinion on Subject feature and recompute the 85.2 percent accuracy with only references dated strictly before each vote; if accuracy drops noticeably, the time-series split is leaking post-vote information. A simpler first look is to verify whether any protocol date in the feature table is later than the corresponding vote date.","supporting_citations":[{"cited_title":"Predicting legislative roll calls from text","cited_arxiv_id":null,"evidence_quote":"Supplies the text-based roll-call prediction approach that motivates the bill-embedding feature."},{"cited_title":"Open Framework for Analyzing Public Parliaments Data","cited_arxiv_id":"2210.00433","evidence_quote":"Provides the generic parliamentary data-collection and parsing framework that VPF extends to voting prediction."},{"cited_title":"Legislative voting in the canadian parliament.Canadian Journal of Political Science/Revue canadienne de science politique, 44(2):367–388, 2011","cited_arxiv_id":null,"evidence_quote":"Supplies the Canadian roll-call analysis and the party-loyalty interpretation used to explain Canada's feature importances."},{"cited_title":"How Predictable is Your State? Leveraging Lexical and Contextual Information for Predicting Legislative Floor Action at the State Level","cited_arxiv_id":"1806.05284","evidence_quote":"Offers a prior roll-call prediction system that VPF generalizes from committee gatekeeping to full votes and bill outcomes."},{"cited_title":"Explaining predic- tion models and individual predictions with feature con- tributions.Knowledge and information systems, 41:647– 665, 2014","cited_arxiv_id":null,"evidence_quote":"Provides the SHAP method used to attribute feature contributions to individual predictions."}],"review_version":1}