{"id":"78ee0f4f-3e1c-46fa-b06d-04921d170301","arxiv_id":"1908.08859","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A network analysis of UK parliamentary votes identifies dissenters via eigenvector centrality, but its reported 94% prediction accuracy is inflated by a class-imbalance baseline.","lead":"This paper uses social network analysis on UK parliamentary voting records to identify rebellious MPs and claims to predict Brexit vote rebellions with 94% accuracy. The accuracy claim is weakened by a trivial baseline and a single-vote validation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 94% accuracy is only 0.94 percentage points above the trivial always-non-rebel baseline of 93.0%, so the >90% predictive claim is not demonstration of forecasting skill.","rationale":"The reader identified the same load-bearing weakness: the accuracy metric is inflated by class imbalance. My independent calculation from the supplementary material confirms this. The central claim of the paper is explicitly predictive: 'using a rebellion metric we predicted how MPs would vote in a forthcoming Brexit deal with over 90% accuracy.' The validation evidence must show that the model predicts rebellion or vote choice better than a plausible null model. It does not. An always-non-rebel classifier yields 93.0% accuracy on the same confusion structure, and the reported model improves on it by only 6 of 639 MPs. No confidence interval, McNemar test, or other significance assessment is reported. Moreover, recall for the actual rebel class is 44%, so the model is not even a sensitive rebel detector. The descriptive network analysis may be a reasonable exploratory contribution, but the headline predictive claim is not established. The central claim therefore fails, and the appropriate verdict is REJECT.","tokens_in":9267,"tokens_out":4124,"duration_ms":49509,"concrete_test":"Recompute the validation on the same October 22, 2019 division using the trivial baseline 'predict no rebellion for all 639 MPs' and compare it to the reported model with McNemar's exact test on the paired binary outcomes. If the baseline accuracy is 594/639 ≈ 93.0%, the model is 600/639 ≈ 93.9%, and the McNemar test gives p > 0.05 (or any non-significant result), the >90% predictive claim is not supported over the party-line baseline.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that the rebellion metric predicted how MPs would vote on the October 22, 2019 timetable division with over 90% accuracy. The only quantitative evidence is the supplementary confusion matrix (Table S1): 600/639 = 93.9% accuracy. But 580 of the 639 MPs are true negatives—MPs predicted not to rebel who did not rebel. A trivial classifier that labels every MP as a non-rebel would achieve 594/639 = 93.0% accuracy, because only 45 of 639 MPs (7.0%) actually rebelled. The model's entire improvement over this baseline is 6 correct classifications, or 0.94 percentage points, with no significance test reported. The model also identifies only 20 of 45 actual rebels (44% recall) and generates 14 false positives. Since 'rebel' is defined as voting against the party/ideological expectation, and the test division has a low rebellion rate, an accuracy figure in the high 90s is almost inevitable. The elbow-method threshold at 0.21 is selected from the full eigenvector centrality distribution, and no out-of-sample threshold validation or pre-registered decision rule is provided. The headline 'predicted how MPs would vote' therefore does not demonstrate predictive power beyond a party-line baseline; it is consistent with simply assuming every MP follows the whip.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper uses Hansard division records from the 57th Parliament (June 2017 to April 2019) to construct cosine-similarity networks among MPs, adjusting the pairwise voting similarities by party and ideological affiliation, and splitting the divisions into Brexit and non-Brexit sets. It visualizes cross-party alliances and within-party conflicts, computes an eigenvector-centrality-based rebellion score for every MP, and reports a top-10 rebel list. The main claims are that Brexit divisions show significantly more dissent than non-Brexit divisions and that the rebellion metric predicted MPs' voting on the October 22, 2019 timetable division with 'over 90% accuracy,' which the supplementary material quantifies as 94%.","tokens_in":9558,"tokens_out":6437,"duration_ms":64271,"significance":"If the predictive claim were supported, the paper would provide an interesting individual-level complement to aggregate party-cohesion metrics, and the use of public parliamentary records with a replication data DOI is a strength. The descriptive network analysis offers a clear way to visualize Brexit-related fragmentation. However, the central result rests on an accuracy calculation that is barely above a trivial always-non-rebel baseline, on a rebellion threshold selected from the full sample, and on a prediction target defined using the same party/ideology labels that enter the construction of the rebellion metric. The authors explicitly acknowledge that the analysis is not causal and that aggregate cohesion measures have their own merits, but they do not address the baseline problem or the limited out-of-sample evidence. As a result, the headline claim of forecasting skill is not supported by the reported analysis.","major_comments":[{"comment":"The reported 94% accuracy does not demonstrate predictive skill because it is measured against a nearly identical trivial baseline. Supplementary Table S1 gives 580 true negatives, 20 true positives, 14 false positives, and 25 false negatives, so the model's accuracy is (580+20)/639 = 93.9%. Since 594 of 639 MPs (93.0%) did not rebel, a classifier that simply labels every MP as a non-rebel also achieves 93.0% accuracy. The model's entire improvement is 6 correct classifications, or 0.94 percentage points, with a recall of only 20/45 = 44.4% for actual rebels and no confidence interval or significance test. The statement in the Validation section that the accuracy score is 94% and that this validates the centrality metric is therefore not supported.","section":"Validation and Table S1"},{"comment":"The rebellion metric is defined as deviation from party- and ideology-adjusted voting similarity, and the validation labels an MP as a rebel if he or she votes against the party/ideology expectation (Table 3). This means the same party and ideology labels are used both to construct the predictor and to define the outcome. The October 2019 division is temporally out-of-sample, which gives some independence, but the outcome is not independent of the labels used to build the model. A stronger test would predict the direction of the vote (Aye/No) directly, or would use a rebellion definition that is not derived from the same party/ideology similarity matrices that generate the eigenvector centralities.","section":"Methods and Validation"},{"comment":"The elbow threshold of 0.21 is selected from the full distribution of all 639 eigenvector centralities, and the entire predictive evaluation rests on a single future division in which only 45 of 639 MPs (7.0%) rebelled. No cross-validation, alternative thresholds, or sensitivity analysis is reported. Given the class imbalance and the data-dependent threshold, the claimed accuracy is not a robust measure of forecasting skill; an out-of-sample threshold selection rule or a pre-registered cut-off would be needed to support the 'over 90% accuracy' claim.","section":"Supplementary Material: Accuracy score calculation"},{"comment":"The abstract's claim that the methodology 'was able to detect a significant difference in eurosceptic behaviour' is an overstatement of what is shown. The Kolmogorov-Smirnov test in this section only establishes that the distributions of pairwise similarity scores differ between Brexit and non-Brexit divisions; it does not establish that the difference is specifically eurosceptic in nature, nor does it quantify the effect size or identify which parties or MPs drive the difference. The histogram suggests more within-party conflict on Brexit divisions, but connecting this to euroscepticism requires additional party-specific evidence rather than a single KS test.","section":"Results: Party and ideology-adjusted voting similarity matrices"}],"minor_comments":[{"comment":"The data period is stated as '21st June, 2017 until 10th April' without a year; this should be corrected to a complete date range.","section":"Data and Methods"},{"comment":"The phrase 'regardless of arbitrary labeling (false positives)' is confusing: an MP detected as a rebel by visual inspection but not by the centrality score is not a statistical false positive. The terminology should be clarified or rephrased.","section":"Results and Figure 4"},{"comment":"The table and its footnotes do not make clear for each MP whether the recorded vote is on the Brexit bill itself or on the expedited timetable division, even though the validation text says the comparison uses the second debate. A column or footnote specifying the division used for each comparison would remove ambiguity.","section":"Table 3"},{"comment":"The confusion matrix labels should explicitly define rows and columns (e.g., 'Predicted rebellion status' versus 'Actual rebellion status') and state that the accuracy formula uses the table orientation shown; the current presentation makes it easy to misread the orientation.","section":"Supplementary Material: Table S1"},{"comment":"The ideology labels for Independent MPs are inferred from prior party affiliation, and for Lady Hermon from a 2010 BBC news report. The sensitivity of the top-10 list and of the accuracy score to these choices is not discussed; a robustness check with alternative ideology assignments would strengthen the analysis.","section":"Data and Methods: Ideology labels"},{"comment":"The data availability statement has a missing space before the URL ('found athttps://'); this should be corrected for readability.","section":"Data Availability"}],"recommendation":"reject","confidential_remarks":"The manuscript has a useful descriptive core: the network visualizations and the individual-level rebellion metric are potentially interesting for legislative studies. However, the central claim of predictive accuracy is not supported by the evidence as presented, because the reported 94% is only 0.94 percentage points above an always-non-rebel baseline and the threshold is selected on the full dataset. Substantial reanalysis and a reframing of the contribution would be needed before the paper could be considered, so I recommend rejection in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one genuinely useful thing: it builds a network-based rebellion score from pairwise voting similarities and releases the data. The top-rebel list for the 57th Parliament looks plausible, and the visualisations are suggestive. The idea of adjusting cosine similarity by party or ideology is a sensible way to look for cross-party alliances and within-party splits.\n\nThe problem is the headline claim. The supplementary confusion matrix gives 94% accuracy on the October 2019 timetable vote, but 580 of 639 MPs are true negatives. A trivial classifier that always predicts 'no rebellion' gets 93% accuracy because only 45 MPs actually rebelled. The model's entire edge is six correct classifications, and it only catches 20 of 45 rebels (44% recall). No significance test is offered. The rebel threshold is selected from the full distribution using an elbow, and there is no out-of-sample threshold validation. On top of that, the rebellion metric is built from deviations from party/ideology labels, and the validation defines rebellion in the same terms, so there is a real circularity concern. The paper's own central claim—that it predicted how MPs would vote—collapses once you compare against the baseline.\n\nThe KS test between Brexit and non-Brexit vote distributions is fine as a descriptive observation, but it does not validate prediction. The authors are honest about some limitations, but they do not acknowledge the baseline problem, which is the most important one. The data and replication materials being available is a real plus, and the method is not wrong as a descriptive tool. It just does not demonstrate forecasting skill.\n\nI would not cite this as evidence of predictive accuracy. But the paper deserves a serious referee, not a desk reject, because the data and method are real and the flaws are fixable. A revision that reframes the contribution as descriptive, adds proper baselines, uses cross-validation for the threshold, and reports precision/recall against the trivial classifier would be a moderately useful contribution. As is, I would reject.","headline":"The network-based rebellion score is a reasonable descriptive tool, but the headline 'over 90% accuracy' is essentially a party-line baseline, so the paper should be revised or rejected as is.","tokens_in":10041,"tokens_out":1298,"would_cite":false,"duration_ms":16016,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Voting networks alone can predict which MPs will rebel on a Brexit deal with 94% accuracy, this paper claims.","keywords":["Social network analysis","Party cohesion","Rebellion","Brexit","House of Commons","Eigenvector centrality","Voting behaviour","Euroscepticism"],"falsifier":"Take the same 639 MPs' centrality scores and apply the same elbow threshold to the next major Brexit-related division in the 57th or 58th Parliament; if the resulting accuracy does not exceed the always-non-rebel baseline (around 93%) by a substantial margin, the claim that the rebellion metric predicts future voting is not supported.","tokens_in":9067,"feed_emoji":"🗳️","tokens_out":5961,"duration_ms":55552,"temperature":0.7,"pith_summary":"This paper argues that individual-level parliamentary rebellion can be measured from public voting records alone, using social network analysis rather than aggregate party-cohesion scores. It constructs a party-adjusted voting-similarity network of MPs and computes an eigenvector-centrality-based 'rebellion metric' for each member. The metric is validated by predicting which MPs would rebel against their party on the October 2019 Brexit deal timetable, with a reported accuracy of 94%. If correct, this offers a way to anticipate legislative dissent as it unfolds, without waiting for historical hindsight.","feed_headline":"MPs' voting networks predict Brexit rebellions with 94% accuracy","feed_subtitle":"A network measure of dissent, based only on past votes, spots which MPs will break party lines.","key_machinery":"The central object is the rebellion metric: the sum of two eigenvector centralities computed on a signed, party-adjusted voting-similarity network. Each MP is a node; edge weights are the difference between the cosine similarity of two MPs' voting vectors and the similarity expected from their party or ideology, so positive edges mark cross-party alliances and negative edges mark within-party conflicts. Eigenvector centrality is computed separately on the positive-edge subnetwork and on the negative-edge subnetwork (with absolute weights), and the two scores are summed. This single number is then thresholded at an elbow in its distribution to label MPs as rebels or loyalists.","core_discovery":"On the paper's own terms, the central discovery is that the structure of an MP's voting relationships carries predictive signal about future rebellion. Building a signed network from pairwise cosine similarities of voting records, subtracting expected party or ideology coherence, and summing eigenvector centralities on the positive and negative edges yields a scalar rebellion score per MP. Using a threshold at the elbow of this score's distribution, the authors classify all 639 MPs as likely rebels or loyalists on the 22 October 2019 vote on the Brexit bill's expedited timetable, and report a confusion-matrix accuracy of 94% (580 true negatives, 20 true positives, 14 false positives, 25 false negatives).","pith_inferences":["The reported 94% accuracy is computed on a single, class-imbalanced division where only about 7% of MPs actually rebelled; a natural extension is to compare the model's accuracy against a trivial always-non-rebel baseline on the same division, which would already score roughly 93%, to quantify the metric's added predictive value.","A more stringent out-of-sample test would fix the rebel threshold using only divisions before 22 October 2019 and then apply it to that vote; the current procedure selects the threshold from the same centrality distribution used for the prediction.","The paper's two-ideology relabeling shows rebellion can be defined at different granularities; an extension would be to let the threshold and ideology labels emerge from clustering the voting network itself, rather than being chosen a priori.","The method is not limited to Brexit; it could be applied to any legislature where similar divisions exist, potentially producing a real-time dissent index for journalists and political analysts."],"forward_implications":["If a parliament's voting record is public, the same pipeline can rank individual MPs by propensity to defect on any upcoming division, without requiring private whip information.","The metric reveals that rebellion on European integration is not just a party-level phenomenon but is concentrated in identifiable individuals, as the paper's networks show blurred party clusters on Brexit divisions.","The method can be re-run continuously across a parliament, so predictions can be updated after each division rather than waiting for end-of-term analysis.","Because the approach uses only recorded votes, it could be extended to other legislatures with digital voting records."],"supporting_citations":[{"why":"Supplies the eigenvector centrality measure used to compute rebellion scores on the signed network.","marker":"Bonacich, 2007"},{"why":"Provides the ForceAtlas2 force-directed layout used for visual identification of rebels in the network projections.","marker":"Jacomy et al. (2014)"},{"why":"Defines the standard Rice score for party cohesion that the paper contrasts with and seeks to extend.","marker":"Rice, 1938"},{"why":"Documents Conservative Party dissent on Europe, providing historical context for the rebellion phenomenon studied.","marker":"Baker et al., 1999"},{"why":"Analyzes European integration as a cross-cutting issue in the House of Commons, framing the substantive focus.","marker":"Tzelgov, 2014"},{"why":"Provides the published vote tallies for the October 2019 Brexit deal timetable used in the validation.","marker":"BBC (2019)"}],"fun_headline_variants":["MP voting networks predict Brexit rebellions with 94% accuracy","Network analysis of votes forecasts which MPs will rebel on Brexit","Voting similarity networks reveal Brexit dissidents among MPs","Eigenvector-based rebellion metric predicts Brexit vote defections"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of over 90% predictive accuracy rests on a single, class-imbalanced future division, with the rebel threshold selected from the same centrality distribution, and no comparison to a trivial always-non-rebel baseline, which would already achieve roughly 93% accuracy on that division.","fun_headline_variants_meta":{"raw":{"variants":["MP voting networks predict Brexit rebellions with 94% accuracy","Network analysis of votes forecasts which MPs will rebel on Brexit","Voting similarity networks reveal Brexit dissidents among MPs","Eigenvector-based rebellion metric predicts Brexit vote defections"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1396,"prompt_tokens":808,"completion_tokens":588,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":424,"tokens_out":588,"duration_ms":7042,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:27:24.566826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 639 MPs' centrality scores and apply the same elbow threshold to the next major Brexit-related division in the 57th or 58th Parliament; if the resulting accuracy does not exceed the always-non-rebel baseline (around 93%) by a substantial margin, the claim that the rebellion metric predicts future voting is not supported.","supporting_citations":[{"cited_title":"Venturini, S","cited_arxiv_id":null,"evidence_quote":"Provides the ForceAtlas2 force-directed layout used for visual identification of rebels in the network projections."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the standard Rice score for party cohesion that the paper contrasts with and seeks to extend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents Conservative Party dissent on Europe, providing historical context for the rebellion phenomenon studied."},{"cited_title":"(2014, March)","cited_arxiv_id":null,"evidence_quote":"Analyzes European integration as a cross-cutting issue in the House of Commons, framing the substantive focus."}],"review_version":1}