{"id":"c516439b-5cd2-463f-bcb9-dc158f77791a","arxiv_id":"2506.16692","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The party composition of a bill's sponsors, along with district area and population, predicts a Korean lawmaker's political affiliation in transportation bills, though the sponsor features are derived from the same affiliation labels.","lead":"This paper combines a large language model and explainable AI to classify transportation bills and predict lawmakers' party affiliation from sponsor and district features. The study finds sponsor-party composition to be the strongest predictor, but the predictor is partly constructed from the very labels being predicted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central finding rests on target leakage: sponsor-party composition features (10–13) are computed from the same affiliation labels the model predicts, making the top SHAP results near-tautological.","rationale":"The reader's weakest-assumption analysis correctly identifies the circularity: sponsor-party features are computed from the same party labels the model predicts. This is the single most load-bearing concern because the paper's central claim about 'critical determinants shaping legislative outcomes' rests almost entirely on the SHAP importance of these leaked features. My independent reading of Section 3.2 and Table 2 confirms the construction: features 10–13 are bill-level sums and proportions of conservative and progressive sponsors, and the prediction target is the individual lawmaker's affiliation. The leave-one-out test I propose would directly separate genuine sponsor-composition signals from target leakage. The dataset-size inconsistency (7,872 vs. 10,060) is a real reporting problem, but it is secondary to the circularity, since even a corrected dataset would not fix the target leakage. Therefore I agree with the reader's rejection: as presented, the explanatory findings are unsupported, although the LLM filtering component could be salvaged with proper validation.","tokens_in":12037,"tokens_out":2436,"duration_ms":31328,"concrete_test":"Re-run the XGBoost and SHAP pipeline with leave-one-out sponsor features: for each lawmaker–bill record, recompute features 10–13 after removing that lawmaker from the bill's sponsor list, then retrain with the same hyperparameters and data split. If the top SHAP rankings and test F1 score remain essentially unchanged, the central finding survives; if sponsor-composition features drop in importance or model performance falls substantially, the reported conclusion is an artifact of label-derived features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's explanatory conclusion—that the number and proportion of conservative and progressive sponsors, along with district size and electoral population, are critical determinants of legislative outcomes—depends on SHAP rankings from an XGBoost model whose target is a lawmaker's party affiliation (Section 3.2). Features 10–13 (number and percentage of conservative and progressive sponsors per bill) are bill-level aggregates derived directly from the party labels being predicted. Since each row is a lawmaker–bill participation record, a lawmaker's own affiliation contributes to the sponsor counts of every bill they join; consequently, the model can predict party affiliation largely by reading the majority-party composition of the bill's sponsor list. SHAP then ranks these leaked aggregates as the most influential features, so the headline result is an artifact of circular feature construction rather than an independent finding about legislative behavior. The abstract additionally overstates the result by claiming implications for 'legislative outcomes' when only party affiliation is modeled. A secondary but related issue is the inconsistent dataset size (7,872 participation instances reported in Section 4.2 versus 10,060 total records used for training and testing in Section 4.3), which further undermines the quantitative claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LegiGPT, a pipeline that combines GPT-4-based multi-stage filtering of transportation-related bills from South Korea's 21st National Assembly with XAI (XGBoost + SHAP) to analyze determinants of lawmakers' party affiliation. From 23,655 bills, the LLM pipeline retains 577 transportation bills; the authors construct 7,872 lawmaker-bill participation instances, train classifiers to predict conservative vs. progressive affiliation, and use SHAP to conclude that sponsor party composition, district size, and electoral population are critical determinants of legislative outcomes. The LLM filtering component is a useful methodological contribution, but the central explanatory claim is compromised by target leakage in the feature construction and by inconsistencies in the reported dataset sizes.","tokens_in":12233,"tokens_out":5191,"duration_ms":55531,"significance":"If valid, the framework would offer a novel way to combine LLM-based text filtering with interpretable machine learning for legislative analysis, and the finding that sponsor composition and district characteristics drive political alignment would extend existing survey-based work. However, the core empirical result is not reliable because features 10-13 are constructed from the same party-affiliation labels the model is asked to predict, making the high predictive accuracy and the SHAP ranking of those features largely tautological. The LLM multi-stage filtering approach itself may be of interest to the text-mining community, but the paper's headline conclusion about legislative determinants does not follow from the current experimental design.","major_comments":[{"comment":"Features 10-13 (number of conservative sponsors, number of progressive sponsors, percentage of conservative sponsors, percentage of progressive sponsors) are bill-level aggregates computed directly from the party affiliations of sponsors. Because each row is a lawmaker-bill participation record and the prediction target is that same lawmaker's party affiliation, the lawmaker's own label contributes to these features. For bills sponsored predominantly by a single party, the feature values almost fully determine the target, so the reported XGBoost F1 of 0.978 and the SHAP ranking of these features as the most important are largely forced by circular feature construction rather than by an independent empirical relationship. This invalidates the central claim that sponsor party composition is a critical determinant of political alignment. Please remove features derived from the target variable or reformulate the prediction task (e.g., predict bill approval or lawmaker participation) so that the features are not functions of the label.","section":"Section 3.2, features 10-13"},{"comment":"The dataset size is stated inconsistently. Section 4.2 reports that the preprocessed dataset contains 7,872 bill participation instances across 577 bills, and Table 2 repeats this count. Section 4.3, however, states that the training set consists of 8,048 records and the test set of 2,012 records, totaling 10,060 records. Since 10,060 exceeds the stated total of 7,872, the quantitative results in Section 4.3 cannot be reproduced from the described dataset. Please clarify which dataset is used for model training and evaluation, and ensure all reported counts are consistent.","section":"Section 4.2 vs Section 4.3"},{"comment":"The claim that the top four features--percentage of conservative sponsors, percentage of progressive sponsors, number of conservative sponsors, and number of progressive sponsors--are 'strongly positively correlated, each exhibiting Pearson correlation coefficients greater than 0.6' is inconsistent with the definitions of the percentage features. For any given row, the percentage of conservative sponsors and the percentage of progressive sponsors sum to 1 (assuming the two-party categorization used throughout the paper), so their Pearson correlation is exactly -1, not a positive value greater than 0.6. This error suggests a problem in the computation or reporting of the correlation matrix and undermines the subsequent interpretation of feature interdependencies.","section":"Section 4.4.1, Figure 5"},{"comment":"The data splitting protocol is not appropriate for the structured nature of the dataset. Rows are lawmaker-bill participation instances, so multiple rows share the same bill and the same lawmaker appears across many bills. A random 85:15 split without grouping by bill or lawmaker allows the same bill or lawmaker to appear in both training and test sets, which can substantially inflate the reported performance metrics via information leakage. Please use grouped or time-based splits (e.g., split by bill ID or by time period) and report performance under such conditions, along with details of how many unique bills and lawmakers appear in each split.","section":"Section 4.3, data splitting"}],"minor_comments":[{"comment":"Subfigures (a) and (b) are both labeled 'proportion of conservative sponsors' in the caption; one of them should presumably refer to the proportion of progressive sponsors, given the accompanying text.","section":"Section 4.4.2, Figure 6 caption"},{"comment":"The phrase 'legislative outcomes' is used to describe the prediction target, but the model actually predicts a lawmaker's party affiliation (conservative vs. progressive), not bill approval or enactment. Please reword the claims to refer to 'political affiliation' or 'political alignment' to avoid overstating the scope.","section":"Abstract and Section 5"},{"comment":"The descriptive differences (e.g., more progressive than conservative participants, more male than female) are presented without any statistical test. Please add appropriate significance tests or explicitly state that these comparisons are descriptive only.","section":"Section 4.2, descriptive statistics"},{"comment":"The text cites 'Lundberg and Lee, 2017' and 'Lee, 2024' without full entries in the reference list; only reference [27] corresponds to Lundberg and Lee. Please ensure all in-text citations have corresponding references.","section":"Section 3.2.2"},{"comment":"The manual validation of the 10% sample is mentioned but no agreement metrics, error counts, or details of the review protocol are provided. Please include quantitative validation results for the multi-stage filtering.","section":"Section 3.1, validation of LLM filtering"}],"recommendation":"reject","confidential_remarks":"The central empirical claim is not supported because of target leakage in features 10-13, which makes the reported accuracy and SHAP rankings near-tautological. The dataset size inconsistency and the impossible positive correlation between percentage-conservative and percentage-progressive further undermine the quantitative results. The LLM filtering pipeline may be a salvageable contribution, but as it stands the paper does not meet the standard for publication in a transportation policy journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's real contribution is the LLM filtering pipeline, not the XAI analysis. The multi-stage keyword-sentence-context approach with English translation is a sensible way to isolate transportation bills from Korean legislative text, and the 577 vs 3,874 comparison is a concrete, useful result. The descriptive statistics on who sponsors transport bills in the 21st Assembly are new data worth having.\n\nThe soft spots are serious. Features 10–13 (number and percentage of conservative/progressive sponsors) are computed from the same party labels the model is trained to predict. Since each row is a lawmaker-bill participation record, the model can read the majority party of the sponsor list, including the lawmaker's own contribution. SHAP then ranks these leaked aggregates as the top drivers. That makes the central conclusion—sponsor party composition shapes ideological alignment—near-tautological. The abstract also overstates the finding by talking about 'legislative outcomes' when the target is party affiliation. And the dataset size is inconsistent: Section 4.2 reports 7,872 participation instances, Section 4.3 says 10,060 total records with 8,048 train and 2,012 test. That is not a trivial discrepancy.\n\nThe XGBoost accuracy (F1 0.978) is high, but that is exactly what you would expect with leakage; it is not evidence that the framework works. The SHAP dependency plots are similarly forced.\n\nMinor things: the 10% manual validation is thin. The 'one keyword' filter vs the described process could be clearer. The citations are fine; the self-cites are to relevant prior work, not padding.\n\nWho is this for? Someone working on LLM-based filtering of legislative corpora would get value from the first half. Anyone wanting to learn about political determinants of transport policy should read the descriptive tables, not the SHAP rankings.\n\nMy recommendation: if this came to me as a referee, I'd reject in current form but encourage a resubmission that either drops the leaking features or reframes the task (e.g., predict party from constituency and personal attributes only), and fixes the dataset count. The LLM part alone might be publishable as a methods note.","headline":"The LLM filtering pipeline is genuinely useful, but the XAI half is sunk by target leakage that makes the headline SHAP findings near-tautological.","tokens_in":12746,"tokens_out":2219,"would_cite":false,"duration_ms":25390,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sponsor party composition and district size are the dominant predictors of a lawmaker's party affiliation in South Korean transportation legislation, according to an LLM-plus-XAI analysis of the 21st National Assembly.","keywords":["party politics","legislative bill","transport policy","large language model","explainable artificial intelligence","South Korea","SHAP","XGBoost"],"falsifier":"Train the same XGBoost and SHAP pipeline on the 577 transportation bills with the four sponsor-party composition features removed; if electoral population and district area still top the SHAP ranking and accuracy stays near baseline, the constituency claim survives, but if accuracy collapses and the remaining features drop out, the sponsor-composition result is an artifact of label leakage.","tokens_in":11783,"feed_emoji":"🚆","tokens_out":8986,"duration_ms":92062,"temperature":0.7,"pith_summary":"This paper introduces LegiGPT, a pipeline that uses GPT-4 and explainable machine learning to study how party politics shapes transportation legislation in South Korea's 21st National Assembly (2020-2024). The authors' central claim is that the strongest determinants of a lawmaker's political alignment are the party composition of a bill's sponsors, especially the number and percentage of conservative and progressive co-sponsors, together with constituency traits such as electoral population and district area. They interpret these patterns as ideological homophily in legislative work: lawmakers predominantly join bills sponsored by their own party, while cross-party participation signals bipartisan cooperation. If the claim holds, the framework provides a scalable and interpretable way to mine legislative text and to anticipate where transportation policy will divide along partisan lines.","feed_headline":"Sponsor mix and district size predict party of transport-bill sponsors","feed_subtitle":"GPT-4 filters 23,655 bills to 577; sponsor mix and district geography dominate the party-alignment ranking.","key_machinery":"The load-bearing machinery is a two-stage pipeline. First, a zero-shot GPT-4 classifier applies a three-step filtering cascade—keyword matching, sentence-level relevance, and context-level focus—to isolate transportation bills, reducing the corpus from 23,655 to 577 bills and improving thematic precision by roughly 85% over keyword-only selection. Second, XGBoost models political affiliation from 19 legislator, constituency, and bill-sponsor features, and SHAP (Shapley additive explanations) decomposes each prediction into additive feature contributions. The sponsor-composition features, counts and percentages of conservative and progressive sponsors, are the ones that dominate the SHAP ranking.","core_discovery":"The paper's central discovery is that, after GPT-4 filters 23,655 South Korean bills to 577 transportation-focused proposals, an XGBoost model predicts a lawmaker's party affiliation (conservative or progressive) with an F1 score of about 0.978, and SHAP attributions rank the share and count of conservative sponsors, the share of progressive sponsors, electoral population, and district area as the top drivers. The modeled outcome is the lawmaker's party label rather than bill passage, and the authors use this as evidence that party-affiliated sponsorship patterns and constituency characteristics are strong predictors of political alignment in transportation policy. The paper further argues that these results reveal bipartisan engagement through different roles: conservatives and progressives both participate in cross-party bills, but their sponsorship profiles remain distinct.","pith_inferences":["The abstract's phrase 'shaping legislative outcomes' is broader than the modeled target, which is the sponsor's party label; a stricter test would set bill approval or passage as the outcome variable.","Because the sponsor-party counts are computed from the same conservative/progressive labels the model predicts, the top SHAP results are partly circular; rerunning with features 10-13 removed would reveal how much signal comes from constituency geography alone.","The same pipeline could be applied to committee votes or amendment co-sponsorship to test whether the homophily pattern holds within individual bills rather than across the corpus.","In other legislatures, the raw sponsor-count features would need party-base-rate normalization, otherwise the model may simply learn the relative sizes of party caucuses."],"forward_implications":["If sponsorship composition reliably marks alignment, analysts can use co-sponsorship networks to map partisan polarization in transport policy without reading full bill text.","The three-stage filtering protocol and the 577-bill transport corpus can serve as a reusable template for applying LLM classifiers to legislative datasets in other policy domains.","The prominence of electoral population and district area implies that rural-versus-urban constituency structure shapes which legislators engage with transport policy, independent of party label.","The SHAP ranking gives policymakers a diagnostic tool for identifying bills with unusual cross-party sponsorship, which the paper interprets as bipartisan cooperation."],"supporting_citations":[{"why":"Supplies the National Assembly legislative activity data and district-level election results that the entire pipeline runs on.","marker":"[25]"},{"why":"Provides SHAP, the model-agnostic interpretation method used to rank feature contributions and produce the paper's core attributions.","marker":"[27]"},{"why":"Provides XGBoost, the prediction model that achieves the best F1 score and is selected for SHAP interpretation.","marker":"[32]"},{"why":"Prior evidence that party partisanship shapes transportation policy preferences, which the paper extends to sponsorship behavior.","marker":"[4]"},{"why":"Shows LLMs can outperform human coders at annotating political text, motivating the zero-shot filtering stage.","marker":"[19]"},{"why":"Documents the conservative-progressive regional divide in South Korea that motivates the district-level hypotheses.","marker":"[2]"},{"why":"Survey evidence on partisan differences in transport taxation and spending priorities, used to connect sponsor patterns to policy ideology.","marker":"[40]"}],"fun_headline_variants":["LegiGPT: sponsor mix and district size predict transport-bill party","GPT-4 filters 23K bills; sponsor mix and district size predict party","Sponsor mix and district size dominate prediction of bill sponsor party","District geography and sponsor mix are key to predicting bill party","LLM-XAI pipeline: sponsor mix and district size flag party of transport bills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the number and percentage of conservative and progressive sponsors can be treated as independent explanatory variables for a lawmaker's party affiliation, even though those counts are derived from the very party labels the model is trained to predict.","fun_headline_variants_meta":{"raw":{"variants":["LegiGPT: sponsor mix and district size predict transport-bill party","GPT-4 filters 23K bills; sponsor mix and district size predict party","Sponsor mix and district size dominate prediction of bill sponsor party","District geography and sponsor mix are key to predicting bill party","LLM-XAI pipeline: sponsor mix and district size flag party of transport bills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00164,"raw_usage":{"total_tokens":6492,"prompt_tokens":890,"completion_tokens":5602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":5507}},"tokens_in":506,"tokens_out":5602,"duration_ms":40498,"temperature":1.0,"reasoning_tokens":5507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:37:13.800031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same XGBoost and SHAP pipeline on the 577 transportation bills with the four sponsor-party composition features removed; if electoral population and district area still top the SHAP ranking and accuracy stays near baseline, the constituency claim survives, but if accuracy collapses and the remaining features drop out, the sponsor-composition result is an artifact of label leakage.","supporting_citations":[{"cited_title":"Public data portal, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the National Assembly legislative activity data and district-level election results that the entire pipeline runs on."},{"cited_title":"Xgboost: A scalable tree boosting system","cited_arxiv_id":null,"evidence_quote":"Provides XGBoost, the prediction model that achieves the best F1 score and is selected for SHAP interpretation."},{"cited_title":"Political partisanship and transportation reform","cited_arxiv_id":null,"evidence_quote":"Prior evidence that party partisanship shapes transportation policy preferences, which the paper extends to sponsorship behavior."},{"cited_title":"Large language models outperform expert coders and supervised classi- fiers at annotating political social media messages","cited_arxiv_id":null,"evidence_quote":"Shows LLMs can outperform human coders at annotating political text, motivating the zero-shot filtering stage."},{"cited_title":"Conservatives and progressives in south korea","cited_arxiv_id":null,"evidence_quote":"Documents the conservative-progressive regional divide in South Korea that motivates the district-level hypotheses."},{"cited_title":"Would americans pay more in taxes for better transportation? answers from seven years of national survey data","cited_arxiv_id":null,"evidence_quote":"Survey evidence on partisan differences in transport taxation and spending priorities, used to connect sponsor patterns to policy ideology."}],"review_version":1}