Pith. sign in

REVIEW 2 major objections 1 minor 42 references

Treating each language model as one rater in a panel produces Krippendorff's alpha of 0.86 for political position scores.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-26 06:21 UTC pith:3KCPN5DR

load-bearing objection The paper shows solid cross-model reproducibility (alpha 0.86) for LLM-panel political coding but correctly flags that this is consistency, not yet validity, so the usability claim for data-sparse regions rests on work the author says is still missing. the 2 major comments →

arxiv 2606.23042 v1 pith:3KCPN5DR submitted 2026-06-22 cs.CY cs.AIcs.CLstat.AP

The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel

Tarek Gara This is my paper
classification cs.CY cs.AIcs.CLstat.AP
keywords political positionslanguage model panelinter-rater reliabilityKrippendorff's alphadata-sparse regionsMiddle East and North Africameasurement methodspanel ratings
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a method for assigning political position scores in regions where manifesto coding, expert surveys, and text-scaling models have not been validated and often cannot be applied. It places each large language model in the role of a single fallible rater whose individual output is pooled with others rather than trusted on its own. An applicability rule keeps a zero score distinct from missing data, and a lens system separates what an actor says from what it does. Across nine models the resulting panel reaches alpha 0.86 on both interval and ordinal scales, with the value unchanged when the panel expands from five to nine raters; blind coding of disagreements attributes about two-thirds to differences in interpretation. The worked example is the Middle East and North Africa, but the design is presented as portable to any region the older tools leave out.

Core claim

The paper establishes that a panel of nine language models drawn from eight laboratories yields political position scores with Krippendorff's alpha of 0.86 on both interval and ordinal metrics, that this level of reproducibility holds steady as the panel grows from five to nine raters, and that the largest observed disagreements are traceable to interpretive differences rather than simple error.

What carries the argument

The language-model panel, in which each model serves as one independent rater and the final score is produced by aggregating their separate judgments.

Load-bearing premise

That reproducibility across different language models can serve as a usable proxy for measurement quality even without human validation of the resulting scores.

What would settle it

A direct comparison of the panel scores against independent human-expert ratings on the same actors in the Middle East and North Africa that shows large systematic divergence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Adding written axis definitions shifts mean scores by 1.8 points on a 21-point scale and reduces mean absolute gaps between raters from 2.81 to 2.50 while raising their correlation from 0.81 to 0.89.
  • Reproducibility measured by alpha remains constant when the panel is expanded beyond five models.
  • The sharpest panel splits occur on actors' stances toward foundational state orders and are largely interpretive rather than erroneous.
  • The applicability rule and lens system allow zero scores to be treated as meaningful and keep statements separate from actions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The stability of alpha after five models suggests that adding still more models may yield diminishing returns on reliability.
  • The same panel structure could be tested on other latent traits such as economic policy orientation once the lens definitions are adapted.
  • The released instrument and data make it feasible for later studies to collect targeted human ratings that directly benchmark the panel outputs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes treating large language models as individual raters in a panel to measure political positions in data-sparse regions such as the Middle East and North Africa, where traditional methods like manifesto coding and expert surveys perform poorly. It introduces an applicability rule and a lens system, reports that adding written axis definitions shifts mean scores by 1.8 points on a 21-point scale and improves inter-rater agreement, achieves a Krippendorff's alpha of 0.86 across nine models from eight laboratories which remains stable from five to nine raters, and finds that most disagreements stem from interpretation rather than error. The paper releases the full instrument and data while acknowledging the lack of human validation.

Significance. Should the reported reproducibility translate into valid measurements of political positions, the approach could offer a valuable tool for analyzing under-studied regions. Strengths include the full release of materials for reproducibility and explicit discussion of limitations. The stability of alpha across panel sizes and the analysis of disagreements provide useful empirical observations.

major comments (2)
  1. [Abstract] Abstract: The central claim that the LLM panel yields usable political-position scores in data-sparse regions rests on treating Krippendorff's alpha of 0.86 as a workable proxy for measurement quality, yet the manuscript explicitly states that human validation of the resulting scores is still lacking; this directly undercuts the usability argument because reproducibility across models does not establish correctness or applicability.
  2. [Results] Results (the three reported findings): The 1.8-point mean shift from adding definitions is presented without error bars, confidence intervals, or a statistical test, and no derivation or formula is supplied for computing Krippendorff's alpha on the interval and ordinal metrics, leaving the reliability claim difficult to evaluate for robustness.
minor comments (1)
  1. [Methods] The applicability rule and lens system are described conceptually but would benefit from a concrete worked example or pseudocode in the methods section to clarify how a score of zero is distinguished from a blank.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the scope and strengthen the empirical presentation. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that the LLM panel yields usable political-position scores in data-sparse regions rests on treating Krippendorff's alpha of 0.86 as a workable proxy for measurement quality, yet the manuscript explicitly states that human validation of the resulting scores is still lacking; this directly undercuts the usability argument because reproducibility across models does not establish correctness or applicability.

    Authors: We agree that reproducibility across models does not establish correctness. The manuscript already distinguishes reliability from validity in both the abstract and results sections, and explicitly flags the absence of human validation as a limitation. The central claim is narrower: the panel provides a reproducible method for generating position scores in regions where manifesto coding and expert surveys are unavailable or unreliable. We do not assert that the scores are valid measures of 'true' positions without further evidence. This framing is consistent with the paper's emphasis on the panel as one rater among several rather than a replacement for human judgment. revision: no

  2. Referee: [Results] Results (the three reported findings): The 1.8-point mean shift from adding definitions is presented without error bars, confidence intervals, or a statistical test, and no derivation or formula is supplied for computing Krippendorff's alpha on the interval and ordinal metrics, leaving the reliability claim difficult to evaluate for robustness.

    Authors: We agree these details would improve evaluability. In revision we will add standard errors or confidence intervals around the 1.8-point mean shift together with a paired statistical test. We will also include the standard Krippendorff formula (as implemented for interval and ordinal scales) and note the software used. Because the full dataset and instrument are already released, these additions can be computed directly from the existing materials. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical alpha and stability are direct computations from external model outputs

full rationale

The paper's reported results consist of empirical computations of Krippendorff's alpha (0.86) and mean absolute gaps directly from the outputs of nine distinct LLMs applied to MENA political texts, with stability observed as panel size increases from five to nine. These are measurements against external inputs rather than quantities derived from any internal fitted parameters, self-definitions, or self-citation chains. The manuscript explicitly separates reliability from validity and flags the missing human validation, so no load-bearing premise reduces to a definitional identity or prior author work. The derivation chain is therefore self-contained and reports observable facts without circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 2 invented entities

The method rests on treating LLMs as interchangeable with human experts for rating purposes and on the assumption that pooled reproducibility can stand in for correctness in the absence of human validation. No numerical free parameters are introduced; the main additions are two new procedural components whose independent evidence is limited to the internal agreement statistics.

axioms (2)
  • domain assumption Large language models can function as fallible but useful raters whose judgments can be pooled like those of human experts in an expert survey.
    This premise is invoked when the paper states it treats the model 'not as a measurement device but as a single, fallible rater in a panel, roughly the way an expert survey treats one expert'.
  • domain assumption High inter-model agreement (Krippendorff's alpha) constitutes evidence that the method produces usable scores even without external human validation.
    The paper reports the alpha result and notes the lack of human validation, yet presents the panel output as a practical measurement approach for data-sparse regions.
invented entities (2)
  • Applicability rule no independent evidence
    purpose: Distinguishes a meaningful score of zero from a blank or missing entry.
    New procedural component introduced to handle data-sparse settings; no independent evidence outside the panel agreement is supplied.
  • Lens system no independent evidence
    purpose: Separates what an actor says from what it does.
    New procedural component introduced to improve measurement validity; no independent evidence outside the panel agreement is supplied.

pith-pipeline@v0.9.1-grok · 5911 in / 1797 out tokens · 26062 ms · 2026-06-26T06:21:40.607123+00:00 · methodology

0 comments
read the original abstract

Most tools for measuring political positions, manifesto coding, expert surveys, text-scaling models, were built and validated on Western party systems, and outside that setting they work poorly, and often not at all. This paper is an attempt at a method for those settings. It treats a large language model not as a measurement device but as a single, fallible rater in a panel, roughly the way an expert survey treats one expert: the value comes from pooling many judges rather than trusting any one of them. I describe the panel, an applicability rule that keeps a score of zero distinct from a blank, and a lens system that separates what an actor says from what it does. I report three results. First, holding a definition-free round fixed, adding written axis definitions moves scores by a mean of 1.8 points on a 21-point scale and tightens agreement between raters (mean absolute gap 2.81 to 2.50; r 0.81 to 0.89); they make two independent raters agree more closely, which an arbitrary steer would not. Second, across nine models from eight laboratories in two countries, Krippendorff's alpha is 0.86 on both an interval and an ordinal metric, and it stayed put as the panel grew from five raters to nine. That is reliability, the reproducibility of a reading, and not validity, its correctness. Third, where the panel does disagree, the disagreement is informative: the sharpest split, a full-scale divergence on an actor's stance toward its state's foundational order, points to a referent problem, and a blind triple-coding puts about two-thirds of it down to interpretation rather than error. I try to be plain about what the method can't do, including the human validation it still lacks, and I release the instrument and data in full. The worked example is the Middle East and North Africa, but I'd expect the method to carry to any region these standard tools leave out.

Figures

Figures reproduced from arXiv: 2606.23042 by Tarek Gara.

Figure 1
Figure 1. Figure 1: The referent problem on regime stance. Each dot is one of the nine models’ scores for the actor; coincident scores are jittered vertically. Both actors split into two clusters separated by most of the scale, not a spread around a centre: the models agree on the facts and disagree on which constitutional order “the regime” denotes. For Hayat Tahrir al-Sham the lower cluster scores its posture toward the Ba’… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 5 canonical work pages · 1 internal anchor

  1. [1]

    Hotelling, H. (1929). Stability in Competition.The Economic Journal, 39(153), 41–57

    1929

  2. [2]

    (1957).An Economic Theory of Democracy

    Downs, A. (1957).An Economic Theory of Democracy. Harper & Brothers

    1957

  3. [3]

    T., & Rosenthal, H

    Poole, K. T., & Rosenthal, H. (1997).Congress: A Political-Economic History of Roll Call V oting. Oxford University Press

    1997

  4. [4]

    (2001).Mapping Policy Preferences: Estimates for Parties, Electors, and Governments, 1945–1998

    Budge, I., Klingemann, H.-D., V olkens, A., Bara, J., & T anenbaum, E. (2001).Mapping Policy Preferences: Estimates for Parties, Electors, and Governments, 1945–1998. Oxford University Press

    2001

  5. [5]

    (2006).Mapping Policy Preferences II: Estimates for Parties, Electors, and Governments in Eastern Europe, European Union and OECD 1990–2003

    Klingemann, H.-D., V olkens, A., Bara, J., Budge, I., & McDonald, M. (2006).Mapping Policy Preferences II: Estimates for Parties, Electors, and Governments in Eastern Europe, European Union and OECD 1990–2003. Oxford University Press

    2006

  6. [6]

    (2021).The Manifesto Project Dataset (MARPOR), V ersion 2021a

    V olkens, A., Burst, T., Krause, W ., Lehmann, P., et al. (2021).The Manifesto Project Dataset (MARPOR), V ersion 2021a. WZB Berlin Social Science Center

    2021

  7. [7]

    Laver, M., Benoit, K., & Garry, J. (2003). Extracting Policy Positions from Political T exts Using W ords as Data. American Political Science Review, 97(2), 311–331

    2003

  8. [8]

    B., & Proksch, S.-O

    Slapin, J. B., & Proksch, S.-O. (2008). A Scaling Model for Estimating Time-Series Party Positions from T exts. American Journal of Political Science, 52(3), 705–722. 19

    2008

  9. [9]

    Lowe, W ., Benoit, K., Mikhaylov, S., & Laver, M. (2011). Scaling Policy Preferences from Coded Political T exts. Legislative Studies Quarterly, 36(1), 123–155

    2011

  10. [10]

    Mikhaylov, S., Laver, M., & Benoit, K. (2012). Coder Reliability and Misclassification in the Human Coding of Party Manifestos.Political Analysis, 20(1), 78–91

    2012

  11. [11]

    Grimmer, J., & Stewart, B. M. (2013). T ext as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political T exts.Political Analysis, 21(3), 267–297

    2013

  12. [12]

    E., & Stewart, B

    Grimmer, J., Roberts, M. E., & Stewart, B. M. (2022).Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press

    2022

  13. [13]

    Bakker, R., de Vries, C., Edwards, E., Hooghe, L., Jolly, S., Marks, G., et al. (2015). Measuring Party Positions in Europe: The Chapel Hill Expert Survey T rend File, 1999–2010.Party Politics, 21(1), 143–152

    2015

  14. [14]

    (2006).Party Policy in Modern Democracies

    Benoit, K., & Laver, M. (2006).Party Policy in Modern Democracies. Routledge

    2006

  15. [15]

    Hooghe, L., Marks, G., & Wilson, C. J. (2002). Does Left/Right Structure Party Positions on European Integration?Comparative Political Studies, 35(8), 965–989

    2002

  16. [16]

    H., Lindberg, S

    Coppedge, M., Gerring, J., Knutsen, C. H., Lindberg, S. I., T eorell, J., et al. (2024).V-Dem Codebook v14. V arieties of Democracy Project

    2024

  17. [17]

    L., T zelgov, E., W ang, Y.-T., Krusell, J., & Miri, F

    Pemstein, D., Marquardt, K. L., T zelgov, E., W ang, Y.-T., Krusell, J., & Miri, F. (2018). The V-Dem Measurement Model: Latent V ariable Analysis for Cross-National and Cross-T emporal Expert-Coded Data.V-Dem W orking Paper 21(3rd ed.)

    2018

  18. [18]

    King, G., Murray, C. J. L., Salomon, J. A., & T andon, A. (2004). Enhancing the V alidity and Cross-Cultural Comparability of Measurement in Survey Research.American Political Science Review, 98(1), 191–207

    2004

  19. [19]

    H., & McKelvey, R

    Aldrich, J. H., & McKelvey, R. D. (1977). A Method of Scaling with Applications to the 1968 and 1972 Presidential Elections.American Political Science Review, 71(1), 111–130

    1977

  20. [20]

    A., Bakker, R., Carroll, R., & Poole, K

    Hare, C., Armstrong, D. A., Bakker, R., Carroll, R., & Poole, K. T. (2015). Using Bayesian Aldrich-McKelvey Scaling to Study Citizens’ Ideological Preferences and Perceptions.American Journal of Political Science, 59(3), 759–774

    2015

  21. [21]

    Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT Outperforms Crowd W orkers for T ext-Annotation T asks.Proceedings of the National Academy of Sciences, 120(30), e2305016120

    2023

  22. [22]

    T örnberg, P. (2023). ChatGPT-4 Outperforms Experts and Crowd W orkers in Annotating Political T witter Messages with Zero-Shot Learning.arXiv:2304.06588

    work page arXiv 2023

  23. [23]

    Ziems, C., Held, W ., Shaikh, O., Chen, J., Zhang, Z., & Y ang, D. (2024). Can Large Language Models T ransform Computational Social Science?Computational Linguistics, 50(1), 237–291

    2024

  24. [24]

    Pangakis, N., W olken, S., & Fasching, N. (2023). Automated Annotation with Generative AI Requires V alidation.arXiv:2306.00176

    work page Pith review arXiv 2023

  25. [25]

    Large Language Models Can Be Used to Estimate the Latent Positions of Politicians

    Wu, P. Y., Nagler, J., Tucker, J. A., & Messing, S. (2023). Large Language Models Can Be Used to Estimate the Latent Positions of Politicians.arXiv:2303.12057

    work page Pith review arXiv 2023

  26. [26]

    Le Mens, G., & Gallego, A. (2025). Positioning Political T exts with Large Language Models by Asking and Averaging.Political Analysis(arXiv:2311.16639). 20

    work page arXiv 2025

  27. [27]

    Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023). Whose Opinions Do Language Models Reflect?Proceedings of the 40th International Conference on Machine Learning (ICML)

    2023

  28. [28]

    Towards Measuring the Representation of Subjective Global Opinions in Language Models

    Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., et al. (2024). T owards Measuring the Representa- tion of Subjective Global Opinions in Language Models.arXiv:2306.16388

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [29]

    J., Park, P

    Atari, M., Xue, M. J., Park, P. S., Blasi, D. E., & Henrich, J. (2023). Which Humans?PsyArXiv preprint

    2023

  30. [30]

    Y., Liu, Y., & T svetkov, Y

    Feng, S., Park, C. Y., Liu, Y., & T svetkov, Y. (2023). From Pretraining Data to Language Models to Downstream T asks: T racking the T rails of Political Biases Leading to Unfair NLP Models.Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 11737–11762

    2023

  31. [31]

    Rozado, D. (2024). The Political Preferences of LLMs.PLOS ONE, 19(7), e0306621

    2024

  32. [32]

    Motoki, F., Pinho Neto, V ., & Rodrigues, V . (2024). More Human than Human: Measuring ChatGPT Political Bias.Public Choice, 198, 3–23

    2024

  33. [33]

    P., Busby, E

    Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of One, Many: Using Language Models to Simulate Human Samples.Political Analysis, 31(3), 337–351

    2023

  34. [34]

    D., Dorff, C., Kenkel, B., & Larson, J

    Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.Political Analysis, 32(4), 401–416

    2024

  35. [35]

    M., Gebru, T., McMillan-Major, A., & Shmitchell, S

    Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be T oo Big?Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT), 610–623

    2021

  36. [36]

    Antoun, W ., Baly, F., & Hajj, H. (2020). AraBER T: T ransformer-based Model for Arabic Language Under- standing.Proceedings of the 4th W orkshop on Open-Source Arabic Corpora and Processing Tools (OSACT4)

    2020

  37. [37]

    (2019).Content Analysis: An Introduction to Its Methodology(4th ed.)

    Krippendorff, K. (2019).Content Analysis: An Introduction to Its Methodology(4th ed.). Sage

    2019

  38. [38]

    F., & Krippendorff, K

    Hayes, A. F., & Krippendorff, K. (2007). Answering the Call for a Standard Reliability Measure for Coding Data.Communication Methods and Measures, 1(1), 77–89

    2007

  39. [39]

    Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales.Educational and Psychological Measurement, 20(1), 37–46

    1960

  40. [40]

    (2016).The New Arab W ars: Uprisings and Anarchy in the Middle East

    Lynch, M. (2016).The New Arab W ars: Uprisings and Anarchy in the Middle East. PublicAffairs

    2016

  41. [41]

    (2001).Syria: Revolution from Above

    Hinnebusch, R. (2001).Syria: Revolution from Above. Routledge

    2001

  42. [42]

    Wickham, C. R. (2013).The Muslim Brotherhood: Evolution of an Islamist Movement. Princeton University Press. 21

    2013