Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

Clearer codebooks raise LLM accuracy on political event coding but do not ensure the models preserve the expert coding logic under changes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 00:51 UTC pith:LMVE54G3

load-bearing objection Clearer codebooks raise accuracy on political event coding but the behavioral tests leave open whether models actually preserve the intended logic. the 2 major comments →

arxiv 2606.06781 v1 pith:LMVE54G3 submitted 2026-06-04 cs.CL

When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding

Zixian He , Bharath Raahul Murugesan , Patrick Brandt , Yibo Hu This is my paper
classification cs.CL
keywords LLM codingpolitical event codingcodebook operationalizationbehavioral reliabilitysource-target classificationsocial science data productionprompt engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether turning expert-written codebooks into clearer LLM prompts—with explicit definitions, examples, and rules for hard cases—improves performance on the task of assigning political event labels to text describing actions between actors. It shows that these clearer versions do lift classification accuracy, with larger gains on fine-grained event types. The same models, however, often change their outputs when the codebook is altered in controlled ways such as swapping label names or reordering entries, even while they can still state the original definitions correctly. This gap matters for social-science applications that need the coded data to reflect the precise logic an expert intended rather than incidental label matching. The authors therefore recommend evaluating LLM coding systems on both raw accuracy and behavioral consistency with the source codebook.

Core claim

Operationalizing expert codebooks into LLM-friendly forms with clearer definitions, examples, retrieved context, and rules for difficult cases substantially improves classification performance on political event coding, especially for fine-grained distinctions, yet these predictive gains do not fully translate into behavioral reliability: models may output valid labels and recover definitions while still failing tests that alter label names, codebook order, or label-definition mappings.

What carries the argument

Behavioral reliability tests that apply controlled perturbations to the codebook (label-name swaps, order changes, definition remappings) and measure whether the model continues to apply the original expert coding logic rather than surface associations.

Load-bearing premise

The controlled codebook perturbations accurately reveal whether the model has internalized the intended relational coding rules instead of learning to match labels by other means.

What would settle it

A model that maintains identical event classifications across the original and perturbed codebooks (with reversed definitions for two similar event types) while still achieving high accuracy on the unperturbed task would falsify the claim that predictive gains do not produce behavioral reliability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Codebook-guided LLM systems must be assessed on behavioral consistency with the source rules, not accuracy alone.
  • Clearer operationalizations improve fine-grained event classification more than coarse classification.
  • Models can state codebook definitions correctly yet still alter outputs when label names or mappings change.
  • Social-science datasets produced by LLMs may contain systematic inconsistencies invisible to standard accuracy metrics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The results suggest LLMs may rely on pattern matching to codebook text rather than on an internal model of actor-action relations.
  • Extending these tests to other structured extraction tasks could show whether the reliability gap appears outside political event coding.
  • One testable extension is whether training objectives that penalize inconsistency under codebook edits close the observed gap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript evaluates LLM performance on political event coding, a source-target relation classification task, using expert codebooks. It tests whether operationalizing codebooks with clearer definitions, examples, retrieved context, and rules for edge cases improves accuracy (especially fine-grained events) and whether these gains produce behavioral reliability under controlled perturbations to label names, codebook order, and label-definition mappings. The central finding is that clearer codebooks raise predictive performance but do not guarantee that models preserve the intended expert coding logic, as they can output valid labels while failing the reliability tests.

Significance. If the results hold after addressing controls, the work is significant for computational social science because it demonstrates that accuracy alone is insufficient to validate LLM coding systems whose outputs are used as structured data. By introducing behavioral reliability tests that probe preservation of coding logic, the paper supplies a concrete, falsifiable criterion beyond standard metrics and applies it to a non-trivial relation-extraction setting.

major comments (2)
  1. [Abstract and behavioral reliability evaluation] The behavioral reliability tests (changes to label names, order, and mappings) are load-bearing for the claim that predictive gains do not translate into preserved coding logic. The manuscript must demonstrate that these perturbations were constructed to hold lexical, length, and token-probability properties constant (or used semantically equivalent rephrasings) so that failures can be attributed to logic rather than general prompt sensitivity; the abstract provides no indication such controls were applied.
  2. [Results on definition recovery] The paper reports that models 'recover definitions' yet still fail the reliability tests. The precise operationalization of 'recover definitions' and the quantitative threshold used to declare recovery versus failure should be stated explicitly, including any statistical tests and sample sizes, because this distinction is central to separating surface-level correctness from logic preservation.
minor comments (2)
  1. Clarify the exact number of events, documents, and annotators in the evaluation dataset; these details are needed to assess statistical power of the reported improvements.
  2. Figure or table captions should explicitly state whether error bars represent standard error, bootstrap intervals, or another measure.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help strengthen the presentation of our behavioral reliability tests. We address each point below and will incorporate clarifications in a revised manuscript.

read point-by-point responses

  1. Referee: [Abstract and behavioral reliability evaluation] The behavioral reliability tests (changes to label names, order, and mappings) are load-bearing for the claim that predictive gains do not translate into preserved coding logic. The manuscript must demonstrate that these perturbations were constructed to hold lexical, length, and token-probability properties constant (or used semantically equivalent rephrasings) so that failures can be attributed to logic rather than general prompt sensitivity; the abstract provides no indication such controls were applied.

    Authors: We agree that explicit documentation of perturbation construction is essential to isolate effects on coding logic. The perturbations were implemented via semantically equivalent rephrasings for label names and mappings, with label order changes preserving the same set of tokens and approximate lengths; token-probability effects were mitigated by using the same underlying vocabulary where feasible. However, these design choices are described only in the methods section and not referenced in the abstract. We will revise the abstract to note the controlled nature of the perturbations and expand the methods with a new subsection detailing the lexical, length, and probability controls, including examples of the rephrasings used. This addresses the concern directly without altering the core findings. revision: yes

  2. Referee: [Results on definition recovery] The paper reports that models 'recover definitions' yet still fail the reliability tests. The precise operationalization of 'recover definitions' and the quantitative threshold used to declare recovery versus failure should be stated explicitly, including any statistical tests and sample sizes, because this distinction is central to separating surface-level correctness from logic preservation.

    Authors: We will make the operationalization fully explicit. 'Recover definitions' is measured as the proportion of cases in which the model, when given only the definition text (without the original label name), selects the intended label from the full codebook; recovery is declared when this proportion exceeds 0.85 on the test set. We will add the exact threshold, the sample size (n=1,248 events), and the statistical comparison (paired t-test against chance level, p<0.001) to both the results section and a new methods paragraph. This clarification will better distinguish surface recovery from the logic-preservation failures shown in the reliability tests. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical evaluation

full rationale

This is an empirical evaluation paper that tests LLM classification performance and behavioral reliability on political event coding tasks using expert codebooks. It reports experimental results on accuracy gains from clearer codebooks and failures on controlled change tests, with no mathematical derivations, equations, parameter fitting, or first-principles predictions. All claims rest on direct measurements against external benchmarks (model outputs vs. codebook rules), which are independent and falsifiable. No self-definitional steps, fitted inputs renamed as predictions, or load-bearing self-citation chains appear in the abstract or described methodology.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no mathematical modeling, derivations, or parameter fitting; no free parameters, axioms, or invented entities are identifiable from the provided text.

pith-pipeline@v0.9.1-grok · 5725 in / 1184 out tokens · 37029 ms · 2026-06-28T00:51:11.672359+00:00 · methodology

0 comments
read the original abstract

High accuracy does not necessarily make an LLM a faithful coder. This issue matters because many social-science studies rely on expert-written codebooks to turn text into structured data. We study this problem in political event coding, a challenging source-target relation classification task beyond ordinary sentence-level classification, where models must determine what one actor did to another using detailed coding rules. We test whether expert codebooks become more effective when operationalized into LLM-friendly forms with clearer definitions, examples, retrieved context, and rules for difficult cases. We then evaluate behavioral reliability under controlled changes to label names, codebook order, and label-definition mappings. Clearer codebooks substantially improve classification performance, especially for fine-grained event classification. However, these predictive gains do not fully translate into behavioral reliability. Models may produce valid labels and recover definitions while still failing behavioral reliability tests under controlled codebook changes. These findings suggest that codebook-guided LLM systems should be evaluated not only by accuracy, but also by whether they preserve the coding logic that makes coded outputs meaningful for social-science research.

Figures

Figures reproduced from arXiv: 2606.06781 by Bharath Raahul Murugesan, Patrick Brandt, Yibo Hu, Zixian He.

Figure 1
Figure 1. Figure 1: Predictive performance and behavioral reliability in codebook-grounded LLM event coding. Expert [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of event-mode distinctions in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prediction–behavioral reliability decoupling [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Local Monitors Miss Compositional Harm: Diagnosing Distributed Backdoors in Multi-Agent Systems

    cs.CR 2026-07 accept novelty 7.0

    Once attack fragments are locally benign and ε-indistinguishable from benign traffic, no local monitor can separate them (TPR−FPR≤ε); the signal reappears only in the right assembled representation.

Reference graph

Works this paper leans on

49 extracted references · cited by 1 Pith paper

  1. [1]

    and Osorio, Javier and D'Orazio, Vito J

    Hu, Yibo and Parolin, Erick Skorupa and Khan, Latifur and Brandt, Patrick T. and Osorio, Javier and D'Orazio, Vito J. , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  2. [2]

    and Freeman, John R

    Brandt, Patrick T. and Freeman, John R. and Schrodt, Philip A. , title =. Conflict Management and Peace Science , volume =. 2011 , doi =

    2011

  3. [3]

    and Schrodt, Philip A

    Gerner, Deborah J. and Schrodt, Philip A. and Yilmaz, Omur and Abu-Jabr, Rajaa , title =

  4. [4]

    2018 , howpublished =

    2018

  5. [5]

    Challenging

    Suzgun, Mirac and Scales, Nathan and Sch. Challenging. Findings of the Association for Computational Linguistics: ACL 2023 , pages =. 2023 , publisher =

    2023

  6. [6]

    , journal =

    Halterman, Andrew and Keith, Katherine A. , journal =. Codebook. 2025 , doi =

    2025

  7. [7]

    2019 , edition =

    Content Analysis: An Introduction to Its Methodology , author =. 2019 , edition =

    2019

  8. [8]

    2017 , edition =

    The Content Analysis Guidebook , author =. 2017 , edition =

    2017

  9. [9]

    Just Read the Codebook!

    Ruckdeschel, Mattes , booktitle =. Just Read the Codebook!. 2025 , publisher =

    2025

  10. [10]

    Advances in Neural Information Processing Systems , volume =

    Language Models are Few-Shot Learners , author =. Advances in Neural Information Processing Systems , volume =

  11. [11]

    Advances in Neural Information Processing Systems , volume =

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems , volume =

  12. [12]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , volume =

  13. [13]

    Sentence-

    Reimers, Nils and Gurevych, Iryna , booktitle =. Sentence-. 2019 , publisher =

    2019

  14. [14]

    Sociological Methods & Research , volume =

    From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models , author =. Sociological Methods & Research , volume =. 2025 , doi =

    2025

  15. [15]

    Sociological Methods & Research , volume =

    Updating ``The Future of Coding'': Qualitative Coding with Generative Large Language Models , author =. Sociological Methods & Research , volume =. 2025 , doi =

    2025

  16. [16]

    International Journal of Forecasting , volume =

    Evaluating Forecasts of Political Conflict Dynamics , author =. International Journal of Forecasting , volume =

  17. [17]

    Proceedings of the First Workshop on Fact Extraction and VERification , pages =

    Zero-Shot Relation Classification as Textual Entailment , author =. Proceedings of the First Workshop on Fact Extraction and VERification , pages =

  18. [18]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages =

    Benchmarking Zero-Shot Text Classification: Datasets, Evaluation and Entailment Approach , author =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages =

    2019

  19. [19]

    and D'Orazio, Vito , booktitle =

    Hu, Yibo and Hosseini, MohammadSaleh and Parolin, Erick Skorupa and Osorio, Javier and Khan, Latifur and Brandt, Patrick T. and D'Orazio, Vito , booktitle =

  20. [20]

    2021 IEEE International Conference on Big Data (Big Data) , pages=

    Come-ke: A new transformers based approach for knowledge extraction in conflict and mediation domain , author=. 2021 IEEE International Conference on Big Data (Big Data) , pages=. 2021 , organization=

    2021

  21. [21]

    and D'Orazio, Vito and Holmes, Jennifer , booktitle =

    Parolin, Erick Skorupa and Khan, Latifur and Osorio, Javier and Brandt, Patrick T. and D'Orazio, Vito and Holmes, Jennifer , booktitle =. 2021 , organization =

    2021

  22. [22]

    and Osorio, Javier and D'Orazio, Vito , booktitle =

    Parolin, Erick Skorupa and Hosseini, MohammadSaleh and Hu, Yibo and Khan, Latifur and Brandt, Patrick T. and Osorio, Javier and D'Orazio, Vito , booktitle =. Multi-. 2022 , doi =

    2022

  23. [23]

    Introducing

    Raleigh, Clionadh and Linke, Andrew and Hegre, H. Introducing. Journal of Peace Research , volume =

  24. [24]

    The Gun Violence Database: A New Task and Data Set for

    Pavlick, Ellie and Ji, Heng and Pan, Xiaoman and Callison-Burch, Chris , booktitle =. The Gun Violence Database: A New Task and Data Set for

  25. [25]

    Findings of the Association for Computational Linguistics , year =

    Few-Shot Upsampling for Protest Size Detection , author =. Findings of the Association for Computational Linguistics , year =

  26. [26]

    Challenges and Applications of Automated Extraction of Socio-Political Events from Text (

    Hurriyetoglu, Ali and Tanev, Hristo and Zavarella, Vanni and Piskorski, Jakub and Yeniterzi, Reyyan and Yuret, Deniz and Villavicencio, Aline , booktitle =. Challenges and Applications of Automated Extraction of Socio-Political Events from Text (

  27. [27]

    Barker, Ken and Awasthy, Parul and Ni, Jian and Florian, Radu , booktitle =

  28. [28]

    , booktitle =

    Radford, Benjamin J. , booktitle =

  29. [29]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages =

    Document-Level Event Argument Extraction by Conditional Generation , author =. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages =

    2021

  30. [30]

    Computational Linguistics , volume =

    Can Large Language Models Transform Computational Social Science? , author =. Computational Linguistics , volume =. 2024 , doi =

    2024

  31. [31]

    Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Zero-Shot Transfer Learning for Event Extraction , author =. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  32. [32]

    , title =

    McClelland, Charles A. , title =. 2006 , howpublished =

    2006

  33. [33]

    The Automatic Content Extraction ( ACE ) Program -- Tasks, Data, and Evaluation

    Doddington, George and Mitchell, Alexis and Przybocki, Mark and Ramshaw, Lance and Strassel, Stephanie and Weischedel, Ralph. The Automatic Content Extraction ( ACE ) Program -- Tasks, Data, and Evaluation. Proceedings of the Fourth International Conference on Language Resources and Evaluation ( LREC ' 04). 2004

    2004

  34. [34]

    and Roy, Joydeep , title =

    Lu, J. and Roy, Joydeep , title =. 2017 , howpublished =

    2017

  35. [35]

    Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages=

    Uncertainty-aware reliable text classification , author=. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages=

  36. [36]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Multidimensional uncertainty-aware evidential neural networks , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  37. [37]

    Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC '20) , publisher=

    Lessons Learned from the Chameleon Testbed , author=. Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC '20) , publisher=

    2020

  38. [38]

    American Political Science Review , volume =

    Measurement Validity: A Shared Standard for Qualitative and Quantitative Research , author =. American Political Science Review , volume =. 2001 , doi =

    2001

  39. [39]

    and Gerner, Deborah J

    Schrodt, Philip A. and Gerner, Deborah J. , title =. Annual Meeting of the American Political Science Association , year =

  40. [40]

    and Yilmaz, Omur and Gerner, Deborah J

    Schrodt, Philip A. and Yilmaz, Omur and Gerner, Deborah J. , title =. Annual Meeting of the International Studies Association , year =

  41. [41]

    and Gerner, Deborah J

    Schrodt, Philip A. and Gerner, Deborah J. and Yilmaz, Omur , title =. Annual Meeting of the International Studies Association , year =

  42. [42]

    Can We Trust the Evaluation on

    Aiyappa, Rachith and An, Jisun and Kwak, Haewoon and Ahn, Yong-Yeol , booktitle =. Can We Trust the Evaluation on. 2023 , publisher =

    2023

  43. [43]

    , journal =

    Azar, Edward E. , journal =. The Conflict and Peace Data Bank (. 1980 , doi =

    1980

  44. [44]

    Craig and Taylor, Charles L

    Bond, Doug and Bond, Joe and Oh, Churl and Jenkins, J. Craig and Taylor, Charles L. , journal =. Integrated Data for Events Analysis (. 2003 , doi =

    2003

  45. [45]

    2015 , howpublished =

    Boschee, Elizabeth and Lautenschlager, Jennifer and O'Brien, Sean and Shellman, Steve and Starz, James and Ward, Michael , title =. 2015 , howpublished =

    2015

  46. [46]

    Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages =

    A Large Annotated Corpus for Learning Natural Language Inference , author =. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages =. 2015 , address =

    2015

  47. [47]

    Analyzing

    Buyukoz, Berfu and Hurriyetoglu, Ali and Ozgur, Arzucan , booktitle =. Analyzing. 2020 , address =

    2020

  48. [48]

    The Annals of Statistics , volume =

    Bootstrap Methods: Another Look at the Jackknife , author =. The Annals of Statistics , volume =. 1979 , doi =

    1979

  49. [49]

    and Ye, Zhiquan and Yuan, Zonggang and Jia, Yantao and Chen, Huajun , booktitle =

    Geng, Yuxia and Chen, Jiaoyan and Chen, Zhuo and Pan, Jeff Z. and Ye, Zhiquan and Yuan, Zonggang and Jia, Yantao and Chen, Huajun , booktitle =. 2021 , doi =

    2021