REVIEW 3 major objections 4 minor 3 cited by
Fine-tuning LLMs on speaker-attributed transcripts with action tags produces civic deliberation simulations that crowdworkers often mistake for real conversations.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 20:52 UTC pith:DNEV45Z2
load-bearing objection Solid engineering-plus-datasets paper whose headline persona-fidelity numbers likely rest on non-meeting-disjoint classifier splits; fixable, but the abstract overstates what is currently shown. the 3 major comments →
Point of Order: Action-Aware LLM Persona Modeling for Data-Grounded Civic Deliberation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that adding structured metadata—especially turn-level pragmatic action tags like [propose_motion] and [call_vote]—to speaker-attributed training data dramatically improves how faithfully a fine-tuned LLM reproduces a specific participant's language and behavior. On held-out test data, fine-tuning with these tags reduces perplexity from 20.37 to 6.64 (-67%), increases Classifier Fool Rate (the fraction of agent-generated utterances that a one-vs-all speaker classifier mistakes for real ones) from 0.29 to 0.64 (+121%), and raises Speaker Attribution Accuracy (a multi-class classifier's ability to identify which speaker the agent is imitating) from 0.23 to 0.45 (+96%). The pape
What carries the argument
The central mechanism is 'action-aware persona modeling': each utterance in the diarized transcript is annotated with a compact set of pragmatic tags (from a domain-specific taxonomy of 15–30 actions), and LLMs are fine-tuned on sequences where these tags are prepended to the text. The tags act as explicit conditioning on communicative intent, constraining the model toward role-consistent behavior. This is combined with a multimodal speaker-linking pipeline that aligns active-speaker boxes, OCR name patches, and audio diarization to assign stable identities across recordings, plus persona profiles and topic summaries extracted from long monologues and transcript chunks. The action tags alone
Load-bearing premise
The fidelity metrics (CFR and SAA) rely on classifiers trained on 70/30 splits that are not speaker-disjoint or meeting-disjoint, so the load-bearing assumption is that these classifiers learn stable speaker style rather than meeting topic or role.
What would settle it
Re-train the CFR and SAA classifiers on speaker-disjoint (and ideally meeting-disjoint) training/test splits, then re-evaluate the fine-tuned models; if the improvements over baseline shrink or vanish (for instance, CFR drops below 0.5), the claimed fidelity gains are largely a leakage artifact.
If this is right
- If the approach is correct, researchers can build realistic simulated civic venues for exploring counterfactual scenarios—changing agendas, participant strategies, or procedural rules—without convening human panels.
- The release of three speaker-labeled government deliberation datasets gives the field a common benchmark for comparing persona-faithful multi-agent dialogue models.
- Action tags are a cheap, transferable annotation that improves persona realism and could be applied to other institutional dialogue domains beyond government meetings.
- Time-aware prompting (attaching simulated timestamps and agenda items) is a lightweight mechanism to keep long simulations on-track and increase the rate at which discussions reach collective decisions.
- The reported near-chance human detection rate suggests that provenance labeling or watermarking may become necessary when publishing simulated deliberations.
Where Pith is reading between the lines
- The reported fidelity gains may be inflated because the CFR/SAA classifiers are trained on 70/30 splits that are not speaker-disjoint or meeting-disjoint; if classifiers pick up meeting topic or role rather than stable speaker style, the gains could reflect leakage rather than genuine persona modeling.
- The pipeline's reliance on Zoom gallery-view recordings limits universality, but the action-tag conditioning idea should transfer to any role-labeled institutional corpus (parliaments, corporate boards) wherever speaker identities can be linked.
- A strong testable extension is to compare downstream deliberative outcomes—vote distributions, decision times, topic shifts—between simulations and real historical meetings, validating functional equivalence rather than just surface style.
- If simulations are as indistinguishable from real transcripts as reported, they could be mistaken for authentic public records; publishers of such data should consider disclosing simulation provenance explicitly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a pipeline for converting public Zoom recordings into speaker-attributed transcripts, releasing three civic-deliberation datasets (DC Court of Appeals, Albemarle School Board, Waipā District Council). It introduces 'action-aware' fine-tuning, in which LLMs are conditioned on persona profiles, topics, and pragmatic action tags. Experiments with LLaMA-3.1-70B, Qwen-2.5-72B, and GPT-OSS-120B report large gains in perplexity, classifier-based persona fidelity (CFR/SAA), and simulation metrics, with a human Turing test indicating that simulated excerpts are often hard to distinguish from real ones. The paper positions the work as a scalable, reproducible foundation for data-grounded civic simulation.
Significance. If the main claims hold, the paper makes a meaningful empirical contribution: it provides three public, human-validated speaker-attributed datasets of real deliberation and demonstrates that fine-tuning on structured metadata can substantially improve the match between simulated and real participant behavior. The reproducibility-oriented design (public datasets, PEFT, standard ASR/diarization tools) is a strength, as are the consistent PPL reductions across multiple model families and domains. However, the headline persona-fidelity claims rest on classifier metrics whose evaluation protocol may overstate genuine stylistic generalization. The paper's central contribution is therefore promising but requires a more rigorous evaluation to be convincing.
major comments (3)
- [Appendix G.1/G.2 and Section 6.1] The CFR and SAA classifiers are trained on a 70/30 random utterance-level split of real transcripts. Since the same meeting topics and speakers appear in both training and test portions, the classifiers can exploit meeting-specific lexical cues rather than stable speaker style. When applied to generated utterances from simulations conditioned on the same meeting agendas, this can inflate fool rate and speaker attribution accuracy. The paper's strongest quantitative claim—that action-aware fine-tuning 'doubles' persona fidelity—therefore needs to be re-evaluated with a meeting-disjoint (or speaker-disjoint) split, and the classifier performance on such splits should be reported. This is essential to rule out topic/context leakage as the source of the reported CFR 0.29→0.64 and SAA 0.23→0.45 gains.
- [Abstract vs. Section 7/Appendix K] The abstract reports 'vote attempts by up to 3.6×' and 'deliberative responsiveness by up to 70%.' These numbers do not appear in the body or appendices. The simulation results in Section 7.2 report topic-coverage changes and aggregate consensus increases (28.4%→40.5%), but neither 3.6× nor 70% is substantiated. The abstract should be corrected to state only numbers that are actually derived from the experiments, or the missing results should be added.
- [Sections 3.2, 6.2, and Appendix K] GPT-5 is used to generate the action tags, persona profiles, and topic summaries that condition the fine-tuned models, and GPT-5 also evaluates topic coverage and vote counts in the simulations. This creates a potential self-consistency loop: the evaluation may partly reward models for matching GPT-5's own annotations rather than human-judged deliberation quality. The paper validates the diarization/transcription quality with MTurk, but no human validation is reported for the action-tag taxonomies or the automatic simulation metrics. I would like to see either human agreement on a sample of action tags and topic-coverage judgments, or an argument for why the GPT-5-based evaluation is not circular.
minor comments (4)
- [Throughout] There are several typos and inconsistencies: 'Albermale' instead of 'Albemarle' in Table 2 and Appendix J; 'Waip ¯a' spacing issues; 'PD' in Figure 16 caption should be 'PPL'; Table 7 captions say 'Fool Rates' but the table mixes one-vs-all and multi-class scores. These should be cleaned up.
- [Section 1 and Section 8] The paper claims to release datasets and code, but no URLs or repository links are provided. Given the stated reproducibility goal, the integration of the datasets/code with the paper should be made explicit (e.g., in a 'Data Availability' statement).
- [Appendix G.1] The one-vs-all classifier setup uses 'an equal number of randomly sampled utterances from other speakers' as negatives. It is unclear whether this sampling is done per meeting or globally, and how this affects the classifier's ability to separate speaker identity from meeting role. More detail on the negative sampling strategy would help.
- [Section 7.1] The text notes that 'GPT models exhibit higher PPL' and provides a plausible explanation, but it would be helpful to also report confidence intervals or variance across the three datasets in the main table, which the table already includes; however, the main text quotes only one dataset's numbers (Albemarle). Consider summarizing the aggregate trend more explicitly.
Circularity Check
Minor self-consistency in automatic simulation-quality metrics; core PPL/CFR/SAA claims are not circular.
specific steps
-
other
[Section 3.2 (Topic Extraction), Section 5.3 (Simulations), Appendix K (Temporal Grounding)]
"GPT-5 then summarizes each chunk into 2–3 concise bullet points. These chunk-level summaries are consolidated by GPT-5 into a small set of topic labels for the meeting. ... To compute this automatically at scale, we use GPT-5 to evaluate whether each agenda topic was substantively discussed within the corresponding transcript segment."
The agenda topics that condition the simulations are produced by GPT-5, and GPT-5 is also the judge of whether the simulated transcript 'substantively discussed' those topics. Topic coverage therefore measures agreement between the simulated text and GPT-5's own topic decomposition, not an externally anchored ground truth. This is a self-consistency check for the automatic simulation-quality metrics. It is not load-bearing for the central persona-fidelity claims, which rest on held-out PPL, independent DeBERTa classifiers (CFR/SAA), and human Turing tests.
full rationale
The paper's central claim—action-aware fine-tuning improves persona fidelity—is supported by three metrics that are not circular: PPL is computed on a held-out set of real transcripts; CFR and SAA use DeBERTa classifiers trained on real utterances and applied to newly generated text; and the human Turing test is an external judgment. The metadata extraction (persona profiles, action tags, topics) uses GPT-5, but this upstream annotation does not define the values of PPL/CFR/SAA. The one soft spot is the automatic simulation-quality evaluation: GPT-5 creates the agenda topics and later judges topic coverage and vote outcomes, so those secondary metrics are partly a self-consistency check with GPT-5's own annotations. This is a minor circularity, not load-bearing for the central result. The skeptical concern about CFR/SAA classifiers using non-meeting-disjoint splits (Appendix G) is a real construct-validity threat—classifiers may exploit topic overlap between real training utterances and generated utterances from models trained on the same meetings—but it is not a circular reduction: the classifiers are not fitted parameters of the generator, and the metrics are not definitionally forced. Overall score reflects the one secondary self-consistency issue.
Axiom & Free-Parameter Ledger
free parameters (4)
- Micro-profile statistics (response length, question rate, politeness rate, sentiment) =
~45 words, ~60%, ~45%, ~0.5
- Utterance length thresholds for evaluation classifiers =
>10 words for CFR, >20 words for SAA
- Number of longest monologues used for persona extraction =
25
- Equal-width vote-count buckets in goal-achievement evaluation =
low / medium / high
axioms (5)
- domain assumption Zoom gallery-view recordings contain a highlighted active-speaker tile with an OCR-readable participant name at the lower-left corner.
- domain assumption Whisper ASR transcripts are sufficiently accurate for persona modeling.
- ad hoc to paper GPT-5-assigned action tags, persona profiles, and topic summaries are reliable without human validation.
- domain assumption DeBERTa classifiers trained on utterance-level splits generalize to unseen generated utterances without exploiting meeting or role leakage.
- domain assumption MTurk majority voting over three workers yields valid ground truth for dataset validation.
read the original abstract
LLM-based simulations can enable controlled studies of civic deliberation, but current systems lack speaker-attributed data and methods for evaluating long-form institutional behavior. ASR transcripts typically use anonymous labels such as $Speaker\_1$, preventing models from learning stable participant behavior across meetings. We present a reproducible pipeline that converts public Zoom recordings into speaker-attributed transcripts enriched with persona profiles, topics, and pragmatic "action tags" such as $[propose\_motion]$. Using this pipeline, we release three public datasets of government deliberation (Appellate Court hearings, School Board meetings, and Municipal Council sessions) and fine-tune LLM personas on this action-aware data. We evaluate simulations along four dimensions: persona fidelity, persona consistency, institutional fidelity, and behavioral coherence. Action-aware fine-tuning cuts perplexity by 67%, doubles classifier-based persona fidelity, increases vote attempts by up to $3.6\times$, and improves deliberative responsiveness by up to 70%. Human evaluations show that simulated excerpts are often hard to distinguish from real deliberations, indicating a practical foundation for data-grounded civic simulation studies.
Figures
Forward citations
Cited by 3 Pith papers
-
When Identity Overrides Incentives: Representational Choices as Governance Decisions in Multi-Agent LLM Systems
Role-based personas in multi-agent LLM systems suppress payoff-aligned behavior, shifting equilibrium selection by up to 90 percentage points in Tragedy of the Commons versus Green Transition scenarios even with full ...
-
MASS: Deep Research for Social Sciences with Memory-Augmented Social Simulation
MASS enhances LLM research generation in social sciences via dynamic social simulations with multi-level norms, multi-disciplinary memory datasets, and Ebbinghaus-inspired forgetting, reporting 6.81% overall quality g...
-
When Identity Overrides Incentives: Representational Choices as Governance Decisions in Multi-Agent LLM Systems
Persona-conditioned LLM agents favor Green outcomes even against explicit Tragedy-dominant payoffs, but the headline 65–90% 'Tragedy equilibrium' recovery is contradicted by the paper's own appendix (0 Tragedy profile...
Reference graph
Works this paper leans on
-
[7]
InProceedings of the 2024 Joint International Con- ference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12712–12723, Torino, Italia
Persona- aware multi-party conversation response generation. InProceedings of the 2024 Joint International Con- ference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12712–12723, Torino, Italia. ELRA and ICCL. OpenAI
2024
-
[8]
Accessed: 2025-10-
Openai cookbook. Accessed: 2025-10-
2025
-
[9]
Accessed: 2025-10-
Introducing gpt-5. Accessed: 2025-10-
2025
-
[10]
Robust speech recognition via large-scale weak su- pervision.arXiv preprint arXiv:2212.04356. Joshua Rosenberg and 1 others
-
[11]
arXiv preprint arXiv:2508.10925
gpt- oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Model card for gpt-oss series, released under Apache 2.0 license. Tianyi Shi and 1 others
-
[12]
Improving factuality and reasoning in language models through multi-agent debate.arXiv preprint arXiv:2305.14325. Amir Taubenfeld and 1 others
-
[13]
Quantifying the persona effect in llm simulations.arXiv preprint arXiv:2402.10811. Hugo Touvron and 1 others
-
[14]
Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971. Waip¯a District Council. Waip¯a district council. https: //www.waipadc.govt.nz/. Accessed: 2025-11-07. Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, and Wangchun- shu Zhou
Pith/arXiv arXiv 2025
-
[15]
Ai persona: Towards life-long per- sonalization of llms.Preprint, arXiv:2412.13103. Qiang Wu and 1 others
-
[16]
Eric Zhongcong Xu, Zeyang Song, Satoshi Tsutsui, Chao Feng, Mang Ye, and Mike Zheng Shou
Autogen: Enabling next- gen llm applications via multi-agent conversation framework.arXiv preprint arXiv:2308.08155. Eric Zhongcong Xu, Zeyang Song, Satoshi Tsutsui, Chao Feng, Mang Ye, and Mike Zheng Shou
-
[17]
Parameter-efficient fine-tuning methods for pretrained language mod- els: A critical review and assessment.Preprint, arXiv:2312.12148. An Yang
-
[18]
Qwen2.5 technical report.arXiv preprint arXiv:2412.15115. yt-dlp contributors
-
[19]
https://github
yt-dlp: A youtube-dl fork with additional features and fixes. https://github. com/yt-dlp/yt-dlp. Accessed: 2025-09-22. Zhang and 1 others
2025
-
[20]
When "a helpful assistant" is not really helpful: Personas in system prompts do not improve performances of large language models.Preprint, arXiv:2311.10054. Hazem Zohny
-
[21]
Simulating ethics: Using llm de- bate panels to model deliberation on medical dilem- mas (adept). ArXiv:2505.21112. A Persona and Topic Extraction Prompts Stage 1: Structured Extraction Prompt System:You are a precise extractor. Only extract information explicitly stated or strongly implied. Return only JSON, nothing else. Task:For each numbered monologue...
Pith/arXiv arXiv 2020
-
[23]
The Court is now in ses- sion
facilitates easy integration with models for classification, pre- diction, and generation tasks. To facilitate cross-dataset analysis, the dataset- specific tagsets were mapped to a set of unified categories: • INFO— providing or clarifying factual or procedural information. • ASK— requesting information, clarification, or explanation. • ARG— making argum...
1973
-
[2015]
https://www.tensorflow
TensorFlow: Large-scale machine learning on heterogeneous systems. https://www.tensorflow. org/. Software available from tensorflow.org. Albemarle County Public Schools. School board. https://www.k12albemarle.org/school-board. Accessed: 2025-11-07. Joshua Ashkinaze
2025
-
[2016]
zoom meeting,
was fine-tuned to classify whether a video thumbnail represented a Zoom gallery-style meeting. Positive examples were sampled from frames of 100 curated gallery-style meetings, while negatives were drawn from thumbnails of diverse YouTube content, including news, comedy, and entertainment. Applying this model reduced the candidate set by roughly 40%, resu...
2020
-
[2021]
Lora: Low-rank adaptation of large language models. ArXiv:2106.09685. Khan and 1 others
-
[2022]
Encoder- decoder based attractors for end-to-end neural di- arization.IEEE/ACM Transactions on Audio, Speech, and Language Processing. EEND-EDA; arXiv:2106.10654. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
-
[2023]
Qlora: Efficient finetuning of quantized llms.Preprint, arXiv:2305.14314. District of Columbia Courts. Court of appeals. https: //www.dccourts.gov/court-of-appeals. Ac- cessed: 2025-11-07. Yingwei Du and 1 others
Pith/arXiv arXiv 2025
-
[2024]
Plurals: A system for guiding llms via simulated social ensembles.arXiv preprint arXiv:2402.04049. Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, and 1 others
-
[2025]
Paddleocr: A practi- cal toolkit for text detection and recognition.arXiv preprint arXiv:2507.05595. J. Flamino and 1 others
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.