Pith. sign in

REVIEW 4 major objections 5 minor 46 references

ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ParliaBench: a benchmark for measuring political authenticity in AI-generated parliamentary speech, with two new embedding-based metrics that score how well a generated speech matches a party's ideology.

desk verdict Useful benchmark resource for parliamentary speech generation, but the headline political authenticity metrics are validated in a circle; worth engaging for the dataset and framework, not for the metric claims. read the letter →

arxiv 2511.08247 v1 pith:SR4RAUDM submitted 2025-11-11 cs.CL cs.CY

Marios Koniaris , Argyro Tsipi , Panayiotis Tsanakas This is my paper · ORCID
classification cs.CLcs.CY
keywords parliamentaryspeechgenerationLLMevaluationpoliticalauthenticityembedding-basedmetricsSpectrumAlignmentPartyQLoRAfine-tuningUKParliament
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ParliaBench is a new benchmark for evaluating large language models when they generate UK parliamentary speeches. The paper argues that standard text-quality metrics miss what matters most in this domain: political authenticity, meaning whether a speech sounds like it genuinely comes from the intended party and ideological position. To fix this, it introduces two embedding-based metrics, Political Spectrum Alignment (PSA) and Party Alignment, which compare a generated speech against centroid embeddings of real parliamentary speeches grouped by ideology or party. The authors fine-tune five models on 448k real UK Parliament speeches, generate 28k speeches, and find that fine-tuning improves most quality metrics, with the new metrics showing the strongest and most consistent improvements. If correct, this provides the first reusable yardstick for political authenticity in LLM output, which matters for any application that simulates or generates partisan political content.

What carries the argument

The central machinery is the two embedding-based metrics: Political Spectrum Alignment (PSA) and Party Alignment. Both construct centroids by embedding many real UK Parliament speeches (grouped by left-right orientation for PSA, or by party for Party Alignment) using sentence transformers, then measuring cosine similarity between a generated speech and the expected centroid, with PSA adding a distance penalty for ideological mismatch. These metrics convert 'does this sound like a Conservative speech?' into a deterministic 0–1 score, replacing subjective judgment with a geometric comparison to real speech distributions.

What would settle it

If human evaluators with parliamentary expertise were shown generated speeches scored 9–10 by the LLM judge and rated them as clearly AI-written or off-party, then the LLM-judge scores—and the conclusions about fine-tuning improving authenticity—would be called into question.

Watch

Extended reading notes

Core claim

The paper claims that political authenticity of generated parliamentary speech can be measured, and that it improves with domain fine-tuning. Novel embedding-based metrics, Political Spectrum Alignment (PSA) and Party Alignment, compute the cosine similarity between a generated speech and centroid embeddings of real speeches by political orientation or party, optionally penalized by ideological distance. On a dataset of 448k speeches from UK Parliament and 28k generated speeches from five fine-tuned LLMs, PSA and Party Alignment show strong discriminative power (ANOVA p<0.001) and significant improvement after fine-tuning across all five models, with effect sizes ranging from small to very l

Load-bearing premise

The entire evaluation rests on the assumption that the LLM judge (Flow-Judge-v0.1) produces valid and unbiased scores for authenticity, appropriateness, and quality — but the paper provides no human agreement measurement, and it explicitly acknowledges this as a limitation.

Editorial extensions

If this is right

  • If PSA and Party Alignment are valid, they give researchers a cheap, reproducible way to audit LLM-generated political text for ideological slant without human labeling.
  • The benchmark provides baseline scores for five common open-weights LLMs, enabling direct comparison of future parliamentary-speech generation models on the same 28k prompts.
  • Fine-tuning on parliamentary data improves political alignment metrics for all tested models, suggesting that domain-specific training is necessary for authentic partisan output.
  • The strong discriminative power of PSA and Party Alignment implies that political authenticity is learnable and measurable, not just a style-based illusion.
  • Cross-context stability scores (91.4–96.2) suggest that fine-tuned models remain consistent across parties, topics, and orientations, which is important for any simulation use-case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because PSA and Party Alignment use centroid similarity, they likely favor generic party-language patterns over distinctive but ideologically accurate arguments; a speech that matches the centroid might be bland rather than genuinely partisan. This is my inference, not the paper's claim.
  • The method could be transferred to other parliaments or political systems by retraining centroids on local corpus data, making the metrics a general toolkit for political authenticity.
  • A direct testable extension: use PSA and Party Alignment as a feedback signal during generation (e.g., through decoding-time steering or reinforcement learning) to see if models can be pushed closer to target ideology without losing fluency.
  • The reliance on embedding centroids means that rare or underrepresented parties with few speeches will have noisy centroids; the paper notes this pattern, and it suggests that performance on smaller parties is a weak spot that targeted data collection could address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ParliaBench, a benchmark for evaluating LLM-generated UK parliamentary speech. It contributes a 447,778-speech dataset derived from ParlaMint, an evaluation framework combining computational metrics (PPL, Dist-N, Self-BLEU, GRUEN, BERTScore, MoverScore) with LLM-as-a-judge scores, and two new embedding-based political authenticity metrics—Political Spectrum Alignment (PSA) and Party Alignment. Five open LLMs are QLoRA-fine-tuned on 80% of the dataset; 27,560 generated speeches from 10 model/type combinations are evaluated. The authors report statistically significant fine-tuning improvements on most metrics and claim that PSA and Party Align show strong discriminative power, leading to the stated contribution of a first validated benchmark for parliamentary speech generation.

Significance. If the central claims hold, ParliaBench would be a useful community resource: the dataset and fine-tuned models are publicly released, the experimental protocol is transparent, and the statistical reporting (t-tests, effect sizes, Bonferroni correction) is above the usual standard for this area. The two proposed political-authenticity metrics address a real gap, since standard NLG metrics ignore ideological positioning. However, the paper's headline claim that these metrics are 'validated' is not currently supported. The construction and validation of PSA and Party Align are circular with the fine-tuning data, and the LLM-judge scores lack any human ground truth. The benchmark's value as a reusable standard therefore depends on resolving these validity issues or substantially tempering the claims.

major comments (4)
  1. [Section 4.1.1, Eqs. (1)-(3); Section 5.2] PSA and Party Align are computed as similarity to centroids c_po and c_p obtained by averaging embeddings of human speeches from ParlaMint; the fine-tuning uses 80% of the same corpus. A fine-tuned model that reproduces party-typical lexical patterns will mechanically score higher on these metrics, so the reported gains are partly by construction rather than evidence of genuine political authenticity. The discriminative validation in Section 6.2 (ANOVA on the metric scores) is also tautological: because the score is defined as similarity to a label-specific nearest centroid, between-group differences are guaranteed whenever centroids differ. This is load-bearing because the abstract and Section 7 claim 'validated metrics' and 'strong discriminative power.' Please add a non-circular validation: e.g., centroids built from temporally disjoint or external data, comparison with established id
  2. [Section 5.3] The evaluation retains only speeches that were successfully generated and fully rated by all 10 model-type combinations (29,220, later 27,560). The paper notes that baseline models had higher failure rates. Restricting to the intersection therefore removes exactly the low-quality baseline outputs that the fine-tuned models could more easily produce, potentially inflating the measured fine-tuning improvements and making the comparison unrepresentative. Please report per-model and per-type failure rates, describe the excluded speeches, and provide robustness analyses (e.g., evaluate all generated outputs with missing-data handling or compare per-model retained subsets).
  3. [Section 4; Appendix 10; Section 7] All LLM-judge metrics (J_Coh, J_Conc, J_Rel, J_Auth, J_PolApp, J_Qual) come from a single judge model, Flow-Judge-v0.1, with no human agreement study and no analysis of judge bias across parties or orientations. Section 7 acknowledges the lack of human validation, but this limitation is not peripheral: the conclusion that fine-tuning improves 'the majority of metrics' depends on the validity of these judge scores. At minimum, provide a human-annotated sample (e.g., 100–200 speeches) with inter-annotator agreement and judge–human correlation, and test whether judge scores are invariant to the party/orientation context in the prompt.
  4. [Abstract; Section 7] The claim that ParliaBench provides 'validated metrics' and that PSA/Party Align have 'strong discriminative power' overstates the evidence. The paper itself states that the evaluation relies entirely on automated metrics without human validation. Discriminative power is demonstrated only by ANOVA on the metric scores themselves, not against any external criterion. Either add an external construct validation (human labels, expert judgments, or comparison with existing political scaling methods) or soften the claims to 'proposed metrics that respond to fine-tuning and separate predefined party/orientation groups.'
minor comments (5)
  1. [Section 4] Typo: 'It inherits it’s architecture' should be 'its architecture.' Also, the paper alternates between 'political authenticity' and 'political appropriateness' without defining the boundary; consider a brief definition.
  2. [Eq. (2)] The orientation coding (Far-left = -6 ... Far-right = +6) and the linear penalty max(0, 1 - Δφ/12) are arbitrary choices. Report sensitivity to these choices, or at least justify them with reference to the RILE scale.
  3. [Table 13] p-values are reported as 0.0000; this is not strictly correct. Use p < 0.0001 or a similarly formatted bound.
  4. [Section 3.1, Step 3] The 1000-speech minimum threshold excludes small parties and affects the party/orientation distribution. Consider reporting how results change when the threshold is varied, especially for Party Align on small groups such as Green Party and Bishops.
  5. [Section 5.3] The paper says 30,000 speeches were planned, but 27,560 were evaluated. Clarify how many were lost at each stage (generation failure, judge failure) and whether those losses correlate with model type or political context.

Circularity Check

2 steps flagged · score 6.0 of 10

PSA/Party Align validation is self-referential: centroids from the same ParlaMint corpus used for fine-tuning make gains and group-discrimination tests partly definitional; no human validation.

  1. fitted input called prediction [Section 4.1.1 (Eqs. 1-3), Section 5.2, Section 6.1/Table 13]
    "We create reference embeddings by grouping parliamentary speeches by political orientations ... computing centroid embeddings using sentence transformers. ... The alignment score measures cosine similarity between generated speech and expected party centroid: Party Align(s, p) = sim(s, cp). ... Training used SFTTrainer from TRL with 80%-20% train-test splits"

    The reference centroids c_po and c_p are computed from the same ParlaMint corpus used to fine-tune the models. A model trained on 80% of those speeches is pushed toward the corpus's party/orientation-typical language, so its outputs score higher on sim(s,c_p) partly because the metric measures distance to the training distribution itself. The reported fine-tuning gains on PSA/Party Align are therefore presented as evidence of 'political authenticity' while the metric's reference is the training data; no independent construct validation is involved.

  2. self definitional [Section 6.2; Section 13.2]
    "Both new political authenticity metrics (PSA and Party Align) successfully discriminate their target political dimensions. Party Align distinguishes parties while PSA distinguishes orientations (both p <0.001)."

    By Eqs. (1)-(3), each PSA/Party Align score is defined as cosine similarity to a centroid built from speeches labeled by that same party/orientation. An ANOVA of these scores across party/orientation groups is structurally a test that the group centroids differ; if they differ at all, significant between-group effects are built into the score definition. Thus the 'discrimination testing' re-states the centroid construction rather than validating that the metric measures ideological positioning. The paper's stated limitation that there is no human validation (Section 7) leaves the construct validity unverified.

full rationale

The circularity is real but localized. The two novel political authenticity metrics are computed as similarity to centroids of the same labeled ParlaMint corpus used for fine-tuning, so both the fine-tuning improvements and the ANOVA 'validation' of discriminative power are partly definitional. However, the paper's additional evidence is not circular: PPL, Dist-N, Self-BLEU, GRUEN, BERTScore, MoverScore, and the Flow-Judge scores are computed with external reference models/references and show independent (if limited) support for fine-tuning gains. No load-bearing self-citation or uniqueness-theorem chain is used. The paper explicitly acknowledges the absence of human validation (Section 7, Limitations), which is a validity gap rather than circularity in itself. On balance, partial circularity in the headline political-authenticity claims warrants a 6, not a higher score, because independent metrics and benchmark resources carry substantive content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark's validity rests on hand-chosen orientation scales, filtering thresholds, an unvalidated LLM judge, and the assumption that embedding centroids represent ideology. These are reasonable heuristics but are not externally grounded; they should be treated as assumptions rather than established facts.

free parameters (4)
  • Political orientation numeric scale = Far-left=-6, Centre=0, Far-right=+6
    Used in PSA Eq. (2) to compute ideological distance; equal spacing is an arbitrary hand-chosen mapping, not derived from data.
  • Minimum affiliation speech threshold = 1000 speeches
    Filter in Section 3.1 reduces 28 to 11 affiliations; affects dataset balance and which parties are evaluated.
  • PSA penalty functional form = max(0, 1 - Δφ/12)
    Linear distance penalty chosen ad hoc; the '12' is just the range of the chosen orientation scale.
  • Generation decoding hyperparameters = temperature=0.7, top_p=0.85, repetition_penalty=1.2
    Hand-selected in Section 5.3; affect output diversity and quality and hence all downstream evaluation.
assumptions (5)
  • domain assumption Political orientation labels assigned to UK parties (e.g., Conservative=Centre-right, Labour=Centre-left) are correct and stable.
    PSA and orientation-level analyses in Sections 6.2-6.4 and Appendix 9.1 assume these labels; mislabeling would invalidate metric interpretation.
  • domain assumption Embedding centroids of party/orientation groups are meaningful representations of ideological position.
    PSA/Party Align (Section 4.1.1) assume cosine similarity to centroid captures ideological closeness, following Rheault & Cochrane (2020) but without validation in this setting.
  • domain assumption Flow-Judge-v0.1's scores are valid proxies for human judgments of speech quality and political authenticity.
    All LLM-judge metrics (J_Coh, J_Conc, J_Rel, J_Auth, J_PolApp, J_Qual) rely on this; no human agreement is measured (Section 7 Limitations).
  • domain assumption The retained 27,560 generated speeches are representative of each model's true output distribution.
    Section 5.3 excludes failed generations and incomplete judge ratings; if failures are systematic, cross-model comparisons are biased.
  • domain assumption EuroVoc topic labels obtained via CAP mapping/KEVLAR are accurate.
    Topic-level results (Section 6.3) assume correct topic classification; mapping errors propagate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech." pith.science (2026). https://pith.science/paper/SR4RAUDM

@misc{pith2026251108247,
  author       = {Pith},
  title        = {Pith review of: ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SR4RAUDM}},
  note         = {Machine review of arXiv:2511.08247}
}
read the original abstract

Parliamentary speech generation presents specific challenges for large language models beyond standard text generation tasks. Unlike general text generation, parliamentary speeches require not only linguistic quality but also political authenticity and ideological consistency. Current language models lack specialized training for parliamentary contexts, and existing evaluation methods focus on standard NLP metrics rather than political authenticity. To address this, we present ParliaBench, a benchmark for parliamentary speech generation. We constructed a dataset of speeches from UK Parliament to enable systematic model training. We introduce an evaluation framework combining computational metrics with LLM-as-a-judge assessments for measuring generation quality across three dimensions: linguistic quality, semantic coherence, and political authenticity. We propose two novel embedding-based metrics, Political Spectrum Alignment and Party Alignment, to quantify ideological positioning. We fine-tuned five large language models (LLMs), generated 28k speeches, and evaluated them using our framework, comparing baseline and fine-tuned models. Results show that fine-tuning produces statistically significant improvements across the majority of metrics and our novel metrics demonstrate strong discriminative power for political dimensions.

Figures

Figures reproduced from arXiv: 2511.08247 by the authors.

Figure 1
Figure 1. Methodology Overview: Data processing (top row) creates Dataset A from UK Parliament proceedings. Model development (middle row) fine￾tunes base architectures using Dataset A. Evalua￾tion pipeline (bottom row) applies the assessment framework to generated speeches (Dataset B) for performance analysis. Similarly, domain-specific benchmarks have emerged for legislative summarization (Kornilova and Eidelman, 2019), opi… view at source ↗
Figure 2
Figure 2. Dataset statistics: (a) Speech length dis [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Absolute performance changes (fine￾tuned vs. baseline) across evaluation categories. All metrics normalized to comparable scales [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Party alignment difficulty scores (0-1 scale) for finetuned models across UK parliamen￾tary parties. Dark green = most difficult showed highest performance. Detailed scores in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 4
Figure 4. Figure 4: Party alignment scores (0-1 scale) for finetuned models across UK parliamentary parties. Color intensity indicates performance level Party alignment Difficulty Analysis. Applying cross-context stability analysis (Eq.4), party diffi￾culty scores ranged narrowly (0.382-0…
Figure 6
Figure 6. Figure 6: Model performance scores (0-1 scale) for finetuned models across topic domains. Color intensity indicates performance level Topic Difficulty Analysis. Different topics posed different challenges ( [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Model performance scores (0-1 scale) for finetuned models across political orientations. Color intensity indicates performance level 7. Discussion and Conclusion To the best of our knowledge, ParliaBench repre￾sents the first benchmark resource addressing the specializ…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 1 linked inside Pith

  1. [1]

    Perplexity(PPL) (Jelinek et al., 1977) mea- sures text naturalness

    Linguistic Quality (a)Computational Metrics i. Perplexity(PPL) (Jelinek et al., 1977) mea- sures text naturalness. We compute perplex- ity using GPT-2 as a fixed reference model across all generated speeches, ensuring cross- model comparability. Lower scores indicate more natural-sounding text according to GPT- 2’s language distribution. (↓better) ii. Dis...

    1977

  2. [2]

    Semantic Coherence (a)Computational Metrics i. GRUEN(Zhu and Bhat, 2020) evaluates lin- guisticqualitythroughgrammaticalcorrectness andsemanticcoherence( ↑better),aggregating four quality dimensions: grammaticality, non- redundancy, focus, and structural coherence. ii. BERTScore(Zhang et al., 2020) measures se- mantic similarity using RoBERTa-large embed-...

    2020

  3. [3]

    We employ two novel embedding-based met- rics for political authenticity assessment: Po- litical Spectrum Alignment (PSA) and Party Alignment (Party Align)

    Political Authenticity (a)Computational Metrics i. We employ two novel embedding-based met- rics for political authenticity assessment: Po- litical Spectrum Alignment (PSA) and Party Alignment (Party Align). Detailed calculation methodology provided in Section 4.1.1. (b)LLM-Judge Metrics i. Authenticity(J_Auth) assesses whether con- tent reflects genuine ...

    2020

  4. [4]

    Evaluation Framework Parliamentary speech evaluation requires assess- ment across multiple levels that generic bench- marks cannot capture. Our framework operates on two levels: (i)speech evaluation metricsmea- suring generation quality across three dimensions, and (ii)consistency evaluation metricsmeasur- ing performance reliability across political cont...

    2023

  5. [5]

    Experimental Setup 5.1. Model Architecture Selection We selected five language models representing distinct architectural approaches to establish base- line performance for parliamentary speech gener- ation:Mistral 7B v0.3(Jiang et al., 2023) uses Grouped Query Attention and Sliding Window At- tention for efficient long-context processing.Llama 3.18B(Dube...

    2023

  6. [6]

    This section presents fine-tuning effectiveness and performance patterns across political parties, topic domains, and ideo- logical orientations

    Results and Analysis We evaluated 27,560 generated speeches using our evaluation framework. This section presents fine-tuning effectiveness and performance patterns across political parties, topic domains, and ideo- logical orientations. Representative examples of generated speeches are provided in Appendix 12. 6.1. Overview and Fine-Tuning Impact Table 2...

  7. [7]

    Par- liaBench provides standardized evaluation proto- cols and baseline performance results that sup- port systematic comparison and reproducible re- search in the field

    Discussion and Conclusion To the best of our knowledge, ParliaBench repre- sents the first benchmark resource addressing the specialized challenges of parliamentary speech generation, comprising dataset, evaluation frame- work,novelmetrics,andbaselinebenchmarks. Par- liaBench provides standardized evaluation proto- cols and baseline performance results th...

  8. [8]

    Bibliographical References Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Jo- han Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, P...

    2024

Show all 46 references
  1. [9]

    AI, :, Alex Young, Bei Chen, Chao Li, Chen- gen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pen...

    2025 arXiv

  2. [10]

    The system employs Flow-Judge-v0.1, a 3.8B parameter evaluation model, to score speeches across six dimensions using a 10-point scale

    LLM-as-a-Judge Evaluation Prompts This appendix documents the automated evaluation system used to assess the quality of generated parliamentary speeches. The system employs Flow-Judge-v0.1, a 3.8B parameter evaluation model, to score speeches across six dimensions using a 10-p...

  3. [11]

    Dataset Statistics Table 3 presents detailed breakdown of speeches by political affiliation in the Parliamentary Debates Benchmark

    Dataset Details 9.1. Dataset Statistics Table 3 presents detailed breakdown of speeches by political affiliation in the Parliamentary Debates Benchmark. Party Orientation Speeches Speakers Active Period Conservative Centre-right 263,513 792 2015–2022 Labour Centre-left 108,831...

    2015

  4. [12]

    Address the debate on Domestic Renewable Energy on ENERGY

    Representative Generated Speeches We present various speeches generated by the models (baseline and finetuned). Baseline Model Speech - Gemma Political Party:Scottish National Party Topic:ENERGY —Section:Domestic Renewable Energy House:House of Commons —Orientation:Centre-left...

    2018

  5. [13]

    Before scoring, analyze the speech structure step-by-step

  6. [14]

    Quote specific evidence or patterns from the speech supporting your assessment

  7. [15]

    Use the full 1-10 range - don’t default to middle scores (5-7) without justification

  8. [16]

    ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text

    Pay attention to how well arguments connect and whether parliamentary conventions are maintained. ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text. - Write the numeric score inside <score> tags, always after the feed...

  9. [17]

    Identify the main argument and check for focus vs digressions

  10. [18]

    Assess if courtesy and formality serve rhetorical purpose (parliamentary context)

  11. [19]

    Detect unnecessary repetition or verbose phrasing

  12. [20]

    Remember: Parliamentary speeches are naturally longer than everyday communication - adjust expectations accordingly

  13. [21]

    {{PROMPT}}

    Quote examples showing appropriate length or excessive verbosity. ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text. - Write the numeric score inside <score> tags, always after the feedback. Please accurately evaluate...

  14. [22]

    Identify what specifically is being asked in the prompt

  15. [23]

    Check coverage of each requirement mentioned in the prompt

  16. [24]

    Assess specificity (concrete details vs vague generalities)

  17. [25]

    Detect tangents and estimate percentage of off-topic content

  18. [26]

    For generic prompts, evaluate if response matches expected political context

  19. [27]

    ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text

    Quote evidence showing relevance or lack thereof. ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text. - Write the numeric score inside <score> tags, always after the feedback. Please accurately evaluate the speech. Str...

  20. [28]

    Read the speech twice: first for overall impression, second for detailed analysis

  21. [29]

    Mr. Speaker

    Check if "Mr. Speaker" and appropriate House references are used naturally (not forced)

  22. [30]

    Assess if language sounds natural or scripted/polished

  23. [31]

    Detect AI patterns: generic phrases, perfect grammar, corporate-speak, marketing tone

  24. [32]

    ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text

    Quote specific evidence showing authenticity or artificiality. ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text. - Write the numeric score inside <score> tags, always after the feedback. Please accurately evaluate th...

  25. [33]

    Consider the party’s known positions on this topic based on their political orientation

  26. [34]

    Check if speech aligns with expected party stance and framing

  27. [35]

    Assess if language and rhetoric are appropriate for this specific party

  28. [36]

    Detect contradictions or use of opposing party’s rhetoric

  29. [37]

    ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text

    Quote evidence showing appropriate or inappropriate party alignment. ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text. - Write the numeric score inside <score> tags, always after the feedback. Please accurately evalu...

  30. [38]

    Assess argumentation quality: Is logic sound? Are claims supported?

  31. [39]

    Evaluate evidence: Does speech use specific data, examples, or reasoning?

  32. [40]

    Check rhetorical devices: Are there memorable phrases or effective persuasion techniques?

  33. [41]

    Consider potential impact: Would this speech influence other parliamentarians?

  34. [42]

    Be tough: Real parliamentary speeches often score 6-8, not 9-10

  35. [43]

    ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text

    Quote evidence showing strong or weak argumentation. ## FORMAT FOR THE EVALUATION - Write your verbal feedback inside <feedback> tags without any surrounding text. - Write the numeric score inside <score> tags, always after the feedback. Please accurately evaluate the speech. ...

  36. [44]

    Setup Implementation Details 11.1. Model Architecture Details Model Quantized Version Memory Reduction Inference Speed Key Features Mistral 7B v0.3 mistral-7b-v0.3-bnb-4bit 62% 2.2×GQA, SWA Llama 3.1 8B Meta-Llama-3.1-8B-bnb-4bit 58% 2.4×128k context Gemma 2 9B gemma-2-9b-bnb-...

  37. [46]

    Party-Specific Performance Table 8 presents detailed performance scores for all UK parliamentary parties across fine-tuned models

    Statistical Analysis and Detailed Results 13.1. Party-Specific Performance Table 8 presents detailed performance scores for all UK parliamentary parties across fine-tuned models. Party Gemma 2 9B Llama 3.1 8B Mistral 7B v0.3 Qwen2 7B YI 6B Average Std Bishops 0.609 0.624 0.550...

  38. [2013]

    In Andrea Volkens, Judith Bara, Ian Budge, Michael D

    Understanding and validating the left-right scale (rile). In Andrea Volkens, Judith Bara, Ian Budge, Michael D. McDonald, and Hans- Dieter Klingemann, editors,Mapping Policy Pref- erences From T exts: Statistical Solutions for Manifesto Analysts, chapter 6, pages 85–105. Oxfor...

    2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.