Pith. sign in

REVIEW 5 major objections 5 minor 14 references

ConVAWG: A Retrieval-Grounded Framework for Controlled Synthetic Dialogue Generation in Violence Against Women and Girls

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces ConVAWG, a retrieval-grounded four-stage pipeline that generates synthetic multi-turn chat dialogues about violence against women and girls, and claims these dialogues score higher on dialogue-level quality than…

desk verdict A genuinely new synthetic VAWG dialogue resource with a careful evaluation, but the realism claim outruns the evidence and the generation machinery is withheld. read the letter →

arxiv 2608.11200 v1 pith:KXPN5R2G submitted 2026-08-11 cs.CL cs.AIcs.LG

Chen Lyu , Xingwei Tan , Simon Cullen , Shelley Wilson , Lois Arthurs , Arshad Jhumka , Gabriele Pergola This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords syntheticdialoguegenerationviolenceagainstwomenandgirlsdomesticabusecoercivecontrolescalationforecastingtoxicityretrieval-groundedLLM-as-Judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ConVAWG, a four-stage pipeline for generating synthetic multi-turn chat dialogues depicting violence against women and girls (VAWG). Its aim is to show that abuse can be modelled as a relational, temporally escalating conversation rather than isolated toxic sentences, and that structured retrieval grounding produces dialogues that read as more realistic and domain-faithful than those written directly by a large language model. The authors argue this matters because real VAWG conversation data is scarce, private, and hard to release, so a high-quality synthetic resource could support research on abuse detection, escalation forecasting, and safety evaluation without exposing victims' data.

What carries the argument

The load-bearing mechanism is the four-stage pipeline: (1) scenario construction from PersonaHub persona seeds subsampled to match ONS victim demographics, CPS crime definitions, and retrieved DHR case patterns; (2) conversion of the scenario outline into a hierarchical directed event graph with 5-8 composite events, each decomposed into 2-4 sub-events carrying a timestamp and an escalation level e in {0,1,2,3,4}; (3) role-play generation of the chat scripts with retrieval-conditioned style notes along persona, escalation, and crime-type axes plus a continuity context block; and (4) targeted toxicity injection that rewrites only LLM-labelled perpetrator escalation utterances using Contrastive Activation Addition (CAA) steering, with strength calibrated to the event's escalation level. The escalation level is the control signal that coordinates event decomposition, temporal spacing, dialogue style, interaction length, and toxicity intensity.

What would settle it

Present human experts in domestic abuse and VAWG with a forced-choice test: pairs of dialogues, one ConVAWG output and one genuinely de-identified or simulated counterpart drawn from real DHR case communications, and ask experts to identify the real one. If experts cannot do better than chance, the realism claim is supported; if they reliably and consistently identify ConVAWG dialogues as synthetic (e.g., above 80% accuracy), the claim of realism would be falsified.

Watch

Extended reading notes

Core claim

ConVAWG claims that generating VAWG dialogues from an explicit scenario specification—persona seeds matched to ONS victim statistics, CPS crime definitions, retrieved Domestic Homicide Review patterns, a hierarchical event graph with escalation levels, and retrieval-conditioned style notes—yields dialogues that outperform eight baselines, including direct GPT-5.2 generation, on dialogue-level quality under four calibrated LLM judges (mean Avg-D 4.75 vs. 4.64 for DiaSynth and 4.57 for GPT-5.2), with human annotation on a shared subset ranking ConVAWG highest (4.62 vs. 4.33 and 4.28). The released dataset, 6,171 dialogue events across 200 scenarios with scenario-, event-, and turn-level metadata, supports controllability validation (toxic behaviour classification with 0.782 macro-F1) and external utility (escalation forecasting that beats persistence on jumps, and cross-corpus transfer above chance to Conversations Gone Awry).

Load-bearing premise

The central claim depends on the assumption that six human annotators and four aligned LLM judges can validly rate 'realism' and 'domain fidelity' on a 1-5 scale, and that those ratings—despite most being 4 or 5—track meaningful differences in how faithfully the synthetic dialogues reflect real VAWG conversational dynamics.

Editorial extensions

If this is right

  • If the framework works as claimed, researchers gain a permission-safe resource for studying abuse as a multi-turn, temporally unfolding phenomenon, with labels (escalation, crime type, behaviour, relationship) attached at scenario, event, and turn granularity.
  • The controllability-validation results imply that generation-conditioning labels are recoverable from dialogue text, so the dataset can serve as training supervision for fine-grained abuse analysis without further annotation.
  • Escalation forecasting that detects imminent jumps (0.548 jump F1, with one third of first severe events flagged at zero false alerts) suggests textual precursors of escalation exist and can be learned, which is relevant to early-warning research.
  • Cross-corpus transfer above chance on Conversations Gone Awry indicates that at least some learned signals of conversational breakdown are artefact-independent, supporting the use of synthetic VAWG dialogues in studies that transfer to real settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ceiling compression in ratings (57.5% of all pooled ratings are exactly 5) implies the 1-5 Likert rubric may not separate strong systems; a reasonable extension would be pair-wise preference judgements or fine-grained critique annotations, which the paper does not report.
  • Because the grounding sources—CPS definitions, ONS statistics, DHR reports—are UK-specific, the framework should transfer to other jurisdictions only after replacing those sources; a testable extension is regenerating scenarios with, say, US or EU crime statistics and checking whether judge scores and downstream task performance remain comparable.
  • The CAA-based toxicity steering is calibrated per model and its vectors are withheld; a possible test is whether the same steering direction transfers across backbone models trained on similar data, which would indicate a generalisable 'toxic register' rather than a model-specific artefact.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces ConVAWG, a four-stage framework for generating synthetic multi-turn chat dialogues about Violence Against Women and Girls (VAWG). Stage 1 constructs persona- and scenario specifications using PersonaHub seeds, ONS statistics, CPS crime definitions, and retrieved Domestic Homicide Review case patterns; Stage 2 plans hierarchical event graphs with escalation levels; Stage 3 role-plays these into online chat dialogues with retrieval-conditioned style notes; Stage 4 selectively rewrites perpetrator utterances using Contrastive Activation Addition for toxicity control. The authors release a dataset of 6,171 dialogue events across 200 scenarios with rich metadata, and evaluate it with human annotation, LLM-as-Judge ratings, ablations, and downstream tasks. The central claims are that ConVAWG generates realistic, CPS-aligned, domain-faithful VAWG dialogues, that it outperforms eight baselines on dialogue-level quality (Avg-D 4.75 vs. 4.64 for DiaSynth and 4.57 for GPT-5.2), and that the dataset supports escalation forecasting, toxic behaviour classification, and transfer to real corpora.

Significance. If the realism and domain-fidelity claims hold, ConVAWG would be a valuable resource for studying abuse as a relational and temporally unfolding conversational phenomenon, and the dataset release would support a range of downstream safety-oriented NLP tasks. The paper has several genuine strengths: it openly separates controllability validation from external utility in Section 5.5; it uses paired persona-level nonparametric tests with Holm correction and reports full rating distributions, explicitly acknowledging ceiling effects in Appendix E; and the human-LLM alignment (AC2 >= 0.925) is carefully quantified. However, the evidence for the realism claim is largely rubric-based and internally referenced, with only modest and partly confounded external anchors. As it stands, the paper convincingly demonstrates internal controllability and high judged consistency, but it does not yet establish that the synthetic dialogues faithfully reflect real VAWG conversational dynamics. The resource is still likely to be useful, but the central claim needs to be either strengthened with external validation or reframed to match the evidence.

major comments (5)
  1. [Section 5.3, 5.5, Appendix F] The claim that ConVAWG produces 'realistic' and 'domain-faithful' dialogues is not anchored in any real VAWG conversational data. The primary evidence is rubric-based: human and LLM judges rate coherence, humanlikeness, persona consistency, toxicity realism, crime fidelity, and scenario realism relative to the generated scenario and CPS taxonomy, so high scores largely reward faithful execution of conditioning signals. The external anchors are weak: zero-shot derailment transfer reaches AUROC 0.643/0.561 (Table 8), and the MentalManip transfer result (Table 9) is confounded because the escalation labels used to define positives (escalation >= 2) were generated by the same pipeline and conditioned the dialogue generation; the monotonic positive rates 0.37/0.56/0.94/0.99/1.00 could reflect the toxicity injection schedule rather than genuine realism. I recommend either tempering the realism claim to conclusively state what is measured (controllability and internal consistency) or adding an evaluation against real conversational data or expert judgments of real-versus-synthetic samples.
  2. [Appendix E, Section 5.3] The ceiling compression is severe: Appendix E reports that 57.5% of all pooled LLM-judge ratings are exactly 5 and 89.3% are 4 or above. With this compression, the 0.11-point Avg-D lead over DiaSynth and the 0.18-point lead over GPT-5.2 are fragile, and the per-metric Holm-corrected tests show no significant advantage on Coherence (Gemini ahead), Persona Consistency (DiaSynth/SPASM tied), Crime Fidelity (GPT-5.2/Gemini tied at the ceiling), or Scenario Realism (GPT-5.2 and Gemini higher). The headline 'better than baselines' claim therefore effectively rests on Humanlikeness and Toxicity Realism. I recommend reporting additional discriminative analyses, such as forced-choice pairwise comparisons, error-rate analyses, or calibration checks, that can separate systems despite the ceiling, and qualifying the abstract's overall-quality claim accordingly.
  3. [Appendices B.1, B.2, D, H] Large parts of the pipeline are withheld: the DHR record schema and retrieval configuration, the scenario output schema and consistency-refinement procedure, the CAA steering layer choice, the level-specific coefficients (alpha1, alpha2, alpha3), the pair-construction constants, and the verbatim Stage 1-3 generation prompts. Because the contribution is a framework, these omissions prevent the reader from reproducing or auditing the pipeline, including the toxicity injection that underlies the Toxicity Realism results. Providing the LLM-as-Judge prompt in full (Appendix H.6) is not sufficient. I ask that the withheld material be made available to reviewers, or at minimum that a detailed technical appendix be supplied, with any dual-use exclusions explicitly scoped to the steering vectors and toxification prompts.
  4. [Section 5.5, 5.6, Conclusion] The controllability validation in Section 5.5 is explicitly circular: models are trained and tested on ConVAWG using labels that conditioned generation, so recovering those labels (0.782 macro-F1 for toxic behaviour classification, 0.629 weighted F1 for relationship prediction) demonstrates that the pipeline implements its control signals, not that the dialogues are externally realistic. The paper itself distinguishes this from external utility, which is good, but the abstract and conclusion nevertheless present 'domain fidelity' and 'realism' as established by the full evaluation. I recommend that the summary statements be rewritten to present the evidence as controllability plus a limited external-transfer signal, and that the circularity be acknowledged at each point where the controllability results are cited in support of realism.
  5. [Section 5.4, Table 10/11/12] The ablations cover style notes, CAA steering, and backbone choice, but there is no ablation that removes the DHR retrieval or the CPS grounding. Since 'retrieval-grounded' and 'real-case-guided' are central design claims, the paper should test whether retrieved DHR patterns actually change the outputs, for example by comparing the full pipeline against a variant with the retrieval context removed while holding everything else fixed. Without this, the reader cannot tell whether the gains over direct generation come from the structured event planning, the persona conditioning, or the retrieval grounding.
minor comments (5)
  1. [Abstract] The abstract contains a grammatical error: 'make it difficult the release of large-scale real conversation datasets' should be 'make it difficult to release large-scale real conversation datasets'.
  2. [Section 5.3, Table 13] The text says ConVAWG 'unfolds each scenario into ~34 dialogues' while Section 5.1 reports an average of 31 dialogues per scenario; Table 13 clarifies that 34.2 is on the matched 50-persona subset and the full release averages 30.9, but the main text should state this explicitly to avoid an apparent inconsistency.
  3. [Abstract and Section 3.1] The phrase 'retrieved Domestic Homicide Review cases' could be read as implying verbatim case reuse; the methodology actually retrieves patterns and structured summaries from DHR reports. Consider rewording to 'patterns from retrieved DHR reports'.
  4. [Appendix F, MentalManip alignment] The definition of ConVAWG positives as dialogues with escalation >= 2 should be justified, because it directly affects the reported positive-rate monotonicity and macro-F1 scores; the paper should state whether this threshold was chosen before or after observing the detector's behaviour.
  5. [Appendix F, Table 5] Table 5 does not mark the best result in each column; adding bolding or a note would improve readability, especially since the text reports Gemini as the strongest zero-shot model while BERT-base is best among fine-tuned encoders.

Circularity Check

3 steps flagged · score 5.0 of 10

Realism evidence is partly self-referential: Scenario Realism scores alignment to the pipeline's own scenario, controllability tasks recover conditioning labels, and the MentalManip monotonicity is confounded with the Stage 4 injection schedule.

  1. self definitional [Appendix C, Evaluation Metrics — 'Scenario Realism' definition; used in Section 4 RQ1 and Table 1]
    "Scenario Realism. Measures the degree to which the generated dialogue aligns with the provided scenario summary. High scores indicate that key scenario elements—such as relationship context, major events, escalation patterns, and outcomes—are faithfully reflected in the dialogue while maintaining plausible narrative progression."

    The 'provided scenario summary' is itself a Stage 1 generated input (persona seeds, DHR retrieval, CPS definitions), and Stage 3 is prompted to realize that summary. A rating of alignment between output and input therefore measures the pipeline's own controllability, not correspondence to real VAWG conversational dynamics. The LLM judge prompt makes the circularity explicit: 'Judge alignment to this specific scenario -- not general VAWG realism.' High Scenario Realism scores cannot, by definition, support the paper's external realism claim; they reward faithful execution of the conditioning scenario.

  2. self definitional [Section 5.5, Downstream Utility — Controllability Validation; Appendix F Tables 5-6]
    "Controllability Validation. We test whether the labels used to condition generation can be recovered from dialogue text through two tasks: message-level toxic behaviour classification and victim-perpetrator relationship prediction. Fine-tuned encoders recover both signals effectively, achieving 0.782 macro-F1 and 0.629 weighted F1, respectively... Because these labels also guided generation, the results indicate that ConVAWG reliably realises its control signals and provides usable supervision for fine-grained abuse analysis."

    The supervision labels (toxic behaviour, relationship type, escalation) are the same signals that conditioned Stage 2-4 generation. Training and testing on ConVAWG with those labels recovers the conditioning inputs, so the reported scores are a controllability check by construction. The paper explicitly acknowledges this and quarantines it from 'external utility,' which mitigates the flaw; nevertheless, the step is self-referential and would be circular if presented as evidence of external validity.

1 more flagged steps
  1. other [Section 5.5.1, External Utility — Real-data transfer and augmentation; Appendix F Table 9; Stage 4 toxicity schedule (Section 3.4)]
    "Presented in Appendix F Table 9, a detector trained only on MentalManip identifies ConVAWG dialogues as manipulative (0.755 macro-F1), and its positive prediction rate increases monotonically with escalation level. This provides independent, human-grounded evidence for both dialogue realism and annotation validity."

    The monotonic escalation-toxicity link is built into the pipeline: Stage 4 'applies activation-steered rewriting only to selected perpetrator utterances associated with escalation' with 'strength aligned with the escalation level of the source event, yielding low, medium, and high toxicity settings for increasingly severe interactions,' and Stage 3 style notes shift perpetrator language with escalation. A toxicity/manipulation detector's positive rate is therefore expected to rise with ConVAWG escalation levels regardless of real-world fidelity. The claimed 'independent, human-grounded evidence' is not independent of the generation controls; the monotonic pattern is partly a design artifact.

full rationale

The paper is not wholly circular. It separates controllability validation from external utility, and it provides genuinely external anchors: zero-shot transfer to Conversations Gone Awry is above chance (best AUROC 0.643/0.561), human annotators on a shared subset rank ConVAWG above two strong direct baselines, and the backbone ablation shows a model-quality gradient. However, the central 'realism and domain fidelity' claim leans on rubric dimensions that are defined relative to the pipeline's own generated scenario and escalation labels. Scenario Realism is explicitly alignment with the provided (generated) scenario; Toxicity Realism is judged against 'appropriate escalation points' that the pipeline itself constructs; and the MentalManip monotonic evidence is confounded with the Stage 4 injection schedule. The LLM-as-Judge rubrics are also anchored to the scenario brief and CPS taxonomy, so high scores partly measure internal consistency rather than external conversational realism. The ceiling compression (57.5% of ratings exactly 5) further weakens the small Avg-D lead, but this is a measurement concern, not circularity. Overall, the derivation chain is partially self-referential but retains independent content through the external transfer experiments and human baseline comparisons, so a moderate circularity score of 5 is appropriate.

Assumptions & free parameters 6 free parameters · 6 assumptions · 1 invented entities

The central contribution is a generation pipeline, so the ledger is dominated by pipeline parameters that are either fitted (CAA coefficients and layer) or chosen by hand (spacing schedule, similarity thresholds, ONS subsampling), many of which are withheld. The key domain assumptions are that ONS/CPS/DHR sources are valid grounding, GPT-5.2 executes the pipeline faithfully, and the human/LLM judge calibration generalizes. The invented escalation-level variable is an internal construct with no independent validation.

free parameters (6)
  • CAA steering coefficients (alpha1, alpha2, alpha3) = withheld
    Calibrated per model and per toxicity level (L1-L3) using RealToxicityPrompts and Detoxify, with layer sweep and alpha sweep (Appendix D). The fitted values are withheld from the preprint.
  • CAA steering layer l* = withheld
    Selected via a per-model layer sweep over targets, the exact choice is withheld (Appendix D.1).
  • Persona lexical filter thresholds and ONS subsampling proportions = not specified
    Stage 1 uses a lightweight lexical filter to retain personas for women/girls and subsamples according to ONS victim characteristics. The exact thresholds and target proportions are not given.
  • Escalation schedule and event spacing parameters = not specified
    Stage 2 uses a schedule mimicking real cases: early events sparse, later high-risk events dense. The quantitative spacing parameters are not given.
  • Sentence-embedding similarity threshold for splitting long responses = not specified
    Stage 3 post-processes overly long responses into shorter chat turns using an adaptive sentence-embedding similarity threshold; the threshold value is not reported.
  • Detoxify threshold and pair-construction constants for CAA = withheld
    Appendix D withholds toxicity thresholds, length/embedding-similarity filters, embedding model, and split seed used to construct toxic-clean contrastive pairs.
assumptions (6)
  • domain assumption ONS victim statistics and CPS VAWG definitions accurately represent real VAWG dynamics in the UK.
    Stage 1 grounds persona and scenario generation in ONS statistics and CPS definitions (Section 3.1). If these sources misrepresent the population or legal categories, the generated scenarios inherit the bias.
  • domain assumption Public DHR reports, as extracted by an LLM information-extraction pipeline, are a valid source of abuse-trajectory patterns.
    DHR records are converted into structured records with an LLM-based extraction pipeline (Appendix B.1). The extraction quality is not evaluated, and the schema is withheld. Errors in extraction propagate to scenario construction.
  • domain assumption GPT-5.2, the backbone model, can reliably perform the four generation stages (scenario construction, event graph extraction, role-play, and rewriting).
    All generative stages use GPT-5.2 (Section 3). The paper's entire pipeline depends on this model behaving as a faithful executor of the stage prompts.
  • domain assumption Toxicity can be validly represented as a single steering direction estimated from Civil Comments pairs and applied via CAA.
    Stage 4 and Appendix D assume that contrastive activation addition on a Civil-Comments-derived direction produces contextually appropriate abusive language across VAWG scenarios. This is a strong assumption about the transfer of a toxicity direction across domains.
  • domain assumption LLM judges, calibrated against six human annotators, provide valid quality scores for the full dataset.
    The paper reports high human-LLM agreement (AC2 >= 0.925) and uses the mean of four judges as the primary evaluation signal (Section 5.2). This assumes the judges do not systematically favor fluent or self-similar outputs in ways the calibration misses.
  • domain assumption Six female annotators are a representative and sufficient panel for judging realism of VAWG dialogues.
    Appendix A states six annotators from a pool, with two excluded. This small panel grounds the calibration of all LLM judges and the human evaluation on the shared subset.
invented entities (1)
  • Escalation level variable e in {0,1,2,3,4}
    purpose: A discrete control signal that drives event decomposition, temporal spacing, style notes, interaction length, and toxicity strength across the pipeline.
    The five-level rubric is defined by the authors and assigned by prompting an LLM with the rubric. It is not derived from an external validated scale, and there is no outside-the-paper validation of the levels themselves.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ConVAWG: A Retrieval-Grounded Framework for Controlled Synthetic Dialogue Generation in Violence Against Women and Girls." pith.science (2026). https://pith.science/paper/KXPN5R2G

@misc{pith2026260811200,
  author       = {Pith},
  title        = {Pith review of: ConVAWG: A Retrieval-Grounded Framework for Controlled Synthetic Dialogue Generation in Violence Against Women and Girls},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KXPN5R2G}},
  note         = {Machine review of arXiv:2608.11200}
}
read the original abstract

Synthetic dialogue generation offers a way to study conversational dynamics in sensitive domains where real data are difficult to access, release, or annotate. The underlying abuse may occur online or offline: threats and coercion can appear directly in messages, while behaviours such as surveillance, isolation, stalking, and physical violence may be planned, disclosed, or referred to conversationally. Privacy and legal constraints make it difficult the release of large-scale real conversation datasets; existing work has mostly focused on sentence-level toxicity of online abuses, leaving a gap in modelling abuse as a relational and temporally unfolding phenomenon. In this work, we focus on modelling Violence Against Women and Girls (VAWG) scenarios as multi-turn dialogues. We introduce ConVAWG, a retrieval-grounded framework for generating CPS-aligned synthetic VAWG chat dialogues. ConVAWG builds scenarios from persona seeds, demographic patterns reported by the UK Office for National Statistics, official crime definitions, and retrieved Domestic Homicide Review cases; converts them into hierarchical event timelines; generates multi-scene role-play dialogues; and applies targeted activation-steered toxicity control to appropriate utterances. We release over 6,000 multi-turn dialogue events across 200 scenarios with rich scenario-, event-, and turn-level metadata. Extensive human evaluation, LLM-as-Judge assessment, ablations, and downstream tasks show strong dialogue quality and domain fidelity.

Figures

Figures reproduced from arXiv: 2608.11200 by the authors.

Figure 1
Figure 1. ConVAWG generated conversations. This data access problem is particularly relevant in under-resourced domains such as Violence Against Women and Girls (VAWG), an umbrella term that covers both physical violence and non-physical offences such as domestic abuse, coercive con￾trol, stalking, harassment, honour-based abuse, and child sexual abuse, as defined by prosecution bod￾ies such as the Crown Prosecution Service (… view at source ↗
Figure 2
Figure 2. Overview of ConVAWG data generation framework. The framework constructs retrieval-grounded [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of LLM-judge ratings (1-5) per evaluation metric, pooled over the four calibrated judges on [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

  1. [1]

    Consent— participants read a Participant In- formation Sheet and confirm informed consent before accessing the task

    work page

  2. [2]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 7828–7840, Bangkok, Thailand

    Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment. InProceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 7828–7840, Bangkok, Thailand. Association for Computational Linguistics. Han Luo and Guy Laban. 2026. SPASM: Stable persona-...

    work page 2026

  3. [3]

    InAdvances in Neural Information Processing Systems 37 (NeurIPS 2024)

    LLM evaluators recognize and favor their own generations. InAdvances in Neural Information Processing Systems 37 (NeurIPS 2024). Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simu- lacra of human behavior. InProceedings of the 36th Annual ACM Symposium ...

    arXiv 2024

  4. [4]

    ATTENTION CHECK— Please flag the follow- ing utterance by clicking the checkbox next to it: ‘〈quoted utterance〉’

    Submission— participants clickComplete & Submitonce all conversations are annotated; the system verifies completeness before accepting the submission. A.2 Annotation Task Each annotated item is aconversation, consist- ing of one or moredialoguesthat correspond to sub-events of a V AWG scenario (e.g., E1.1 pre, E1.2during). Annotators were provided with th...

    work page arXiv 2024

  5. [5]

    4https://streamlit.io

    Guidelines— a step-by-step guidelines page explains the rating dimensions, the flagging task, the rating scale, and the attention-check proce- dure. 4https://streamlit.io

    work page

  6. [6]

    Progress is auto-saved every minute, allowing participants to pause and resume

    Annotation— participants annotate conversa- tions in any order via a sidebar navigation panel. Progress is auto-saved every minute, allowing participants to pause and resume

    work page

  7. [9]

    - Appropriate responses to previous utterances - Topic continuity - Absence of contradictions or abrupt shifts Judge internal conversational logic ONLY -- not scenario accuracy

    coherence Turn-level logical consistency and conversational flow. - Appropriate responses to previous utterances - Topic continuity - Absence of contradictions or abrupt shifts Judge internal conversational logic ONLY -- not scenario accuracy

    work page

  8. [10]

    humanlikeness Naturalness and conversational authenticity. - Linguistic fluency, natural phrasing - Emotional appropriateness - Realistic conversational behaviour in online chatting context - Treat typical chat traits (short replies, mild typos, slang, imperfect grammar) as normal unless they significantly harm clarity Focus on language quality and intera...

    work page

Show all 14 references
  1. [11]

    - Tone, vocabulary, behavioural patterns - Emotional reactions consistent with assigned profile - Alignment with character description Consider both speakers across all turns

    persona_consistency Stability of character traits and speaking style. - Tone, vocabulary, behavioural patterns - Emotional reactions consistent with assigned profile - Alignment with character description Consider both speakers across all turns. 23

  2. [12]

    coherence

    toxicity_realism Realism of toxic / abusive language (if any). - Emerges at contextually appropriate points - Progresses in intensity in a believable way - Aligns with relationship dynamics and situation Judge REALISM of timing and expression only -- NOT morality or toxicity l...

  3. [13]

    crime_fidelity Domain-level realism of VAWG dynamics across ALL dialogues. - Coercion and control behaviours - Escalation trajectories - Power imbalance - Victim-perpetrator interaction patterns Focus on domain authenticity -- not writing quality or scenario alignment

  4. [14]

    crime_fidelity

    scenario_realism Faithful and plausible alignment with the GIVEN SCENARIO across ALL dialogues. - Key elements reflected (relationship, ages, cultural background, vulnerabilities) - Logical consistency with described events - Realistic pacing and progression Judge alignment to...

    2015

  5. [2020]

    Higher values indicate greater detected toxic content

    toxicity scores across generation systems on the 50 matched personas. Higher values indicate greater detected toxic content. 21 H Pipeline Prompts To prevent premature replication of the generation pipeline ahead of the peer-reviewed release, the verbatim stage prompts are wit...

  6. [2024]

    Shengzhi Li and Nima Tajbakhsh

    Multi-document grounded multi-turn synthetic dialog generation.arXiv preprint arXiv:2409.11500. Shengzhi Li and Nima Tajbakhsh. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs.arXiv preprint arXiv:2308.03349. Zi Lin, Zihan ...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.