Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

The paper claims PACT, a synthetic personality-tagged argumentation-based negotiation corpus, is a high-quality training resource: fine-tuning on it matches human-human negotiation data and yields up to a 27.2% personality-accuracy gain.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:05 UTC pith:HZXZ6A3D

load-bearing objection Useful synthetic negotiation dataset and benchmark, but the 'high-quality' claim rests on partly self-confirming evaluation; worth refereeing, not uncritical adoption. the 4 major comments →

arxiv 2509.11118 v1 pith:HZXZ6A3D submitted 2025-09-14 cs.CL cs.AI

We Argue to Agree: Towards Personality-Driven Argumentation-Based Negotiation Dialogue Systems for Tourism

Priyanshu Priya , Saurav Dudhate , Desai Vishesh Yasheshbhai , Asif Ekbal This is my paper
classification cs.CL cs.AI
keywords argumentation-based negotiationpersonality-driven dialoguesynthetic datasettourism negotiationdialogue act predictionresponse generationlarge language modelspersonality recognition
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes a new task, Personality-driven Argumentation-based Negotiation Dialogue Generation (PAN-DG), and a supporting dataset, PACT, containing over 8,600 synthetic two-party tourism negotiations. Each dialogue encodes three personality profiles (argumentation, preference, buying style) and a sequence of 23 argumentation-specific dialogue acts, so the corpus is structured to teach models both personalization and persuasive reasoning. The authors' central claim is that PACT is high-quality enough to serve as a training resource: fine-tuning on it matches or beats fine-tuning on human-human negotiation data, and injecting personality, dialogue-act, and factual package knowledge during fine-tuning yields the strongest gains, including a 27.2% increase in personality accuracy and a 17.6% increase in dialogue-act accuracy for Vicuna. A sympathetic reader would care because personalized, justifiable negotiation is needed in tourism and other service domains, and the paper offers a cheap, repeatable way to generate such training data.

Core claim

The central discovery is that an entirely synthetic benchmark can, at least under this paper's evaluation, behave like a high-quality human negotiation corpus. PACT's filtering step keeps only conversations that GPT-4 experts rate perfect on quality, personality consistency, negotiation efficacy, argumentation efficacy, and non-toxicity, and manual evaluation corroborates the dataset's quality (e.g., personality relevance 3.62/4). When the authors fine-tune open LLMs on PACT, the models outperform pre-trained baselines and match or exceed models fine-tuned on the Job Interview human-human negotiation corpus. The best configuration, FT-Ours, couples fine-tuning with retrieval-augmented packag

What carries the argument

The load-bearing mechanism is the PACT generation pipeline: a curated database of 10 travel packages; random assignment of personality profiles (4 argumentation profiles, 10 preference profiles, 3 buying-style profiles); an ABN pathway encoder that converts negotiation dynamics into a sequence of 23 dialogue acts; one-shot prompting of Gemini-1.5-Flash to realize each act as an utterance; filtering by a GPT-4 mixture-of-experts that discards any conversation not rated perfect; and quality checks. This pipeline encodes the paper's core assumption, that a structured act-level plan plus a strict filter yields rational, personality-consistent negotiation behavior. The companion fine-tuning setup

Load-bearing premise

The labels and quality scores that define PACT are produced by the generating-and-filtering pipeline itself, so the paper's improvements may reflect models learning to match synthetic artifacts rather than true negotiation and personality competence.

What would settle it

Select a random subset of PACT and independently re-label traveler personality and dialogue acts by human annotators who never see the synthetic labels; if human-model agreement on personality is near chance (or far below the P-ACC classifier's 61.57% accuracy), the benchmark's central metric is measuring pipeline self-consistency rather than human-recognizable personality. Alternatively, run live negotiations where human users interact with FT-Ours models and rate personality alignment and satisfaction; if ratings do not exceed those for human-human data-trained models, the headline gains do

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Fine-tuning on PACT matches or outperforms fine-tuning on human-human negotiation data, making synthetic data a viable alternative when human corpora are scarce.
  • Injecting personality, dialogue-act, and package knowledge during fine-tuning produces large gains in personality alignment and dialogue-act consistency (e.g., Vicuna P-ACC +27.2%, D-ACC +17.6%), and reduces hallucination as measured by FEQA and QuestEval.
  • The three sub-tasks (personality recognition, dialogue act prediction, response generation) form a reusable benchmark for evaluating negotiation agents in tourism.
  • The generation-filtering-assessment pipeline is a template for creating similar synthetic dialogue datasets in other domains and languages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's personality and dialogue-act labels are products of the same LLM pipeline that generates the conversations, so the headline accuracy gains may partly measure how well a model mimics the pipeline's own stylistic conventions rather than human psychological traits; a human-label validation study would clarify.
  • The strict 'perfect score from all experts' filter likely prunes away rare but realistic negotiation behaviors; relaxing it with diversity-aware selection could broaden coverage of acts like Justify-X, which currently accounts for only 0.1% of dialogue acts.
  • Testable extension: apply FT-Ours-style fine-tuning to out-of-domain negotiation corpora (e.g., job offers or real-estate bargaining) to see whether PACT's benefits transfer beyond tourism.
  • The same act-sequence encoding could serve as an interpretable control interface for other goal-oriented dialogue systems, letting developers steer generated responses toward desired argumentation structures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a new task, Personality-driven Argumentation-based Negotiation Dialogue Generation (PAN-DG), and a new dataset, PACT, consisting of 8,687 LLM-generated tourism negotiation dialogues annotated with three personality profile families (Argumentation, Preference, Buying Style) and 23 ABN-specific dialogue acts. The construction pipeline uses Gemini-1.5-Flash for generation, a GPT-4 mixture-of-experts filter that retains only conversations receiving maximum quality scores, and then automatic plus manual quality assessment. The paper reports that PACT is high quality and that fine-tuning on PACT with additional background, personality, and dialogue-act information (FT-Ours) substantially improves Personality Recognition, Dialog Act Prediction, and Response Generation, with Vicuna achieving, e.g., P-ACC 0.789, D-ACC 0.841, and a human Personality Relevance score of 3.92/4.

Significance. If the claims are validated, PACT would be a useful resource: it is the first tourism-domain negotiation dataset with an argumentation-based dialogue-act taxonomy, integrates multiple personality facets beyond Big Five, and provides open resources and a reproducible generation pipeline. The paper also contributes strong baselines and a multi-task framework for the three sub-tasks. These are real potential strengths. However, the headline claims rest in large part on evaluation instruments that are trained on or filtered by the same synthetic pipeline they are meant to validate. The evidence for the central 'high-quality' and 'personality-driven' claims is therefore currently self-confirming, and the human evaluation is not independent of dataset creation. The contribution would be significant if the authors supply external validation of the dataset and the evaluation metrics.

major comments (4)
  1. [§3.2 and §3.3, Table 1] The quality assessment of PACT is not independent of the filtering procedure. Section 3.2 retains only conversations that receive the maximum score from all five GPT-4 experts, and Section 3.3 then reports high FED and G-Eval scores on the remaining data. FED and G-Eval are themselves LLM-based judgments, so the scores in Table 1 largely confirm that the retained data matches the same judge family that selected it. This is a circularity concern for the claim that PACT is 'high-quality'. Please provide evidence that the quality scores are not an artifact of selection, e.g., by reporting FED/G-Eval on the pre-filtering sample, or by comparing against independent human judgments on a held-out sample.
  2. [Appendix G.5, Table 5] P-ACC and D-ACC, which carry the claim of personality-driven generation, are computed by RoBERTa classifiers fine-tuned on PACT's own synthetic labels. The P-ACC classifier reaches only 61.57% accuracy and 58.31% macro-F1 on the original labels, yet FT-Ours Vicuna reports P-ACC 0.789. This suggests that the metric rewards conformity to the fitted label stereotype rather than externally verifiable personality alignment. The D-ACC classifier is better (80.59%) but still trained on the same annotation conventions. Please report classifier-human agreement, calibrate the metrics, or use an independent judge (e.g., human ratings or a model not trained on PACT). Without this, the reported gains in P-ACC/D-ACC are ambiguous.
  3. [Appendix H.1 and §3.3] The human evaluation is not external to the dataset construction. Appendix H.1 states that the RG human evaluation is conducted by 'the same four evaluators involved in the dataset creation process,' and Appendix G.3 indicates the same evaluators annotated the PACT quality and the human-human comparison set. The high human ratings (e.g., PR 3.92 for Vicuna FT-Ours, Table 10) therefore cannot be taken as independent confirmation of the automatic results. Please use evaluators who were not involved in any stage of dataset creation, or report whether the evaluators were blind to system identity and dataset provenance. The paper's central claim needs at least one genuinely independent human assessment.
  4. [Limitations and §3.1] The Limitations section concedes that prompting remains an 'uncontrolled generation process' and that the approach 'cannot fully replicate realistic negotiation conversations.' This is relevant to the external validity of the dataset. The 23-act taxonomy and the four argumentation profiles are introduced as ad-hoc, paper-specific constructs, and no validation is provided that these categories correspond to actual human negotiation behavior. Since the task definition itself depends on these categories, the paper should at least compare the distribution and usage of these acts/profiles against a human-human negotiation corpus, or report an expert-based validation of the taxonomy. This would strengthen the claim that the task is not an artifact of the generation prompt.
minor comments (6)
  1. [§3.1] The text says 'four distinct categories' but then enumerates five categories of erroneous conversations. Also, the sentence defining the Argumentation Profile is duplicated: 'For both travel agent and traveler, we define an Argumentation Profile...' appears twice in a row.
  2. [Table 12] In the Diversity prompt, the rating scale is written as '1 signifies low diversity and 1 indicates strong diversity'; the second '1' should presumably be '3'.
  3. [Appendix E.1] The text says FED evaluates 18 aspects, but the list contains 19 items (Interesting through Inquisitive). Please correct the count or the list.
  4. [Figure 3] All three subcaptions are labeled 'Preference Profile', although (b) shows argumentation profile distributions and (c) shows buying style profile distributions. The captions should be corrected.
  5. [Appendix G.2.2] There is a typo: 'used aa a signal' should be 'used as a signal.'
  6. [§5] The text refers to 'ChatGPT' when comparing with the baseline, but Table 5 and the experimental setup list GPT-4.1-mini. Please use consistent model names.

Circularity Check

3 steps flagged

Headline P-ACC/D-ACC gains are measured by RoBERTa classifiers trained on PACT's own labels, and the human validation is conducted by the dataset's creators; the 'high-quality personality-driven' claim is supported by self-confirming evidence.

specific steps
  1. fitted input called prediction [Appendix G.5 (Evaluation Metrics Details); Section 5 (Evaluation Results); Table 5]
    "To evaluate P-ACC and D-ACC, we fine-tune two distinct RoBERTa-base models (Liu et al., 2019) on our dataset for 8 epochs. The classifier used for computing P-ACC achieves an accuracy of 61.57% and a macro-F1 score of 58.31%."

    P-ACC, the metric used to support the claim that fine-tuned models generate personality-driven responses ('compared to FT-Vanilla counterpart, it obtains a 27.2% increase in P-ACC ... for Vicuna'), is computed by a RoBERTa classifier trained on PACT's own synthetic labels. The same PACT labels define the task and are also used to fine-tune the evaluated models. Thus a high P-ACC measures agreement with a classifier fitted to the same distribution the generator was trained on, not an external property of personality. With the judge itself only 61.57% accurate on the original labels, the reported 0.789 P-ACC for Vicuna FT-Ours is a self-referential consistency score rather than an independent measure of personality alignment.

  2. other [Appendix H.1 (Human Evaluation for the Response Generation Task); Section 3.3 (PACT Quality Assessment)]
    "Human evaluation for the RG task is conducted by the same four evaluators involved in the dataset creation process. ... It is noted that dataset annotation involves the same human evaluators as those involved in manual assessment of PACT dataset quality (§3.3)."

    The manual assessments that the paper presents as evidence that PACT 'comprises high-quality dialogues' and that FT-Ours responses are superior (e.g., PR 3.92 for Vicuna FT-Ours) are carried out by the same four people who built, annotated, and filtered PACT. The evaluators are therefore not external or blind; their scores partly reflect their own design choices and expectations. The purported independent human validation is actually an in-group self-assessment, so the high quality and improvement claims are not independently corroborated.

  3. other [Section 3.2 (PACT Filtering) and Section 3.3 (PACT Quality Assessment)]
    "The conversations that obtain a score of ‘3’ from all experts and ‘1’ from the Consistency and Toxicity experts are retained. ... The result presented in Table 1 reveal high FED and G-Eval scores, indicating that the generated conversations mimic key aspects of human-to-human interactions."

    The dataset is filtered by GPT-4-based experts to keep only conversations receiving the maximum possible rating, and the subsequent quality assessment includes G-Eval, an LLM-as-judge metric. The same model family that selected the 'perfect' conversations is then used again to certify their quality, so the high scores are partly guaranteed by the selection rule rather than by an independent measurement. The quality claim is therefore at least partially an artifact of the filtering criterion and the evaluation's reliance on LLM judgments.

full rationale

The paper's central empirical claims are not independently established. The automatic metrics carrying the headline result—P-ACC and D-ACC—are computed by RoBERTa classifiers fine-tuned on PACT itself, so a model fine-tuned on PACT is being scored by a judge trained on the same synthetic labels; the reported gains (e.g., +27.2% P-ACC for Vicuna) partly measure conformity to the dataset's own label conventions. The 'manual' validation is performed by the same four evaluators who created and annotated the dataset, so the high quality scores are not external confirmation. Additionally, the dataset was filtered by GPT-4 experts to retain only conversations with perfect scores, and the subsequent quality assessment relies on LLM-based metrics, making the quality certification partially a same-judge loop. These are real circularities in the evaluation chain, though the paper does construct a useful resource and the core generation pipeline is not itself circular. No load-bearing self-citation was found.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central claims rest on the synthetic generation loop: hand-set concession constants and turn caps, an assumption that LLM paraphrase preserves the scripted content, and evaluator instruments trained on the same labels they measure. The package database (10 real-world-sourced packages, Section 3.1) is treated as exogenous input and is not counted as a parameter. No external behavioral benchmark exists, so all effectiveness claims are internal to the synthetic distribution.

free parameters (5)
  • Concession rate c = 1.2 (low budget), 0.9 (moderate), 0.6 (high budget)
    Hand-set per budget band (Section 3.1, Table 7). Governs how quickly agent and traveler prices converge in generated dialogues and varies with the initial price gap; chosen by the authors, not estimated from data.
  • Tolerance threshold phi = not reported numerically
    Acceptance rule: agent accepts when traveler's price is at most Pa - phi*Pa (Section 3.1, Figure 4). Set per price gap and negotiation deadline; the numeric value is not disclosed, so the acceptance behavior cannot be audited.
  • Budget and tolerance distribution parameters = not reported
    Traveler budget and tolerance limits are drawn from a normal distribution (Section 3.1) with unspecified mean and standard deviation; these determine initial offers and how quickly the traveler concedes.
  • Turn caps for price and argumentation phases = not reported
    Predefined turn limits, varied by argumentation profile (Section 3.1). They decide when the dialogue shifts from price negotiation to argumentation and when the agent issues 'Accept'.
  • One-shot prompt utterances for each dialog act = manually authored
    The sample utterance for each dialog act is 'manually generated based on task overview' (Section 3.1, Table 14). These templates bound the language style and content of the entire dataset.
axioms (5)
  • domain assumption The three profile families capture negotiation-relevant personality variation: Argumentation (Ag/Di/Om/Ar), Preference (10 types from Park et al. 2010), and Buying Style (QC/BC/B&QC from Oumayma and Ez-Zohra 2023).
    Invoked throughout Section 3.1 and in the filtering expert prompts (Table 13). If these profiles do not track real negotiation behavior, the PR sub-task and the P-ACC claims rest on synthetic labels with no external validity.
  • ad hoc to paper The 23 ABN-specific dialogue acts with the negotiation/argumentation/general taxonomy faithfully represent argumentation-based negotiation.
    Defined in Section 3.1 and Appendix C; used to script the entire conversation flow and to supervise the DAP sub-task. Acts such as Justify-price and Disagree-X are the paper's own schema; no evidence links them to a validated ABN dialogue-act ontology (the cited ABN works are agent-level, not dialogue-act level).
  • domain assumption GPT-4 mixture-of-experts filtering with perfect scores selects high-quality dialogues rather than selecting texts that match GPT-4's stylistic preferences.
    Section 3.2. This single judge shapes the final dataset: cumulative survival is 63.9%, so roughly 36% of generated conversations are discarded based on the judge's ratings. No analysis shows what the judge's criteria correlate with.
  • domain assumption Gemini-1.5-Flash, prompted one-shot with the scripted flow, produces fluent, personality-consistent, package-faithful utterances.
    Section 3.1 and Appendix D. The preliminary analysis compares only 50 hand-scored dialogues across three models. The sample conversation in Table 15 contains details (Emirates flights, a 'Beach Bum' package) not present in the package database shown in Appendix A and Table 11, so faithfulness is not achieved even in the illustrative example.
  • standard math The Faratin et al. (1998) concession dynamics, parameterized exactly as printed in Section 3.1, generate realistic price paths.
    As printed, the traveler's equation Ptk = Pak-1 + (Pak-1 - Ptk-1)e^{-ck} places the traveler's counter-offer above the agent's previous price whenever the agent's price exceeds the traveler's, contradicting the sample dialogue in Table 15. This is likely an index transcription error, but the simulation model as written is internally inconsistent.
invented entities (2)
  • 23-act ABN dialogue-act taxonomy (Negotiate-price-increase ... Acknowledge-acceptance) no independent evidence
    purpose: Structures the conversation flow, provides supervision for the DAP sub-task, and is claimed to embody argumentation-based negotiation.
    The acts are defined in this paper and validated only through in-house checks and distribution statistics (Table 4). No external validation links the taxonomy to real human negotiation behavior or to established argumentation frameworks beyond the cited agent-level papers.
  • Argumentation Profile family (Agreeable, Disagreeable, Open-minded, Argumentative) no independent evidence
    purpose: Claims to capture how a negotiator persuades, accepts, or rejects offers; used to assign traveler and agent stances during generation and to supervise the PR sub-task.
    The names are borrowed from Amgoud and Parsons (2001) and Heras et al. (2013), but the four-way operationalization and its mapping onto dialogue behavior is new here and is never checked against human annotators outside the loop; the PR classifier itself reaches only about 61.6% accuracy, so the labels are noisy.

pith-pipeline@v1.3.0-alltime-deepseek · 36368 in / 23957 out tokens · 254027 ms · 2026-08-04T17:05:05.279125+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of We Argue to Agree: Towards Personality-Driven Argumentation-Based Negotiation Dialogue Systems for Tourism." pith.science (2026). https://pith.science/paper/HZXZ6A3D

@misc{pith2026250911118,
  author       = {Pith},
  title        = {Pith review of: We Argue to Agree: Towards Personality-Driven Argumentation-Based Negotiation Dialogue Systems for Tourism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HZXZ6A3D}},
  note         = {Machine review of arXiv:2509.11118}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Integrating argumentation mechanisms into negotiation dialogue systems improves conflict resolution through exchanges of arguments and critiques. Moreover, incorporating personality attributes enhances adaptability by aligning interactions with individuals' preferences and styles. To advance these capabilities in negotiation dialogue systems, we propose a novel Personality-driven Argumentation-based Negotiation Dialogue Generation (PAN-DG) task. To support this task, we introduce PACT, a dataset of Personality-driven Argumentation-based negotiation Conversations for Tourism sector. This dataset, generated using Large Language Models (LLMs), features three distinct personality profiles, viz. Argumentation Profile, Preference Profile, and Buying Style Profile to simulate a variety of negotiation scenarios involving diverse personalities. Thorough automatic and manual evaluations indicate that the dataset comprises high-quality dialogues. Further, we conduct comparative experiments between pre-trained and fine-tuned LLMs for the PAN-DG task. Multi-dimensional evaluation demonstrates that the fine-tuned LLMs effectively generate personality-driven rational responses during negotiations. This underscores the effectiveness of PACT in enhancing personalization and reasoning capabilities in negotiation dialogue systems, thereby establishing a foundation for future research in this domain.

Figures

Figures reproduced from arXiv: 2509.11118 by Asif Ekbal, Desai Vishesh Yasheshbhai, Priyanshu Priya, Saurav Dudhate.

Figure 1
Figure 1. Figure 1: A conversation snippet showing how personality-driven, argumentation-based negotiation aligns with user preferences and employs ABN-specific act justify-price to improve the negotiation outcome. transactional tasks, tourism negotiations involve diverse user preferences, conflicting constraints, and planning across interdependent variables (e.g., price, accommodation, destination), areas where current LLMs … view at source ↗
Figure 2
Figure 2. Figure 2: The overall pipeline of PACT dataset construction. tion, and specifications on amenities, services, and pricing, ensuring that dialogues reflect real-world tourism scenarios. The list of packages can be found in Appendix A. Negotiators Profile Setting. In real-world nego￾tiation, negotiators exhibit varied personalities that influence their interactions. Experiments show that when a negotiator’s personalit… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of conversations across different profile types. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Flow chart illustrating PACT dataset generation process. C ABN-specific Dialog Acts Definition C.1 Negotiation Dialog Acts • Negotiate-price-increase: Used by the travel agent to negotiate a higher price for a package or its related features and/or services during the deal. • Negotiate-price-decrease: Used by the traveler to negotiate a lower price for a package or ser￾vice. • Negotiate-price-nochange: Sug… view at source ↗
Figure 5
Figure 5. Figure 5: Diversity and coverage of negotiation aspects. [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PRISMA: Preference-Reinforced Self-Training Approach for Interpretable Emotionally Intelligent Negotiation Dialogues

    cs.CL 2026-04 unverdicted novelty 4.0

    PRISMA augments self-training with direct preference optimization and an emotion-aware negotiation strategy chain-of-thought to produce more interpretable and effective negotiation dialogues on two new datasets.

Reference graph

Works this paper leans on

30 extracted references · 6 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Personality Relevance: To what extent does the generated conversation reflect the traveler’s and travel agent’s given personalities? Options - 1: Completely irrelevant 2: Mostly irrelevant 3: Mostly relevant 4: Completely relevant

  2. [2]

    Options - 1: Completely ineffective 2: Mostly ineffective 3: Mostly effective 4: Completely effective

    Bargaining Efficacy: To what extent the trav- eler/agent presents offers, justifies their posi- tion, and makes concessions or compelling ar- guments that impact the other party’s decisions in the negotiation. Options - 1: Completely ineffective 2: Mostly ineffective 3: Mostly effective 4: Completely effective

  3. [3]

    arXiv preprint arXiv:2305.10142

    Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142. Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2305.19165

    Strategic reasoning with language models. arXiv preprint arXiv:2305.19165. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli

    Pith/arXiv arXiv

  5. [5]

    Proceedings of the National Academy of Sciences, 120(30):e2305016120

    Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. Atharva Gundawar, Mudit Verma, Lin Guan, Karthik Valmeekam, Siddhant Bhambri, and Subbarao Kamb- hampati. 2024. Robust planning with llm-modulo framework: Case study in travel planning. arXiv preprint arXiv:2405.20625. Ji-Eun ...

    Pith/arXiv arXiv 2024

  6. [6]

    Humanness Do you think this conversation is from the LLM or a human? Options - 1: Definitely a model 2: Probably a model 3: Probably a human 4: Definitely a human

  7. [7]

    Two of the evaluators hold Ph.D

    Fluency Is the conversation seems contextually natural? Are you able to comprehend the con- versation clearly? Options - 1: Very unnatural 2: Mostly unnatural 3: Mostly natural 4: Very natural E.3 Human Evaluators Expertise All four evaluators possess advanced expertise in dialogue systems and negotiation concepts, with strong proficiency in the English l...

    2025

  8. [12]

    Outcome Fairness To what extent does the final negotiation outcome reflect fairness and equity, resulting in a win-win scenario for both parties? Options - 1: Entirely unfair 2: Mostly unfair 3: Mostly fair 4: Entirely fair

  9. [13]

    Engagineness How engaging and content-rich is the overall conversation? Options - 1: Not at all engaging 2: Somewhat engaging 3: Mostly engaging 4: Very engaging

  10. [14]

    Coherent How well-structured and logical is the overall conversation? Options - 1: Completely incoherent 2: Mostly incoherent 3: Mostly coherent 4: Completely coherent

  11. [18]

    This annotated dataset is re- ferred to as ANN -JI, hereafter

    Manual annotation of dialogue samples: It involves randomly sampling 100 dialogues from the dataset and manually annotating the dialogues with recruiter’s and applicant’s per- sonality profile labels and the corresponding exaplanations. This annotated dataset is re- ferred to as ANN -JI, hereafter

  12. [19]

    These labels are ac- companied by natural language explanations to provide insight into the model’s predictions

    Few-shot prompting for personality profile labeling: In this step, LLaMA-3.1-8B (Tou- vron et al., 2023) is prompted in a few-shot setting, using the manually annotated sam- ples as few-shot demonstrations, to generate personality profile labels. These labels are ac- companied by natural language explanations to provide insight into the model’s prediction...

    2023

  13. [20]

    Annotation Guidelines

    Manual Verification and Correction: It in- volves verifying and re-annotating the dia- logues by the human subjects to create gold- standard datasets. Annotation Guidelines. Each dialogue is anno- tated by 3 annotators. The annotators are instructed to analyze the overall interaction style of the inter- locutors rather than focusing on individual utter- a...

  14. [22]

    We fine-tune the LLMs with LoRA adaption (Hu et al., 2021) using 4*V100 GPUs, with batch size of 2 per GPU, and learning rate of 5e−5 for three epochs

    with p = 0.95 and temperature T = 1.0. We fine-tune the LLMs with LoRA adaption (Hu et al., 2021) using 4*V100 GPUs, with batch size of 2 per GPU, and learning rate of 5e−5 for three epochs. The fine-tuning of all the LLMs are based on the SFT trainer of Transformer Reinforcement Learning (TRL)2 (von Werra et al., 2020). All the implementation is done usi...

    2021

  15. [23]

    Personality Relevance (PR) evaluates whether the generated response aligns with the estab- lished personality profile. Instruction: To what extent does the generated response reflect the established personality pro- file of the traveler and travel agent? Scale: 1 (not at all aligned with the personality profile) to 4 (perfectly aligned with the person- al...

  16. [24]

    Bargaining Efficacy (BE) measures the effec- tiveness of the generated response in presenting compelling offers and arguments that influence the other party’s decisions, ultimately achieving favorable outcomes. Instruction: How effectively does the generated response present persuasive offers and argu- ments during the negotiation? Scale: 1 (ineffective a...

  17. [25]

    Outcome Fairness (OF) evaluates the equity of the negotiation results by analyzing how well the generated response reflects an agreement that meets the interests of both the traveler and the agent. Instruction: To what extent does the generated response indicate a fair and equitable agree- ment between the parties? Scale: 1 (entirely unfair, heavily favor...

  18. [26]

    Fluency (F) evaluates the grammatical correct- ness of the generated response. Preprint Version Paper is accepted at EMNLP (Findings) 2025 Instruction: How grammatically correct and fluent is the generated response? Scale: 1 (severely flawed with numerous errors) to 4 (fluent and grammatically correct)

    2025

  19. [27]

    Coherence (C) examines how well the gener- ated responses align with the dialogue context. Instruction: To what extent does the generated response maintain logical flow and relevance within the conversation? Scale: 1 (completely incoherent and irrelevant) to 4 (perfectly coherent and contextually rele- vant)

  20. [28]

    Instruction: How captivating and interesting is the generated response in retaining the user’s attention? Scale: 1 (not engaging at all) to 4 (extremely engaging and captivating)

    Engagingness (E) measures the extent to which the generated response is engaging, compelling, and capable of retaining users’ attention. Instruction: How captivating and interesting is the generated response in retaining the user’s attention? Scale: 1 (not engaging at all) to 4 (extremely engaging and captivating)

  21. [29]

    Faithfulness (Faith.) and Hallucinated (Hal.) evaluate whether the response is faithful or hal- lucinated. Instruction: Whether the generated response is faithful or hallucinated? A response is faith- ful if it aligns with the dialogue context and package information, whereas it is labeled as hallucinated if it either contradicts the provided sources or l...

  22. [30]

    packageX

    is computed to measure the inter-evaluator agreement. The average Kappa scores of 0.83, 0.83, 0.80, 0.78, 0.77, 0.85, 0.83 are obtained for PR, BE, OF, F, C, E, Faith. (or Hal.), respectively, implying substantial agreement among the evaluators. H.3 Case Study We analyze a few samples from the test set and present the responses generated by the top two Pr...

    2025

  23. [1992]

    given its universality and wide applicability. Based on the recent advancements in leveraging the LLMs as annotators to significantly reduce anno- tation costs and human effort (Gilardi et al., 2023; He et al., 2023), we utilize the LLM - LLaMA- 3.1-8B (Touvron et al., 2023) due to its strong rea- soning capability followed by a human-in-the-loop approach...

    2023

  24. [1998]

    Robotics and Autonomous Systems , 24(3- 4):159–182

    Negotiation decision functions for autonomous agents. Robotics and Autonomous Systems , 24(3- 4):159–182. Mauajama Firdaus, Naveen Thangavelu, Asif Ekbal, and Pushpak Bhattacharyya. 2022. I enjoy writing and playing, do you?: a personalized and emotion grounded dialogue agent using generative adversarial network. IEEE Transactions on Affective Computing, ...

    2022

  25. [2012]

    It is noted that dataset annotation involves the same human evaluators as those involved in manual assessment of PACT dataset quality (§3.3)

    agreement ratios of 79.2% in Step 1 and 81.8% Step 3 among the annotators. It is noted that dataset annotation involves the same human evaluators as those involved in manual assessment of PACT dataset quality (§3.3). G.4 Implementation Details We split the dataset into train, validation, and test with the ratio of 7:1:2. All the results are com- puted on ...

  26. [2018]

    EMNLP 2018, page 87

    Retrieve and refine: Improved sequence gener- ation models for dialogue. EMNLP 2018, page 87. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.0...

    Pith/arXiv arXiv 2018

  27. [2019]

    In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 367–378

    A dynamic strategy coach for effective nego- tiation. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 367–378. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & ...

    2018

  28. [2020]

    arXiv preprint arXiv:2010.09954

    Improving dialog systems for negotia- tion with personality modeling. arXiv preprint arXiv:2010.09954. Denis Yarats and Mike Lewis. 2018. Hierarchical text generation and planning for strategic dialogue. In In- ternational Conference on Machine Learning, pages 5591–5599. PMLR. Haolan Zhan, Yufei Wang, Zhuang Li, Tao Feng, Yuncheng Hua, Suraj Sharma, Lizhe...

    Pith/arXiv arXiv 2010

  29. [2021]

    In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 745–757

    Dialogue act-based breakdown detection in negotiation dialogues. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 745–757. Runzhe Yang, Jingxiao Chen, and Karthik Narasimhan

  30. [2023]

    URL https://lmsys

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-03-30-vicuna, 3(5). Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, and Tat-Seng Chua. 2023. Plug-and-play policy planner for large language model powered dialogue agents. In The Twelfth International Conference on Learning Representati...

    Pith/arXiv arXiv 2023