Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

AgreeMate: Teaching LLMs to Haggle

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Larger Llama models negotiate more agreeably and fairly, while chain-of-thought makes them explore more, according to a 21-model study of prompt engineering, fine-tuning, and personality prompts in buyer-seller bargaining.

desk verdict A clearly written course project with a genuine empirical bug: the Fairness metric divides by zero in every run, so the headline fairness results are undefined. read the letter →

arxiv 2412.18690 v1 pith:QRVGZ7GW submitted 2024-12-24 cs.CL cs.LG

Ainesh Chatterjee , Samuel Miller , Nithin Parepally This is my paper · ORCID
classification cs.CLcs.LG
keywords LLMnegotiationstrategicbargainingchain-of-thoughtpromptingpersonalityLoRAfine-tuningattentionprobingCraigslistDatasetDealorNo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models can serve as strategic price negotiators in a decoupled buyer–seller architecture, and that their bargaining behavior is systematically controlled by three levers: model scale, training (prompt engineering, fine-tuning, chain-of-thought), and role personality prompts. The study runs 30 Craigslist negotiation scenarios across 21 Llama 3.2 models and 52 combinations, reporting that larger models are more agreeable, fairer, less aggressive, and shorter in dialogue; that chain-of-thought makes agents explore by inquiring more and conceding less; and that personality prompts shift outcomes, with aggressive buyers dominating and passive sellers conceding. It also fine-tunes 3B models into buyer, seller, and generalist specialists with memory-efficient LoRA and quantization. If these results hold, negotiation behavior in LLMs can be steered cheaply through prompts and targeted fine-tuning rather than hand-built strategy modules, and attention probing can identify internal token relationships tied to negotiation.

What carries the argument

The load-bearing object is the AgreeMate framework itself: a decoupled bargaining architecture with no parser, in which the LLM serves as both manager (selecting from eleven coarse actions) and generator (producing the utterance), with outputs formatted as action-utterance pairs. A negotiation runner orchestrates buyer and seller agents over at most 15 turns on 30 fixed scenarios, and a tester records prices and actions for metric calculation. The metrics that carry the evaluation are agreement rate, a midpoint-deviation fairness score, a seller-target bias score, aggressiveness, concession rate, relative efficiency, and probing ratio. Role specialization is produced by LoRA fine-tuning (rank 32, alpha 16) with 4-bit nf4 quantization, gradient checkpointing, cyclic learning-rate scheduling, EMA loss tracking, and layerwise learning-rate decay, keeping memory around 3.56GB. Attention probing of specific heads then serves as the internal-evidence component.

What would settle it

Recompute every metric in Tables 2 and 3 from raw conversation histories using a defined fairness denominator (for example, seller target minus buyer target), and independently verify that the parsed accepted prices and action labels match the raw model text; the scale and CoT conclusions stand only if the corrected fairness rankings still show larger models as fairer and CoT as more exploratory.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an LLM can replace both the manager and generator of a modular negotiation system, choosing coarse actions (offer, counter-price, insist, accept, inform, inquire) and producing utterances, and that this design exposes clear, regular effects. Larger Llama models show higher agreement rates, lower aggressiveness, and fairer outcomes, while the 8B models are an outlier with low agreement and long dialogues. Chain-of-thought prompting produces exploratory behavior—higher probing ratios, more aggressive initial offers, and smaller average concessions—that improves agreement rates and relative efficiency but makes small models much more buyer-biased and unfair. Personality prompts act as another control: aggressive buyers paired with fair sellers reach the highest agreement rate, and passive sellers exhibit the highest buyer bias. Attention probing finds a layer-1 head that attends to negotiation verbs regardless of fine-tuning, plus a layer-2 head that attends to 'sell' differently between buyer and seller specialists, which the paper reads as evidence that models track negotiation-relevant semantics.

Load-bearing premise

All conclusions rest on the assumption that the fairness metric is well-defined (its printed formula divides by seller target minus seller target, which is zero for every run) and that the prices and action labels parsed from model outputs accurately reflect what the agents said.

Editorial extensions

If this is right

  • Larger Llama models (70B over 3B) can be expected to produce higher agreement rates, fairer outcomes, less aggressiveness, and shorter dialogues in the same bargaining setup.
  • Chain-of-thought prompting shifts negotiation style toward exploration: more inquiry acts, more aggressive price moves, and smaller concessions, with agreement rates and relative efficiency improving even as small-model bias and fairness degrade.
  • Personality prompting works as a control on the whole conversation: aggressive buyers dominate and shorten dialogues, passive sellers concede most, and aggressive-plus-fair pairings give the highest agreement rate.
  • Role-specialized negotiation agents (buyer, seller, generalist) can be fine-tuned at 3B scale under tight memory constraints, suggesting specialized bargaining can run on modest hardware.
  • Attention-head patterns tied to negotiation verbs indicate that even without a dedicated parser, the models track negotiation-relevant semantics in their input.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One untested extension is to raise the 15-turn cap for CoT agents; the paper's own relative-efficiency result suggests longer horizons could convert exploratory inquiry into fairer agreements, but the paper does not run this comparison.
  • The fine-tuning study only covers the 3B scale, leaving open whether role specialization at 8B or 70B would reproduce the same buyer/seller asymmetries or interact with the scale effects the paper reports.
  • The attention-head finding suggests a direct control experiment the paper does not attempt: steering or masking the layer-1 negotiation-verb head should measurably alter concession behavior if the semantic-attention claim is causal.
  • Because personality prompts were applied to base 3B models rather than the fine-tuned specialists, a natural follow-up is to cross personality prompts with fine-tuned weights to separate prompt-driven from weight-driven strategy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. AgreeMate introduces a framework in which an LLM acts as both negotiation manager and generator, producing action-utterance pairs under a turn limit. The paper evaluates Llama 3.2 3B/8B/70B models with and without chain-of-thought prompting, personality-prompted 3B models, and three fine-tuned 3B specialists (buyer, seller, generalist) on 30 scenarios from the Craigslist Bargaining Dataset. It defines several custom metrics, including fairness, bias, aggressiveness, concession rate, relative efficiency, and probing ratio, and reports that larger models are more agreeable and fairer, that chain-of-thought prompting induces exploratory bargaining behavior, that personality prompts shift negotiation outcomes, and that certain attention heads track negotiation-related verbs. The central quantitative claims rest on the custom metrics and on automatic extraction of prices and actions from model outputs; as written, the fairness metric is undefined and the extraction is not validated.

Significance. If the empirical results were valid, the paper would provide a useful comparison of model scale, prompting, and fine-tuning in a strategic dialogue setting, and the release of code on GitHub together with the use of public datasets would aid reproducibility. Strengths include the breadth of the model zoo tested, the clear separation of the evaluation set (Craigslist) from the fine-tuning set (Deal or No Deal), and the transparency about the course-project context. However, the headline claims are not currently supported: the fairness formula in Section 6.2 divides by zero as printed, the fine-tuning claim in the abstract is never tested against a baseline, and all metric-based comparisons depend on an unvalidated parser. These are load-bearing issues rather than presentation slips, although they are identifiable and, in principle, correctable.

major comments (5)
  1. [Section 6.2] The Fairness formula is written as Fairness = 1 - 2*|accepted - (seller_target + buyer_target)/2| / (seller_target - seller_target). The denominator is identically zero for every negotiation, so every Fairness value in Tables 2 and 3, Figures 7 and 18, and all fairness-based conclusions in Sections 6.3.3, 6.4.3, and 6.4.6 are undefined as printed. Please correct the denominator (presumably to seller_target - buyer_target) and recompute all affected results; the abstract's 'fairer' claims depend directly on this metric.
  2. [Section 6.1] The paper states that 'relevant information like prices and actions are recorded' from model outputs, but it reports no validation of this extraction. Because Agreement Rate, Bias, Aggressiveness, Concession Rate, and Probing Ratio are all computed from these extracted prices and action labels, any parsing error corrupts every metric reported. Please provide parsing accuracy on a labeled sample, or compare the extracted actions with the Craigslist dataset's provided intents, so that the main measurements are trustworthy.
  3. [Section 5.4 and Section 6.1] The three fine-tuned agents (buyer specialist, seller specialist, generalist) are listed in Section 6.1, but no subsequent table or figure reports their negotiation metrics, and there is no comparison with the base Llama 3.2 3B model. Section 5.4 reports only training loss convergence. Therefore the abstract's claim that 'fine-tuning ... enhances model performance' is not demonstrated by the experiments. Please add a direct comparison of the fine-tuned specialists against the corresponding baseline on the Section 6.2 metrics.
  4. [Section 6.1] All results are based on 30 randomly selected scenarios, and the paper reports no confidence intervals or significance tests. Differences such as Table 3's With-CoT versus Without-CoT Bias (1.0813 vs 0.1671) or Fairness (-0.4385 vs 0.3021), as well as the agreement rates in Table 1, could plausibly reflect noise at n=30. Please add bootstrap confidence intervals or a suitable statistical test for each headline comparison.
  5. [Section 6.2] Several metrics, including Aggressiveness, Bias, Fairness, Relative Efficiency, and Concession Rate, are defined in terms of an accepted price, but agreement rates in Table 1 and Figure 16 are below 1. The paper does not state whether rejected or turn-limit-truncated negotiations are excluded from the metric averages or assigned a specific value. This decision affects every reported mean and should be made explicit.
minor comments (6)
  1. [Throughout] The dataset name is consistently misspelled as 'Craiglist'; it should be 'Craigslist'.
  2. [Section 6.2] The heading 'Aggreement Rate' contains a typo; it should be 'Agreement Rate'.
  3. [Section 6.1] The phrase 'aggressive, fair, and passive passive personalities' duplicates 'passive'; one instance should be removed.
  4. [Section 3] The sentence 'To goal being, to see if LLMs can infer common negotiation strategies...' is ungrammatical and should be rewritten.
  5. [References] The Llama 3.2 models are cited to Touvron et al. (2023), which describes Llama 1; please cite the correct Llama 3.2 release or model card.
  6. [Section 6.5] The probing analysis draws conclusions such as 'this specific attention head was not impacted by fine-tuning' from visual inspection of a single conversation; this claim should be labeled as anecdotal or supported by quantitative comparisons across multiple inputs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the empirical chain is independent of its own conclusions, though §6.2's Fairness formula and unvalidated parsing are serious validity problems, not circularity.

full rationale

I examined the derivation chain from the abstract's claims through Sections 5–6.5. The main claims are (1) model scale, CoT prompting, personality prompts, and fine-tuning affect negotiation behavior, and (2) certain attention heads track negotiation semantics. The evaluation uses the Craigslist Negotiation Dataset, while fine-tuning uses the separate Deal or No Deal dataset; no metric or parameter is fitted to the evaluation outcomes. The custom metrics in §6.2 are computed directly from recorded prices, targets, and action counts, and they are not used as training objectives, so they do not make the conclusions true by construction. The attention probing is a post-hoc observation of attention weights on a single conversation, not a metric derived from the conclusion. The paper contains no self-citations that carry a load-bearing premise; its architecture is inherited from He et al. (2018) and the base model from Touvron et al. (2023), both external and independently available. Therefore no circular step can be exhibited. For completeness, §6.2's Fairness formula has a zero denominator (seller_target - seller_target) and §6.1 reports no validation of the price/action parser; these are serious validity and reproducibility problems, but they are not circularity, because the metrics are not defined in terms of the conclusions and are not fitted to them.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on assumptions about metric definitions, automatic parsing of LLM outputs, representativeness of 30 scenarios, and the validity of personality prompts. None of these assumptions are validated in the paper, and one, the fairness denominator, is visibly false as written.

free parameters (3)
  • Negotiation turn limit = 15
    Hand-picked cap on dialogue length in Section 6.1; agreement rate and dialogue length metrics depend on it.
  • Test scenario count = 30
    Randomly selected from the Craigslist test split in Section 6.1; all aggregate metrics are based on this small sample, with no seed given.
  • Fine-tuning hyperparameters = rank 32, alpha 16, batch 32, seq 512, 4k steps
    Hand-chosen training settings in Section 5.3; no ablation or sensitivity analysis is reported, and the fine-tuned models are never compared with baseline models on negotiation metrics.
assumptions (5)
  • domain assumption LLM-generated action labels and extracted prices accurately reflect the true negotiation state
    Section 6.1 says relevant prices and actions are recorded per turn, but no validation, manual audit, or inter-annotator agreement is reported; all metrics rely on these parses.
  • ad hoc to paper The fairness formula is meaningful and its denominator is nonzero
    Section 6.2 defines Fairness with denominator (seller target - seller target), which is zero; the paper silently relies on a corrected or intended formula.
  • domain assumption 30 randomly chosen scenarios from the Craigslist test set are representative for model comparisons
    Section 6.1 selects 30 scenarios without a seed, and no confidence intervals or significance tests are reported.
  • domain assumption Personality prompt labels correspond to distinct agent strategies
    Section 6.3 labels models as aggressive, fair, or passive based on prompts and attributes behavioral differences to personality, but no manipulation check verifies that the models actually follow the intended strategy.
  • domain assumption Visual inspection of attention weights from one conversation is sufficient evidence of semantic negotiation processing
    Section 6.5 draws general conclusions from selected attention heads in a single sample listing without quantitative measurements or statistical tests.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AgreeMate: Teaching LLMs to Haggle." pith.science (2026). https://pith.science/paper/QRVGZ7GW

@misc{pith2026241218690,
  author       = {Pith},
  title        = {Pith review of: AgreeMate: Teaching LLMs to Haggle},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRVGZ7GW}},
  note         = {Machine review of arXiv:2412.18690}
}
read the original abstract

We introduce AgreeMate, a framework for training Large Language Models (LLMs) to perform strategic price negotiations through natural language. We apply recent advances to a negotiation setting where two agents (i.e. buyer or seller) use natural language to bargain on goods using coarse actions. Specifically, we present the performance of Large Language Models when used as agents within a decoupled (modular) bargaining architecture. We demonstrate that using prompt engineering, fine-tuning, and chain-of-thought prompting enhances model performance, as defined by novel metrics. We use attention probing to show model attention to semantic relationships between tokens during negotiations.

Figures

Figures reproduced from arXiv: 2412.18690 by the authors.

Figure 1
Figure 1. Training Loss Metrics. Left: EMA-smoothed [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Steps Since Improvement. Minimal fluctu [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Evaluation Loss and Throughput Metrics. Left: Validation loss trends stabilize around 4.05. Right: Evaluation throughput demonstrates consistent samples per second [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Learning Rate and Gradient Behavior. Left: [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Agreement Rates Across Personality Com￾binations. Aggressive buyers paired with fair sellers achieved the highest success, while passive combina￾tions exhibited moderate agreement rates. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Bias Toward Buyers/Sellers Across Person [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Price Progression Heatmap for Aggressive [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 7
Figure 7. Figure 7: Fairness Scores Across Personality Combina [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 12
Figure 12. Figure 12: Price Progression Trends Over Turns. Comparative trends reveal aggressive buyers converge quickly, while passive negotiations take longer to stabi￾lize [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Action Distribution Across Personality Com [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 14
Figure 14. Figure 14: Top Actions for Aggressive Buyer + Passive [PITH_FULL_IMAGE:figures/full_fig_p009_14.png]
Figure 16
Figure 16. Figure 16: Agreement Rate across combinations. Shows the number of negotiations closed / ended in an accept act for each combination 6.4.2 Average Bias As seen in 17, smaller models tend to be more bi￾ased towards the buyer while larger models tend to be more biased towards the …
Figure 17
Figure 17. Figure 17: Average bias across combinations. A pos [PITH_FULL_IMAGE:figures/full_fig_p009_17.png]
Figure 18
Figure 18. Figure 18: Average Fairness across all combinations [PITH_FULL_IMAGE:figures/full_fig_p010_18.png]
Figure 19
Figure 19. Figure 19: Average Aggressiveness across all combina [PITH_FULL_IMAGE:figures/full_fig_p010_19.png]
Figure 20
Figure 20. Figure 20: Average Dialogue Length across combina￾tions 6.4.6 Size & CoT Summary Larger models tend to be more agreeable, fairer, less aggressive, and produce shorter dialogues. With the exception of certain results pertaining to model combinations involving 8B parameter mod￾els…
Figure 21
Figure 21. Figure 21: Selected attention weights of Layer 1 (count [PITH_FULL_IMAGE:figures/full_fig_p011_21.png]
Figure 22
Figure 22. Figure 22: Selected attention weights on Layer 1 (count [PITH_FULL_IMAGE:figures/full_fig_p011_22.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EvoEmo: Towards Evolved Emotional Policies for Adversarial LLM Agents in Multi-Turn Price Negotiation

    cs.AI 2025-09 reject novelty 6.0 of 10

    EvoEmo evolves emotion-transition policies for buyer LLM agents and reports higher savings, success rates, and efficiency than vanilla or fixed-emotion baselines in simulated price negotiations.

  2. State-Inference-Based Prompting for Natural Language Trading with Game NPCs

    cs.AI 2025-07 conditional novelty 5.0 of 10

    A prompt framework that makes LLM game merchants follow a six-state trading flow achieves over 97% state compliance, over 95% item accuracy, and 99.7% price accuracy in simulated dialogues.

Reference graph

Works this paper leans on

7 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Simon Martin Breum, Daniel V dele Egdal, Victor Gram Mortensen, Anders Giovanni M ller, and Luca Maria Aiello. 2023. https://arxiv.org/abs/2312.15523 The persuasive power of large language models . Computing Research Repository, arXiv:2312.15523

    arXiv 2023

  4. [4]

    He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. https://doi.org/10.18653/v1/D18-1256 Decoupling strategy and generation in negotiation dialogues . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333--2343, Brussels, Belgium. Association for Computational Linguistics

    doi:10.18653/v1/d18-1256 2018

  5. [5]

    Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017. https://arxiv.org/abs/1706.05125 Deal or no deal? end-to-end learning for negotiation dialogues . Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Published by Meta AI, focusing on negotiation tasks that combine reasoning and linguistic ...

    arXiv 2017

  6. [6]

    Deepa Seetharaman. 2024. https://www.wsj.com/tech/ai/open-ai-division-for-profit-da26c24b Turning OpenAI into a real business is tearing it apart . The Wall Street Journal

    work page 2024

  7. [7]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama: Open and efficient foundation language models . arXiv preprint arXiv:2302.13971...

    arXiv 2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.