{"id":"6ee3a393-8b12-4e62-91de-c1e3666731f3","arxiv_id":"2412.18690","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"AgreeMate reports that larger Llama models, chain-of-thought prompting, and personality prompts change price-negotiation outcomes, but the supporting metrics contain errors and the fine-tuning claim is not evaluated.","lead":"This paper introduces AgreeMate, a framework for letting large language models bargain over prices in natural language, and tests how model size, personality prompts, and chain-of-thought reasoning change buyer-seller negotiations. The central empirical claims are undermined by a division-by-zero fairness metric, a missing fine-tuning comparison, and no error bars or released code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §6.2 Fairness formula divides by zero for every run, so all fairness scores and fairness-based conclusions in Tables 2–3 are undefined as printed; the unvalidated price/action parser additionally makes every metric-based comparison unreliable.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the custom metrics are not reliable, starting with a Fairness formula whose denominator is always zero, and the LLM-generated action labels and prices are never validated. I agree with that framing. The zero denominator is a concrete internal inconsistency, not a matter of taste or consensus: it makes the fairness numbers in Tables 2 and 3 undefined, and those numbers feed the abstract's claim that performance was enhanced per novel metrics. The parser validation issue is just as central because every quantitative result, including agreement rate and bias, depends on extracting the correct accepted price and action from free-form model output. The paper also promises a fine-tuning comparison but never reports one, so even if the metrics were fixed, the 'fine-tuning enhances performance' part of the abstract would remain unsupported. I do not see a way to interpret the current text that preserves the central empirical claims. The judgment of REJECT is therefore appropriate, and the paper should be resubmitted only after the formulas are corrected, the parser is validated, and the fine-tuned agents are actually compared with baselines.","tokens_in":11088,"tokens_out":3059,"duration_ms":30406,"concrete_test":"Recompute every metric for the 30 retained scenarios from the raw recorded negotiation transcripts using explicit formulas with (seller_target - buyer_target) as the Fairness denominator, then compare the corrected Tables 2 and 3 with the paper's printed values. If any corrected fairness value changes sign or ranking, the fairness conclusions fail. As a secondary check, hand-annotate a random sample of at least 100 action/price extractions and measure parser accuracy; the pipeline should report agreement rates on these validated labels before any metric-based claim is accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AgreeMate's central claim is that prompt engineering, fine-tuning, and chain-of-thought prompting enhance negotiation performance as measured by its stated metrics. The load-bearing requirement is that those metrics are correctly computed from true negotiation outcomes. Section 6.2 defines Fairness as 1 - 2*|accepted - (seller_target + buyer_target)/2| / (seller_target - seller_target). The denominator is identically zero, so every Fairness value in Tables 2 and 3, Figures 7 and 18, and all 'fairer/less fair' conclusions are undefined as written. Since the abstract and Section 7 lean heavily on fairness, the headline empirical claim has no valid support. This is compounded by the absence of any validation of the parser that records 'prices and actions' from raw model outputs in Section 6.1: if action labels or accepted-price extraction are wrong, then agreement rates, bias, aggressiveness, concession rate, probing ratio, and relative efficiency are all computed from corrupted records. Finally, the fine-tuned agents are listed in Section 6.1, but no baseline-versus-fine-tuned comparison appears in any reported results table or figure, so the abstract's fine-tuning claim is not actually demonstrated. These are not presentation issues; the central empirical argument depends directly on the metrics and the parsing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AgreeMate introduces a framework in which an LLM acts as both negotiation manager and generator, producing action-utterance pairs under a turn limit. The paper evaluates Llama 3.2 3B/8B/70B models with and without chain-of-thought prompting, personality-prompted 3B models, and three fine-tuned 3B specialists (buyer, seller, generalist) on 30 scenarios from the Craigslist Bargaining Dataset. It defines several custom metrics, including fairness, bias, aggressiveness, concession rate, relative efficiency, and probing ratio, and reports that larger models are more agreeable and fairer, that chain-of-thought prompting induces exploratory bargaining behavior, that personality prompts shift negotiation outcomes, and that certain attention heads track negotiation-related verbs. The central quantitative claims rest on the custom metrics and on automatic extraction of prices and actions from model outputs; as written, the fairness metric is undefined and the extraction is not validated.","tokens_in":11450,"tokens_out":9015,"duration_ms":79606,"significance":"If the empirical results were valid, the paper would provide a useful comparison of model scale, prompting, and fine-tuning in a strategic dialogue setting, and the release of code on GitHub together with the use of public datasets would aid reproducibility. Strengths include the breadth of the model zoo tested, the clear separation of the evaluation set (Craigslist) from the fine-tuning set (Deal or No Deal), and the transparency about the course-project context. However, the headline claims are not currently supported: the fairness formula in Section 6.2 divides by zero as printed, the fine-tuning claim in the abstract is never tested against a baseline, and all metric-based comparisons depend on an unvalidated parser. These are load-bearing issues rather than presentation slips, although they are identifiable and, in principle, correctable.","major_comments":[{"comment":"The Fairness formula is written as Fairness = 1 - 2*|accepted - (seller_target + buyer_target)/2| / (seller_target - seller_target). The denominator is identically zero for every negotiation, so every Fairness value in Tables 2 and 3, Figures 7 and 18, and all fairness-based conclusions in Sections 6.3.3, 6.4.3, and 6.4.6 are undefined as printed. Please correct the denominator (presumably to seller_target - buyer_target) and recompute all affected results; the abstract's 'fairer' claims depend directly on this metric.","section":"Section 6.2"},{"comment":"The paper states that 'relevant information like prices and actions are recorded' from model outputs, but it reports no validation of this extraction. Because Agreement Rate, Bias, Aggressiveness, Concession Rate, and Probing Ratio are all computed from these extracted prices and action labels, any parsing error corrupts every metric reported. Please provide parsing accuracy on a labeled sample, or compare the extracted actions with the Craigslist dataset's provided intents, so that the main measurements are trustworthy.","section":"Section 6.1"},{"comment":"The three fine-tuned agents (buyer specialist, seller specialist, generalist) are listed in Section 6.1, but no subsequent table or figure reports their negotiation metrics, and there is no comparison with the base Llama 3.2 3B model. Section 5.4 reports only training loss convergence. Therefore the abstract's claim that 'fine-tuning ... enhances model performance' is not demonstrated by the experiments. Please add a direct comparison of the fine-tuned specialists against the corresponding baseline on the Section 6.2 metrics.","section":"Section 5.4 and Section 6.1"},{"comment":"All results are based on 30 randomly selected scenarios, and the paper reports no confidence intervals or significance tests. Differences such as Table 3's With-CoT versus Without-CoT Bias (1.0813 vs 0.1671) or Fairness (-0.4385 vs 0.3021), as well as the agreement rates in Table 1, could plausibly reflect noise at n=30. Please add bootstrap confidence intervals or a suitable statistical test for each headline comparison.","section":"Section 6.1"},{"comment":"Several metrics, including Aggressiveness, Bias, Fairness, Relative Efficiency, and Concession Rate, are defined in terms of an accepted price, but agreement rates in Table 1 and Figure 16 are below 1. The paper does not state whether rejected or turn-limit-truncated negotiations are excluded from the metric averages or assigned a specific value. This decision affects every reported mean and should be made explicit.","section":"Section 6.2"}],"minor_comments":[{"comment":"The dataset name is consistently misspelled as 'Craiglist'; it should be 'Craigslist'.","section":"Throughout"},{"comment":"The heading 'Aggreement Rate' contains a typo; it should be 'Agreement Rate'.","section":"Section 6.2"},{"comment":"The phrase 'aggressive, fair, and passive passive personalities' duplicates 'passive'; one instance should be removed.","section":"Section 6.1"},{"comment":"The sentence 'To goal being, to see if LLMs can infer common negotiation strategies...' is ungrammatical and should be rewritten.","section":"Section 3"},{"comment":"The Llama 3.2 models are cited to Touvron et al. (2023), which describes Llama 1; please cite the correct Llama 3.2 release or model card.","section":"References"},{"comment":"The probing analysis draws conclusions such as 'this specific attention head was not impacted by fine-tuning' from visual inspection of a single conversation; this claim should be labeled as anecdotal or supported by quantitative comparisons across multiple inputs.","section":"Section 6.5"}],"recommendation":"major_revision","confidential_remarks":"This manuscript has a useful experimental framework and the flaws I found are identifiable and correctable, which is why I chose major_revision rather than reject. If the authors cannot produce corrected fairness values, add the missing fine-tuning comparison, and validate the parser, then rejection would be appropriate. The current abstract and conclusion substantially overstate what the experiments establish."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a course project that builds a clean haggling harness and runs a lot of Llama configurations, but the headline results don't survive contact with the metrics. The Fairness formula in §6.2 has a zero denominator for every run, so Tables 2–3 and Figure 18 are undefined. That's not a presentation nit; it's the abstract's \"fairer\" claim. The stress-test note is right, and I checked the formula: 1 - 2*|accepted - midpoint| / (seller_target - seller_target). Denom is always zero. So all fairness-based conclusions are void as printed.\n\nWhat's actually new: the systematic comparison of CoT, personality prompts, and model scale in price haggling, which the cited negotiation papers don't cover. The architecture is sensible (LLM as manager+generator, action-utterance pairs), the fine-tuning setup is described concretely (LoRA, 4-bit, ~3.5GB memory), and the attention probing, though exploratory, is honest about being non-comprehensive. The paper also does not hide its course-project origin, which is fine.\n\nSoft spots beyond the fairness bug: the fine-tuned specialists are announced in §6.1 but never compared to baselines in any results table or figure, so the abstract's fine-tuning claim is not demonstrated. The parser that records \"prices and actions\" from raw model outputs is unvalidated; every metric depends on it. There are no error bars or significance tests on the 30-scenario evaluations, so the qualitative size/CoT findings are suggestive at best. The attention probing is visual inspection of one conversation, which supports only a weak \"there is evidence\" claim, not the general statement in §7. The paper says code is on GitHub but gives no link or commit hash.\n\nThese are fixable, but they're load-bearing: you'd need to repair the metric, validate the parser, actually run the fine-tuned vs. baseline comparison, and release the code. The descriptive observations about larger models being more agreeable and CoT inducing exploratory behavior might survive a rerun, but the numeric claims and the fairness story shouldn't be cited.\n\nMy recommendation: desk reject, with encouragement for the authors to fix the metrics, add the missing comparison, and resubmit. As it stands, the central empirical claims are unsupported. A serious editor should not spend referee time on this version.","headline":"A clearly written course project with a genuine empirical bug: the Fairness metric divides by zero in every run, so the headline fairness results are undefined.","tokens_in":11881,"tokens_out":1723,"would_cite":false,"duration_ms":17042,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Larger Llama models negotiate more agreeably and fairly, while chain-of-thought makes them explore more, according to a 21-model study of prompt engineering, fine-tuning, and personality prompts in buyer-seller bargaining.","keywords":["LLM negotiation","strategic bargaining","chain-of-thought prompting","personality prompting","LoRA fine-tuning","attention probing","Craigslist Bargaining Dataset","Deal or No Deal dataset"],"falsifier":"Recompute every metric in Tables 2 and 3 from raw conversation histories using a defined fairness denominator (for example, seller target minus buyer target), and independently verify that the parsed accepted prices and action labels match the raw model text; the scale and CoT conclusions stand only if the corrected fairness rankings still show larger models as fairer and CoT as more exploratory.","tokens_in":10852,"feed_emoji":"🤝","tokens_out":7887,"duration_ms":67964,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models can serve as strategic price negotiators in a decoupled buyer–seller architecture, and that their bargaining behavior is systematically controlled by three levers: model scale, training (prompt engineering, fine-tuning, chain-of-thought), and role personality prompts. The study runs 30 Craigslist negotiation scenarios across 21 Llama 3.2 models and 52 combinations, reporting that larger models are more agreeable, fairer, less aggressive, and shorter in dialogue; that chain-of-thought makes agents explore by inquiring more and conceding less; and that personality prompts shift outcomes, with aggressive buyers dominating and passive sellers conceding. It also fine-tunes 3B models into buyer, seller, and generalist specialists with memory-efficient LoRA and quantization. If these results hold, negotiation behavior in LLMs can be steered cheaply through prompts and targeted fine-tuning rather than hand-built strategy modules, and attention probing can identify internal token relationships tied to negotiation.","feed_headline":"Bigger Llama models make fairer, more agreeable haggling agents","feed_subtitle":"A 21-model study shows scale, personality prompts, and chain-of-thought change how LLMs negotiate price.","key_machinery":"The load-bearing object is the AgreeMate framework itself: a decoupled bargaining architecture with no parser, in which the LLM serves as both manager (selecting from eleven coarse actions) and generator (producing the utterance), with outputs formatted as action-utterance pairs. A negotiation runner orchestrates buyer and seller agents over at most 15 turns on 30 fixed scenarios, and a tester records prices and actions for metric calculation. The metrics that carry the evaluation are agreement rate, a midpoint-deviation fairness score, a seller-target bias score, aggressiveness, concession rate, relative efficiency, and probing ratio. Role specialization is produced by LoRA fine-tuning (rank 32, alpha 16) with 4-bit nf4 quantization, gradient checkpointing, cyclic learning-rate scheduling, EMA loss tracking, and layerwise learning-rate decay, keeping memory around 3.56GB. Attention probing of specific heads then serves as the internal-evidence component.","core_discovery":"On its own terms, the paper's discovery is that an LLM can replace both the manager and generator of a modular negotiation system, choosing coarse actions (offer, counter-price, insist, accept, inform, inquire) and producing utterances, and that this design exposes clear, regular effects. Larger Llama models show higher agreement rates, lower aggressiveness, and fairer outcomes, while the 8B models are an outlier with low agreement and long dialogues. Chain-of-thought prompting produces exploratory behavior—higher probing ratios, more aggressive initial offers, and smaller average concessions—that improves agreement rates and relative efficiency but makes small models much more buyer-biased and unfair. Personality prompts act as another control: aggressive buyers paired with fair sellers reach the highest agreement rate, and passive sellers exhibit the highest buyer bias. Attention probing finds a layer-1 head that attends to negotiation verbs regardless of fine-tuning, plus a layer-2 head that attends to 'sell' differently between buyer and seller specialists, which the paper reads as evidence that models track negotiation-relevant semantics.","pith_inferences":["One untested extension is to raise the 15-turn cap for CoT agents; the paper's own relative-efficiency result suggests longer horizons could convert exploratory inquiry into fairer agreements, but the paper does not run this comparison.","The fine-tuning study only covers the 3B scale, leaving open whether role specialization at 8B or 70B would reproduce the same buyer/seller asymmetries or interact with the scale effects the paper reports.","The attention-head finding suggests a direct control experiment the paper does not attempt: steering or masking the layer-1 negotiation-verb head should measurably alter concession behavior if the semantic-attention claim is causal.","Because personality prompts were applied to base 3B models rather than the fine-tuned specialists, a natural follow-up is to cross personality prompts with fine-tuned weights to separate prompt-driven from weight-driven strategy."],"forward_implications":["Larger Llama models (70B over 3B) can be expected to produce higher agreement rates, fairer outcomes, less aggressiveness, and shorter dialogues in the same bargaining setup.","Chain-of-thought prompting shifts negotiation style toward exploration: more inquiry acts, more aggressive price moves, and smaller concessions, with agreement rates and relative efficiency improving even as small-model bias and fairness degrade.","Personality prompting works as a control on the whole conversation: aggressive buyers dominate and shorten dialogues, passive sellers concede most, and aggressive-plus-fair pairings give the highest agreement rate.","Role-specialized negotiation agents (buyer, seller, generalist) can be fine-tuned at 3B scale under tight memory constraints, suggesting specialized bargaining can run on modest hardware.","Attention-head patterns tied to negotiation verbs indicate that even without a dedicated parser, the models track negotiation-relevant semantics in their input."],"supporting_citations":[{"why":"Supplies the modular decoupling architecture this paper adapts and the Craigslist Bargaining Dataset used for evaluation.","marker":"He et al., 2018"},{"why":"Supplies the Deal or No Deal negotiation dialogues used to fine-tune role-specialized agents.","marker":"Lewis et al., 2017"},{"why":"Defines the LLaMA family whose 3B, 8B, and 70B checkpoints are the base models tested and fine-tuned.","marker":"Touvron et al., 2023"}],"fun_headline_variants":["Scale and chain-of-thought shift LLM negotiation tactics","Llama size and personality prompts alter haggling outcomes","AgreeMate: CoT explores, scale agrees, prompts bias","Attention probing reveals negotiation-aware LLM heads","Bigger LLMs haggle fairer, but CoT makes them bold"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All conclusions rest on the assumption that the fairness metric is well-defined (its printed formula divides by seller target minus seller target, which is zero for every run) and that the prices and action labels parsed from model outputs accurately reflect what the agents said.","fun_headline_variants_meta":{"raw":{"variants":["Scale and chain-of-thought shift LLM negotiation tactics","Llama size and personality prompts alter haggling outcomes","AgreeMate: CoT explores, scale agrees, prompts bias","Attention probing reveals negotiation-aware LLM heads","Bigger LLMs haggle fairer, but CoT makes them bold"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1291,"prompt_tokens":848,"completion_tokens":443,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":359}},"tokens_in":464,"tokens_out":443,"duration_ms":4643,"temperature":1.0,"reasoning_tokens":359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:33:38.805627+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute every metric in Tables 2 and 3 from raw conversation histories using a defined fairness denominator (for example, seller target minus buyer target), and independently verify that the parsed accepted prices and action labels match the raw model text; the scale and CoT conclusions stand only if the corrected fairness rankings still show larger models as fairer and CoT as more exploratory.","supporting_citations":[],"review_version":1}