{"id":"38326c5c-866c-405b-845a-24336da4cd61","arxiv_id":"2507.09157","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"PU-Lie reports a macro F1 of 0.60 for deception detection in Diplomacy dialogues using only 1,345 trainable parameters, via PU learning over frozen BERT and handcrafted features.","lead":"A lightweight deception detector for Diplomacy game messages combines frozen BERT embeddings, handcrafted language and game features, and positive-unlabeled learning, reporting a macro F1 of 0.60 with only 1,345 trainable parameters. The paper is easy to read but does not release code, does not describe the data split, and does not compare with the published benchmark it claims to beat.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.60 macro F1 is unverifiable because the paper never states whether the train/test split is by message, dialogue, or game; with only 12 games, a message-level split would let game-specific features leak context into test and inflate the headline result.","rationale":"The paper's central empirical claim is a new state-of-the-art macro F1 of 0.60 with 1,345 trainable parameters. For that claim to hold, the evaluation must reflect generalization to unseen strategic dialogues. The manuscript never says how the 17,289 messages from 12 games are divided into training and test sets. The dataset description in §III-A explicitly includes game-level metadata—game id, absolute message index, game score delta, seasons, and player roles—and §III-B1 feeds game metadata into PU-Lie. If the split is random at the message level, training and test contain messages from the same games and often from the same dialogue, so game-specific features can act as a shortcut. With only 12 games, this is a severe leakage risk. The reader's weakest assumption identifies exactly this issue, and I agree with it. I also note the secondary problem that threshold tuning is described in §IV-A without stating that it is validation-only. These are correctness risks in the evaluation protocol, not disagreements with consensus or attacks on the authors. The paper provides no code, no split description, and no comparison to the original benchmark, so the reported number cannot be independently verified. The verdict of REJECT with low confidence remains appropriate because the evaluation details are missing and the headline number is therefore unsupported. I would not change the reader's verdict; the concern reinforces it. The concrete test—a strict game-level split with validation-only threshold selection—would settle whether the concern lands. If the held-out-game macro F1 remains around 0.60, the central claim would be substantially strengthened. If it drops materially or the baseline ranking changes, the reported 0.60 would be attributable to leakage rather than to the model's design.","tokens_in":6224,"tokens_out":4662,"duration_ms":61272,"concrete_test":"Obtain from the authors the exact split construction (or code) and rerun PU-Lie and the strongest baselines under a strict game-level protocol: train on 10 complete games, tune the decision threshold on one validation game, and test on one held-out game, rotating across all 12 games or using leave-one-game-out. Report the game IDs used and the per-game macro-F1 distribution. If the game-level macro F1 drops by more than about 0.05 from the reported 0.60, or if PU-Lie no longer beats the BERT+Game+LSTM baseline, then the message-level split inflated the headline result. If the held-out-game macro F1 remains near 0.60, the leakage concern is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that PU-Lie achieves a new state-of-the-art macro F1 of 0.60 with only 1,345 trainable parameters—requires that the test set measure generalization to unseen strategic dialogues. The paper never describes the split. The dataset description in §III-A lists game id, absolute message index, game score delta, seasons, and player roles as available metadata, and §III-B1 explicitly feeds game metadata such as season, game score delta, and player roles into PU-Lie. If the split is random at the message level, then training and test contain messages from the same games and often from the same dialogue. Under those conditions, game-specific features encode shared game state, and the model can exploit that shared context rather than learning a general deception detector. Because only 12 complete games exist, this is not a minor implementation detail: a proper game-level split leaves few test games and must be reported explicitly. The paper also does not state whether the precision-recall threshold tuning described in §IV-A is performed on a validation set or on the test set, which compounds the risk of an inflated macro F1. The absence of code, split identifiers, or a comparison to the original Peskov et al. benchmark makes the number impossible to audit. This is a correctness risk in the evaluation protocol, not a disagreement with any field consensus. The parameter-efficiency claim is irrelevant to this concern: even a 1,345-parameter model can overfit to game identity or message-level context if the split leaks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PU-Lie, a lightweight deception detection model for the Diplomacy dataset, combining frozen BERT embeddings, handcrafted linguistic and game-specific features, and a Positive-Unlabeled (PU) learning objective. The authors report a macro F1 of 0.60 with only 1,345 trainable parameters, claiming a 650x reduction over prior models and a new state-of-the-art result. The paper compares seven models on the task, reporting macro F1, parameter counts, training time, and inference time, and includes ablation-style discussion. The central claim is that PU learning with a compact architecture outperforms heavier supervised baselines on a highly imbalanced strategic-dialogue task.","tokens_in":6493,"tokens_out":5006,"duration_ms":54531,"significance":"If the reported result is reproducible and methodologically sound, PU-Lie would be an interesting demonstration that a PU-learning objective combined with frozen BERT and handcrafted features can handle extreme class imbalance in deception detection with a remarkably small trainable parameter count. The paper addresses a relevant and under-studied task, and its architecture is simple and interpretable. The claimed efficiency and performance would be of practical interest if the evaluation protocol is confirmed. However, the current manuscript does not provide sufficient detail to establish the headline result, and several load-bearing evaluation choices are underspecified.","major_comments":[{"comment":"The manuscript never states whether the train/test split is performed at the message, dialogue, or game level. Because PU-Lie consumes game-specific features such as season, game score delta, and player roles, and the dataset comprises only 12 complete games, a message-level split would allow the model to exploit shared game context between training and test, inflating the reported macro F1 of 0.60. The authors must specify the split protocol, report the number of messages and dialogues in each split, and provide a game-level split result to demonstrate that the model generalizes to unseen strategic dialogues.","section":"§III-A, §IV-A"},{"comment":"The description of precision-recall curve-based threshold tuning is ambiguous: it does not state whether the threshold is selected on a held-out validation set or on the test set. If the threshold is tuned on the test set, the reported macro F1 is an optimistic estimate of generalization. Please clarify the validation procedure, report the chosen threshold value, and include precision and recall for both the deceptive and truthful classes.","section":"§IV-A"},{"comment":"The PU objective assumes that labeled deceptive messages are a random subset of all deceptive messages (the SCAR assumption), and the class prior is fixed to π=0.05, which matches the dataset's overall deceptive rate. The paper does not justify this random-subset assumption for manually annotated deception labels, and the limitation section (Section VI) only mentions the static prior without addressing the random-subset assumption or the threshold-tuning dependence. Provide a discussion of the annotation process in support of SCAR, and include a sensitivity analysis of π to show that the headline macro F1 is not an artifact of this prior choice.","section":"§III-B3, §VI"},{"comment":"The evaluation reports only macro F1 and standard deviation; the paper gives no per-class precision or recall, no statistical significance tests, and no direct comparison against the original Peskov et al. benchmark or other published systems on the Diplomacy deception task. The 'new state-of-the-art' claim is unsupported without such comparisons, and the reported macro F1 could be driven by the threshold-tuning procedure rather than by the model's learned representations. Please add per-class metrics, significance testing across the five seeds, and a comparison with prior published numbers on this dataset.","section":"Table I, §IV-A"},{"comment":"The abstract and introduction state that PU-Lie reduces trainable parameters by 'over 650x,' but Table I reports baseline trainable parameter counts of 1,101,954 to 2,300,000; the ratio to PU-Lie's 1,345 parameters is approximately 800x–1,700x, not 650x. This numerical inconsistency undermines the efficiency claim as stated. The reduction factor should be corrected and the reference model for the comparison should be identified explicitly.","section":"Abstract, Introduction, Table I"}],"minor_comments":[{"comment":"The paragraph beginning 'To this end, we introduce PU-Lie...' appears twice verbatim in Section I; the duplicate should be removed.","section":"Introduction"},{"comment":"Figure 1 is referenced in the text but no figure content is included in the provided manuscript; ensure the architecture figure is present and properly labeled in the final version.","section":"III-B2, Figure 1"},{"comment":"The 'ablation studies' in Section V-B largely repeat the model descriptions from Section IV-B rather than presenting a structured quantitative ablation. Consider reorganizing the section to show the contribution of each feature group (BERT, linguistic features, game features, PU loss) through controlled variations.","section":"§IV-B, §V-B"},{"comment":"The hardware and implementation details (GPU/CPU model, deep-learning framework, and batch sizes for the baseline models) are not reported, making the training and inference time comparisons difficult to interpret.","section":"Table I"},{"comment":"Reference [15] (Zhou et al., 'Learning with local and global consistency') does not correspond to TF-IDF + Logistic Regression; an appropriate citation for the classical baseline should be provided.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The split ambiguity in the evaluation protocol is the most serious issue in this manuscript. If the authors cannot provide a leakage-free evaluation (e.g., a game-level split) or at least a clear split description with per-class metrics, the headline claim will not be credible. The numeric inconsistency in the claimed parameter reduction factor is also concerning and should be corrected. I recommend major revision rather than rejection at this stage because the identified problems are in principle fixable with additional experiments and reporting, but the current evidence is insufficient to support the stated state-of-the-art claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PU-Lie is a reasonable idea with an unverifiable headline number. The paper applies positive-unlabeled learning to deception detection in Diplomacy, using frozen BERT embeddings plus handcrafted linguistic and game features. That combination is new for this task, and the design is sensible: PU learning is a natural fit for a setting with roughly 4.5% positive labels, and keeping BERT frozen gives a model with only 1,345 trainable parameters. Running seven baselines and ablations is also more than many short submissions do.\n\nThe soft spot is the evaluation protocol, and it is load-bearing. The paper never states whether the train/test split is by message, dialogue, or game. With only 12 complete games, this is not a detail: game-specific features like season, score delta, and player roles can encode shared game state, so a random message-level split would let the model exploit context from the same game in the test set, inflating the reported 0.60 macro F1. The precision-recall threshold tuning is described in Section IV-A without saying whether it is done on a validation set or the test set. The paper also reports no per-class precision or recall, no comparison to the original Peskov et al. benchmark, and no code or split identifiers. These omissions make the central claim impossible to audit.\n\nThe paper does acknowledge the static class prior limitation, but it does not discuss the SCAR assumption (whether annotated deceptive messages are a random subset of all deceptive messages). That is a second, quieter problem. The duplicated paragraphs in the introduction and contributions suggest a rushed write-up, but that is fixable.\n\nWho is this for? People working on deception detection or PU learning in NLP. The direction is worth taking seriously, but as submitted the evidence does not support the state-of-the-art claim. I would not cite the 0.60 number. A serious referee could extract a solid paper if the authors provide the missing split and threshold details, plus a comparison to the existing benchmark. On the current version, I would desk reject with encouragement to resubmit after those experiments.","headline":"PU-Lie pairs a sensible PU-learning approach with an unverifiable headline F1: missing split details and threshold tuning make the 0.60 macro F1 impossible to audit.","tokens_in":7069,"tokens_out":2963,"would_cite":false,"duration_ms":36202,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 1,345-parameter model, PU-Lie, pairs frozen BERT with handcrafted features and a positive-unlabeled objective to report a macro F1 of 0.60 on Diplomacy deception detection — a new best with 650x fewer trainable parameters.","keywords":["deception detection","positive-unlabeled learning","Diplomacy dataset","class imbalance","macro F1","lightweight model","BERT embeddings","linguistic features"],"falsifier":"Retrain PU-Lie under a strict game-level split — train on some of the 12 games and test only on held-out games — and compare the macro F1 with the reported 0.60; a sizable drop would indicate the original score exploited shared game context rather than deception cues. A second check tests the PU assumption directly: if labeled deceptive messages have systematically different feature distributions from the deceptive messages the model finds in the unlabeled pool, the assumption that labels are a random sample of deceptive content is violated and the risk estimate is biased.","tokens_in":6023,"feed_emoji":"🕵️","tokens_out":16422,"duration_ms":158801,"temperature":0.7,"pith_summary":"Deception in strategic dialogue is hard to detect because lies are rare — under 5% of the annotated messages in the Diplomacy dataset — and expensive to miss. This paper argues that the right response is not a bigger supervised model but a different learning objective: treat the task as positive-unlabeled (PU) learning, in which only the few labeled deceptive messages are trusted and everything else is unlabeled. The resulting PU-Lie model reports a macro F1 of 0.60, the best in its seven-model comparison, while training only 1,345 parameters — about 650x fewer than the BERT+LSTM baselines — and cutting training time to under two minutes. If the claims hold, the lesson is that under extreme class imbalance, explicitly modeling the rare class with a PU objective plus modest handcrafted features beats much heavier fully supervised architectures, and that prioritizing deception over truth is the right design choice for high-stakes dialogue.","feed_headline":"New best lie-detection score from a 1,345-parameter model","feed_subtitle":"Frozen BERT plus handcrafted features and positive-unlabeled learning beat supervised baselines 650x larger.","key_machinery":"The load-bearing component is the positive-unlabeled (PU) learning objective: instead of binary cross-entropy over both classes, the model estimates the risk of misclassification using only the labeled deceptive messages and the unlabeled set, with a class prior $\\pi = 0.05$, following the standard PU risk formulation from the literature it cites. Around that objective the architecture stays deliberately small — frozen bert-base-uncased embeddings supply context, a parallel extractor computes linguistic features (pronouns, hedges, assertiveness, sentiment) and game-specific metadata (season, score delta, player roles), a small linear-ReLU-dropout FeatureNet fuses the handcrafted features, and a single-logit linear classifier outputs the score. Only the FeatureNet and classifier weights are trained, giving 1,345 trainable parameters, and a precision-recall threshold sweep replaces the 0.5 decision boundary to favor recall on the deceptive class.","core_discovery":"The paper's central claim is that deception detection in the Diplomacy dataset is better posed as a positive-unlabeled problem than as standard binary classification. Because only about 4.5% of messages carry deception labels and the rest are unlabeled, PU-Lie estimates classification risk from the known deceptive messages plus the unlabeled pool, using a class prior set to $\\pi = 0.05$, instead of assuming reliable negative labels that the dataset does not really provide. The model fuses frozen BERT sentence embeddings with handcrafted linguistic cues (pronoun ratios, hedge usage, assertiveness, VADER sentiment) and game metadata (season, score delta, player roles), trains only the 1,345-parameter FeatureNet and classifier with a PU loss, and tunes its decision threshold on the precision-recall curve. Across five seeded runs it reports macro F1 of 0.60, ahead of six baselines that include BERT+LSTM hybrids, an oversampled variant, and a graph attention network, while reducing trainable parameters by over 650x and cutting training and inference time to 1.7 minutes and 5 seconds. The authors present this as evidence that a lightweight, interpretable model tailored to the rare deceptive class is the better route than heavier models trained for balanced classification.","pith_inferences":["If the train/test split is at message level rather than game level, the season, score-delta, and role features could leak shared context between the two sides, and the reported 0.60 would not reflect performance on unseen games; re-running under a game-level split would settle this.","Manual annotation likely labels the most conspicuous lies, which violates the PU assumption that labeled positives are a random subset of all deceptive messages and would bias the risk estimate; comparing the features of labeled deceptive messages with those the model discovers in the unlabeled pool would expose the bias.","The 650x parameter reduction counts trainable parameters only, whereas inference still executes the full bert-base-uncased encoder, so the practical efficiency gain depends on how the frozen backbone is accounted for.","The paper's own future-work list — dynamic class prior, cross-domain transfer, real-time detection — is also the natural test battery: a dynamic estimator would directly address the static $\\pi = 0.05$ limitation the paper acknowledges."],"forward_implications":["Explicit positive-unlabeled risk estimation with a $\\pi = 0.05$ prior outperforms fully supervised baselines on the 4.5% deceptive class, so the binding constraint in this task is the learning objective, not model capacity.","A trainable head of 1,345 parameters suffices once frozen BERT and handcrafted features carry the representation, which makes strong deception detection compatible with real-time and low-resource deployment.","The seven-model comparison implies that oversampling, focal-loss-style weighting, and graph structure are weaker remedies for extreme imbalance than treating unlabeled messages as a mixture of deceptive and truthful content.","Because missing a lie is costlier than a false alarm, deception-first evaluation — macro F1 and precision-recall threshold tuning — is the appropriate lens for model selection in strategic dialogue settings."],"supporting_citations":[{"why":"Introduces the Diplomacy dataset of 17,289 annotated messages that defines the task and its baseline; the text cites this source but the reference list omits the entry.","marker":"Peskov et al. (2020)"},{"why":"Supplies the positive-unlabeled learning framework and risk estimation that the PU loss module implements.","marker":"[7]"},{"why":"Provides the class-prior approach for learning classifiers from only positive and unlabeled data that the objective builds on.","marker":"[8]"},{"why":"Defines BERT, whose frozen bert-base-uncased encoder produces the sentence embeddings.","marker":"[3]"},{"why":"The prior NLP application of PU learning, to review spam, that motivates using it for deception detection.","marker":"[14]"},{"why":"Focal loss, the imbalance remedy the paper contrasts with PU learning.","marker":"[6]"},{"why":"Defines the graph attention mechanism behind the GNN baseline PU-Lie is compared against.","marker":"[10]"}],"fun_headline_variants":["PU-Lie: tiny 1,345-param model beats big baselines","Positive-unlabeled learning yields best lie-detection score","Deception detection: PU learning with 650x fewer parameters","1,345 parameters: new best macro F1 for lie detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 0.60 macro F1 rests on the unstated premise that the training and test sets draw from different messages without shared game context — if messages from the same game appear on both sides, the game-specific features could leak information and the score would overstate how well PU-Lie detects lies in new dialogues.","fun_headline_variants_meta":{"raw":{"variants":["PU-Lie: tiny 1,345-param model beats big baselines","Positive-unlabeled learning yields best lie-detection score","Deception detection: PU learning with 650x fewer parameters","1,345 parameters: new best macro F1 for lie detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1488,"prompt_tokens":999,"completion_tokens":489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":427}},"tokens_in":615,"tokens_out":489,"duration_ms":6016,"temperature":1.0,"reasoning_tokens":427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:02:19.996561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain PU-Lie under a strict game-level split — train on some of the 12 games and test only on held-out games — and compare the macro F1 with the reported 0.60; a sizable drop would indicate the original score exploited shared game context rather than deception cues. A second check tests the PU assumption directly: if labeled deceptive messages have systematically different feature distributions from the deceptive messages the model finds in the unlabeled pool, the assumption that labels are a random sample of deceptive content is violated and the risk estimate is biased.","supporting_citations":[{"cited_title":"Learning from positive and unlabeled data: A survey,","cited_arxiv_id":null,"evidence_quote":"Supplies the positive-unlabeled learning framework and risk estimation that the PU loss module implements."},{"cited_title":"Learning classifiers from only positive and unlabeled data,","cited_arxiv_id":null,"evidence_quote":"Provides the class-prior approach for learning classifiers from only positive and unlabeled data that the objective builds on."},{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"Defines BERT, whose frozen bert-base-uncased encoder produces the sentence embeddings."},{"cited_title":"Learning to identify review spam,","cited_arxiv_id":null,"evidence_quote":"The prior NLP application of PU learning, to review spam, that motivates using it for deception detection."},{"cited_title":"Graph attention networks,","cited_arxiv_id":null,"evidence_quote":"Defines the graph attention mechanism behind the GNN baseline PU-Lie is compared against."}],"review_version":1}