PU-Lie: Lightweight Deception Detection in Imbalanced Diplomatic Dialogues via Positive-Unlabeled Learning

Bhavinkumar Vinodbhai Kuwar; Bikrant Bikram Pratap Maurya; Nitin Choudhury; Priyanshu Gupta

REVIEW 5 major objections 5 minor 15 references

PU-Lie: Lightweight Deception Detection in Imbalanced Diplomatic Dialogues via Positive-Unlabeled Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 1,345-parameter model, PU-Lie, pairs frozen BERT with handcrafted features and a positive-unlabeled objective to report a macro F1 of 0.60 on Diplomacy deception detection — a new best with 650x fewer trainable parameters.

desk verdict PU-Lie pairs a sensible PU-learning approach with an unverifiable headline F1: missing split details and threshold tuning make the 0.60 macro F1 impossible to audit. read the letter →

arxiv 2507.09157 v1 pith:KLJWINPM submitted 2025-07-12 cs.CL

Bhavinkumar Vinodbhai Kuwar , Bikrant Bikram Pratap Maurya , Priyanshu Gupta , Nitin Choudhury This is my paper · ORCID

classification cs.CL

keywords deceptiondetectionpositive-unlabeledlearningDiplomacydatasetclassimbalancemacroF1lightweightmodelBERTembeddingslinguisticfeatures

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deception in strategic dialogue is hard to detect because lies are rare — under 5% of the annotated messages in the Diplomacy dataset — and expensive to miss. This paper argues that the right response is not a bigger supervised model but a different learning objective: treat the task as positive-unlabeled (PU) learning, in which only the few labeled deceptive messages are trusted and everything else is unlabeled. The resulting PU-Lie model reports a macro F1 of 0.60, the best in its seven-model comparison, while training only 1,345 parameters — about 650x fewer than the BERT+LSTM baselines — and cutting training time to under two minutes. If the claims hold, the lesson is that under extreme class imbalance, explicitly modeling the rare class with a PU objective plus modest handcrafted features beats much heavier fully supervised architectures, and that prioritizing deception over truth is the right design choice for high-stakes dialogue.

What carries the argument

The load-bearing component is the positive-unlabeled (PU) learning objective: instead of binary cross-entropy over both classes, the model estimates the risk of misclassification using only the labeled deceptive messages and the unlabeled set, with a class prior $\pi = 0.05$, following the standard PU risk formulation from the literature it cites. Around that objective the architecture stays deliberately small — frozen bert-base-uncased embeddings supply context, a parallel extractor computes linguistic features (pronouns, hedges, assertiveness, sentiment) and game-specific metadata (season, score delta, player roles), a small linear-ReLU-dropout FeatureNet fuses the handcrafted features, and a single-logit linear classifier outputs the score. Only the FeatureNet and classifier weights are trained, giving 1,345 trainable parameters, and a precision-recall threshold sweep replaces the 0.5 decision boundary to favor recall on the deceptive class.

What would settle it

Retrain PU-Lie under a strict game-level split — train on some of the 12 games and test only on held-out games — and compare the macro F1 with the reported 0.60; a sizable drop would indicate the original score exploited shared game context rather than deception cues. A second check tests the PU assumption directly: if labeled deceptive messages have systematically different feature distributions from the deceptive messages the model finds in the unlabeled pool, the assumption that labels are a random sample of deceptive content is violated and the risk estimate is biased.

Watch

Extended reading notes

Core claim

The paper's central claim is that deception detection in the Diplomacy dataset is better posed as a positive-unlabeled problem than as standard binary classification. Because only about 4.5% of messages carry deception labels and the rest are unlabeled, PU-Lie estimates classification risk from the known deceptive messages plus the unlabeled pool, using a class prior set to $\pi = 0.05$, instead of assuming reliable negative labels that the dataset does not really provide. The model fuses frozen BERT sentence embeddings with handcrafted linguistic cues (pronoun ratios, hedge usage, assertiveness, VADER sentiment) and game metadata (season, score delta, player roles), trains only the 1,345-parameter FeatureNet and classifier with a PU loss, and tunes its decision threshold on the precision-recall curve. Across five seeded runs it reports macro F1 of 0.60, ahead of six baselines that include BERT+LSTM hybrids, an oversampled variant, and a graph attention network, while reducing trainable parameters by over 650x and cutting training and inference time to 1.7 minutes and 5 seconds. The authors present this as evidence that a lightweight, interpretable model tailored to the rare deceptive class is the better route than heavier models trained for balanced classification.

Load-bearing premise

The reported 0.60 macro F1 rests on the unstated premise that the training and test sets draw from different messages without shared game context — if messages from the same game appear on both sides, the game-specific features could leak information and the score would overstate how well PU-Lie detects lies in new dialogues.

Editorial extensions

If this is right

Explicit positive-unlabeled risk estimation with a $\pi = 0.05$ prior outperforms fully supervised baselines on the 4.5% deceptive class, so the binding constraint in this task is the learning objective, not model capacity.
A trainable head of 1,345 parameters suffices once frozen BERT and handcrafted features carry the representation, which makes strong deception detection compatible with real-time and low-resource deployment.
The seven-model comparison implies that oversampling, focal-loss-style weighting, and graph structure are weaker remedies for extreme imbalance than treating unlabeled messages as a mixture of deceptive and truthful content.
Because missing a lie is costlier than a false alarm, deception-first evaluation — macro F1 and precision-recall threshold tuning — is the appropriate lens for model selection in strategic dialogue settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the train/test split is at message level rather than game level, the season, score-delta, and role features could leak shared context between the two sides, and the reported 0.60 would not reflect performance on unseen games; re-running under a game-level split would settle this.
Manual annotation likely labels the most conspicuous lies, which violates the PU assumption that labeled positives are a random subset of all deceptive messages and would bias the risk estimate; comparing the features of labeled deceptive messages with those the model discovers in the unlabeled pool would expose the bias.
The 650x parameter reduction counts trainable parameters only, whereas inference still executes the full bert-base-uncased encoder, so the practical efficiency gain depends on how the frozen backbone is accounted for.
The paper's own future-work list — dynamic class prior, cross-domain transfer, real-time detection — is also the natural test battery: a dynamic estimator would directly address the static $\pi = 0.05$ limitation the paper acknowledges.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

PU-Lie is a reasonable idea with an unverifiable headline number. The paper applies positive-unlabeled learning to deception detection in Diplomacy, using frozen BERT embeddings plus handcrafted linguistic and game features. That combination is new for this task, and the design is sensible: PU learning is a natural fit for a setting with roughly 4.5% positive labels, and keeping BERT frozen gives a model with only 1,345 trainable parameters. Running seven baselines and ablations is also more than many short submissions do.

The soft spot is the evaluation protocol, and it is load-bearing. The paper never states whether the train/test split is by message, dialogue, or game. With only 12 complete games, this is not a detail: game-specific features like season, score delta, and player roles can encode shared game state, so a random message-level split would let the model exploit context from the same game in the test set, inflating the reported 0.60 macro F1. The precision-recall threshold tuning is described in Section IV-A without saying whether it is done on a validation set or the test set. The paper also reports no per-class precision or recall, no comparison to the original Peskov et al. benchmark, and no code or split identifiers. These omissions make the central claim impossible to audit.

The paper does acknowledge the static class prior limitation, but it does not discuss the SCAR assumption (whether annotated deceptive messages are a random subset of all deceptive messages). That is a second, quieter problem. The duplicated paragraphs in the introduction and contributions suggest a rushed write-up, but that is fixable.

Who is this for? People working on deception detection or PU learning in NLP. The direction is worth taking seriously, but as submitted the evidence does not support the state-of-the-art claim. I would not cite the 0.60 number. A serious referee could extract a solid paper if the authors provide the missing split and threshold details, plus a comparison to the existing benchmark. On the current version, I would desk reject with encouragement to resubmit after those experiments.

Referee Report

5 major / 5 minor

Summary. The paper proposes PU-Lie, a lightweight deception detection model for the Diplomacy dataset, combining frozen BERT embeddings, handcrafted linguistic and game-specific features, and a Positive-Unlabeled (PU) learning objective. The authors report a macro F1 of 0.60 with only 1,345 trainable parameters, claiming a 650x reduction over prior models and a new state-of-the-art result. The paper compares seven models on the task, reporting macro F1, parameter counts, training time, and inference time, and includes ablation-style discussion. The central claim is that PU learning with a compact architecture outperforms heavier supervised baselines on a highly imbalanced strategic-dialogue task.

Significance. If the reported result is reproducible and methodologically sound, PU-Lie would be an interesting demonstration that a PU-learning objective combined with frozen BERT and handcrafted features can handle extreme class imbalance in deception detection with a remarkably small trainable parameter count. The paper addresses a relevant and under-studied task, and its architecture is simple and interpretable. The claimed efficiency and performance would be of practical interest if the evaluation protocol is confirmed. However, the current manuscript does not provide sufficient detail to establish the headline result, and several load-bearing evaluation choices are underspecified.

major comments (5)

[§III-A, §IV-A] The manuscript never states whether the train/test split is performed at the message, dialogue, or game level. Because PU-Lie consumes game-specific features such as season, game score delta, and player roles, and the dataset comprises only 12 complete games, a message-level split would allow the model to exploit shared game context between training and test, inflating the reported macro F1 of 0.60. The authors must specify the split protocol, report the number of messages and dialogues in each split, and provide a game-level split result to demonstrate that the model generalizes to unseen strategic dialogues.
[§IV-A] The description of precision-recall curve-based threshold tuning is ambiguous: it does not state whether the threshold is selected on a held-out validation set or on the test set. If the threshold is tuned on the test set, the reported macro F1 is an optimistic estimate of generalization. Please clarify the validation procedure, report the chosen threshold value, and include precision and recall for both the deceptive and truthful classes.
[§III-B3, §VI] The PU objective assumes that labeled deceptive messages are a random subset of all deceptive messages (the SCAR assumption), and the class prior is fixed to π=0.05, which matches the dataset's overall deceptive rate. The paper does not justify this random-subset assumption for manually annotated deception labels, and the limitation section (Section VI) only mentions the static prior without addressing the random-subset assumption or the threshold-tuning dependence. Provide a discussion of the annotation process in support of SCAR, and include a sensitivity analysis of π to show that the headline macro F1 is not an artifact of this prior choice.
[Table I, §IV-A] The evaluation reports only macro F1 and standard deviation; the paper gives no per-class precision or recall, no statistical significance tests, and no direct comparison against the original Peskov et al. benchmark or other published systems on the Diplomacy deception task. The 'new state-of-the-art' claim is unsupported without such comparisons, and the reported macro F1 could be driven by the threshold-tuning procedure rather than by the model's learned representations. Please add per-class metrics, significance testing across the five seeds, and a comparison with prior published numbers on this dataset.
[Abstract, Introduction, Table I] The abstract and introduction state that PU-Lie reduces trainable parameters by 'over 650x,' but Table I reports baseline trainable parameter counts of 1,101,954 to 2,300,000; the ratio to PU-Lie's 1,345 parameters is approximately 800x–1,700x, not 650x. This numerical inconsistency undermines the efficiency claim as stated. The reduction factor should be corrected and the reference model for the comparison should be identified explicitly.

minor comments (5)

[Introduction] The paragraph beginning 'To this end, we introduce PU-Lie...' appears twice verbatim in Section I; the duplicate should be removed.
[III-B2, Figure 1] Figure 1 is referenced in the text but no figure content is included in the provided manuscript; ensure the architecture figure is present and properly labeled in the final version.
[§IV-B, §V-B] The 'ablation studies' in Section V-B largely repeat the model descriptions from Section IV-B rather than presenting a structured quantitative ablation. Consider reorganizing the section to show the contribution of each feature group (BERT, linguistic features, game features, PU loss) through controlled variations.
[Table I] The hardware and implementation details (GPU/CPU model, deep-learning framework, and batch sizes for the baseline models) are not reported, making the training and inference time comparisons difficult to interpret.
[References] Reference [15] (Zhou et al., 'Learning with local and global consistency') does not correspond to TF-IDF + Logistic Regression; an appropriate citation for the classical baseline should be provided.

Circularity Check

1 steps flagged · score 3.0 of 10

No self-citation or derivation-level circularity, but the headline 0.60 macro F1 is partly constructed by selecting the decision threshold on the test PR curve.

fitted input called prediction [Section IV-A (Evaluation Protocol); headline claim in Abstract and Section I]
"For PU-Lie, we apply precision-recall curve-based threshold tuning to better capture deceptive messages by maximizing recall at an optimal precision."

The reported macro F1 of 0.60 is not the performance of a fixed decision rule evaluated on held-out data; the decision threshold is selected from the test PR curve, and the F1 is then computed on the same test set. This makes the headline number, by construction, the F1 at a threshold chosen to optimize recall on the test labels rather than an independent prediction. The paper does not state that the threshold tuning is performed on a separate validation set, so the reported result re-imports test-label information into the model's decision rule.

full rationale

The paper contains no self-citations that are load-bearing, no imported uniqueness theorems, and no ansatz smuggled in via prior work. The PU learning objective, frozen BERT embeddings, and handcrafted features are standard components, and the central claim that a lightweight PU model can outperform heavier baselines is not equivalent to its inputs by definition. The only partial circularity is the PR-curve threshold tuning in Section IV-A: because the threshold is chosen using the test set and the headline macro F1 is reported from that same test set, the 0.60 figure is partly an optimized fit rather than an unbiased estimate. A separate correctness concern, not circularity, is the unstated train/test split level; if messages from the same game appear in both training and test, game-specific features such as season, score delta, and roles could leak shared context and inflate the number. That concern does not change the circularity score but reinforces that the 0.60 result should be treated with caution.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. Its central claim rests on two data-derived numbers (class prior and decision threshold), the untested SCAR assumption, and an unstated split protocol. These are the main burden on the reported result.

free parameters (3)

Class prior pi = 0.05
Set to approximate the reported 4.5% deceptive rate and used in the PU risk estimate in Section III-B.3. A wrong prior biases the loss and the final F1.
Decision threshold = not reported
Tuned via precision-recall curve in Section IV-A to maximize recall at an optimal precision. The final macro F1 depends on this threshold, and it is unclear whether tuning is done on validation or test.
FeatureNet dimensions and dropout = not reported
The feed-forward subnetwork is described only as a linear layer with ReLU and dropout; dimensions are not given, so exact reproduction is impossible.

assumptions (3)

domain assumption Selected Completely At Random (SCAR): labeled deceptive messages are a random subset of all deceptive messages.
PU learning's unbiased risk estimate requires this assumption. It enters in Section III-B.3 where the PU loss treats labeled positives and the unlabeled set as a mixture. The authors do not test or discuss the plausibility of SCAR for manual deception annotations.
domain assumption Static class prior equal to 0.05.
The authors state in Section VI that PU-Lie assumes a static class prior. If the true deception rate differs from 0.05, the loss is biased.
domain assumption Train/test split does not leak game-level information.
The paper never states whether the split is at message level or game/dialogue level. If same-game messages appear in both train and test, game features such as season, score delta, and roles leak information, invalidating the reported F1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PU-Lie: Lightweight Deception Detection in Imbalanced Diplomatic Dialogues via Positive-Unlabeled Learning." pith.science (2026). https://pith.science/paper/KLJWINPM

@misc{pith2026250709157,
  author       = {Pith},
  title        = {Pith review of: PU-Lie: Lightweight Deception Detection in Imbalanced Diplomatic Dialogues via Positive-Unlabeled Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KLJWINPM}},
  note         = {Machine review of arXiv:2507.09157}
}

read the original abstract

Detecting deception in strategic dialogues is a complex and high-stakes task due to the subtlety of language and extreme class imbalance between deceptive and truthful communications. In this work, we revisit deception detection in the Diplomacy dataset, where less than 5% of messages are labeled deceptive. We introduce a lightweight yet effective model combining frozen BERT embeddings, interpretable linguistic and game-specific features, and a Positive-Unlabeled (PU) learning objective. Unlike traditional binary classifiers, PU-Lie is tailored for situations where only a small portion of deceptive messages are labeled, and the majority are unlabeled. Our model achieves a new best macro F1 of 0.60 while reducing trainable parameters by over 650x. Through comprehensive evaluations and ablation studies across seven models, we demonstrate the value of PU learning, linguistic interpretability, and speaker-aware representations. Notably, we emphasize that in this problem setting, accurately detecting deception is more critical than identifying truthful messages. This priority guides our choice of PU learning, which explicitly models the rare but vital deceptive class.

Figures

Figures reproduced from arXiv: 2507.09157 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

[1]

Lying for money: How people lie differently when they are paid more,

J. T. Hancock, L. E. Curry, S. Goorha, and M. Woodworth, “Lying for money: How people lie differently when they are paid more,” Personality and Social Psychology Bulletin , vol. 34, no. 4, pp. 536–548, 2008

work page 2008
[2]

Finding deceptive opinion spam by any stretch of the imagination,

M. Ott, Y . Choi, C. Cardie, and J. T. Hancock, “Finding deceptive opinion spam by any stretch of the imagination,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics , 2011

work page 2011
[3]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” NAACL- HLT, 2019

work page 2019
[4]

Csi: A hybrid deep model for fake news detection,

N. Ruchansky, S. Seo, and Y . Liu, “Csi: A hybrid deep model for fake news detection,” in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , 2017, pp. 797–806

work page 2017
[5]

Linguistic deception detection using neural networks,

M. V ollmer and S. Adali, “Linguistic deception detection using neural networks,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021

work page 2021
[6]

Focal loss for dense object detection,

T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 2980–2988

2017
[7]

Learning from positive and unlabeled data: A survey,

J. Bekker and J. Davis, “Learning from positive and unlabeled data: A survey,” Machine Learning, vol. 109, pp. 719–760, 2020

work page 2020
[8]

Learning classifiers from only positive and unlabeled data,

C. Elkan and K. Noto, “Learning classifiers from only positive and unlabeled data,” Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pp. 213–220, 2008

work page 2008

Show all 15 references

[9]

Explain yourself! leveraging language models for commonsense reasoning,

N. F. Rajani, B. McCann, C. Xiong, and R. Socher, “Explain yourself! leveraging language models for commonsense reasoning,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

2019
[10]

Graph attention networks,

P. Veli ˇckovi´c, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y . Bengio, “Graph attention networks,” in International Conference on Learning Representations (ICLR) , 2018

2018
[11]

Structure-aware dialogue state tracking,

T. Zhang, R. Xu, Q. Zhang, and Z. Wu, “Structure-aware dialogue state tracking,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) , 2021

2021
[12]

Syntactic stylometry for deception detection,

S. Feng, R. Banerjee, and Y . Choi, “Syntactic stylometry for deception detection,” ACL, 2012

2012
[13]

Smote: Synthetic minority over-sampling technique,

N. V . Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: Synthetic minority over-sampling technique,” Journal of artificial intelligence research, vol. 16, pp. 321–357, 2002

2002
[14]

Learning to identify review spam,

F. Li, M. Huang, Y . Yang, and X. Zhu, “Learning to identify review spam,” in Proceedings of the Twenty-Second international joint conference on Artificial Intelligence, 2011

2011
[15]

Learning with local and global consistency,

D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Sch ¨olkopf, “Learning with local and global consistency,” in Advances in neural information processing systems, 2004

2004

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Lying for money: How people lie differently when they are paid more,

J. T. Hancock, L. E. Curry, S. Goorha, and M. Woodworth, “Lying for money: How people lie differently when they are paid more,” Personality and Social Psychology Bulletin , vol. 34, no. 4, pp. 536–548, 2008

work page 2008

[2] [2]

Finding deceptive opinion spam by any stretch of the imagination,

M. Ott, Y . Choi, C. Cardie, and J. T. Hancock, “Finding deceptive opinion spam by any stretch of the imagination,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics , 2011

work page 2011

[3] [3]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” NAACL- HLT, 2019

work page 2019

[4] [4]

Csi: A hybrid deep model for fake news detection,

N. Ruchansky, S. Seo, and Y . Liu, “Csi: A hybrid deep model for fake news detection,” in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , 2017, pp. 797–806

work page 2017

[5] [5]

Linguistic deception detection using neural networks,

M. V ollmer and S. Adali, “Linguistic deception detection using neural networks,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021

work page 2021

[6] [6]

Focal loss for dense object detection,

T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 2980–2988

2017

[7] [7]

Learning from positive and unlabeled data: A survey,

J. Bekker and J. Davis, “Learning from positive and unlabeled data: A survey,” Machine Learning, vol. 109, pp. 719–760, 2020

work page 2020

[8] [8]

Learning classifiers from only positive and unlabeled data,

C. Elkan and K. Noto, “Learning classifiers from only positive and unlabeled data,” Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pp. 213–220, 2008

work page 2008