REVIEW 3 major objections 6 minor 20 references
ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper introduces ViTOED, a manually annotated Vietnamese dataset of 10,985 social media comments with 21,244 opinion quadruples, and shows that current language models struggle to attribute emotions to their targets.
desk verdict Genuinely needed Vietnamese target-oriented emotion dataset, but the annotation reliability and internal consistency need serious work before it can be used. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing representation is the opinion quadruple (source, target, expression, polarity), where each component is annotated as a token span inside a Vietnamese comment. The baseline mechanism is the structured sentiment graph from [4], which converts each sentence into a dependency graph whose nodes are spans and whose edges carry source, target, and expression relations with polarity, using either head-first or head-final root choices. The graph is produced by a BiLSTM over word, POS-tag, character, and transformer-contextualized embeddings, scored with a bilinear transformation. This machinery turns emotion detection into a joint span-extraction and relation-extraction problem, and its failure modes become the paper's error taxonomy.
What would settle it
Have fresh annotators independently re-annotate a random sample of the test set using only the published guidelines, then compute pairwise F1 against the released source spans; if the agreement is no better than the original 0.32, or if training a model with source spans removed matches the full model's Targeted F1, then the source component is not carrying real signal and the quadruple claim is not supported.
Extended reading notes
Core claim
ViTOED is a human-annotated Vietnamese dataset of 10,985 user comments collected from social networks, containing 21,244 opinion quadruples of the form (source, target, expression, polarity), with source, target, and expression recorded as token spans and polarity as positive, negative, or neutral. The paper claims that this resource captures Vietnamese-specific annotation phenomena that sentence-level sentiment labels miss: 52.10% of opinions have an implicit source (category T-E only), targets are mentioned less often than opinions, and common pronouns such as 'tao' and 'mày' appear on both sides of the quadruple, creating vocabulary ambiguity. Using a structured sentiment graph baseline from [4] over nine pretrained language models, the paper reports that current models succeed at expression span detection but struggle with source and target spans, graph root placement, and relation alignment. The error analysis attributes these failures to three causes: vocabulary complexity in span recognition, root and edge misprediction, and source-target confusion caused by Vietnamese morphology. The intended payoff is a benchmark on which future Vietnamese target-oriented emotion detection systems can be measured, plus an explicit list of error types to attack.
Load-bearing premise
The dataset's usefulness rests on the assumption that the annotators' opinion quadruples are reliable enough to learn from; in particular, if the low source-span agreement (F1 = 0.32) means source annotations are effectively arbitrary, the quadruple structure that defines ViTOED loses its validity.
Editorial extensions
If this is right
- Vietnamese target-oriented emotion detection now has a public train/dev/test benchmark with 21,244 quadruples and fixed splits.
- Because 52.10% of opinions in ViTOED lack an explicit source, any successful Vietnamese model must learn to infer the commenter as an implicit source rather than only detecting explicit mentions.
- The structured sentiment graph baseline shows that both multilingual and monolingual Vietnamese language models leave substantial headroom on this task, particularly on source and target spans.
- The head-final graph representation and the +inlabel extension offer concrete starting points for improving expression and span extraction on this dataset.
- The paper's error taxonomy gives future work three defined targets: span misalignment, root and edge misprediction, and source-target confusion in Vietnamese morphology.
Reading between the lines
- Since source-span agreement is only F1 = 0.32, part of the reported performance gap on sources may reflect annotation noise rather than model weakness; a natural next test is to score models only on sentences where annotators agreed on the source span.
- The same quadruple annotation protocol could transfer to other subject-drop languages such as Japanese, Korean, or Thai, where implicit sources predominate, enabling cross-lingual comparison if those datasets adopt the same graph parser.
- A simple and testable extension suggested by the data is to add a Vietnamese first-person and second-person pronoun feature to the graph parser, which might directly reduce the source-target confusion the paper identifies.
- The paper reports per-component agreement but not exact quadruple-level agreement; computing the fraction of fully matching (source, target, expression, polarity) tuples would give a stricter, and possibly much lower, reliability estimate.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ViTOED, a new Vietnamese dataset for target-oriented emotion detection. It claims to contain 10,985 user comments and 21,244 manually annotated opinion quadruples, each consisting of source, target, expression, and polarity. The authors describe an annotation process with guidelines and four rounds of annotator training, analyze dataset statistics including opinion categories and word frequencies, and evaluate a structured sentiment graph baseline (following Barnes et al., 2021) with several Vietnamese pre-trained language models. The main contributions are stated as the dataset itself, an analysis of Vietnamese-specific phenomena such as implicit sources and targets, and baseline results that reveal challenges in span detection and relation extraction.
Significance. If the dataset is reliable, it would fill a real gap in Vietnamese target-oriented emotion detection resources and provide a benchmark that exposes how poorly current models connect emotions to specific targets in Vietnamese social media. The annotation of opinion quadruples is a new contribution, and the baseline experiments with multiple Vietnamese PLMs are a useful starting point. However, the central value of the paper depends entirely on the trustworthiness of the manual annotation, and the current evidence for that trustworthiness is weak.
major comments (3)
- [III.C, Table III] The paper's central claim of a dataset with 'strict guidelines' and reliable opinion quadruples is not supported by the reported inter-annotator agreement. After four training rounds, Source F1 is 0.32 and Polarity agreement is 0.47; these are the final reported values, measured only on a 210-sentence sample. No IAA is reported for the final corpus, and the adjudication procedure is only described as 'cross-checking among annotators' without detail. The statement that 'the level of agreement indicates that the annotators understood the labeling guidelines and followed the instructions accurately' directly contradicts the low Source and Polarity figures. Please report IAA on the full corpus, describe how disagreements were resolved, and either justify why Source F1 of 0.32 is sufficient or revise the claims about annotation quality.
- [III.B, Table I, Example 2] The annotation guidelines in Section III.B state that expressions containing profanity, rudeness, and sensitive aspects are classified as Negative, and that humor is Positive only if it does not contain hateful or offensive content. Example 2 in Table I, '7 thằng này không phải con người chúng mày ạ' ('These 7 guys are not human'), is labeled Positive even though calling people 'not human' is an offensive statement under any ordinary reading. This is an internal contradiction within the paper's own example. Either the example's label is wrong, or the guideline needs an explicit exception; as published, it undermines the claim of consistent annotation.
- [III.A/III.D, Tables II and IV] The dataset statistics are internally inconsistent. Section III.A reports 6,000 sentences from UIT-VSMEC plus 5,010 collected comments, totaling 11,010, but Sections III.D and the abstract state 10,985 comments. Summing Table II gives source counts 2,359+385+572=3,316, while the source-bearing categories in Table IV (S-T-E + S-E) total 2,243+1,527=3,770; the corresponding target counts are 8,632+1,160+2,127=11,919 versus 2,243+11,069=13,312. These numbers must agree because they count the same opinion quadruples. Additionally, the sentence 'The T-E demonstrated that an opinion missing the Source accounts for 52.10% of the total opinions' is misleading: T-E is only one of the two categories without a Source (the other is E), and the true proportion of opinions missing Source is (11,069+6,405)/21,244 = 82.25%. Please reconcile all counts and correct the interpretation.
minor comments (6)
- [I] There is a typo in 'sentencesin human language'; also, the notation 'opinion tuples O={o1, o2, ..., on}' should say 'quadruples' for consistency with the rest of the paper.
- [II] The phrase 'DSU nis' appears to be a typo or incorrect rendering of the dataset name; please check the official name of the dataset introduced in reference [6].
- [Figure 2] The label 'BERTologies' should likely be 'BERT' or 'BERT embeddings'.
- [III.C] The text cites 'Examples #3 and #4 in Table I' as cases lacking a source, but Table I contains only three examples and Example #3 has an explicit source ('tao'). Please correct the reference or extend the table.
- [III.C, Eq. (2)] The notation in Equation (2) uses the same symbols A and B for annotators and for token sets, which is confusing. Please rewrite with distinct notation, e.g., annotators 1 and 2 and sets X and Y.
- [V, Table VII] The table title 'Results of graph annotation by opinion components by F1(%)' is unclear, and the column layout mixes entity types with polarity classes; please clarify what each column and row represents.
Circularity Check
No circularity: ViTOED's quadruple annotations are new manual labels, the baseline is an external graph-parsing model, and the evaluation metrics are standard.
full rationale
The paper's central contribution is a manually annotated dataset, not a derived prediction or fitted constant. The annotation quadruples are produced from raw comments by human annotators according to stated guidelines; no quantity is defined in terms of another quantity that it is then said to predict. The baseline model is taken from the external work of Barnes et al. [4], and the metrics are the standard Span F1, Targeted F1, Parsing Graph F1, and Sentiment Graph F1 of that external framework. The only author-group citations are non-load-bearing: [3] supplies 6,000 source sentences (reused data, not a result), [11] supplies the F1 agreement formula, and [15]–[16] are among the evaluated language models. None of these citations forbids alternatives or supplies the conclusion. The IAA evidence in Section III.C is a reliability limitation rather than a circular step: the paper reports Source F1=0.32 and Polarity=0.47 after four rounds and no final-corpus agreement, so the assertion that 'the annotators understood the labeling guidelines' is under-supported, but this is an empirical support problem, not a reduction of a claim to its own input. Table I Example 2 also appears to contradict the stated polarity guideline, and the Source counts in Tables II and IV do not reconcile; these internal inconsistencies affect dataset quality, not circularity. The paper is therefore self-contained against external baselines and contains no derivation chain that reduces to its inputs.
Assumptions & free parameters
assumptions (4)
- domain assumption Target-oriented emotion detection is well-defined as extracting quadruples <source, target, expression, polarity> from each sentence.
- domain assumption The 6,000 UIT-VSMEC sentences plus 5,010 unspecified platform comments form a suitable sample of Vietnamese social media.
- standard math Token-span F1 and the pairwise label overlap measure in Equations 1 and 2 are valid estimators of annotator agreement.
- domain assumption The structured sentiment graph formalism of Barnes et al. is an appropriate baseline for the ViTOED task.
Cite this review
Pith. "Pith review of ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts." pith.science (2026). https://pith.science/paper/XB7BWCHN
@misc{pith2026260812776,
author = {Pith},
title = {Pith review of: ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts},
year = {2026},
howpublished = {\url{https://pith.science/paper/XB7BWCHN}},
note = {Machine review of arXiv:2608.12776}
}
read the original abstract
This paper introduces ViTOED, a novel dataset for target-oriented emotion detection in Vietnamese social media texts. The ViTOED comprises 10,985 user comments and 21,244 manually annotated opinion quadruples (source, target, expression, polarity) that follow strict guidelines. The dataset reveals Vietnamese-specific phenomena, such as implicit sources and targets and vocabulary ambiguities, enabling deeper analysis of user emotions toward entities. We propose a baseline using structured sentiment graphs and evaluate various Vietnamese pre-trained language models. The empirical results highlight challenges in span detection and relation extraction and indicate substantial room for model improvement in Vietnamese Target-Oriented Emotion Detection tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Liu,Sentiment analysis and opinion mining
B. Liu,Sentiment analysis and opinion mining. Springer Nature, 2022
work page 2022
-
[2]
Z. Li, Y . Zou, C. Zhang, Q. Zhang, and Z. Wei, “Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training,” inProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, Nov. 2021, pp. 246–256
work page 2021
-
[3]
Emotion recognition for vietnamese social media text,
V . A. Ho, D. H.-C. Nguyen, D. H. Nguyen, L. T.-V . Pham, D.-V . Nguyen, K. V . Nguyen, and N. L.-T. Nguyen, “Emotion recognition for vietnamese social media text,” inComputational Linguistics: 16th International Conference of the Pacific Association for Computational Linguistics, PACLING 2019, Hanoi, Vietnam, October 11–13, 2019. Springer, 2020, pp. 319–333
work page 2019
-
[4]
Structured sentiment analysis as dependency graph parsing,
J. Barnes, R. Kurtz, S. Oepen, L.Øvrelid, and E. Velldal, “Structured sentiment analysis as dependency graph parsing,” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, 2021, pp. 3387–3402
work page 2021
-
[5]
The sentiment problem: A critical survey towards deconstructing sentiment analysis,
P . Venkit, M. Srinath, S. Gautam, S. Venkatraman, V . Gupta, R. Pas- sonneau, and S. Wilson, “The sentiment problem: A critical survey towards deconstructing sentiment analysis,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Dec. 2023, pp. 13 743–13 763
work page 2023
-
[6]
Sentence and expression level annotation of opinions in user-generated discourse,
C. Toprak, N. Jakob, and I. Gurevych, “Sentence and expression level annotation of opinions in user-generated discourse,” inProceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, Jul. 2010, pp. 575–584
work page 2010
-
[7]
J. Barnes, T. Badia, and P . Lambert, “MultiBooked: A corpus of Basque and Catalan hotel reviews annotated for aspect-level sentiment classification,” inProceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018
work page 2018
-
[8]
A systematic literature review on vietnamese aspect-based sentiment analysis,
D. Van Thin, D. N. Hao, and N. L.-T. Nguyen, “A systematic literature review on vietnamese aspect-based sentiment analysis,”ACM Trans. Asian Low-Resour. Lang. Inf. Process., vol. 22, no. 8, aug 2023
work page 2023
Show all 20 references
-
[9]
Vlsp shared task: sentiment analysis,
H. T. Nguyen, H. V . Nguyen, Q. T. Ngo, L. X. Vu, V . M. Tran, B. X. Ngo, and C. A. Le, “Vlsp shared task: sentiment analysis,”Journal of Computer Science and Cybernetics, vol. 34, no. 4, pp. 295–310, 2018
2018
-
[10]
Sa2sl: From aspect-based sentiment analysis to social listening system for business intelligence,
L. Luc Phan, P . Huynh Pham, K. Thi-Thanh Nguyen, S. Khai Huynh, T. Thi Nguyen, L. Thanh Nguyen, T. Van Huynh, and K. Van Nguyen, “Sa2sl: From aspect-based sentiment analysis to social listening system for business intelligence,” inKnowledge Science, Engineering and Management...
2021
-
[11]
Span detection for aspect-based sentiment analysis in Vietnamese,
K. N. T. Thanh, S. H. Khai, P . P . Huynh, L. P . Luc, D.-V . Nguyen, and K. N. Van, “Span detection for aspect-based sentiment analysis in Vietnamese,” inProceedings of the 35th Pacific Asia Conference on Language, Information and Computation, Shanghai, China, 11 2021, pp. 318–328
2021
-
[12]
BERT: Pre- training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W . Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nol...
2019
-
[13]
Unsu- pervised cross-lingual representation learning at scale,
A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Unsu- pervised cross-lingual representation learning at scale,” inProceedings of the 58th Annual Meeting of the Association for Computational Linguist...
2020
-
[14]
PhoBERT: Pre-trained language models for Vietnamese,
D. Q. Nguyen and A. Tuan Nguyen, “PhoBERT: Pre-trained language models for Vietnamese,” inFindings of the Association for Computa- tional Linguistics: EMNLP 2020, Online, Nov. 2020, pp. 1037–1042
2020
-
[15]
ViSoBERT: A pre- trained language model for Vietnamese social media text processing,
N. Nguyen, T. Phan, D.-V . Nguyen, and K. Nguyen, “ViSoBERT: A pre- trained language model for Vietnamese social media text processing,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Dec. 2023, pp. 5191–5207
2023
-
[16]
VLUE: A new benchmark and multi-task knowledge transfer learning for Vietnamese natural language understanding,
P . Do, S. Tran, P . Hoang, K. Nguyen, and N. Nguyen, “VLUE: A new benchmark and multi-task knowledge transfer learning for Vietnamese natural language understanding,” inFindings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, Jun. 2024, pp. 211–222
2024
-
[17]
ViT5: Pretrained text-to- text transformer for Vietnamese language generation,
L. Phan, H. Tran, H. Nguyen, and T. H. Trinh, “ViT5: Pretrained text-to- text transformer for Vietnamese language generation,” inProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Stud...
2022
-
[18]
Bartpho: pre-trained sequence- to-sequence models for vietnamese,
N. L. Tran, D. M. Le, and D. Q. Nguyen, “Bartpho: pre-trained sequence- to-sequence models for vietnamese,”arXiv preprint arXiv:2109.09701, 2021
2021 arXiv
-
[19]
mT5: A massively multilingual pre-trained text-to-text transformer,
L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mT5: A massively multilingual pre-trained text-to-text transformer,” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics...
2021
-
[20]
Multilingual denoising pre-training for neural machine translation,
Y . Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,”Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, 2020
2020
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.