Pith. sign in

REVIEW 3 major objections 6 minor 20 references

ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read The paper introduces ViTOED, a manually annotated Vietnamese dataset of 10,985 social media comments with 21,244 opinion quadruples, and shows that current language models struggle to attribute emotions to their targets.

desk verdict Genuinely needed Vietnamese target-oriented emotion dataset, but the annotation reliability and internal consistency need serious work before it can be used. read the letter →

arxiv 2608.12776 v1 pith:XB7BWCHN submitted 2026-08-13 cs.CL

classification cs.CL
keywords target-orientedemotiondetectionVietnamesesocialmediaopinionquadrupleannotationstructuredsentimentgraphspananalysislow-resourceNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ViTOED is a manually annotated Vietnamese dataset built from 10,985 social media comments and containing 21,244 opinion quadruples of the form (source, target, expression, polarity). The paper's central claim is that this resource captures how emotions attach to specific targets in Vietnamese, including phenomena sentence-level sentiment labels miss: over half of the opinions have no explicit source, and the same Vietnamese pronouns can serve as source, target, or both. It then evaluates a structured sentiment graph baseline on the dataset and reports that current language models extract expression spans fairly well but misalign source-target relations and confuse the two roles, leaving a wide performance gap. A sympathetic reader would care because this gives Vietnamese target-oriented emotion detection a reusable benchmark and a concrete list of error types that future models must address.

What carries the argument

The load-bearing representation is the opinion quadruple (source, target, expression, polarity), where each component is annotated as a token span inside a Vietnamese comment. The baseline mechanism is the structured sentiment graph from [4], which converts each sentence into a dependency graph whose nodes are spans and whose edges carry source, target, and expression relations with polarity, using either head-first or head-final root choices. The graph is produced by a BiLSTM over word, POS-tag, character, and transformer-contextualized embeddings, scored with a bilinear transformation. This machinery turns emotion detection into a joint span-extraction and relation-extraction problem, and its failure modes become the paper's error taxonomy.

What would settle it

Have fresh annotators independently re-annotate a random sample of the test set using only the published guidelines, then compute pairwise F1 against the released source spans; if the agreement is no better than the original 0.32, or if training a model with source spans removed matches the full model's Targeted F1, then the source component is not carrying real signal and the quadruple claim is not supported.

Watch

Extended reading notes

Core claim

ViTOED is a human-annotated Vietnamese dataset of 10,985 user comments collected from social networks, containing 21,244 opinion quadruples of the form (source, target, expression, polarity), with source, target, and expression recorded as token spans and polarity as positive, negative, or neutral. The paper claims that this resource captures Vietnamese-specific annotation phenomena that sentence-level sentiment labels miss: 52.10% of opinions have an implicit source (category T-E only), targets are mentioned less often than opinions, and common pronouns such as 'tao' and 'mày' appear on both sides of the quadruple, creating vocabulary ambiguity. Using a structured sentiment graph baseline from [4] over nine pretrained language models, the paper reports that current models succeed at expression span detection but struggle with source and target spans, graph root placement, and relation alignment. The error analysis attributes these failures to three causes: vocabulary complexity in span recognition, root and edge misprediction, and source-target confusion caused by Vietnamese morphology. The intended payoff is a benchmark on which future Vietnamese target-oriented emotion detection systems can be measured, plus an explicit list of error types to attack.

Load-bearing premise

The dataset's usefulness rests on the assumption that the annotators' opinion quadruples are reliable enough to learn from; in particular, if the low source-span agreement (F1 = 0.32) means source annotations are effectively arbitrary, the quadruple structure that defines ViTOED loses its validity.

Editorial extensions

If this is right

  • Vietnamese target-oriented emotion detection now has a public train/dev/test benchmark with 21,244 quadruples and fixed splits.
  • Because 52.10% of opinions in ViTOED lack an explicit source, any successful Vietnamese model must learn to infer the commenter as an implicit source rather than only detecting explicit mentions.
  • The structured sentiment graph baseline shows that both multilingual and monolingual Vietnamese language models leave substantial headroom on this task, particularly on source and target spans.
  • The head-final graph representation and the +inlabel extension offer concrete starting points for improving expression and span extraction on this dataset.
  • The paper's error taxonomy gives future work three defined targets: span misalignment, root and edge misprediction, and source-target confusion in Vietnamese morphology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since source-span agreement is only F1 = 0.32, part of the reported performance gap on sources may reflect annotation noise rather than model weakness; a natural next test is to score models only on sentences where annotators agreed on the source span.
  • The same quadruple annotation protocol could transfer to other subject-drop languages such as Japanese, Korean, or Thai, where implicit sources predominate, enabling cross-lingual comparison if those datasets adopt the same graph parser.
  • A simple and testable extension suggested by the data is to add a Vietnamese first-person and second-person pronoun feature to the graph parser, which might directly reduce the source-target confusion the paper identifies.
  • The paper reports per-component agreement but not exact quadruple-level agreement; computing the fraction of fully matching (source, target, expression, polarity) tuples would give a stricter, and possibly much lower, reliability estimate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces ViTOED, a new Vietnamese dataset for target-oriented emotion detection. It claims to contain 10,985 user comments and 21,244 manually annotated opinion quadruples, each consisting of source, target, expression, and polarity. The authors describe an annotation process with guidelines and four rounds of annotator training, analyze dataset statistics including opinion categories and word frequencies, and evaluate a structured sentiment graph baseline (following Barnes et al., 2021) with several Vietnamese pre-trained language models. The main contributions are stated as the dataset itself, an analysis of Vietnamese-specific phenomena such as implicit sources and targets, and baseline results that reveal challenges in span detection and relation extraction.

Significance. If the dataset is reliable, it would fill a real gap in Vietnamese target-oriented emotion detection resources and provide a benchmark that exposes how poorly current models connect emotions to specific targets in Vietnamese social media. The annotation of opinion quadruples is a new contribution, and the baseline experiments with multiple Vietnamese PLMs are a useful starting point. However, the central value of the paper depends entirely on the trustworthiness of the manual annotation, and the current evidence for that trustworthiness is weak.

major comments (3)
  1. [III.C, Table III] The paper's central claim of a dataset with 'strict guidelines' and reliable opinion quadruples is not supported by the reported inter-annotator agreement. After four training rounds, Source F1 is 0.32 and Polarity agreement is 0.47; these are the final reported values, measured only on a 210-sentence sample. No IAA is reported for the final corpus, and the adjudication procedure is only described as 'cross-checking among annotators' without detail. The statement that 'the level of agreement indicates that the annotators understood the labeling guidelines and followed the instructions accurately' directly contradicts the low Source and Polarity figures. Please report IAA on the full corpus, describe how disagreements were resolved, and either justify why Source F1 of 0.32 is sufficient or revise the claims about annotation quality.
  2. [III.B, Table I, Example 2] The annotation guidelines in Section III.B state that expressions containing profanity, rudeness, and sensitive aspects are classified as Negative, and that humor is Positive only if it does not contain hateful or offensive content. Example 2 in Table I, '7 thằng này không phải con người chúng mày ạ' ('These 7 guys are not human'), is labeled Positive even though calling people 'not human' is an offensive statement under any ordinary reading. This is an internal contradiction within the paper's own example. Either the example's label is wrong, or the guideline needs an explicit exception; as published, it undermines the claim of consistent annotation.
  3. [III.A/III.D, Tables II and IV] The dataset statistics are internally inconsistent. Section III.A reports 6,000 sentences from UIT-VSMEC plus 5,010 collected comments, totaling 11,010, but Sections III.D and the abstract state 10,985 comments. Summing Table II gives source counts 2,359+385+572=3,316, while the source-bearing categories in Table IV (S-T-E + S-E) total 2,243+1,527=3,770; the corresponding target counts are 8,632+1,160+2,127=11,919 versus 2,243+11,069=13,312. These numbers must agree because they count the same opinion quadruples. Additionally, the sentence 'The T-E demonstrated that an opinion missing the Source accounts for 52.10% of the total opinions' is misleading: T-E is only one of the two categories without a Source (the other is E), and the true proportion of opinions missing Source is (11,069+6,405)/21,244 = 82.25%. Please reconcile all counts and correct the interpretation.
minor comments (6)
  1. [I] There is a typo in 'sentencesin human language'; also, the notation 'opinion tuples O={o1, o2, ..., on}' should say 'quadruples' for consistency with the rest of the paper.
  2. [II] The phrase 'DSU nis' appears to be a typo or incorrect rendering of the dataset name; please check the official name of the dataset introduced in reference [6].
  3. [Figure 2] The label 'BERTologies' should likely be 'BERT' or 'BERT embeddings'.
  4. [III.C] The text cites 'Examples #3 and #4 in Table I' as cases lacking a source, but Table I contains only three examples and Example #3 has an explicit source ('tao'). Please correct the reference or extend the table.
  5. [III.C, Eq. (2)] The notation in Equation (2) uses the same symbols A and B for annotators and for token sets, which is confusing. Please rewrite with distinct notation, e.g., annotators 1 and 2 and sets X and Y.
  6. [V, Table VII] The table title 'Results of graph annotation by opinion components by F1(%)' is unclear, and the column layout mixes entity types with polarity classes; please clarify what each column and row represents.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ViTOED's quadruple annotations are new manual labels, the baseline is an external graph-parsing model, and the evaluation metrics are standard.

full rationale

The paper's central contribution is a manually annotated dataset, not a derived prediction or fitted constant. The annotation quadruples are produced from raw comments by human annotators according to stated guidelines; no quantity is defined in terms of another quantity that it is then said to predict. The baseline model is taken from the external work of Barnes et al. [4], and the metrics are the standard Span F1, Targeted F1, Parsing Graph F1, and Sentiment Graph F1 of that external framework. The only author-group citations are non-load-bearing: [3] supplies 6,000 source sentences (reused data, not a result), [11] supplies the F1 agreement formula, and [15]–[16] are among the evaluated language models. None of these citations forbids alternatives or supplies the conclusion. The IAA evidence in Section III.C is a reliability limitation rather than a circular step: the paper reports Source F1=0.32 and Polarity=0.47 after four rounds and no final-corpus agreement, so the assertion that 'the annotators understood the labeling guidelines' is under-supported, but this is an empirical support problem, not a reduction of a claim to its own input. Table I Example 2 also appears to contradict the stated polarity guideline, and the Source counts in Tables II and IV do not reconcile; these internal inconsistencies affect dataset quality, not circularity. The paper is therefore self-contained against external baselines and contains no derivation chain that reduces to its inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper adds an empirical resource rather than a theory. The ledger records the task ontology, the convenience sample, and the borrowed baseline formalism as domain assumptions; there are no fitted free parameters or invented entities.

assumptions (4)
  • domain assumption Target-oriented emotion detection is well-defined as extracting quadruples <source, target, expression, polarity> from each sentence.
    Defines the annotation ontology and the task itself; introduced in Section I and the annotation guidelines in Section III.B.
  • domain assumption The 6,000 UIT-VSMEC sentences plus 5,010 unspecified platform comments form a suitable sample of Vietnamese social media.
    Data creation process in Section III.A; the sample is convenience-based without stratification or platform disclosure.
  • standard math Token-span F1 and the pairwise label overlap measure in Equations 1 and 2 are valid estimators of annotator agreement.
    Section III.C; this is a standard approach to span-level IAA, following [11].
  • domain assumption The structured sentiment graph formalism of Barnes et al. is an appropriate baseline for the ViTOED task.
    Section IV; the baseline is directly reused from [4] without task-specific modification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts." pith.science (2026). https://pith.science/paper/XB7BWCHN

@misc{pith2026260812776,
  author       = {Pith},
  title        = {Pith review of: ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XB7BWCHN}},
  note         = {Machine review of arXiv:2608.12776}
}
read the original abstract

This paper introduces ViTOED, a novel dataset for target-oriented emotion detection in Vietnamese social media texts. The ViTOED comprises 10,985 user comments and 21,244 manually annotated opinion quadruples (source, target, expression, polarity) that follow strict guidelines. The dataset reveals Vietnamese-specific phenomena, such as implicit sources and targets and vocabulary ambiguities, enabling deeper analysis of user emotions toward entities. We propose a baseline using structured sentiment graphs and evaluate various Vietnamese pre-trained language models. The empirical results highlight challenges in span detection and relation extraction and indicate substantial room for model improvement in Vietnamese Target-Oriented Emotion Detection tasks.

Figures

Figures reproduced from arXiv: 2608.12776 by the authors.

Figure 1
Figure 1. A sample of target-oriented emotion detection from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The sentiment graph baseline model. as the root and other tokens within the span as the dependents (as illustrated in Figure 3a). On the other hand, the head-final sets the final token in the span as the root, and other tokens in the spans as dependents (as shown in Figure 3b). Finally, we employ two types of BERTology models for Vietnamese —multilingual and monolingual —to represent contextual embeddings. The multi… view at source ↗
Figure 3
Figure 3. The example of the parsing graph representation. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Error sample: the edges are misaligned between tokens. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 18 canonical work pages

  1. [1]

    Liu,Sentiment analysis and opinion mining

    B. Liu,Sentiment analysis and opinion mining. Springer Nature, 2022

  2. [2]

    Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training,

    Z. Li, Y . Zou, C. Zhang, Q. Zhang, and Z. Wei, “Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training,” inProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, Nov. 2021, pp. 246–256

  3. [3]

    Emotion recognition for vietnamese social media text,

    V . A. Ho, D. H.-C. Nguyen, D. H. Nguyen, L. T.-V . Pham, D.-V . Nguyen, K. V . Nguyen, and N. L.-T. Nguyen, “Emotion recognition for vietnamese social media text,” inComputational Linguistics: 16th International Conference of the Pacific Association for Computational Linguistics, PACLING 2019, Hanoi, Vietnam, October 11–13, 2019. Springer, 2020, pp. 319–333

  4. [4]

    Structured sentiment analysis as dependency graph parsing,

    J. Barnes, R. Kurtz, S. Oepen, L.Øvrelid, and E. Velldal, “Structured sentiment analysis as dependency graph parsing,” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, 2021, pp. 3387–3402

  5. [5]

    The sentiment problem: A critical survey towards deconstructing sentiment analysis,

    P . Venkit, M. Srinath, S. Gautam, S. Venkatraman, V . Gupta, R. Pas- sonneau, and S. Wilson, “The sentiment problem: A critical survey towards deconstructing sentiment analysis,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Dec. 2023, pp. 13 743–13 763

  6. [6]

    Sentence and expression level annotation of opinions in user-generated discourse,

    C. Toprak, N. Jakob, and I. Gurevych, “Sentence and expression level annotation of opinions in user-generated discourse,” inProceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, Jul. 2010, pp. 575–584

  7. [7]

    MultiBooked: A corpus of Basque and Catalan hotel reviews annotated for aspect-level sentiment classification,

    J. Barnes, T. Badia, and P . Lambert, “MultiBooked: A corpus of Basque and Catalan hotel reviews annotated for aspect-level sentiment classification,” inProceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018

  8. [8]

    A systematic literature review on vietnamese aspect-based sentiment analysis,

    D. Van Thin, D. N. Hao, and N. L.-T. Nguyen, “A systematic literature review on vietnamese aspect-based sentiment analysis,”ACM Trans. Asian Low-Resour. Lang. Inf. Process., vol. 22, no. 8, aug 2023

Show all 20 references
  1. [9]

    Vlsp shared task: sentiment analysis,

    H. T. Nguyen, H. V . Nguyen, Q. T. Ngo, L. X. Vu, V . M. Tran, B. X. Ngo, and C. A. Le, “Vlsp shared task: sentiment analysis,”Journal of Computer Science and Cybernetics, vol. 34, no. 4, pp. 295–310, 2018

  2. [10]

    Sa2sl: From aspect-based sentiment analysis to social listening system for business intelligence,

    L. Luc Phan, P . Huynh Pham, K. Thi-Thanh Nguyen, S. Khai Huynh, T. Thi Nguyen, L. Thanh Nguyen, T. Van Huynh, and K. Van Nguyen, “Sa2sl: From aspect-based sentiment analysis to social listening system for business intelligence,” inKnowledge Science, Engineering and Management...

  3. [11]

    Span detection for aspect-based sentiment analysis in Vietnamese,

    K. N. T. Thanh, S. H. Khai, P . P . Huynh, L. P . Luc, D.-V . Nguyen, and K. N. Van, “Span detection for aspect-based sentiment analysis in Vietnamese,” inProceedings of the 35th Pacific Asia Conference on Language, Information and Computation, Shanghai, China, 11 2021, pp. 318–328

  4. [12]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W . Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nol...

  5. [13]

    Unsu- pervised cross-lingual representation learning at scale,

    A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Unsu- pervised cross-lingual representation learning at scale,” inProceedings of the 58th Annual Meeting of the Association for Computational Linguist...

  6. [14]

    PhoBERT: Pre-trained language models for Vietnamese,

    D. Q. Nguyen and A. Tuan Nguyen, “PhoBERT: Pre-trained language models for Vietnamese,” inFindings of the Association for Computa- tional Linguistics: EMNLP 2020, Online, Nov. 2020, pp. 1037–1042

  7. [15]

    ViSoBERT: A pre- trained language model for Vietnamese social media text processing,

    N. Nguyen, T. Phan, D.-V . Nguyen, and K. Nguyen, “ViSoBERT: A pre- trained language model for Vietnamese social media text processing,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Dec. 2023, pp. 5191–5207

  8. [16]

    VLUE: A new benchmark and multi-task knowledge transfer learning for Vietnamese natural language understanding,

    P . Do, S. Tran, P . Hoang, K. Nguyen, and N. Nguyen, “VLUE: A new benchmark and multi-task knowledge transfer learning for Vietnamese natural language understanding,” inFindings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, Jun. 2024, pp. 211–222

  9. [17]

    ViT5: Pretrained text-to- text transformer for Vietnamese language generation,

    L. Phan, H. Tran, H. Nguyen, and T. H. Trinh, “ViT5: Pretrained text-to- text transformer for Vietnamese language generation,” inProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Stud...

  10. [18]

    Bartpho: pre-trained sequence- to-sequence models for vietnamese,

    N. L. Tran, D. M. Le, and D. Q. Nguyen, “Bartpho: pre-trained sequence- to-sequence models for vietnamese,”arXiv preprint arXiv:2109.09701, 2021

  11. [19]

    mT5: A massively multilingual pre-trained text-to-text transformer,

    L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mT5: A massively multilingual pre-trained text-to-text transformer,” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics...

  12. [20]

    Multilingual denoising pre-training for neural machine translation,

    Y . Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,”Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, 2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.