Annotation and modeling of emotions in a textual corpus: an evaluative approach

Jonas Noblet (LIDILEM)

REVIEW 3 major objections 6 minor 13 references

Annotation and modeling of emotions in a textual corpus: an evaluative approach

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Disagreement among emotion annotators is a stable, textually grounded intensity gradient, not noise: a fine-tuned language model reproduces the averaged +1/0/−1 annotation proportions on projects held out of training.

desk verdict New French EMINOSA corpus with genuine annotation-nuance data, but the headline claim about modeling intensity gradients is under-supported because held-out projects are confounded with topics and no baseline is offered. read the letter →

arxiv 2509.01260 v1 pith:O27IYBBH submitted 2025-09-01 cs.CL

Jonas Noblet (LIDILEM) This is my paper

classification cs.CL

keywords emotionannotationappraisaltheoryinter-annotatoragreementKrippendorffalphalanguagemodelfine-tuningFrenchcorpusopinionminingvariability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that disagreement among annotators labeling emotions in text is not noise but a stable, readable signal. On an industrial corpus of 4,980 French opinion segments about innovative products, six judges marked four appraisal dimensions — familiarity, agreeableness, utility, legitimacy — with +1/0/−1, and agreed poorly overall (Krippendorff alpha between 0.29 and 0.53). The author's hypothesis, carried over from an earlier study, is that the more intensely a dimension is expressed in a text, the more annotators will label it, so the averaged judgment is a continuous intensity target. A multilingual transformer fine-tuned on those averaged labels reproduces the annotation proportions on projects held out of training, with positive-class recall of 55.2% for agreeableness and 71.8% for utility. If the paper is right, low inter-rater agreement in emotion annotation is a measurable property of the text, and language models can carry an evaluative, appraisal-based conception of emotion — the underused theoretical approach the paper advocates.

What carries the argument

The central object is the averaged annotation proportion: each verbatim's mean of six +1/0/−1 human judgments per dimension, which turns a coarse three-point vote into a continuous quantity that can encode graded intensity. Carrying the argument is the intensity-gradient hypothesis, first stated in the author's earlier study and restated in Section 4: the more clearly and intensely a dimension is linguistically marked, the more annotators will label it, so disagreement follows a gradient rather than chance. The probe is a fine-tuned transformer — the multilingual GTE model with its last layer trained on the averaged labels — evaluated by cross-validation at the level of whole projects so tha

What would settle it

A project-level cross-validation against a cheap surface baseline — a logistic regression over raw word frequencies, or a model conditioned only on project identity and per-project label rates — would settle it: if that baseline matches the fine-tuned GTE's positive-class recall (about 55–72%), the model is recovering topic or base-rate structure, not the intensity gradient. A second, direct test is a fresh annotation round in which the same verbatims receive both the +1/0/−1 labels and a continuous intensity rating; the gradient hypothesis predicts the averaged discrete labels track the conti

Watch

Extended reading notes

Core claim

The paper's central claim is that the labeling process itself can be modeled: variability in six annotators' +1/0/−1 judgments across the four evaluative dimensions follows stable statistical trends guided by underlying linguistic features. The evidence has three parts. First, although global agreement is low (alpha between 0.29 and 0.53 depending on the dimension), agreement on polarity alone is very high (alpha ≥ 0.80), so disagreement is about whether a dimension is present in a text, not about whether the opinion is positive or negative. Second, example verbatims fall into smoothly graded average annotation proportions from 0.17 to 1.00, consistent with an intensity continuum. Third, a G

Load-bearing premise

The load-bearing premise is that the six annotators' disagreement reflects a real intensity gradient in the text, so that averaging their +1/0/−1 judgments yields a meaningful continuous target; if most disagreement is task ambiguity or annotator idiosyncrasy instead, the model's success could track topics, wording, or base rates rather than emotional intensity.

Editorial extensions

If this is right

Inter-annotator disagreement in emotion labeling can be treated as a continuous, learnable quantity instead of a nuisance to be removed by majority vote or adjudication.
A model trained on averaged +1/0/−1 judgments can serve as an automatic annotator that reproduces human labeling trends on projects never seen in training, at least for positive judgments.
Keeping the corpus unfiltered — with 45% to 76% of verbatims unlabeled per dimension — is workable: the model learns to emit the neutral label for non-relevant segments rather than needing a pre-selected emotion-bearing subset.
The trained model reproduces the corpus's strong class imbalance, with low recall on negative labels except for familiarity, showing that negative evaluations of innovative concepts are rare in this data and are not yet reliably captured.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the gradient interpretation holds, a cheaper annotation design becomes testable: fewer judges plus a continuous intensity rating per text, checking whether the averaged discrete labels converge to those continuous ratings on held-out material.
The sharp split between high polarity agreement and low relevance agreement suggests annotation decomposes into two cognitive operations; an interface or model that separates 'is the dimension present?' from 'positive or negative?' might reduce the dominant source of disagreement.
A transfer experiment into other French registers — narrative, interpersonal, news — would show whether the linguistic grounding of evaluative emotion is general or specific to product-opinion discourse; the paper's own conclusion notes that interpersonal sources of emotion are excluded from this corpus.
The author plans interpretability analyses; a concrete next step would test whether the model's gradient predictions align with linguistic markers of intensity such as confidence adverbs, explicit appraisal lexicon, and negation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a short French-language communication that ships a genuinely new resource — 4,980 French verbatims about 22 innovation concepts, annotated by six raters on four EMINOSA appraisal dimensions — and a reasonable hypothesis that annotator disagreement tracks an intensity gradient. The modeling experiment, however, doesn't yet demonstrate that. Because cross-validation is done at project level and each project is a different concept, the model could be keying on topic words and simply reproducing project-level label base rates. There is no text-free baseline (project-mean predictor, TF-IDF), no within-project item-level evaluation, and no error bars across folds. So the abstract's claim that 'variability is guided by underlying linguistic features' is underdetermined — true only in the weak sense that some words correlate with some labels.

What's good: the corpus design is a real contribution. Six annotators on the same units, no pre-filtering, and the decision to keep the non-annotated examples is sensible if you want ecological validity. The authors also report the alpha decomposition: global alpha is low (0.29–0.53) but polarity agreement is high (0.80–0.93), which is a useful diagnostic. They explicitly call their results 'approximatifs' and say no definitive conclusion. That honesty is worth respecting.

Soft spots beyond the confound: the reported metrics use arbitrary thresholds (±1/3) on averaged continuous outputs, and there is no per-fold breakdown, so we can't tell how stable the 4-project split is. The FlauBERT figure in the appendix looks like it has more diagonal structure than the text claims, though the figure is hard to read. These are minor compared with the baseline problem.

Who should read it: people building or using French emotion corpora, and anyone who wants a compact example of why project-level CV can be misleading when topic is confounded with unit. I'd send it to review — the resource deserves scrutiny — but I'd ask for a baseline comparison and within-project results before taking the central claim seriously.

Referee Report

3 major / 6 minor

Summary. The paper presents a French-language corpus of opinions about innovative products/concepts, collected on an industrial platform. Six annotators labeled 4,980 verbatims for four evaluative (EMINOSA) dimensions (Familiarity, Agreeableness, Utility, Legitimacy) with +1/0/-1. The paper documents low overall inter-annotator agreement but high agreement on polarity, posits that annotator variability follows a stable intensity gradient, and tests this by fine-tuning Transformer language models on averaged annotations under project-level cross-validation. Confusion matrices and threshold-based precision/recall scores are reported for fine-tuned GTE, and the paper concludes that the labeling process is modelable and that variability is driven by underlying linguistic features. The manuscript also reviews related emotion corpora and theoretical frameworks.

Significance. If the central claim is established, the paper would provide a useful contribution: a new French appraisal-based emotion corpus with multiple annotators, a direct treatment of annotator disagreement as signal rather than noise, and evidence that averaged discrete annotations can be predicted from text. The theoretical framing around evaluative approaches to emotion is a real complement to category- and dimension-based work. The main strength is the idea of treating inter-annotator variation as a stable, textually grounded gradient. However, the current experimental design does not yet distinguish this interpretation from a much weaker one: that the model reproduces project-level label base rates or topic signals. The reported metrics are pooled over held-out projects, with no baselines, no within-project evaluation, and no variance over folds, so the central inference is underdetermined. The paper itself acknowledges that the results are 'encore trop approximatifs' (Section 5), but the conclusion in the abstract is nevertheless stronger than the evidence supports.

major comments (3)

[Section 4, 'Modélisation des annotations par réseaux Transformers' (Table 4, Figure 2, Figure 1)] The evaluation is performed at the project level ('validation croisée à l'échelle des projets'), while Figure 1 shows that the proportion of positive labels varies strongly across projects. A model that recognizes topic/project-related vocabulary could predict the project-level mean label and thereby reproduce the pooled diagonal pattern without computing any per-verbatim intensity. No baseline is reported: neither a text-free project-mean predictor, a majority-class predictor, nor a TF-IDF/logistic regression. Since Table 4 reports pooled precision/recall over all held-out projects, the positive recalls (e.g., 71.8 for Utility) may largely reflect correct project-level guesses. This missing control is load-bearing for the abstract claim that variability is 'guidée par des caractéristiques linguistiques sous-jacentes.'
[Section 4, Figure 2 and Table 4] All metrics are aggregated over the held-out projects and no within-project scores are reported. The continuous averaged annotations are discretized with arbitrary thresholds at -1/3 and +1/3, and the paper acknowledges these thresholds are arbitrary. This makes the precision/recall values in Table 4 threshold-dependent and not directly interpretable as evidence for a continuous intensity gradient. To support the central claim, the paper should report per-verbatim correlation or regression metrics between predicted continuous scores and averaged gold labels, within-project performance, and the spread across cross-validation folds. Without such numbers, 'retrouver les proportions d'annotation' is not quantitatively established.
[Section 4, 'Statistiques générales' (Table 2) and averaging procedure] The target variable is the mean of six discrete +1/0/-1 judgments, and the paper's interpretation assumes this mean reflects an underlying intensity gradient. However, global Krippendorff alpha is only 0.29-0.53, and about 45-76% of verbatims are unannotated for a given dimension; the averaged labels are therefore heavily zero-inflated. The paper does not validate that the averaged target carries information beyond base rates or topic-related lexical cues. A direct validation would be an annotator split-half test: train on the average of half the annotators and predict the other half, or predict individual annotator labels. This would show whether the 'gradient' generalizes across annotators, rather than merely reflecting project-level label frequencies. As it stands, the conclusion that variability is governed by stable linguistic features rests on an untested assumption about the avera

minor comments (6)

[Section 4, cross-validation description] The description of the project-level cross-validation is incomplete: it is not stated how many folds there are, how the 21 projects were partitioned, or whether participants can appear in multiple projects, which could create leakage. This information should be provided.
[Table 4 and Figure 2] Precision/recall numbers are dominated by the majority zero class. Reporting macro-F1, Matthews correlation, or a continuous regression metric would make the comparison across dimensions more meaningful.
[Table 2] Krippendorff's alpha values are reported without confidence intervals. Given the small number of annotators, CIs would be useful.
[Throughout] There are small typographical issues, e.g., 'Kripendorff' in Table 2 caption, 'illusrée' in Figure 2 caption, and inconsistent use of French-English labels. These are cosmetic but should be fixed.
[Section 3] The dimension definitions are clear for Agreeableness and Utility, but Familiarity and Legitimacy would benefit from more precise guidelines, especially given the low agreement on these dimensions.
[Section 4] The paper mentions that Flaubert alone is unable to distinguish annotation values, but the corresponding annex figures (Figures 3 and 4) are not discussed in detail; a sentence explaining the comparison would help the reader interpret them.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: held-out project prediction is a genuine generalization test; only minor non-load-bearing self-citation.

full rationale

The paper's central derivation chain is: annotators assign +1/0/-1 on Ecog dimensions; these are averaged to form continuous intensity labels; Transformers are fine-tuned on these averaged labels; and the models are evaluated by cross-validation at the project level. The load-bearing evaluation is genuinely held-out: '4 projets ont servi successivement de tests tandis que les autres ont contribué à l'entraînement' (Section 4, Modélisation des annotations par réseaux Transformers). Thus reported recalls (e.g., 71.8% for positive Utility in Table 4) are out-of-sample predictions, not reconstructions of the training labels, and the supervised task is not circular by construction. The self-citations are contextual rather than load-bearing. Noblet (2025) is cited for the annotation protocol and for the earlier formulation of the probabilistic hypothesis, and Noblet (2026) for the thesis context; but the empirical test performed here—fine-tuning GTE on averaged annotations and testing on unseen projects—is new and does not reduce to those prior publications. EMINOSA is attributed to Sbai (2013), Dupré (2016), and Loeser (2019), which is external prior work. Concerns about missing baselines (e.g., a project-mean predictor), low Krippendorff alpha, and the possibility that the model exploits project-level topic base rates are validity and interpretation risks, not circularity. The paper itself is cautious, stating its results are 'encore trop approximatifs pour constituer une conclusion définitive.' The core prediction is a standard supervised generalization test, so no circular step can be exhibited.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The empirical core is a new corpus and a new annotation-scheme application. The modeling pipeline relies on the EMINOSA framework (prior same-lab theses), the annotation protocol (Noblet 2025), and the assumptions above. No invented entities. The two ad hoc numeric choices (thresholds, hidden hyperparameters) directly shape the headline performance numbers.

free parameters (2)

Thresholds for discretizing predictions (+/-1/3) = +1/3 and -1/3
Section 4, Table 4: continuous averaged predictions are cut into negative/zero/positive bins at +/-1/3. These thresholds are chosen ad hoc after seeing the model outputs and directly determine the reported precision and recall figures; the author acknowledges the assumption of a continuous scale.
Fine-tuning hyperparameters = not reported
Learning rate, epochs, batch size, and the exact assignment of the 21 projects into training and test folds are not given, so the results' sensitivity to these choices cannot be checked.

assumptions (3)

domain assumption The EMINOSA Ecog dimensions (Familiarity, Agreeableness, Utility, Legitimacy) are valid constructs for reactions to innovative concepts and are identifiable in text.
Section 3: the framework is inherited from prior theses (Sbai 2013, Dupre 2016, Loeser 2019) and Scherer et al. 2001; the paper does not revalidate the construct or the text-to-dimension mapping.
domain assumption Annotator disagreement reflects an intensity gradient in linguistic expression rather than noise or ambiguity.
Section 4 and Conclusion: this hypothesis is what allows predictive success to be interpreted as evidence about emotion intensity; with global alpha of 0.29-0.53 (Table 2), low reliability could also indicate task ambiguity.
ad hoc to paper The mean of six discrete judgments (+1/0/-1) is a meaningful continuous target for training.
Section 4: models are trained on averaged annotations; treating the arithmetic mean of three-level judgments as a scaled intensity (from -100 to +100) is assumed without justification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Annotation and modeling of emotions in a textual corpus: an evaluative approach." pith.science (2026). https://pith.science/paper/O27IYBBH

@misc{pith2026250901260,
  author       = {Pith},
  title        = {Pith review of: Annotation and modeling of emotions in a textual corpus: an evaluative approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O27IYBBH}},
  note         = {Machine review of arXiv:2509.01260}
}

read the original abstract

Emotion is a crucial phenomenon in the functioning of human beings in society. However, it remains a widely open subject, particularly in its textual manifestations. This paper examines an industrial corpus manually annotated following an evaluative approach to emotion. This theoretical framework, which is currently underutilized, offers a different perspective that complements traditional approaches. Noting that the annotations we collected exhibit significant disagreement, we hypothesized that they nonetheless follow stable statistical trends. Using language models trained on these annotations, we demonstrate that it is possible to model the labeling process and that variability is driven by underlying linguistic features. Conversely, our results indicate that language models seem capable of distinguishing emotional situations based on evaluative criteria.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

[1]

Nous réservons le terme projet pour désigner spéciﬁquement les études de test de concepts – bien ou service – innovants conduites par Ixiade ; chaque projet est associé à un concept innovant, porté par un client donné (Noblet, 2025)

work page 2025
[2]

1994 ; Barrett, 2018 ; Siegel et al., 2018)

La notion d’acceptabilité désigne l’usage potentiel d’u ne innovation telle que l’individu ciblé se la représente, avant les premières manipulations ( Terrade et al., 2009). 1994 ; Barrett, 2018 ; Siegel et al., 2018). De là naît l’approche constructiviste : celle- ci afﬁrme que l’émotion est le produit d’un apprentissage en lien avec un contexte sociocul...

work page 2009
[3]

Corpus Granularité # Annotation Taille Source Corpus anglophones Tales phrases 6 C + V 15 302 Alm et al

En particulier, certains corpus font appel à une variante du modèle d’Ekman développée par Plutchik (1980, 2001). Corpus Granularité # Annotation Taille Source Corpus anglophones Tales phrases 6 C + V 15 302 Alm et al. (2005) CrowdFlower tweets 14 C 40 000 V an Pelt & Sorokin (2012) TEC tweets 8 C 21 051 Mohammad (2012) DailyDialog conversation 6 C 13 118...

work page 1980
[4]

https://webapp.yoomaneo.com/

work page
[5]

Segment de texte, généralement de la taille d’une phrase

work page
[6]

Perso nnalité, Cognition, Changement Social

Laboratoire Inter-universitaire de Psychologie. Perso nnalité, Cognition, Changement Social. (2013), Dupré (2016), et Loeser (2019). Dans ses fondements, il est inspiré du modèle multi-componentiel de l’émotion développé par Scherer et al. (2001) 7. Ce dernier di- vise l’émotion en cinq composantes qui interagissent entre elles : évaluative, conative, exp...

work page 2013
[7]

D’autres travaux ont également servi de référence pour EMINOSA , dont ceux de Frijda (1986) et Roseman (2013)

work page 1986
[8]

La composante conative intègre la projection au mouvemen t – l’action telle qu’elle est conçue en amont ; la composante motrice les mouvements involontaires (rythme cardiaque, sudation) et la composante expressive les mouvements volontaires (expression faciale, posture)

work page

Show all 13 references

[9]

— 0 si la dimension est absente ou présente de manière ambigüe

Cette déﬁnition inclut les propriétés ergonomiques et la facilité / difﬁculté d’utilisation. — 0 si la dimension est absente ou présente de manière ambigüe . Pour l’Exemple 1, un annotateur aura tendance à associer la valeur +1 à l’Agré abilité (intuitif, facile d’utilisation)...

2017
[10]

Le projet écarté est l’un des premiers ayant été déployé s ur la plate-forme, selon des modalités nota- blement différentes de ceux qui ont suivi. FamiliaritéAgréabilitéUtilité Légitimité 0 10 20 30 40 50 60 Verbatims annotés (en %) (a) V erbatims annotés positivement Familiar...
[11]

objet connecté

En se référant à la typologie d’ Étienne et al. (2020), l’appellation dimensions désignées pourrait être employée . Légitimité Exemples 1,00 (6/6) Cette solution s’appuie sur une approche "objet connecté", clairement dans l’air du temps. 0,83 (5/6) De plus, si ce produit peut ...

2020
[12]

mesure de produire des représentations qui permettent déjà de discriminer grossière- ment des gradients d’annotation

Comme le modèle GTE, il est toutefois possible de l’afﬁne r pour trouver des performances correctes. mesure de produire des représentations qui permettent déjà de discriminer grossière- ment des gradients d’annotation. Nous pouvons en effet cons tater que les annotations produ...

2005 doi arXiv
[2011]

Notons toutefois que ce désaccord n’est pas total

est représenté dans la première ligne du Tableau 2. Notons toutefois que ce désaccord n’est pas total. Si nous ignorons les annotations nulles et que nous jugeons seulement l’accord entre annotations positives et négativ es (deuxième ligne du Tableau 2), l’accord est très bon ...

2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Nous réservons le terme projet pour désigner spéciﬁquement les études de test de concepts – bien ou service – innovants conduites par Ixiade ; chaque projet est associé à un concept innovant, porté par un client donné (Noblet, 2025)

work page 2025

[2] [2]

1994 ; Barrett, 2018 ; Siegel et al., 2018)

La notion d’acceptabilité désigne l’usage potentiel d’u ne innovation telle que l’individu ciblé se la représente, avant les premières manipulations ( Terrade et al., 2009). 1994 ; Barrett, 2018 ; Siegel et al., 2018). De là naît l’approche constructiviste : celle- ci afﬁrme que l’émotion est le produit d’un apprentissage en lien avec un contexte sociocul...

work page 2009

[3] [3]

Corpus Granularité # Annotation Taille Source Corpus anglophones Tales phrases 6 C + V 15 302 Alm et al

En particulier, certains corpus font appel à une variante du modèle d’Ekman développée par Plutchik (1980, 2001). Corpus Granularité # Annotation Taille Source Corpus anglophones Tales phrases 6 C + V 15 302 Alm et al. (2005) CrowdFlower tweets 14 C 40 000 V an Pelt & Sorokin (2012) TEC tweets 8 C 21 051 Mohammad (2012) DailyDialog conversation 6 C 13 118...

work page 1980

[4] [4]

https://webapp.yoomaneo.com/

work page

[5] [5]

Segment de texte, généralement de la taille d’une phrase

work page

[6] [6]

Perso nnalité, Cognition, Changement Social

Laboratoire Inter-universitaire de Psychologie. Perso nnalité, Cognition, Changement Social. (2013), Dupré (2016), et Loeser (2019). Dans ses fondements, il est inspiré du modèle multi-componentiel de l’émotion développé par Scherer et al. (2001) 7. Ce dernier di- vise l’émotion en cinq composantes qui interagissent entre elles : évaluative, conative, exp...

work page 2013

[7] [7]

D’autres travaux ont également servi de référence pour EMINOSA , dont ceux de Frijda (1986) et Roseman (2013)

work page 1986

[8] [8]

La composante conative intègre la projection au mouvemen t – l’action telle qu’elle est conçue en amont ; la composante motrice les mouvements involontaires (rythme cardiaque, sudation) et la composante expressive les mouvements volontaires (expression faciale, posture)

work page