Active Annotation: bootstrapping annotation lexicon and guidelines for supervised NLU learning

Alessandra Cervone; Evgeny A. Stepanov; Federico Marinelli; Giuliano Tortoreto; Giuseppe Di Fabbrizio; Giuseppe Riccardi

REVIEW 4 major objections 4 minor 21 references

Active Annotation: bootstrapping annotation lexicon and guidelines for supervised NLU learning

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cluster-guided annotation produces intent-label training data about ten times faster than one-by-one labeling, while improving downstream classifier accuracy.

desk verdict A genuinely new annotation workflow, but the order-of-magnitude speed claim rides on unmeasured cluster purity and an unfair baseline comparison. read the letter →

arxiv 1908.04092 v1 pith:RMH4IQ5T submitted 2019-08-12 cs.CL cs.LG

Federico Marinelli , Alessandra Cervone , Giuliano Tortoreto , Evgeny A. Stepanov , Giuseppe Di Fabbrizio , Giuseppe Riccardi This is my paper · ORCID

classification cs.CLcs.LG

keywords activeannotationintentclassificationnaturallanguageunderstandinghuman-in-the-loopsentenceembeddingsk-meansclusteringspeedpredicate-argumentstructure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the slowest part of building a Natural Language Understanding system—hand-labeling sentences with intents—can be made roughly an order of magnitude faster by having annotators label small clusters of similar sentences instead of one sentence at a time. The proposed Active Annotation paradigm starts with no predefined label set: it embeds the unlabeled utterances, clusters them, shows an annotator a few sentences near a cluster center, and lets the annotator create or confirm an intent label on the spot. Nearby sentences are then proposed in batches, and the annotator checks each one with a yes/no decision rather than typing or choosing a label. In a 25-minute annotation experiment on movie-ticket booking dialogues, the method labelled 999 sentences per session versus 119 for the baseline, and the resulting training data supported a CNN intent classifier with higher test-set F1 and slightly higher annotator agreement. If the results hold, the main cost of supervised NLU—annotating a new domain from scratch—drops from many person-hours to a fraction of that.

What carries the argument

The load-bearing mechanism is a pipeline of unsupervised representation and human binary verification. Each sentence is converted to a vector with the Universal Sentence Encoder, projected with PCA, and grouped by k-means++; the elbow method picks the number of clusters. Within a cluster, the three points closest to the centroid serve as pivots, and a predicate-argument label is extracted by taking the most frequent verb and object from subject-verb-object parses of the cluster's sentences. The annotator's confirmation of that label authorizes a batch of the five nearest neighbors, retrieved by approximate nearest-neighbour search, to be accepted or rejected with a single binary check. The speed gain comes from converting most annotation decisions from reading-plus-labeling into reading-plus-confirming.

What would settle it

Run the same clustering and pivot selection on a labeled intent dataset, then compute the proportion of nearest-neighbor batches in which all sentences carry the same ground-truth label; if that proportion is not high, the Active Annotation speedup in a real workflow would degrade toward single-sentence decisions.

Watch

Extended reading notes

Core claim

The paper's central claim is that an iterative human-machine loop can replace the standard "define the label schema first, then annotate" workflow. In the proposed Active Annotation paradigm, sentence embeddings are computed, reduced by PCA, and clustered with k-means++; annotators explore randomly chosen clusters by looking at three pivot sentences nearest the centroid, then either supply a label, accept the automatically proposed predicate-argument label, or skip. Once a label is accepted, the tool retrieves the five nearest neighbors of those pivots and the annotator only checks whether each proposed sentence fits the label. Across four annotators and 25-minute sessions, the annotation rate was 999.3 sentences (sd 171.4) for Active Annotation versus 118.6 (sd 18.5) for the one-at-a-time baseline; a CNN trained on the resulting data reached test-set F1 of 0.89 (sd 0.03) versus 0.81 (sd 0.04), and Cohen's kappa against ground truth was 0.64 versus 0.61. The paper further claims the approach produces open label sets that can be extended over time.

Load-bearing premise

The entire speed advantage rests on the assumption that sentences close to each other in the embedding space usually share the same intent label, so one confirmation can correctly label several sentences at once; the paper never measures how often that holds.

Editorial extensions

If this is right

A new NLU domain can be bootstrapped without a pre-existing intent taxonomy, because annotators create labels while exploring clusters.
The annotation rate jump implies that building an intent-classification training set of a few thousand utterances can shrink from days of hand labeling to a few hours of guided confirmation.
Because labels are created on the fly, the paradigm admits open intent classes that can be added as new clusters appear after deployment.
The higher F1 at higher speed suggests the batch check does not introduce extra label noise relative to item-by-item annotation, at least in this movie-ticket domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The decisive unmeasured quantity for transferring the method is cluster label purity: how often the five nearest neighbors of a pivot actually share the pivot's intent label. If purity is high in general, the speed advantage is real; if it is low, the binary checks become per-item decisions and the speed gain disappears.
The baseline already receives the same automatic predicate-argument label for each sentence, so the speed gap isolates the effect of clustering and batch confirmation rather than the automatic label extractor.
A direct test of the cognitive mechanism would be to replay the same batches to annotators one sentence at a time after the guided pass and measure how much of the speedup is mechanical (fewer label selections) versus attentional (similar sentences are easier to judge).
The paper reports it could manually map only about 80% of annotator labels to ground-truth labels, suggesting the open-label procedure produces synonyms or finer distinctions that a fixed schema would miss, which is an advantage for explorative annotation but a cost for strict evaluation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Here is my take on Marinelli et al. (arXiv:1908.04092). The genuinely new piece is the workflow: embed utterances, cluster them, show annotators a few pivot sentences near the centroid, let them define the intent label on the fly, then confirm or reject five similar neighbors with checkmarks. That combination, plus an open label schema that can adapt as data come in, is not in Raymond et al. 2008, despite the overlapping author and name. It is worth taking seriously as a practical annotation tool.

What the paper does well: the baseline is not a straw man in one respect, because both tools pre-compute the same predicate-argument label, so the comparison isolates annotation methodology from automatic labeling. The four annotators are counterbalanced, and the paper reports speed, label counts, cross-validation F1, test F1, and Kappa. On a matched random subsample, the AA labels get CV F1 0.91 versus 0.83 for baseline, which is at least suggestive.

Now the soft spots. The order-of-magnitude speed claim is not self-explanatory. The baseline shows one random sentence at a time; AA batches five similar sentences. The entire speed gain is attributed to cluster-guided batching, but cluster purity is never measured, even though the authors had ground-truth labels for all 2,000 sentences and could have computed the precision of the KNN proposals and the annotators' acceptance rate directly. If the five neighbors are not label-homogeneous, the binary check degrades into per-item decisions and the speed advantage shrinks or vanishes. The reported numbers are equally compatible with a UI effect. The stress-test note is right on this point.

Second, the test-set F1 comparison is uneven: AA annotators produced about 999 sentences per session, baseline about 119, and the classifier trained on all labeled data is evaluated on the same test set. That compares data volume, not label quality. The CV F1 on the matched subsample is fairer, but it comes from one random subsample and there are no significance tests across four annotators. The Kappa difference (0.61 versus 0.64) is small and does not support the claim of higher quality training data. Also, only 80% of annotator labels could be mapped to ground truth; the rest are dropped, which could bias the quality comparison.

Net: the idea has real practical value, the system is described clearly, and I would not desk-reject it. But the headline claims are not supported as presented. A serious referee should ask for a fair speed comparison (same UI density or per-sentence timing), cluster-purity and acceptance-rate measurements, and significance testing.

My recommendation: send to peer review, expect major revision. The paper is worth engaging; the current evidence is not enough to accept the abstract's claims.

Referee Report

4 major / 4 minor

Summary. The paper proposes Active Annotation, an iterative human-in-the-loop annotation procedure for NLU intent classification. Unlabeled utterances are embedded with Universal Sentence Encoder, reduced with PCA, clustered with k-means++, and then annotated in two phases: annotators first define a label for N pivot sentences near a cluster centroid, with an automatically proposed predicate-argument label, and subsequently verify K nearest neighbors via binary checkmarks in batches of five. The authors report a user study with four annotators in 25-minute sessions, comparing this tool against a baseline that presents random sentences one at a time. They claim an order-of-magnitude annotation-speed gain (999.3 vs. 118.6 sentences in Table 1), better F1 (0.91/0.89 vs. 0.83/0.81), and higher agreement (0.64 vs. 0.61).

Significance. If the claims were supported, the contribution would be practically valuable: an annotation paradigm that defines labels on the fly and batches similar sentences could cut the cost of building intent training sets by roughly an order of magnitude while maintaining or improving quality. The paper has real strengths: it reports a genuine annotation study with counterbalanced order, compares against a concrete baseline, and the predicate-argument label proposal is a sensible way to bootstrap an open label space. The authors also had gold labels for all 2,000 unlabeled utterances, which means the key assumptions are testable. However, the current evidence does not support the abstract's speed and quality claims because the comparison is confounded and the mechanisms behind the speed gain are not measured.

major comments (4)

[§5.2, §5.4, Table 1] The throughput comparison is not controlled. The baseline presents one random sentence at a time, whereas the AA tool presents five similar sentences and asks for binary checkmarks (§4.3). The 8.4x difference in sentences labelled (118.6 vs. 999.3 in Table 1) can therefore be due to batch size and UI design rather than to cluster-guided selection. The authors should add a controlled condition that presents the same batch-of-five interface with random sentences, or report per-sentence decision times and check-mark acceptance/rejection rates for the AA condition.
[§4.1, §5.3] The speed mechanism rests on cluster homogeneity: the paper states that pivot points nearest the centroid 'most probably have the same label,' but it never measures the purity of pivot sets or the precision@K of the KNN proposals. Since ground-truth labels exist for all 2,000 sentences in D, this can be computed directly. The authors should report cluster purity, KNN precision@K, and annotator acceptance rates; without these measurements the order-of-magnitude speed gain is unexplained and could be an artifact of batch presentation or of annotators confirming without careful inspection.
[Table 2, §5.5] The test-set F1 comparison is confounded by training-set size. The equal-size random extraction mentioned before Table 1 does not appear to apply to the test-set evaluation in Table 2, which says classifiers were trained 'on the labelled data' without subsampling. Because the AA condition produced roughly eight times more sentences than the baseline, the higher test-set F1 (0.89 vs. 0.81) may simply reflect more training data. Please clarify whether subsampling was used for Table 2 and, if not, re-evaluate with equal-size subsamples or report learning curves.
[§3–§4] The method's behavior depends on several free parameters—k, PCA variance, N, and K—but no sensitivity analysis or final selected values are reported beyond N=3 and K=5. Cluster granularity k is particularly load-bearing because it directly determines whether proposed batches are label-homogeneous. The authors should report the chosen k and PCA settings and vary k to show that the speed and quality conclusions are robust to these choices.

minor comments (4)

[Table 2] The kappa difference between AA and baseline (0.64 vs. 0.61) appears within one standard deviation of the reported variability; the paper should report confidence intervals or a significance test before claiming higher annotation quality.
[§5.3] The relationship between the original e2e labels and the new predicate-argument labels needs clarification. The text says 'We annotated this data with new intent labels' and later mentions manually mapping annotator labels to ground truth 'where possible' with 80% coverage; the mapping procedure should be described explicitly.
[Algorithm 3] The pseudocode 'string(Predicate Argument)' omits a separator, although examples such as 'inform none' suggest one is needed. It would also help to specify which spaCy pipeline components are used for SVO extraction.
[§5.4] With only four annotators and short sessions, the paper should report per-annotator results or ranges and briefly discuss possible learning or fatigue effects, even though the order was counterbalanced.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical measurements, not derivations that reduce to their inputs.

full rationale

The central claim of the paper is an empirical comparison of two annotation methodologies, not a mathematical derivation. The order-of-magnitude speed advantage (999.3 vs. 118.6 sentences in Table 1) is a directly measured outcome of a timed experiment, and the quality comparison is based on training a CNN classifier on the resulting labels, with the cross-validation condition balanced by randomly extracting the same number of baseline-annotated sentences from the Active Annotation pool, as stated in Section 5.5. The cluster-purity intuition in Section 4.1 is an assumption about the data, and its possible failure would weaken the mechanism behind the speed gain, but the paper does not define the speed result in terms of that assumption; this is a validity threat, not a circular reduction. The automatic predicate-argument label style is used both in the tools and in the ground-truth labels, but the ground-truth labels were explicitly verified by three expert annotators (Section 5.3), and the comparison is between two annotation procedures sharing the same automatic labelling component, which the authors explicitly acknowledge by saying 'only the annotation methodology is compared and not the automatic labelling algorithm' (Section 5.2). The only self-citation, Raymond et al. 2008 [14], appears in the related-work discussion to situate the term Active Annotation; it is not load-bearing for the paper's new results. No equation, fitted parameter, or claimed prediction reduces by construction to an input, so the paper receives a circularity score of 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central evaluation rests on several hand-set hyperparameters (N, K, and the elbow-selected number of clusters k) and on unvalidated assumptions: that embeddings cluster by intent, that predicate-argument phrases are good intent names, and that the manually built test set is a trustworthy reference. No new physical or conceptual entities are introduced beyond a workflow, so the invented_entities list is empty.

free parameters (4)

N (number of pivot sentences) = 3
Hand-set in experiments; affects cognitive load and cluster label reliability.
K (number of proposed sentences per annotation step) = 5 (adjustable up to a threshold)
Arbitrarily set; larger K increases batch throughput but requires cluster purity.
k (number of clusters) = determined per dataset via Elbow Method
The elbow method is a heuristic; the optimal k is not validated against label distribution.
PCA components / explained variance = not specified
Paper says PCA is applied but does not state the number of components; this is a free input to the pipeline.

assumptions (4)

domain assumption Universal Sentence Encoder embeddings capture enough semantic similarity for clustering to be meaningful.
Section 3 relies on USE to map utterances into a space where k-means forms coherent groups.
ad hoc to paper Data points inside the same cluster are likely to have the same label; clusters are label-homogeneous enough for batch verification.
Section 4.1 states 'it is likely that data points inside the same cluster have the same label'; the speed gain depends on this and it is never measured.
domain assumption Predicate-argument (verb-object) structures are a good representation of user intents.
Section 4.2 uses verbs and objects to construct label names; no comparison with other intent taxonomies is made.
domain assumption The manually built 140-sentence test set with 14 expert-verified labels is a valid ground truth for evaluation.
Section 5.3 describes the test set; Section 5.5 manually maps 80% of annotator labels to it, assuming that mapping is correct and unbiased.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Active Annotation: bootstrapping annotation lexicon and guidelines for supervised NLU learning." pith.science (2026). https://pith.science/paper/RMH4IQ5T

@misc{pith2026190804092,
  author       = {Pith},
  title        = {Pith review of: Active Annotation: bootstrapping annotation lexicon and guidelines for supervised NLU learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RMH4IQ5T}},
  note         = {Machine review of arXiv:1908.04092}
}

read the original abstract

Natural Language Understanding (NLU) models are typically trained in a supervised learning framework. In the case of intent classification, the predicted labels are predefined and based on the designed annotation schema while the labelling process is based on a laborious task where annotators manually inspect each utterance and assign the corresponding label. We propose an Active Annotation (AA) approach where we combine an unsupervised learning method in the embedding space, a human-in-the-loop verification process, and linguistic insights to create lexicons that can be open categories and adapted over time. In particular, annotators define the y-label space on-the-fly during the annotation using an iterative process and without the need for prior knowledge about the input data. We evaluate the proposed annotation paradigm in a real use-case NLU scenario. Results show that our Active Annotation paradigm achieves accurate and higher quality training data, with an annotation speed of an order of magnitude higher with respect to the traditional human-only driven baseline annotation methodology.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages

[1]

programming by example

Introduction Supervised methods are arguably one of the most popular and used techniques for a wide range of tasks in Machine Learning (ML), especially Natural Language tasks. This class of tech - niques is based on the notion of “programming by example”, where the annotation guidelines let the designers specify t he output that they are looking for. Even...

work page
[2]

Background In this section, we ﬁrst discuss relevant work from the emerg ing ﬁeld of human-in-the-loop computational architectures ap plied for data annotation, then we review existing work methodolo g- ically related to our proposed framework. Human-in-the-loop systems, or human-machine hybrid sys- tems, are aimed at exploiting the complementarity betwee...

work page
[3]

Universal Sen- tence Encoder

Active Annotation The Active Annotation methodology employs both human and machine intelligence to create data for machine learning mo d- els. The ﬁnal objective is to shorten the time to obtain effec tive machine learning models from scratch. Algorithm 1 shows a general overview of the Active Annotation methodology that we are going to present. Algorithm...

work page
[4]

I’d like to add those items to the shopping-cart

Algorithms 4.1. Guidelines Deﬁnition Procedure Algorithm 2: Guidelines Deﬁnition Pseudo Algorithm Input = CE′ – Clustered input data; Select one random cluster c inside CE′ ; Select the N most informative data points in c as Pivots; Compute a cluster label Lc; Propose the N Pivots data points and Lc to the annotator; if Annotator provides a label then ret...

work page
[5]

Dash Plotly

Experiments 5.1. Active Annotation Web-tool We deployed the Active Annotation paradigm as an annotation web-tool where the annotator can annotate the unlabelled da ta points. The annotation web-tool needs to have several requi re- ments in order to enable fast, accurate and reliable annotat ions. In particular, we designed a well-structured User Interfac ...

work page
[6]

Conclusions We presented an Active Annotation (AA) paradigm where we combined unsupervised learning in the embedding space, a human-in-the-loop methodology and linguistic insights to cre- ate data for machine learning models. This methodology was evaluated in a real use-case Natural Language Understandin g scenario: four internal annotators were enrolled ...

work page
[7]

Principal component an al- ysis

Herv´ e Abdi and Lynne J Williams. Principal component an al- ysis. Wiley interdisciplinary reviews: computational statisti cs, 2(4):433–459, 2010

work page 2010
[8]

K-means++: The a dvan- tages of careful seeding

David Arthur and Sergei V assilvitskii. K-means++: The a dvan- tages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’07, 2007

work page 2007

Show all 21 references

[9]

Ebk-means: A clust ering technique based on elbow method and k-means in wsn

Purnima Bholowalia and Arvind Kumar. Ebk-means: A clust ering technique based on elbow method and k-means in wsn. Interna- tional Journal of Computer Applications , 105(9), 2014

2014
[10]

John, Noah Constant, Mario Guajardo-Cesped es, Steve Y uan, Chris Tar, Y un-Hsuan Sung, Brian Strope, and Ray Kurzweil

Daniel Cer, Yinfei Y ang, Sheng yi Kong, Nan Hua, Nicole Li mti- aco, Rhomni St. John, Noah Constant, Mario Guajardo-Cesped es, Steve Y uan, Chris Tar, Y un-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder. CoRR, abs/1803.11175, 2018

2018 arXiv
[11]

Cohn, Zoubin Ghahramani, and Michael I

David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. Ac- tive learning with statistical models. J. Artif. Intell. Res. , 4:129– 145, 1994

1994
[12]

Hybrid humanmachine information s ystems: Challenges and opportunities

Gianluca Demartini. Hybrid humanmachine information s ystems: Challenges and opportunities. Computer Networks , 90:5 – 13,
[13]

Michael Heilman and Noah A. Smith. Rating computer-gene rated questions with mechanical turk. In Mturk@HLT-NAACL, 2010

2010
[14]

Convolutional neural networks for sentence cl assiﬁ- cation

Y oon Kim. Convolutional neural networks for sentence cl assiﬁ- cation. In EMNLP, 2014

2014
[15]

M i- crosoft dialogue challenge: Building end-to-end task-com pletion dialogue systems

Xiujun Li, Sarah Panda, Jingjing Liu, and Jianfeng Gao. M i- crosoft dialogue challenge: Building end-to-end task-com pletion dialogue systems. arXiv preprint arXiv:1807.11125, 2018

2018 arXiv
[16]

Recaptcha: Human-based character reco g- nition via web security measures

Colin McMillen David Abraham Luis V on Ahn, Benjamin Mau - rer and Manuel Blum. Recaptcha: Human-based character reco g- nition via web security measures. Science 321, 5895 (2008), 14651468, 2008

2008
[17]

Y . A. Malkov and D. A. Y ashunin. Efﬁcient and robust appr oxi- mate nearest neighbor search using Hierarchical Navigable Small World graphs. arXiv e-prints, March 2016

2016
[18]

Interrater reliability: the kappa stati stic

Mary L McHugh. Interrater reliability: the kappa stati stic. Bio- chemia medica: Biochemia medica , 22(3):276–282, 2012

2012
[19]

Costa-Juss` a, and Rafael Banchs

Bart Mellebeek, Francesc Benavent, Jens Grivolla, Joa n Codina, Marta R. Costa-Juss` a, and Rafael Banchs. Opinion mining of spanish customer comments with non-expert annotations on m e- chanical turk. In Proceedings of the NAACL HLT 2010 W orkshop on Creating Speech and Langu...

2010
[20]

Active annotation in the luna italian corpus of spont aneous dialogues

Christian Raymond, Kepa Joseba Rodr´ ıguez, and Giusep pe Ric- cardi. Active annotation in the luna italian corpus of spont aneous dialogues. In LREC, 2008

2008
[21]

Using predicate-argument structures for informa tion ex- traction

Mihai Surdeanu, Sanda Harabagiu, John Williams, and Pa ul Aarseth. Using predicate-argument structures for informa tion ex- traction. In Proceedings of the 41st Annual Meeting of the Asso- ciation for Computational Linguistics , 2003

2003

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

programming by example

Introduction Supervised methods are arguably one of the most popular and used techniques for a wide range of tasks in Machine Learning (ML), especially Natural Language tasks. This class of tech - niques is based on the notion of “programming by example”, where the annotation guidelines let the designers specify t he output that they are looking for. Even...

work page

[2] [2]

Background In this section, we ﬁrst discuss relevant work from the emerg ing ﬁeld of human-in-the-loop computational architectures ap plied for data annotation, then we review existing work methodolo g- ically related to our proposed framework. Human-in-the-loop systems, or human-machine hybrid sys- tems, are aimed at exploiting the complementarity betwee...

work page

[3] [3]

Universal Sen- tence Encoder

Active Annotation The Active Annotation methodology employs both human and machine intelligence to create data for machine learning mo d- els. The ﬁnal objective is to shorten the time to obtain effec tive machine learning models from scratch. Algorithm 1 shows a general overview of the Active Annotation methodology that we are going to present. Algorithm...

work page

[4] [4]

I’d like to add those items to the shopping-cart

Algorithms 4.1. Guidelines Deﬁnition Procedure Algorithm 2: Guidelines Deﬁnition Pseudo Algorithm Input = CE′ – Clustered input data; Select one random cluster c inside CE′ ; Select the N most informative data points in c as Pivots; Compute a cluster label Lc; Propose the N Pivots data points and Lc to the annotator; if Annotator provides a label then ret...

work page

[5] [5]

Dash Plotly

Experiments 5.1. Active Annotation Web-tool We deployed the Active Annotation paradigm as an annotation web-tool where the annotator can annotate the unlabelled da ta points. The annotation web-tool needs to have several requi re- ments in order to enable fast, accurate and reliable annotat ions. In particular, we designed a well-structured User Interfac ...

work page

[6] [6]

Conclusions We presented an Active Annotation (AA) paradigm where we combined unsupervised learning in the embedding space, a human-in-the-loop methodology and linguistic insights to cre- ate data for machine learning models. This methodology was evaluated in a real use-case Natural Language Understandin g scenario: four internal annotators were enrolled ...

work page

[7] [7]

Principal component an al- ysis

Herv´ e Abdi and Lynne J Williams. Principal component an al- ysis. Wiley interdisciplinary reviews: computational statisti cs, 2(4):433–459, 2010

work page 2010

[8] [8]

K-means++: The a dvan- tages of careful seeding

David Arthur and Sergei V assilvitskii. K-means++: The a dvan- tages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’07, 2007

work page 2007