Predict Emoji Combination with Retrieval Strategy

Jianmin Wu; Tianhuang Su; Tinghsuan Chao; Weitsung Lin

REVIEW 3 major objections 5 minor 9 references

Predict Emoji Combination with Retrieval Strategy

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Emoji combinations are best predicted by retrieving from a mined dictionary of frequent combinations, which raises F1 from 0.141 to 0.204.

desk verdict A simple retrieval re-ranking idea for a new emoji-combination task, but the reported F1 gains rest on an evaluation that never rules out candidate-list leakage. read the letter →

arxiv 1908.07761 v1 pith:VKADLA6T submitted 2019-08-21 cs.CL

Weitsung Lin , Tinghsuan Chao , Jianmin Wu , Tianhuang Su This is my paper · ORCID

classification cs.CL

keywords emojipredictioncombinationretrievalstrategyBERTTwittermulti-emojifrequencydictionarycross-entropyranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that emoji combinations, consecutive sequences of one to three emojis, can be predicted from short Twitter text by treating the combinations as a finite dictionary of frequent phrases and retrieving the best-matching one. The authors build a BERT-based model that predicts a probability for each of 500 emojis, then rank 30,000 pre-mined emoji combinations by cross-entropy against that distribution. On 10,000 test tweets this retrieval strategy raises F1 from 0.141 with a naive top-3 baseline to 0.204, and to 0.236 when a length penalty is added. The significance, if correct, is that multi-emoji prediction can be handled without open-ended generation, relying instead on a mined inventory plus a unigram approximation of combination probability.

What carries the argument

The retrieval strategy is a ranking score, $S_j = -\sum_i \text{Candidate}_j(y_i) \log P(y_i|W)$, where $\text{Candidate}_j(y_i)$ is the unigram emoji distribution in a candidate combination and $P(y_i|W)$ is the BERT model's predicted per-emoji probability. This score turns emoji-combination prediction into nearest-phrase retrieval over a 30,000-entry dictionary, with frequency counts breaking ties between orderings and an optional size penalty trading recall against precision.

What would settle it

Re-run the experiment with candidate mining and frequency estimation performed only on the training portion, keeping the 10,000 test tweets out; if the retrieval F1 does not stay close to 0.204 (0.236 with penalty), the reported improvement is an artifact of leakage.

Watch

Extended reading notes

Core claim

The paper claims that predicting an emoji combination is best framed as dictionary retrieval: from a pre-mined list of the 30,000 most frequent emoji combinations in the corpus, rank each candidate by the cross-entropy between its unigram emoji distribution and a fine-tuned BERT's predicted per-emoji probability, then output the top-ranked candidate, with an optional length penalty. On a 10,000-tweet test set this raises F1 from 0.141 (naive top-3) to 0.204, and to 0.236 with a penalty that balances precision and recall. The authors also present this as the first work to study emoji combination prediction, extending single-emoji prediction to sequences while keeping the model's output space constrained to a mined vocabulary.

Load-bearing premise

The 10,000 test tweets are assumed to be independent of the candidate emoji combinations and their frequency counts used by the retrieval ranking; the paper does not state that these were computed without the test set, so the reported gains could be inflated by test-set leakage.

Editorial extensions

If this is right

If the central claim holds, emoji-combination prediction can be treated as closed-vocabulary retrieval: the 30k mined candidates define the entire output space, so any novel combination is by construction unpredicted.
The reported F1 gain from 0.141 to 0.204 implies that the cross-entropy ranking, not the underlying BERT probabilities alone, is the main driver of performance; swapping in any per-emoji probability model should reproduce most of the gain.
The penalty result (0.236 at pen=0.3) shows that a simple size-based precision-recall knob tunes the system in a predictable way, which is useful for practical deployment.
The unigram approximation in Eq. 2 means order within a combination is only captured by candidate frequencies, not by sequence modeling, so the method's quality ceiling is set by how well frequency ties match natural ordering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If candidate mining were truly train-only, the retrieval strategy would likely transfer to any finite inventory of short sequences, such as multi-word hashtags or meme phrases, where ranking by cross-entropy replaces generation.
The paper leaves open whether the 30k candidate-set size itself is the bottleneck; testing smaller or larger dictionaries would reveal whether emoji combinations form a Zipfian vocabulary like words.
A natural testable extension is to replace the unigram product with a bigram or trigram candidate score; a large improvement would mean the unigram approximation is the limiting factor, while a small gain would confirm that frequency-based retrieval already captures most ordering information.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper defines a genuinely new task — predicting ordered 1–3 emoji sequences — and applies a simple retrieval re-ranking over BERT probabilities that plausibly helps. The F1 numbers as reported, though, are not yet trustworthy.

What is new and good: previous work predicts single emojis; here the target is an emoji combination as a sequence, closer to actual usage. The retrieval strategy treats frequent combinations as a phrase dictionary and ranks candidates by cross-entropy against the per-emoji distribution, breaking ties by frequency. That is a straightforward but reasonable idea. The paper is clearly written, the comparison keeps the same model weights across strategies, and fine-tuning BERT on Twitter text is a sensible baseline choice. The math is just cross-entropy ranking, and the citation pattern to prior single-emoji work looks adequate.

The soft spots are concentrated in the evaluation. The test set is 10,000 random samples from the same 9M-sample Twitter dataset, and the retrieval dictionary is described as the “30k most frequently used emojis in dataset.” Nothing says the test samples were excluded from candidate mining or from the frequency tie-break. If they were not, the dictionary likely contains many gold combinations from the test set, and the naive and greedy baselines cannot use that information. That would inflate the retrieval F1. The penalty is also tuned on the test set (best at pen=0.3 in Table 1), with no validation set or error bars. The penalty effect is only about 3 F1 points; the split ambiguity is the load-bearing issue, not the tuning. There is also no code or released data, and the candidate list is not specified as train-only.

My take: the central idea is plausible and the task is new, so this deserves a serious referee rather than a desk reject. For readers working on emoji prediction or retrieval-augmented decoding for constrained output spaces, there is something to take from it. I would ask the authors to state explicitly whether candidate mining and frequency estimation came only from the training partition, add a validation set to select the penalty, and report variance or significance. If the split is clean, the reported result is believable enough for a small applied NLP venue.

Referee Report

3 major / 5 minor

Summary. The paper studies emoji combination prediction: given a short text, predict a consecutive sequence of 1–3 emojis. The authors build a ~9M-sample Twitter dataset, fine-tune a BERT-based model to produce per-emoji probabilities, and then compare three decoding strategies: Naive Top-3, Greedy Top-3, and a proposed Retrieval Strategy that ranks a pre-mined dictionary of 30k frequent emoji combinations by cross-entropy with the predicted distribution, using frequency counts as a tie-breaker and an optional size penalty. In Table 1, the Retrieval Strategy reports an F1 of 0.204 versus 0.141 for Naive Top-3, rising to 0.236 with a penalty of 0.3.

Significance. If the reported gains are valid, the paper offers a simple and intuitive contribution: treating emoji combinations as phrases and retrieving them from a mined dictionary rather than generating them independently. The approach is clearly described, and the comparison across multiple thresholds and penalty values is transparent. The central quantitative claim, however, rests entirely on the evaluation protocol: the candidate dictionary, the frequency tie-breaker, and the penalty selection must be independent of the test set. Because the manuscript does not establish that independence, the significance of the result is currently uncertain.

major comments (3)

[Section 3 and Section 5] The paper never states that the 10,000 test samples are excluded from the candidate mining and frequency estimation. Section 5 says the Retrieval Strategy ranks '30k most frequently used emojis in dataset' and uses frequency of emoji combination in 'training data' for tie-breaking, while Section 3 describes only a single 9M-sample Twitter dataset from which the test set is randomly selected. If the candidate dictionary and frequency counts are built from the full dataset, the dictionary is likely to contain the exact gold combinations of many test samples, and the frequency tie-breaker is influenced by test labels. This would give Retrieval an unfair advantage over Naive and Greedy, which do not access such a gold-inclusive dictionary, and would invalidate the reported F1 improvement. The authors must specify the exact data flow and, ideally, re-run candidate mining and frequency estimation on the training portion only.
[Table 1] The penalty value that yields the best reported F1 (pen=0.3) is selected after looking at the test-set results. No validation set or cross-validation is described, and no error bars or significance tests are reported for any of the numbers in Table 1. Because the penalty is a free parameter tuned on the test set, the 0.236 F1 is an optimistic, in-sample estimate rather than a reliable measure of generalization. The authors should tune the penalty on a validation split and report test performance for the fixed penalty, along with variance estimates such as bootstrap confidence intervals.
[Section 4.2, Eq. (3)] The tie-breaking frequency is said to come from 'training data,' but the manuscript does not specify what the training data are in relation to the 10K test samples or the 9M-sample dataset. It is unclear whether the model is fine-tuned on the remaining samples after removing the test set, and whether the frequency counts are computed on the same remainder or on the full dataset. This ambiguity is load-bearing for the leakage concern and must be resolved by a precise description of the train/test split and the provenance of both the candidate set and the frequency counts.

minor comments (5)

[Section 5] The phrase '30k most frequently used emojis in dataset' should say 'emoji combinations,' since the retrieval candidates are combinations of emojis, not individual emojis.
[Section 4.1, Eq. (1) and Eq. (2)] The product notation in Eqs. (1) and (2) is confusing: Eq. (1) starts with P(y0|W) followed by a product over i=1 to n, while Eq. (2) writes the product over i=0 to n; please clarify the indexing and whether y0 is the first emoji or a separate start token.
[References] Reference [9] has garbled author names; it should be Çağrı Çöltekin and Taraka Rama.
[Author block] The author affiliation line contains placeholder-like email handles ('v liweicong, v zhaotingxuan, wujianmin01, sutianhuang') that do not match the author names listed above; this should be corrected.
[Figure 2] Figure 2 is captioned '500 emoji classes,' but the task is to predict emoji combinations; the caption should clarify whether the figure shows the 500 individual emojis that form the combination vocabulary.

Circularity Check

1 steps flagged · score 6.0 of 10

Retrieval dictionary is built from the same dataset that supplies the test labels; the reported F1 gain reduces to a lookup in a test-inclusive dictionary.

self definitional [Section 5, Experiment (dataset split and Retrieval Strategy description; Table 1)]
"we randomly select 10000 samples from the Twitter dataset for testing ... In Retrieval Strategy , we use Eq.3 to rank 30k most frequently used emojis in dataset"

The candidate list for retrieval is defined as the '30k most frequently used emojis in dataset', while the test set is '10000 samples from the Twitter dataset'. Since the test samples are randomly drawn from that same 9M-sample dataset and no exclusion is stated, the gold emoji combination of each test sample is, whenever frequent enough to enter the top 30k, already contained in the dictionary used at prediction time. Retrieval thus reduces to looking up labels that were used to build the dictionary; the reported F1 gap over Naive/Greedy baselines (which have no such dictionary) measures train/test contamination rather than generalization. The frequency tie-break in the retrieval scorer uses corpus frequencies from the same data, encoding test label statistics as well.

full rationale

The paper does not engage in self-citation circularity: the cited works are all external (BERT, Transformer, GPT, and prior emoji-prediction papers), and no uniqueness theorem or self-supporting citation chain is invoked. The central circularity concern is the construction of the retrieval dictionary. The paper defines a 9M-sample Twitter dataset, randomly draws 10,000 test samples from it, and separately ranks '30k most frequently used emojis in dataset' to form the retrieval candidate set. Nowhere is it stated that candidate mining or frequency estimation excludes the test 10,000. Consequently, by the paper's own procedure, the dictionary used for prediction is defined from the same data that contains the test labels. Whenever a test gold combination is among the 30k most frequent combinations, the retrieval strategy can output it without any generalization from text, and the improvement from 0.141 to 0.204 (and 0.236 with penalty) is at least partly a self-fulfilling lookup. The additional selection of the best penalty value from several test-set F1 numbers (pen=0.2, 0.3, 0.4; best 0.3) is an evaluation overfitting issue rather than a circular derivation, but it compounds the inflation of the headline result. Because the paper's central quantitative claim depends on this holdout gap, the empirical claim is partially circular by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two fitted parameters (retrieval penalty and greedy threshold) and several domain assumptions about the dataset and the unigram approximation. The most fragile assumption is that the test set does not leak into the candidate/frequency construction.

free parameters (2)

Retrieval penalty = 0.3
Introduced in Section 5 as 'add an emoji size penalty'. The penalty value 0.3 gives the highest F1 in Table 1; it is selected by evaluating the test set, so it is effectively fitted to the evaluation data.
Greedy Top-3 threshold = 0.3 (also 0.2 and 0.4 evaluated)
The threshold for the Greedy Top-3 baseline is varied on the test set; the paper reports the best F1 at threshold 0.3. This is another parameter tuned on the test set.

assumptions (4)

domain assumption The 10,000 test samples are independent of the candidate emoji combinations and their frequency counts.
The paper does not explicitly state that candidate mining and frequency estimation are restricted to training data; this assumption is necessary for the reported F1 to reflect generalization.
domain assumption The unigram approximation P(Y|W) ≈ product of P(y_i|W) is adequate for ranking emoji combinations.
Eq.2 ignores dependencies between emojis and ordering; the paper acknowledges this but still bases the retrieval score on this approximation.
domain assumption The Twitter corpus and the 500 most frequent emoji classes are representative of emoji combination usage.
The dataset is filtered to US geo-location, English tweets, and only emojis among the 500 most frequent; results may not generalize to other languages or platforms.
standard math Cross-entropy is a valid scoring function for ranking candidates.
Eq.3 uses cross-entropy between a one-hot-like candidate distribution and the model's unigram probabilities; this is a standard divergence measure.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Predict Emoji Combination with Retrieval Strategy." pith.science (2026). https://pith.science/paper/VKADLA6T

@misc{pith2026190807761,
  author       = {Pith},
  title        = {Pith review of: Predict Emoji Combination with Retrieval Strategy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKADLA6T}},
  note         = {Machine review of arXiv:1908.07761}
}

read the original abstract

As emojis are widely used in social media, people not only use an emoji to express their emotions or mention things but also extend its usage to represent complicate emotions, concepts or activities by combining multiple emojis. In this work, we study how emoji combination, a consecutive emoji sequence, is used like a new language. We propose a novel algorithm called Retrieval Strategy to predict what emoji combination follows given a short text as context. Our algorithm treats emoji combinations as phrase in language, ranking sets of emoji combinations like retrieving words from dictionary. We show that our algorithm largely improves the F1 score from 0.141 to 0.204 on emoji combination prediction task.

Figures

Figures reproduced from arXiv: 1908.07761 by the authors.

**Figure 1.** Emoji combination use cases Recent work [2] has shown that textual information can be used to predict single emoji associated to text. [3] further extended the emoji prediction task to 300 emoji classes and used a multi-task approach, predicting emojis and emoji categories to improve prediction accuracy. In this paper, we extend the single emoji prediction task to a setup much closer to real world usage. First, we… view at source ↗

**Figure 2.** 500 emoji classes in this work and written in English. In the preprocessing stage, we removed all hyperlinks, mentions and hashtags from tweet. Also we normalize emoji by removing skin tone modifier to focus on meaning of emoji. To build the dataset, whose sample consists of a pair of textual context (feature) and emoji combination (target), we process Tweets as follow: (1) Select tweets include emojis in the 500 mo… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 7 canonical work pages

[1]

Multimodal emoji prediction

Francesco Barbieri, Miguel Ballesteros, Francesco Ronzano, and Horacio Saggion. Multimodal emoji prediction. In NAACL-HLT, 2018

work page 2018
[2]

Are emojis predictable? In EACL, 2017

Francesco Barbieri, Miguel Ballesteros, and Ho- racio Saggion. Are emojis predictable? In EACL, 2017

work page 2017
[3]

Multi-task emoji learning

Francesco Barbieri, L´ us Marujo, Pradeep Karu- turi, and William Brendel. Multi-task emoji learning. 2018

work page 2018
[4]

Ntua- slp at semeval-2018 task 2: Predicting emo- jis using rnns with context-aware attention

Christos Baziotis, Nikos Athanasiou, Georgios Paraskevopoulos, Nikolaos Ellinas, Athanasia Kolovou, and Alexandros Potamianos. Ntua- slp at semeval-2018 task 2: Predicting emo- jis using rnns with context-aware attention. In SemEval@NAACL-HLT, 2018

work page 2018
[5]

The New Modality: Emoji Challenges in Prediction, Anticipation, and Retrieval

Spencer Cappallo, Stacey Svetlichnaya, Pierre Garrigues, Thomas Mensink, and Cees Snoek. The new modality: Emoji challenges in pre- diction, anticipation, and retrieval. CoRR, abs/1801.10253, 2018

work page Pith review arXiv 2018
[6]

Bert: Pre-training of deep bidirectional transformers for language under- standing

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04805, 2018

arXiv 2018
[7]

Improving language understand- ing by generative pre-training

Alec Radford. Improving language understand- ing by generative pre-training. 2018

work page 2018
[8]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017

2017

Show all 9 references

[9]

T¨ ubingen-oslo at semeval-2018 task 2: Svms perform better than rnns in emoji prediction

agri ¨ oltekin and Taraka Rama. T¨ ubingen-oslo at semeval-2018 task 2: Svms perform better than rnns in emoji prediction. In SemEval@NAACL- HLT, 2018

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Multimodal emoji prediction

Francesco Barbieri, Miguel Ballesteros, Francesco Ronzano, and Horacio Saggion. Multimodal emoji prediction. In NAACL-HLT, 2018

work page 2018

[2] [2]

Are emojis predictable? In EACL, 2017

Francesco Barbieri, Miguel Ballesteros, and Ho- racio Saggion. Are emojis predictable? In EACL, 2017

work page 2017

[3] [3]

Multi-task emoji learning

Francesco Barbieri, L´ us Marujo, Pradeep Karu- turi, and William Brendel. Multi-task emoji learning. 2018

work page 2018

[4] [4]

Ntua- slp at semeval-2018 task 2: Predicting emo- jis using rnns with context-aware attention

Christos Baziotis, Nikos Athanasiou, Georgios Paraskevopoulos, Nikolaos Ellinas, Athanasia Kolovou, and Alexandros Potamianos. Ntua- slp at semeval-2018 task 2: Predicting emo- jis using rnns with context-aware attention. In SemEval@NAACL-HLT, 2018

work page 2018

[5] [5]

The New Modality: Emoji Challenges in Prediction, Anticipation, and Retrieval

Spencer Cappallo, Stacey Svetlichnaya, Pierre Garrigues, Thomas Mensink, and Cees Snoek. The new modality: Emoji challenges in pre- diction, anticipation, and retrieval. CoRR, abs/1801.10253, 2018

work page Pith review arXiv 2018

[6] [6]

Bert: Pre-training of deep bidirectional transformers for language under- standing

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04805, 2018

arXiv 2018

[7] [7]

Improving language understand- ing by generative pre-training

Alec Radford. Improving language understand- ing by generative pre-training. 2018

work page 2018

[8] [8]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017

2017