Don't Just Scratch the Surface: Enhancing Word Representations for Korean with Hanja

Kang Min Yoo; Sang-goo Lee; Taeuk Kim

REVIEW 4 major objections 4 minor 28 references

Don't Just Scratch the Surface: Enhancing Word Representations for Korean with Hanja

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hanja-level SISG adds Hanja n-gram vectors to the subword skip-gram scoring function, initializes them with Chinese character embeddings, and improves Korean word representations on analogy and headline generation.

desk verdict Modest, honest Korean-embedding work whose main analogy claim rests on a weak metric; deserves review if the authors fix the evaluation. read the letter →

arxiv 1908.09282 v3 pith:7W5PT35H submitted 2019-08-25 cs.CL cs.LG

Kang Min Yoo , Taeuk Kim , Sang-goo Lee This is my paper · ORCID

classification cs.CLcs.LG

keywords KoreanwordrepresentationsHanjaskip-gramsubwordn-gramscross-lingualtransferlearninganalogyheadlinegenerationSino-Korean

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that Korean word embeddings can be improved by explicitly adding Hanja, the Chinese-origin logograms used in written Korean, as an extra signal in the subword skip-gram objective. The central idea is that Hanja syllables carry more lexical meaning than Hangul phonograms, so their n-gram vectors should help the model separate the meanings of Sino-Korean words that look identical in Hangul. The paper also tests whether these Hanja vectors can be initialized with pre-trained Chinese character embeddings, which would mean that knowledge transfers between languages at the level of individual characters. If the approach is right, static Korean word representations become more useful for analogy tasks and for generating news headlines, which draw heavily on formal Sino-Korean vocabulary, while remaining competitive on colloquial sentiment analysis.

What carries the argument

The central mechanism is the extended scoring function $s^{(h)}(w,c) = s^{(j)}(w,c) + \sum_{h \in G^{(h)}_w} z_h v_c$, where $G^{(h)}_w$ is the set of Hanja n-grams extracted from the Hanja annotations of word $w$, $z_h$ is the learned vector for a Hanja n-gram, and $v_c$ is the context-word vector. This equation adds a semantically richer logogram channel to the existing character- and jamo-level channels, so that predicting a context word requires the model to align the meaning carried by Hanja characters with the surrounding discourse; the Hanja vectors are initialized with Chinese character embeddings so that cross-lingual knowledge enters at the start of training.

What would settle it

A controlled run on a fixed corpus with gold-standard, human-verified Hanja annotations instead of automatic ones would settle the question: if the analogy and headline-generation gains vanish or shrink to noise when the annotations are correct, the reported improvements depend on tagger noise rather than on Hanja semantics.

Watch

Extended reading notes

Core claim

The paper proposes Hanja-level SISG, a modification of the subword-information skip-gram model in which every Korean word is annotated with Hanja sequences, and the scoring function is extended to sum dot products between Hanja n-gram vectors and the context-word vector alongside the usual character and jamo n-grams. It reports that this extension improves performance on the Korean word analogy test over both plain skip-gram and the jamo-level baseline, and that initializing the Hanja n-gram vectors with Chinese character embeddings gives a further improvement over random initialization, demonstrating character-level cross-lingual transfer. On a newly collected Korean news headline generation task, the Hanja-initialized embeddings lead to higher BLEU scores and lower perplexity than prior embeddings, while on a movie-review sentiment task the method is competitive but not superior, which the paper attributes partly to errors in the automatic Hanja annotation.

Load-bearing premise

The automatic Hanja tagger used to annotate the entire training corpus produces labels accurate enough that the Hanja n-grams correspond to true Sino-Korean meaning; the paper itself reports seeing annotation errors that can hurt performance.

Editorial extensions

If this is right

Korean word analogy accuracy should improve whenever the training corpus contains Sino-Korean words with Hanja annotations, because the extra logogram channel disambiguates homophonous Hangul forms.
Character-level cross-lingual initialization from Chinese embeddings is a viable way to inject knowledge into Korean word representations without parallel text or shared vocabulary.
Downstream tasks that involve formal written Korean, such as news headline generation, stand to benefit more from Hanja-informed embeddings than tasks on colloquial or spoken-style text.
The approach is a drop-in extension of the subword skip-gram scoring function, so it can be combined with other subword vocabularies and training objectives without changing the underlying architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the same character-level transfer idea could be tested for Japanese Kanji, Vietnamese Han-Nom, or other scripts that share etymological roots with Chinese characters; the paper only demonstrates it for Korean Hanja, so the broader cross-linguistic scope remains an open extension.
The observed trade-off between gains on analogy and losses on similarity suggests that Hanja injects categorical, relational structure rather than graded relatedness; a future model might add Hanja information only for words with high Sino-Korean confidence to avoid hurting similarity judgements.
An end-to-end model that learns Hanja annotation jointly with embeddings could remove the current dependence on an external tagger; the paper names this as future work, and one testable version would compare supervised tagger-free training against the pipelined approach on the same analogies and headline task.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version. The paper does one genuinely new thing: it adds Hanja n-grams to a Korean subword skip-gram and initializes them from Chinese character embeddings. That is a sensible, well-motivated idea, and the authors tested it cleanly, including an ablation with random initialization. Credit where due: the code is public, they re-ran their baselines with their own evaluation script rather than quoting published numbers, and they openly report the results that hurt them (sentiment gets worse, similarity gets worse). That is real evidence-gathering behavior.

The problem is the load-bearing positive result. The analogy improvement in Table 1 is reported as average cosine distance to the true answer (1−cos(v_b−v_a+v_c, v_d)), not as the standard rank-based accuracy. That metric can improve when the true answer moves marginally closer even if it is still rarely the nearest neighbor. The paper gives no rank accuracy, no error bars, and no count of solved quadruples. Since the other results are flat or negative—WS353 Spearman drops from .671 to .633, NSMC F1 drops from 83.50 to 81.75, and headline BLEU-1 goes from 31.78 to 32.03—the analogy number is doing most of the work for the abstract's claim. I don't think the stress-test is off base; this is the main soft spot.

Secondary but real: the method depends on an automatic Hanja tagger, and the authors concede it makes errors, especially on colloquial text. That explains the sentiment result but it also means the trained model inherits whatever systematic noise the tagger has. Fair enough, but it's another reason to treat the mixed results as exactly what they are: mixed.

The Hanja n-gram size is a hand-chosen hyperparameter, but there's no sign it was tuned to the test sets, so I'm not worried about circularity.

Who benefits: anyone working on Korean lexical semantics or on subword embeddings for morphologically rich languages. It is not a breakthrough and it is not a large evaluation. But the idea is clear, the write-up is honest, and the code is available. I would send it to a serious referee, with the explicit request that the authors report rank-based analogy accuracy and significance tests. If the analogy result survives that check, the paper earns its place. If it doesn't, the abstract needs major revision.

Recommendation: peer-review it, but make the evaluation demand explicit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Hanja-level SISG, an extension of Korean subword skip-gram models that adds Hanja n-gram vectors to the scoring function and initializes those vectors with pre-trained Chinese character embeddings. The authors evaluate the resulting embeddings on Korean word analogy and similarity tests and on two downstream tasks, including a newly collected Korean news headline generation dataset. They report improvements on the word analogy test and small BLEU gains on headline generation, while similarity correlations and NSMC sentiment F1 are worse than the jamo-level baseline. The paper acknowledges that the automatic Hanja tagger introduces errors and that the approach depends on an external annotator.

Significance. If the central claim were established, the paper would offer a simple and linguistically motivated method for injecting etymological knowledge into Korean word embeddings, together with a reusable new headline-generation benchmark and publicly available code and models. The cross-lingual initialization idea is also potentially interesting for other scripts with historical character relations. However, the evidence as presented is not yet convincing: the main positive result rests on a nonstandard analogy metric, the downstream gains are small and without significance testing, and the similarity and sentiment results move in the opposite direction. The paper's own limitation statement about Hanja-tagger reliability is honest but only partially mitigates these concerns.

major comments (4)

[§3.2.1 and Table 1] The word analogy evaluation is reported as mean cosine distance 1−cos(v_a+v_b−v_c, v_d), with lower values claimed to be better, but the baseline rows marked † are taken from Park et al. (2018), whose standard protocol is rank-based analogy accuracy (higher is better). Comparing SISG(cjh3)'s 0.395 with SISG(cj)†'s 0.432 as an improvement is therefore comparing two different quantities. Even setting aside the † rows, mean distance to the correct answer is not a measure of whether the model actually solves analogies: it can improve by uniformly shrinking distances without increasing the frequency with which the correct word is the nearest neighbor. Please report standard top-1/top-k rank accuracy with the query words properly excluded, and provide error bars or a significance test. This is load-bearing because the similarity, sentiment, and BLEU results in Tables 1–3 are otherwise mixed or negative.
[§3.3.1 and Table 2] The headline generation gains are very small and are not shown to be statistically meaningful. BLEU-1 increases from 31.78 to 32.03, BLEU-2 from 11.17 to 11.25, and BLEU-3 from 4.80 to 4.83, while the reported per-word perplexity is essentially unchanged (3.938 vs 3.941, where lower is better). No variance across random seeds or significance tests are reported for this task. Given that BLEU differences of this size are often within run-to-run noise, the downstream effectiveness claim is not established by the current evidence.
[§3.2.2, §3.3.2, and Tables 1, 3] The negative results are acknowledged but not adequately accounted for. WS353 Spearman correlation drops from 0.671 (SISG(cj)) to 0.633 (SISG(cjh3)), and NSMC F1 drops from 83.50 to 81.75. The paper attributes the sentiment drop to Hanja-tagger errors, but no tagger error analysis is provided; moreover, the same automatic tagger is used for all datasets, including the positive analogy and headline results. The authors should either quantify tagger accuracy on the relevant text types or demonstrate that the positive results are robust to annotation noise, rather than treating the tagger limitation as confined to the sentiment task.
[§2.4 and Table 1] The evidence for the cross-lingual transfer contribution is thin. The only comparison supporting it is SISG(cjhr) versus SISG(cjh3): on the analogy metric the pretrained initialization improves by 0.014 (0.409 vs 0.395), but on the similarity test the randomly initialized version is better (Spearman 0.647 vs 0.633). No significance testing is provided. Since the abstract explicitly credits cross-lingual transfer as part of the method, this claim needs stronger support, ideally with the standard analogy accuracy metric and multiple runs.

minor comments (4)

[Table 1] The column header 'Pr.' is used without explanation; it should be written as 'Pearson' and 'Spearman' for clarity.
[§3.1.2] There is a typo in the notation 'SISG(cjh))' with an extra closing parenthesis; it should read 'SISG(cjh)'.
[§A.1] The phrase '20m bucket size' is ambiguous; if it means 20 million, it should be written as '20M' or '20,000,000'.
[§3.3.1] The sentence 'no equivalent exists for Korean' about ROUGE is inaccurate, since ROUGE is a language-independent n-gram overlap measure and can be applied to Korean with appropriate tokenization; please rephrase or justify.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central SISG(cjh) claim is an independent extension of SISG, evaluated on external benchmarks, with only minor non-load-bearing self-citations in related work.

full rationale

The paper's central derivation is self-contained against external inputs. Equation (5) defines Hanja-level SISG by adding a Hanja n-gram term to the existing SISG/Jamo scoring function; this is a model proposal, not an equivalence to the evaluation. Hanja n-gram vectors are initialized with external Chinese embeddings (Li et al., 2018), and the intrinsic and downstream evaluations all use externally constructed benchmarks: the Park et al. (2018) Korean analogy and WS353 similarity sets, the NSMC sentiment corpus, and a newly collected headline-generation set. No test-set quantity is used as a training signal or fitted parameter: the Hanja n-gram lengths are presented as model variants (cjh3, cjh4, cjhr), and the character/jamo n-gram ranges are taken from prior work via ablation rather than tuned on the evaluation. The only author-overlapping citations (Choi et al. 2016 and 2017) appear in the related-work overview of syllable/Jamo methods and are not load-bearing for the proposed model's correctness. The skeptical concern about the analogy metric (average cosine distance rather than rank accuracy) is a measurement-validity issue, not a circularity: it does not make Equation (5)'s output equivalent to its input by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central mechanism is an extension of subword skip-gram with one extra subword channel (Hanja) and an external initialization from Chinese embeddings. The main domain assumption is that Hanja annotation is accurate and semantically useful, which the authors themselves qualify in the sentiment analysis section.

free parameters (1)

Hanja n-gram size = 3 and 4 (cjh3, cjh4)
The length of Hanja n-grams is a hyperparameter (Section 2.3); the paper reports results for values 3 and 4 without a systematic search or selection mechanism.

assumptions (4)

domain assumption Sino-Korean words form a substantial portion of Korean vocabulary and Hanja logograms carry more lexical meaning than Hangul phonograms.
Motivates the entire approach; stated in Section 1 as historical and linguistic background.
domain assumption Hanjaro tagger produces sufficiently accurate Hanja annotations for the training corpus.
All datasets are automatically annotated with Hanjaro (Section 3.1.1); the authors later report observed tagging errors (Section 3.3.2).
domain assumption Pretrained Chinese character embeddings from Li et al. (2018) can be mapped one-to-one to Hanja and transfer semantically to Korean.
Used to initialize Hanja n-gram vectors (Section 2.4); no separate validation of mapping quality is provided.
standard math Skip-gram with negative sampling is a valid model for learning word representations.
External baseline from Mikolov et al. (2013) accepted without proof.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Don't Just Scratch the Surface: Enhancing Word Representations for Korean with Hanja." pith.science (2026). https://pith.science/paper/7W5PT35H

@misc{pith2026190809282,
  author       = {Pith},
  title        = {Pith review of: Don't Just Scratch the Surface: Enhancing Word Representations for Korean with Hanja},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7W5PT35H}},
  note         = {Machine review of arXiv:1908.09282}
}

read the original abstract

We propose a simple yet effective approach for improving Korean word representations using additional linguistic annotation (i.e. Hanja). We employ cross-lingual transfer learning in training word representations by leveraging the fact that Hanja is closely related to Chinese. We evaluate the intrinsic quality of representations learned through our approach using the word analogy and similarity tests. In addition, we demonstrate their effectiveness on several downstream tasks, including a novel Korean news headline generation task.

Figures

Figures reproduced from arXiv: 1908.09282 by the authors.

**Figure 1.** An example of a Korean word showing its form and multi-level meanings. The Sino-Korean word consists of Hangul phonograms (KR) and Hanja logograms (HJ). Although annotation of Hanja is optional, it offers deeper insight into the word meaning due to its association with the Chinese characters (CN). Information Skip-Gram), for capturing the semantics of Hanja and subword structures of Korean and introducing them in… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 16 canonical work pages

[1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

arXiv 2014
[2]

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the ACL, 5:135--146

work page 2017
[3]

Jihun Choi, Jonghem Youn, and Sang-goo Lee. 2016. A grapheme-level approach for constructing a korean morphological analyzer without linguistic knowledge. In 2016 IEEE International Conference on Big Data (Big Data), pages 3872--3879. IEEE

work page 2016
[4]

Sanghyuk Choi, Taeuk Kim, Jinseok Seol, and Sang-goo Lee. 2017. A syllable-based technique for word embeddings of korean words. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 36--40

work page 2017
[5]

Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053

arXiv 2018
[6]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018
[7]

Daniel Edmiston and Karl Stratos. 2018. Compositional morpheme embeddings with affixes as functions and stems as arguments. In Proceedings of the Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP, pages 1--5

work page 2018
[8]

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393

arXiv 2016

Show all 28 references

[9]

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693--1701

2015
[10]

Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

1997
[11]

Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence

2016
[12]

Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. https://doi.org/10.18653/v1/P17-4012 Open NMT : Open-source toolkit for neural machine translation . In Proc. ACL

2017 doi
[13]

Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291

2019 arXiv
[14]

Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. https://openreview.net/forum?id=H196sainb Word translation without parallel data . In ICLR

2018
[15]

Ki-Moon Lee and S Robert Ramsey. 2011. A history of the Korean language. Cambridge University Press

2011
[16]

Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the ACL), volume 2, pages 138--143

2018
[17]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111--3119

2013
[18]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on ACL, pages 311--318. Association for Computational Linguistics

2002
[19]

Sungjoon Park, Jeongmin Byun, Sion Baek, Yongseok Cho, and Alice Oh. 2018. Subword-level word vector representations for korean. In Proceedings of the 56th Annual Meeting of the ACL, volume 1, pages 2429--2438

2018
[20]

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the NAACL: HLT, Volume 1 (Long Papers), volume 1, pages 2227--2237

2018
[21]

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of ACL, volume 1, pages 1715--1725

2016
[22]

Ho-Min Sohn. 2001. The Korean Language. Cambridge University Press

2001
[23]

Jae Jung Song. 2006. The Korean language: Structure, use and context. Routledge

2006
[24]

Karl Stratos. 2017. A sub-character architecture for korean language processing. In Proceedings of the 2017 Conference on EMNLP, pages 721--726

2017
[25]

Insup Taylor. 1997. Psycholinguistic reasons for keeping chinese characters in korean and japanese. Cognitive processing of Chinese and related Asian languages, 319

1997
[26]

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649--657

2015
[27]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
[28]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

arXiv 2014

[2] [2]

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the ACL, 5:135--146

work page 2017

[3] [3]

Jihun Choi, Jonghem Youn, and Sang-goo Lee. 2016. A grapheme-level approach for constructing a korean morphological analyzer without linguistic knowledge. In 2016 IEEE International Conference on Big Data (Big Data), pages 3872--3879. IEEE

work page 2016

[4] [4]

Sanghyuk Choi, Taeuk Kim, Jinseok Seol, and Sang-goo Lee. 2017. A syllable-based technique for word embeddings of korean words. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 36--40

work page 2017

[5] [5]

Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053

arXiv 2018

[6] [6]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018

[7] [7]

Daniel Edmiston and Karl Stratos. 2018. Compositional morpheme embeddings with affixes as functions and stems as arguments. In Proceedings of the Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP, pages 1--5

work page 2018

[8] [8]

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393

arXiv 2016