{"id":"8234c3ae-d3f2-447d-b449-6a80d92ff281","arxiv_id":"1908.09282","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Hanja-level subword embeddings with Chinese pretrained initialization improve Korean word analogy and headline generation at the cost of similarity and sentiment performance.","lead":"This paper adds Hanja, the Chinese-derived characters used in Korean, to subword skip-gram embeddings and initializes them with Chinese character embeddings. The method improves Korean word analogy and news headline generation, but hurts similarity and sentiment tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main analogy improvement is reported as mean cosine distance to the true answer, not standard rank accuracy; this metric can improve without improving actual analogy solving.","rationale":"The central claim is that Hanja n-grams, initialized from Chinese embeddings, improve Korean word representations. For that claim to hold, the positive evaluation must be trustworthy. Similarity and sentiment results go the wrong way, and the headline BLEU gain is tiny, so the analogy result carries the weight. Section 3.2.1 says the metric is average 1−cos to the true target, lower better. Prior work (Park et al. 2018) uses rank accuracy, and the paper's own Table 1 footnote acknowledges that their evaluation script gives different numbers from the original authors. Average distance to the true answer is not monotonically related to top-1 rank accuracy, so the reported 'improvement' may be a summary-statistic artifact. The concrete test is to recompute the same models with top-1 rank accuracy; if the ordering flips, the central claim is unsupported. I therefore disagree with the reader that the Hanja tagger is the weakest assumption. The tagger is a real and self-acknowledged limitation (Section 3.3.2), but the metric issue is prior: it would undermine the paper even if the tagger were perfect. Until the rank-accuracy check is run, the evidence for the central claim is insufficient, so I recommend UNVERDICTED rather than CONDITIONAL or REJECT.","tokens_in":8633,"tokens_out":14032,"duration_ms":149059,"concrete_test":"Re-run the word analogy evaluation for SG, SISG(c), SISG(cj), SISG(cjh3), SISG(cjh4), and SISG(cjhr) using the standard rank-based protocol: for each quadruple a:b::c:d, compute v_b − v_a + v_c, score every vocabulary word x by cosine similarity (or 1 − distance) to this vector, exclude a, b, and c from the candidate set, and record whether d is ranked first. Report top-1 accuracy overall and by category. If SISG(cjh3) does not beat SISG(cj) under this metric, the paper's headline improvement is an artifact of the average-distance summary and the central claim loses its principal support.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's principal positive evidence is the word analogy improvement in Table 1 (SISG(cjh3) overall .395 vs SISG(cj)‡ .451). Section 3.2.1 defines the metric as the average cosine distance 1−cos(v_b−v_a+v_c, v_d) over quadruples, with 'lower is better.' This is not the standard analogy evaluation used by the baseline they build on, Park et al. (2018), which is rank-based accuracy: is d the nearest neighbor among the vocabulary? Average distance to the true d can decrease even when d is rarely the nearest neighbor, if vectors cluster; conversely a model with higher average distance can have higher top-1 accuracy. The paper reports no rank accuracy, no error bars, and no count of solved quadruples. This matters because the other results are negative or marginal: WS353 similarity degrades (SISG(cj) Spearman .671 vs SISG(cjh3) .633), NSMC sentiment F1 drops (83.50 to 81.75), and headline BLEU-1 improves only from 31.78 to 32.03. The analogy score is thus the load-bearing evidence for the abstract's claim. I also flag the paper's own Section 3.3.2 limitation admitting Hanja-tagger errors; it is real but secondary, because even perfect Hanja annotations would not rescue the central claim if the headline metric does not measure analogical reasoning.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hanja-level SISG, an extension of Korean subword skip-gram models that adds Hanja n-gram vectors to the scoring function and initializes those vectors with pre-trained Chinese character embeddings. The authors evaluate the resulting embeddings on Korean word analogy and similarity tests and on two downstream tasks, including a newly collected Korean news headline generation dataset. They report improvements on the word analogy test and small BLEU gains on headline generation, while similarity correlations and NSMC sentiment F1 are worse than the jamo-level baseline. The paper acknowledges that the automatic Hanja tagger introduces errors and that the approach depends on an external annotator.","tokens_in":8940,"tokens_out":5599,"duration_ms":59563,"significance":"If the central claim were established, the paper would offer a simple and linguistically motivated method for injecting etymological knowledge into Korean word embeddings, together with a reusable new headline-generation benchmark and publicly available code and models. The cross-lingual initialization idea is also potentially interesting for other scripts with historical character relations. However, the evidence as presented is not yet convincing: the main positive result rests on a nonstandard analogy metric, the downstream gains are small and without significance testing, and the similarity and sentiment results move in the opposite direction. The paper's own limitation statement about Hanja-tagger reliability is honest but only partially mitigates these concerns.","major_comments":[{"comment":"The word analogy evaluation is reported as mean cosine distance 1−cos(v_a+v_b−v_c, v_d), with lower values claimed to be better, but the baseline rows marked † are taken from Park et al. (2018), whose standard protocol is rank-based analogy accuracy (higher is better). Comparing SISG(cjh3)'s 0.395 with SISG(cj)†'s 0.432 as an improvement is therefore comparing two different quantities. Even setting aside the † rows, mean distance to the correct answer is not a measure of whether the model actually solves analogies: it can improve by uniformly shrinking distances without increasing the frequency with which the correct word is the nearest neighbor. Please report standard top-1/top-k rank accuracy with the query words properly excluded, and provide error bars or a significance test. This is load-bearing because the similarity, sentiment, and BLEU results in Tables 1–3 are otherwise mixed or negative.","section":"§3.2.1 and Table 1"},{"comment":"The headline generation gains are very small and are not shown to be statistically meaningful. BLEU-1 increases from 31.78 to 32.03, BLEU-2 from 11.17 to 11.25, and BLEU-3 from 4.80 to 4.83, while the reported per-word perplexity is essentially unchanged (3.938 vs 3.941, where lower is better). No variance across random seeds or significance tests are reported for this task. Given that BLEU differences of this size are often within run-to-run noise, the downstream effectiveness claim is not established by the current evidence.","section":"§3.3.1 and Table 2"},{"comment":"The negative results are acknowledged but not adequately accounted for. WS353 Spearman correlation drops from 0.671 (SISG(cj)) to 0.633 (SISG(cjh3)), and NSMC F1 drops from 83.50 to 81.75. The paper attributes the sentiment drop to Hanja-tagger errors, but no tagger error analysis is provided; moreover, the same automatic tagger is used for all datasets, including the positive analogy and headline results. The authors should either quantify tagger accuracy on the relevant text types or demonstrate that the positive results are robust to annotation noise, rather than treating the tagger limitation as confined to the sentiment task.","section":"§3.2.2, §3.3.2, and Tables 1, 3"},{"comment":"The evidence for the cross-lingual transfer contribution is thin. The only comparison supporting it is SISG(cjhr) versus SISG(cjh3): on the analogy metric the pretrained initialization improves by 0.014 (0.409 vs 0.395), but on the similarity test the randomly initialized version is better (Spearman 0.647 vs 0.633). No significance testing is provided. Since the abstract explicitly credits cross-lingual transfer as part of the method, this claim needs stronger support, ideally with the standard analogy accuracy metric and multiple runs.","section":"§2.4 and Table 1"}],"minor_comments":[{"comment":"The column header 'Pr.' is used without explanation; it should be written as 'Pearson' and 'Spearman' for clarity.","section":"Table 1"},{"comment":"There is a typo in the notation 'SISG(cjh))' with an extra closing parenthesis; it should read 'SISG(cjh)'.","section":"§3.1.2"},{"comment":"The phrase '20m bucket size' is ambiguous; if it means 20 million, it should be written as '20M' or '20,000,000'.","section":"§A.1"},{"comment":"The sentence 'no equivalent exists for Korean' about ROUGE is inaccurate, since ROUGE is a language-independent n-gram overlap measure and can be applied to Korean with appropriate tokenization; please rephrase or justify.","section":"§3.3.1"}],"recommendation":"major_revision","confidential_remarks":"The crux is the analogy metric in Table 1: if the † rows are indeed accuracies from Park et al. (2018), the headline comparison is apples-to-oranges. This is fixable by rerunning the evaluation with rank-based accuracy on all models and reporting variance, so I would not reject the paper outright. The method is simple, the resources are useful, and the new headline dataset is a positive contribution, but the current manuscript does not yet support its central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The paper does one genuinely new thing: it adds Hanja n-grams to a Korean subword skip-gram and initializes them from Chinese character embeddings. That is a sensible, well-motivated idea, and the authors tested it cleanly, including an ablation with random initialization. Credit where due: the code is public, they re-ran their baselines with their own evaluation script rather than quoting published numbers, and they openly report the results that hurt them (sentiment gets worse, similarity gets worse). That is real evidence-gathering behavior.\n\nThe problem is the load-bearing positive result. The analogy improvement in Table 1 is reported as average cosine distance to the true answer (1−cos(v_b−v_a+v_c, v_d)), not as the standard rank-based accuracy. That metric can improve when the true answer moves marginally closer even if it is still rarely the nearest neighbor. The paper gives no rank accuracy, no error bars, and no count of solved quadruples. Since the other results are flat or negative—WS353 Spearman drops from .671 to .633, NSMC F1 drops from 83.50 to 81.75, and headline BLEU-1 goes from 31.78 to 32.03—the analogy number is doing most of the work for the abstract's claim. I don't think the stress-test is off base; this is the main soft spot.\n\nSecondary but real: the method depends on an automatic Hanja tagger, and the authors concede it makes errors, especially on colloquial text. That explains the sentiment result but it also means the trained model inherits whatever systematic noise the tagger has. Fair enough, but it's another reason to treat the mixed results as exactly what they are: mixed.\n\nThe Hanja n-gram size is a hand-chosen hyperparameter, but there's no sign it was tuned to the test sets, so I'm not worried about circularity.\n\nWho benefits: anyone working on Korean lexical semantics or on subword embeddings for morphologically rich languages. It is not a breakthrough and it is not a large evaluation. But the idea is clear, the write-up is honest, and the code is available. I would send it to a serious referee, with the explicit request that the authors report rank-based analogy accuracy and significance tests. If the analogy result survives that check, the paper earns its place. If it doesn't, the abstract needs major revision.\n\nRecommendation: peer-review it, but make the evaluation demand explicit.","headline":"Modest, honest Korean-embedding work whose main analogy claim rests on a weak metric; deserves review if the authors fix the evaluation.","tokens_in":9420,"tokens_out":2805,"would_cite":false,"duration_ms":28591,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hanja-level SISG adds Hanja n-gram vectors to the subword skip-gram scoring function, initializes them with Chinese character embeddings, and improves Korean word representations on analogy and headline generation.","keywords":["Korean word representations","Hanja","skip-gram","subword n-grams","cross-lingual transfer learning","word analogy","headline generation","Sino-Korean"],"falsifier":"A controlled run on a fixed corpus with gold-standard, human-verified Hanja annotations instead of automatic ones would settle the question: if the analogy and headline-generation gains vanish or shrink to noise when the annotations are correct, the reported improvements depend on tagger noise rather than on Hanja semantics.","tokens_in":8459,"feed_emoji":"🈶","tokens_out":4966,"duration_ms":51727,"temperature":0.7,"pith_summary":"This paper proposes that Korean word embeddings can be improved by explicitly adding Hanja, the Chinese-origin logograms used in written Korean, as an extra signal in the subword skip-gram objective. The central idea is that Hanja syllables carry more lexical meaning than Hangul phonograms, so their n-gram vectors should help the model separate the meanings of Sino-Korean words that look identical in Hangul. The paper also tests whether these Hanja vectors can be initialized with pre-trained Chinese character embeddings, which would mean that knowledge transfers between languages at the level of individual characters. If the approach is right, static Korean word representations become more useful for analogy tasks and for generating news headlines, which draw heavily on formal Sino-Korean vocabulary, while remaining competitive on colloquial sentiment analysis.","feed_headline":"Hanja n-grams sharpen Korean word embeddings","feed_subtitle":"Adding Chinese-initialized Hanja to skip-gram lifts analogy scores and news headline quality.","key_machinery":"The central mechanism is the extended scoring function $s^{(h)}(w,c) = s^{(j)}(w,c) + \\sum_{h \\in G^{(h)}_w} z_h v_c$, where $G^{(h)}_w$ is the set of Hanja n-grams extracted from the Hanja annotations of word $w$, $z_h$ is the learned vector for a Hanja n-gram, and $v_c$ is the context-word vector. This equation adds a semantically richer logogram channel to the existing character- and jamo-level channels, so that predicting a context word requires the model to align the meaning carried by Hanja characters with the surrounding discourse; the Hanja vectors are initialized with Chinese character embeddings so that cross-lingual knowledge enters at the start of training.","core_discovery":"The paper proposes Hanja-level SISG, a modification of the subword-information skip-gram model in which every Korean word is annotated with Hanja sequences, and the scoring function is extended to sum dot products between Hanja n-gram vectors and the context-word vector alongside the usual character and jamo n-grams. It reports that this extension improves performance on the Korean word analogy test over both plain skip-gram and the jamo-level baseline, and that initializing the Hanja n-gram vectors with Chinese character embeddings gives a further improvement over random initialization, demonstrating character-level cross-lingual transfer. On a newly collected Korean news headline generation task, the Hanja-initialized embeddings lead to higher BLEU scores and lower perplexity than prior embeddings, while on a movie-review sentiment task the method is competitive but not superior, which the paper attributes partly to errors in the automatic Hanja annotation.","pith_inferences":["Beyond the paper, the same character-level transfer idea could be tested for Japanese Kanji, Vietnamese Han-Nom, or other scripts that share etymological roots with Chinese characters; the paper only demonstrates it for Korean Hanja, so the broader cross-linguistic scope remains an open extension.","The observed trade-off between gains on analogy and losses on similarity suggests that Hanja injects categorical, relational structure rather than graded relatedness; a future model might add Hanja information only for words with high Sino-Korean confidence to avoid hurting similarity judgements.","An end-to-end model that learns Hanja annotation jointly with embeddings could remove the current dependence on an external tagger; the paper names this as future work, and one testable version would compare supervised tagger-free training against the pipelined approach on the same analogies and headline task."],"forward_implications":["Korean word analogy accuracy should improve whenever the training corpus contains Sino-Korean words with Hanja annotations, because the extra logogram channel disambiguates homophonous Hangul forms.","Character-level cross-lingual initialization from Chinese embeddings is a viable way to inject knowledge into Korean word representations without parallel text or shared vocabulary.","Downstream tasks that involve formal written Korean, such as news headline generation, stand to benefit more from Hanja-informed embeddings than tasks on colloquial or spoken-style text.","The approach is a drop-in extension of the subword skip-gram scoring function, so it can be combined with other subword vocabularies and training objectives without changing the underlying architecture."],"supporting_citations":[{"why":"Provides the base skip-gram objective and negative sampling that all scoring functions in this paper extend.","marker":"Mikolov et al., 2013"},{"why":"Introduces subword-information skip-gram with character n-gram vectors, the starting point of the SISG family compared here.","marker":"Bojanowski et al., 2017"},{"why":"Supplies the jamo-level SISG baseline, the Korean training corpus, the word analogy dataset, and the WS353 similarity dataset used throughout the evaluation.","marker":"Park et al., 2018"},{"why":"Provides the pre-trained Chinese character embeddings used to initialize Hanja n-gram vectors, enabling the character-level cross-lingual transfer.","marker":"Li et al., 2018"},{"why":"Defines the CNN/Daily Mail-style article and headline pairing that motivates the new Korean news headline generation task.","marker":"Hermann et al., 2015"},{"why":"Supplies the soft attention mechanism used in the headline generation encoder-decoder model that measures downstream effectiveness.","marker":"Bahdanau et al., 2014"},{"why":"Provides the BLEU metric used to evaluate the generated Korean news headlines.","marker":"Papineni et al., 2002"}],"fun_headline_variants":["Hanja n-grams with Chinese init sharpen Korean embeddings","Cross-lingual Hanja boosts Korean word representations","Korean embeddings improved by Hanja and Chinese transfer","Hanja-initialized skip-gram lifts Korean analogy and headline tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatic Hanja tagger used to annotate the entire training corpus produces labels accurate enough that the Hanja n-grams correspond to true Sino-Korean meaning; the paper itself reports seeing annotation errors that can hurt performance.","fun_headline_variants_meta":{"raw":{"variants":["Hanja n-grams with Chinese init sharpen Korean embeddings","Cross-lingual Hanja boosts Korean word representations","Korean embeddings improved by Hanja and Chinese transfer","Hanja-initialized skip-gram lifts Korean analogy and headline tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2772,"prompt_tokens":781,"completion_tokens":1991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":397,"completion_tokens_details":{"reasoning_tokens":1924}},"tokens_in":397,"tokens_out":1991,"duration_ms":14494,"temperature":1.0,"reasoning_tokens":1924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:15:59.816045+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled run on a fixed corpus with gold-standard, human-verified Hanja annotations instead of automatic ones would settle the question: if the analogy and headline-generation gains vanish or shrink to noise when the annotations are correct, the reported improvements depend on tagger noise rather than on Hanja semantics.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces subword-information skip-gram with character n-gram vectors, the starting point of the SISG family compared here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the jamo-level SISG baseline, the Korean training corpus, the word analogy dataset, and the WS353 similarity dataset used throughout the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BLEU metric used to evaluate the generated Korean news headlines."}],"review_version":1}