Pith. sign in

REVIEW 2 major objections 6 minor 17 references

CA-EHN: Commonsense Analogy from E-HowNet

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CA-EHN, a 90,505-item Chinese commonsense word analogy benchmark, is built by comparing definitions in the E-HowNet ontology, and retrofitting commonsense knowledge into embeddings lifts its accuracy by up to 179 percent.

desk verdict The CA-EHN dataset is a genuinely useful new Chinese analogy resource, but the headline 'great indicator' claim rests on a confounded retrofit experiment and needs independent validation. read the letter →

arxiv 1908.07218 v5 pith:65QLERKD submitted 2019-08-20 cs.CL

Peng-Hsuan Li , Tsan-Yu Yang , Wei-Yun Ma This is my paper · ORCID
classification cs.CL
keywords commonsensereasoningwordanalogyembeddingsE-HowNetontologyChinesebenchmarklexicaldatabase
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that commonsense knowledge can be reduced to word-level analogical reasoning and measured by a dedicated benchmark. It introduces CA-EHN, which it calls the first commonsense word analogy dataset, with 90,505 analogies over 5,656 Chinese words and 763 relations, extracted by comparing structured definitions in the E-HowNet ontology. Existing Chinese analogy datasets are mostly morphological or named-entity relations; CA-EHN instead draws on ordinary lexical knowledge such as sound-origin, organ-disabled, and juvenile-adult relations. The paper's central evidence is that retrofitting E-HowNet taxonomy into word embeddings raises CA-EHN accuracy by up to 179 percent while improving or hardly changing other benchmarks, which it interprets as showing that the dataset tests how well embeddings store commonsense knowledge.

What carries the argument

The central object is the E-HowNet definition graph: each word sense is parsed into a directed graph whose nodes are words, concepts, or functions and whose edges are attribute modifiers, so a sense like 'laboratory' becomes an InstitutePlace concept modified by a telic relation to research or experiment. The extraction mechanism is graph comparison: two definition graphs that differ in exactly one concept node yield the analogy w1:c1 = w2:c2. Synonym expansion then turns the concept analogy into word-level questions, with the right-hand answer kept as a synset so that the standard vector arithmetic v1 + v2 - v3 can be evaluated as a membership test.

What would settle it

Take a random sample of CA-EHN analogies, remove the ontology definitions, and have native speakers judge whether the relation between the two word pairs is a meaningful commonsense analogy; if judged correctness falls far below the paper's reported annotation agreement for concept analogies, the extraction criterion is admitting spurious relations. A second check is to build a control benchmark from random single-node perturbations of the same definition graphs and see whether retrofitting improves accuracy on the control as much as it does on CA-EHN.

Watch

Extended reading notes

Core claim

The paper's central claim is that a commonsense analogy can be read off an ontology by comparing definition graphs: two word senses form an analogy when their parsed definitions differ in exactly one concept node. Using E-HowNet's structured sense definitions and taxonomy, the authors extract concept analogies, expand the left concept into synonymous words to form analogy questions, and keep the right expansion as an accepted synset so embeddings can be scored by whether their nearest vector falls in that synset. After filtering for concrete concepts and frequent words and after linguist annotation of concept analogies and synsets, the result is a benchmark of 90,505 analogies covering 5,656 words and 763 relations. The paper further claims this benchmark is a useful indicator of commonsense knowledge in embeddings because injecting commonsense ontology structure via retrofitting produces much larger gains on CA-EHN than on existing analogy sets.

Load-bearing premise

One load-bearing premise is that E-HowNet's structured definitions are accurate and complete enough that two definition graphs differing in exactly one concept node correspond to a human-meaningful commonsense analogy; if the ontology is noisy or the one-node-difference rule admits coincidental pairs, the dataset's relation labels are not valid commonsense relations.

Editorial extensions

If this is right

  • CA-EHN can serve as an intrinsic evaluation for Chinese word embeddings that targets commonsense relations rather than morphology or named entities.
  • Because retrofitting E-HowNet taxonomy improves CA-EHN accuracy by up to 179 percent, the dataset can be used to test whether a representation method actually absorbs ontology-structured commonsense knowledge.
  • The 763 discovered relations offer a much finer inventory of commonsense relation types than the dozens of predefined relations in existing benchmarks.
  • Since E-HowNet senses carry English translations, the same analogy questions can be projected to English multi-word expressions, giving a path to cross-lingual commonsense analogy evaluation.
  • The extraction procedure, if applied to other ontologies with structured definitions, would produce comparable analogy benchmarks in other languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the single-node-difference criterion could be run in reverse: automatically propose new commonsense relations by clustering analogies whose differing concept pairs share a taxonomy path, turning CA-EHN into a relation-induction resource.
  • The benchmark's sensitivity to retrofitting suggests CA-EHN could be used diagnostically, e.g., comparing how much ontology knowledge different training objectives preserve, rather than only as a ranking of final accuracies.
  • A testable extension is to build a control benchmark by randomly perturbing one concept node in the same definition graphs; if retrofitting improves accuracy on that control as much as on CA-EHN, then the benchmark is picking up generic graph-structure effects rather than commonsense specifically.
  • The paper's coverage of Chinese common words hints that a frequency-matched analogical benchmark for other languages could be derived from equivalent structured lexicons, allowing direct cross-lingual comparison of commonsense embedding quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces CA-EHN, a Chinese word analogy dataset automatically extracted from the E-HowNet ontology by comparing structured word-sense definition graphs that differ in exactly one concept node. The extraction is filtered to concrete concepts and common words, and the resulting concept analogies are partially validated by linguists (κ=0.76 on 1,000 items), yielding 90,505 word-level analogies covering 5,656 words and 763 derived relation types. The paper then evaluates distributed word embeddings on this benchmark and reports that retrofitting with the E-HowNet taxonomy improves CA-EHN accuracy by up to 179%, while retrofitting with HIT-Thesaurus improves it by up to 88%. On this basis it claims that CA-EHN is a 'great indicator' of how well word representations embed commonsense knowledge.

Significance. If validated, CA-EHN would be a valuable resource: it is substantially larger and relationally richer than existing Chinese analogy benchmarks, it is derived from a structured ontology rather than hand-written templates, and the dataset is publicly released. The extraction pipeline is described concretely with transparent filters, the inter-annotator agreement is reported, and the benchmark is evaluated across multiple embedding configurations. However, the central claim that CA-EHN specifically measures commonsense embedding quality is currently supported by a retrofit experiment that shares its source ontology with the benchmark, so the significance hinges on whether additional source-independent validation confirms the claim.

major comments (2)
  1. [Section 4.1 and Section 4.3] The central claim that CA-EHN is 'a great indicator of how well word representations embed commonsense knowledge' is supported mainly by the retrofit experiment, but that experiment is confounded by source overlap. CA-EHN analogies are extracted from E-HowNet definition graphs (Section 4.1), while the retrofitting lexicon is the E-HowNet taxonomy (Section 3.2). Because the definition graphs and the taxonomy are two views of the same ontology, the injected knowledge directly moves the representations of words and concepts that appear in the analogy questions; the up-to-179% improvement may therefore reflect the embedding's exposure to E-HowNet structure rather than a general sensitivity to commonsense knowledge. The HIT-Thesaurus result (up to 88%) is a useful partial control, but it does not rule out sharing of words or concepts with CA-EHN. I ask for a source-independent validation: for example, retrofit with a lexicon that has no ontology overlap with E-HowNet, or evaluate on a split of CA-EHN from which all words and concepts appearing in the retrofitting resource have been removed, and report the gains on that split. Without such a control, the differential gain cannot be interpreted as evidence for the benchmark's construct validity.
  2. [Section 4.1 and Section 4.3] The extraction criterion that two definition graphs differing in exactly one concept node form an analogy is a syntactic rule; its validity as a proxy for human commonsense analogies is only partially established. The paper reports inter-annotator agreement of κ=0.76 on 1,000 of 36,100 concept analogies, but it does not state how the 25,010 surviving concept analogies were selected from the annotations (e.g., majority vote, unanimous agreement, or some other threshold), nor the distribution of labels. It also does not explicitly state whether the final 90,505 word-level analogies were re-validated after left/right expansion, or only the concept-level analogies. Please provide the annotation guideline, label distribution, selection rule, and a clear statement of which units (concept analogies vs. word analogies) were validated. This is necessary for users to judge the reliability of the dataset's relation labels.
minor comments (6)
  1. [Section 2] The text says 'the later is translated from English'; 'later' should be 'latter'.
  2. [Section 5.2 and Table 3] The '763 relations' statistic is computed by grouping word pairs into equivalence classes, not by annotating each analogy with a relation label; the manuscript should state this explicitly in the abstract and in the Table 3 caption to avoid implying that the dataset contains 763 labelled relation types.
  3. [Section 4.3] In the synset annotation example, the paper says 'the annotator' judged camellia, lavender, and iris as hyponyms; please specify how many annotators performed this step, whether the synset refinement was applied to all synsets or a sample, and report the number of synsets that were refined.
  4. [Section 5.3, Table 4] The coverage row for CA-EHN reports 90,505 for every embedding; please clarify whether this means all CA-EHN analogy questions were fully covered by all embedding vocabularies, and if so, how this follows from the construction pipeline.
  5. [Section 5.2] The example 'Beijin : Peking University' contains a typo: 'Beijin' should be 'Beijing'.
  6. [Section 5.4] The phrase 'infused some structure knowledge' might be better expressed as 'retrofitted with structured knowledge' for consistency with the retrofitting terminology used elsewhere.

Circularity Check

1 steps flagged · score 6.0 of 10

CA-EHN's 'great indicator' claim rests on a closed loop: the benchmark is extracted from E-HowNet, and its main validation injects E-HowNet, so the 179% gain is partly forced by source overlap.

  1. self definitional [Sections 4.1 (Analogy Extraction) and 5.4 (Commonsense Benchmarking, Tables 5 and 6)]
    "We extract word analogies with rich coverage of words and commonsense relations by comparing word sense definitions (Section 3.1.). ... The two definition graphs are compared to see if they differ only in one concept node. ... Performance increases by up to 179% and 88% by infusing E-HowNet taxonomy and HIT-Thesaurus respectively. This shows that CA-EHN is a great indicator of how well word representations embed commonsense knowledge."

    CA-EHN analogies are constructed from E-HowNet word sense definitions, whose concept nodes are taken from the E-HowNet taxonomy. The validation experiment then injects the same E-HowNet taxonomy into embeddings via retrofitting. The benchmark and the intervention therefore share the same ontology by construction: the words and concepts that determine CA-EHN analogy questions are exactly the items whose vectors are moved by the E-HowNet retrofit. Consequently, the up-to-179% accuracy gain on CA-EHN is partly a mechanical effect of source overlap rather than an independent demonstration that CA-EHN captures general commonsense knowledge.

full rationale

The dataset construction itself is not circular: analogies are extracted from E-HowNet definitions by a stated graph-comparison rule, then filtered by frequency and checked by linguists. The circularity enters at the validation step. Section 5.4 tests the paper's central claim by retrofitting embeddings with the E-HowNet taxonomy. Since CA-EHN was extracted from E-HowNet word sense definitions, the benchmark and the injected resource share the same ontology. The large CA-EHN gain is therefore partly expected: the retrofit directly moves vectors of words and concepts that define the benchmark questions. The HIT-Thesaurus result (+88%) provides some evidence that the benchmark responds to lexical knowledge generally, and the linguist annotation supports the accuracy of the extracted analogies, so the paper is not wholly circular. However, no source-independent control or overlap analysis is provided, so the headline conclusion that CA-EHN is a great indicator of commonsense embedding quality is not independently established. The shared-source confound is a concrete reduction of the claimed evidence, not a merely stylistic or citation-based concern.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities, particles, or forces. Its assumptions are about the reliability of the E-HowNet ontology, the validity of the graph-difference analogy criterion, the applicability of the vector-offset evaluation, and the representativeness of the annotation sample.

free parameters (1)
  • minimum word frequency in ASBC = 5 occurrences
    Chosen as a heuristic filter to remove uncommon words; the value is arbitrary and affects dataset coverage, but it is not fitted to optimize a target metric.
assumptions (4)
  • domain assumption E-HowNet's structured word sense definitions accurately encode commonsense knowledge.
    All analogies are extracted from these definitions; if E-HowNet definitions are noisy or wrong, the dataset inherits those errors.
  • ad hoc to paper Two definition graphs that differ in exactly one concept node yield a valid analogy.
    This is the core extraction criterion in Section 4.1, Graph Comparison, and is not derived from an independent linguistic theory.
  • domain assumption Word embedding analogy evaluation via vector offset v3+v2-v1 is a valid measure of relational knowledge.
    This standard intrinsic evaluation assumption is adopted in Section 4.2; if the offset method does not reflect the underlying relation, the benchmark scores are not meaningful.
  • domain assumption Linguist annotation of a subset of concept analogies generalizes to the full dataset.
    The paper reports high agreement on 1,000 quadruple-annotated concept analogies, but the remaining 24,010 concept analogies may rely on fewer annotators, so annotation quality is only partially verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CA-EHN: Commonsense Analogy from E-HowNet." pith.science (2026). https://pith.science/paper/65QLERKD

@misc{pith2026190807218,
  author       = {Pith},
  title        = {Pith review of: CA-EHN: Commonsense Analogy from E-HowNet},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/65QLERKD}},
  note         = {Machine review of arXiv:1908.07218}
}
read the original abstract

Embedding commonsense knowledge is crucial for end-to-end models to generalize inference beyond training corpora. However, existing word analogy datasets have tended to be handcrafted, involving permutations of hundreds of words with only dozens of pre-defined relations, mostly morphological relations and named entities. In this work, we model commonsense knowledge down to word-level analogical reasoning by leveraging E-HowNet, an ontology that annotates 88K Chinese words with their structured sense definitions and English translations. We present CA-EHN, the first commonsense word analogy dataset containing 90,505 analogies covering 5,656 words and 763 relations. Experiments show that CA-EHN stands out as a great indicator of how well word representations embed commonsense knowledge. The dataset is publicly available at https://github.com/ckiplab/CA-EHN.

Figures

Figures reproduced from arXiv: 1908.07218 by the authors.

Figure 1
Figure 1. shows the partially expanded taxonomy. The words beside each node have senses defined trivially by that concept. For example, one definition of 東西 is sim￾ply {thing|萬物}. In the experiments, we infuse E-HowNet taxonomy to distributed word representations and analyze performance changes across word analogy benchmarks (Section 5.4.). 4. Commonsense Analogy We extract word analogies with rich coverage of words and commo… view at source ↗
Figure 2
Figure 2. Commonsense analogy extraction. (a) 實驗室 (laboratory). (b) 外公 (maternal grandpa) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Sample parsed definition graphs. and 馬|horse, we extract the following concept analogy – 良材:wood|木=駿馬:馬|horse. Left Expansion The left concept in the concept analogy is expanded into synonym words, i.e., words that have one sense defined trivially by it. For example, there is only one word 木頭 defined as {wood|木}. Thus after expansion, there is still only one analogy: 良材:木頭=駿馬:馬|horse. Most of the time, this step yie… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Some definition graphs that leads to the sound-origin relation. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Some definition graphs that leads to the organ-disabled relation. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Some definition graphs that leads to the painter-instrument relation. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Some definition graphs that leads to the doctor-patient relation. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages

  1. [1]

    R., Angeli, G., Potts, C., and Manning, C

    Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

    work page 2015

  2. [2]

    and Ma, W.-Y

    Chen, C.-Y. and Ma, W.-Y. (2018). Word embedding evaluation datasets and wikipedia title embedding for chinese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)

    work page 2018

  3. [3]

    Chen, K.-J., Huang, S.-L., Shih, Y.-Y., and Chen, Y.-J. (2005). Extended- H ow N et: A representational framework for concepts. In Proceedings of O nto L ex 2005 - Ontologies and Lexical Resources

    work page 2005

  4. [4]

    and Dong, Q

    Dong, Z. and Dong, Q. (2003). Hownet - a hybrid language and knowledge resource. In Proceedings of the International Conference on Natural Language Processing and Knowledge Engineering

    work page 2003

  5. [5]

    and Chen, K

    Graff, D. and Chen, K. (2003). Chinese gigaword ldc2003t09. Linguistic Data Consortium

    work page 2003

  6. [6]

    Li, S., Zhao, Z., Hu, R., Li, W., Liu, T., and Du, X. (2018). Analogical reasoning on chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

    work page 2018

  7. [7]

    and Shih, Y.-Y

    Ma, W.-Y. and Shih, Y.-Y. (2018). Extended hownet 2.0 -- an entity-relation common-sense representation model. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)

    work page 2018

  8. [8]

    Ma, W.-Y., Hsieh, Y.-M., Yang, C.-H., and Chen, K.-J. (2001). Design of management system for chinese corpus construction. In Proceedings of Research on Computational Linguistics Conference XIV

    work page 2001

Show all 17 references
  1. [9]

    Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

    2013 arXiv

  2. [10]

    Miller, G. A. (1995). Wordnet: a lexical database for english. Communications of the ACM

    1995

  3. [11]

    and Havasi, C

    Speer, R. and Havasi, C. (2012). Representing general relational knowledge in conceptnet 5. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012)

    2012

  4. [12]

    Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V...

    2018

  5. [13]

    Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

    2017

  6. [14]

    K., Dyer, C., Hovy, E., and Smith, N

    Faruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E., and Smith, N. A. (2015). Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

    2015

  7. [15]

    S., and Dean, J

    Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26

    2013

  8. [16]

    Pennington, J., Socher, R., and Manning, C. (2014). GloVe : Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP )

    2014

  9. [17]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.