Pith. sign in

REVIEW 5 major objections 6 minor 22 references

SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sememe Entanglement Encoding compresses Transformer embedding layers by 10x to 80x while keeping translation BLEU within about 2.5 points on IWSLT17 ZH-EN.

desk verdict A plausible MorphTE extension with decent MT numbers, but the implementation ambiguity about whether the dense embedding table is materialized undermines the headline compression claim. read the letter →

arxiv 2412.12204 v1 pith:JPAWXZT3 submitted 2024-12-15 cs.LG cs.AIcs.CL

Jing Zhang , Shuzhen Sun , Peng Zhang , Guangxing Cao , Hui Gao , Xindian Ma , Nan Xu , Yuexian Hou This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords modelcompressionembeddinglayersememeHowNettensorproductquantumentanglementlow-rankapproximationknowledgedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a Transformer's token embedding layer, which can be 20% to 80% of the model parameters, can be heavily compressed by replacing each word vector with a sum of tensor products of much smaller vectors that encode the word's morphemes and sememes (the smallest units of meaning, drawn from the HowNet knowledge base). Instead of storing $|V| \times d$ numbers, the method stores only $(|S| + |M|) \times d^{1/o} \times m$ numbers, where $|S| + |M|$ is the number of sememe and morpheme types, $o$ is the number of factors in each tensor product, and $m$ is a small multiplicity. If true, this matters because embedding layers are a major memory bottleneck, and the semantic units give an interpretable, expert-guided meaning to the rank of the low-rank approximation. The reported translation results support the claim: at 10x embedding compression on IWSLT17 ZH-EN, BLEU drops about 0.4 points, and at 80x about 2.5 points, slightly better than the MorphTE baseline.

What carries the argument

The central object is the sememe-entangled embedding layer, in which each word is represented as a sum over $m$ multiplicities and $r$ ranks of a tensor product of $o$ low-dimensional vectors: $e = \sum_{i=1}^{m} \sum_{j=1}^{r} \otimes_{k=1}^{o} v_{jk,i}$. A sememe is the smallest indivisible unit of meaning, and a morpheme is the smallest structural unit; every word is assumed to contain one morpheme group plus $r-1$ senses, each sense containing $o$ sememes. The tensor-product combination, inspired by quantum entanglement, lets semantically related words share low-dimensional components while the final high-dimensional embedding is precomputed at initialization, so inference-time lookup cost is unchanged.

What would settle it

Inspect the actual BPE vocabularies used in the WMT17 and IWSLT17 translation runs and count how many of the 40K subword tokens have a sememe record in HowNet or a valid morpheme split; if a nontrivial fraction lacks such a decomposition, then the reported 10x to 80x parameter counts cannot be achieved without an unstated fallback, and the central compression claim would need to be revised.

Watch

Extended reading notes

Core claim

The central claim is that word embeddings can be reconstructed from low-dimensional sememe and morpheme vectors through a generalized quantum entanglement operation, compressing the embedding layer from $|V| \times d$ to $(|S| + |M|) \times d^{1/o} \times m$ parameters while maintaining stable downstream performance. On IWSLT17 ZH-EN, the method loses 0.4 BLEU at 10x compression and 2.54 BLEU at 80x compression, compared with a 3.2 BLEU loss for MorphTE; on Phi3-3B with 5x embedding compression, average ARC accuracy drops 0.6%. The authors interpret these results as evidence that the fine-grained semantic knowledge supplied by sememes compensates for the parameters removed by the low-rank entanglement construction.

Load-bearing premise

Every token in the vocabulary can be decomposed into exactly $r \times o$ basic units—one morpheme group and $r-1$ senses, each sense containing $o$ sememes—and HowNet contains sememe annotations for every one of those units, including BPE subwords that are not dictionary words.

Editorial extensions

If this is right

  • Embedding parameters for a Transformer can be cut by 10x to 80x using the parameter formula $(|S| + |M|) \times d^{1/o} \times m$, with no extra per-token computation at inference because the entangled vectors are precomputed during initialization.
  • At 10x compression on IWSLT17 ZH-EN the model stays within 0.4 BLEU of the full Transformer, and at 80x it stays within 2.54 BLEU, beating MorphTE's 3.2 BLEU drop at the same high compression.
  • Rank becomes a semantic concept rather than a black-box hyperparameter: adding more senses (increasing rank) does not increase the number of embedding parameters.
  • For a large language model, 5x embedding compression with the proposed multi-stage distillation costs only 0.6% average accuracy on ARC-c and ARC-e, suggesting the approach is feasible beyond training-from-scratch translation models.
  • The method couples low-rank embedding compression with a distillation recipe—MSE on embeddings and hidden states first, then logit KL divergence plus task cross-entropy—providing a concrete procedure for compressing pretrained Transformers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same entanglement construction could plausibly be applied to encoder-only and decoder-only models beyond translation, but the paper only demonstrates translation tasks plus two ARC benchmarks, so its generalization to other architectures and tasks is an untested extension.
  • Because rank does not affect parameter count, the method creates a cheap design lever: models could be given higher rank for semantically rich words at zero parameter cost; however, the paper's sensitivity analysis shows that order, not rank, is the setting that most strongly affects BLEU, which suggests the per-sememe vector dimension is the real capacity bottleneck.
  • A practical deployment extension would be to learn sememe assignments or build fallback decompositions for BPE subwords absent from HowNet, since the central assumption that every vocabulary token has the same $r \times o$ decomposition is not guaranteed for subword tokenizations.
  • If the method is correct, the reported 80x compression on a 42.9% embedding share implies an overall model compression of roughly 1.7x even before other modules are touched; the paper does not state this aggregate figure explicitly, but it follows from the reported proportions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Sememe Entanglement Encoding (SEE), a method for compressing the embedding layer of Transformer-based models. SEE represents each token through morphemes and sememes from HowNet, stores low-dimensional vectors for these basic units, and reconstructs the token embedding as a sum of tensor products of these vectors. The authors report experiments on WMT17 ZH-EN and IWSLT17 ZH-EN translation with 10x–80x embedding compression, comparisons to Word2Ket, MorphTE, matrix factorization, and tensor-train baselines, and an application to Phi-3 with a multi-stage distillation loss.

Significance. If the reported results hold, SEE would be a useful contribution to embedding compression: it reduces both vocabulary size and embedding dimension, makes the rank semantically interpretable, and leverages the external HowNet knowledge base. The paper also provides a concrete parameter-count formula and compares against several established baselines. However, the central claims of parameter and computation savings are currently obscured by an implementation ambiguity, and the evaluation has gaps (missing variance estimates, and a test set drawn from the training set for the Phi-3 experiments). These issues are substantial but addressable, so the work merits a major revision rather than rejection.

major comments (5)
  1. [Methodology, Eq. (4) and the note following it] The paper states that 'embedding calculations ... are completed during the model initialization' while also claiming that inference 'does not consume extra computational time.' This leaves a critical ambiguity: if the full |V|×d embedding table is materialized at initialization, the deployed model still stores |V|×d parameters and the compression ratios in Table 3 are not realized; if the table is not materialized, each forward lookup requires O(m·r·d) multiplications rather than a single gather, contradicting the claim of no extra computation. Please specify which implementation is used and report both the actual parameter count and the inference cost.
  2. [Methodology, 'The Representation of Basic Units'] The definition that 'a word has r × o basic units, specifically 1 × o morphemes, r − 1 senses, with each sense containing o sememe' conflicts with Table 1, where the number of senses varies across words (e.g., 'power' has five senses), and with BPE subwords, which are not dictionary words and may not have HowNet annotations. No fallback is given for tokens with fewer senses or for tokens absent from HowNet. Please state how such tokens are handled and whether the compression formula (|S|+|M|)×d^(1/o)×m still applies under that fallback.
  3. [Compression on Phi3, first paragraph] The paper states that 'due to the special nature of embedding compression ... we selected some data from the training set as a test set.' Because the test set overlaps with the fine-tuning training data, the reported ARC-c and ARC-e accuracy drops (0.9% and 0.2%) cannot be interpreted as evidence of maintained generalization. Please evaluate on a held-out or standard split, or explicitly reframe the Phi-3 result as a feasibility demonstration without claims of preserved benchmark performance.
  4. [Table 4 and 'Compression on Phi3'] Table 4 is not self-contained: the row labeled '22.7%' is unexplained, the column header 'model parameter' is ambiguous, and the text refers to 'Phi3-8B' while the section title says 'Phi3-3B' and the conclusion says '1B model.' Without a clear specification of which parameters are counted and which model is actually used, the large-model compression claim cannot be assessed. Please correct the model name and explain the reported fraction.
  5. [Main Results, Table 2] The differences between SEE and MorphTE at 10x and 20x compression on IWSLT are 0.59 and 0.13 BLEU, and the drop from the full model at 10x is 0.4 BLEU. These differences are small enough that they could be noise, yet no error bars, multiple seeds, or significance tests are reported. Please provide variance estimates or additional runs for the main compression comparison.
minor comments (6)
  1. [Title and throughout] The title contains a typo ('Transformer-bases' should be 'Transformer-based'), and the dataset name is written inconsistently as both 'IWLST' and 'IWSLT'.
  2. [Equation (3)] The notation is inconsistent: Eq. (3) uses v_{jk,i} while the surrounding text says 'vij,m'; please align the subscripts.
  3. [Abstract] The abstract claims that the approach compresses 'computational costs,' but this claim is not currently supported unless the materialization question in Major Comment 1 is resolved; please temper or substantiate it.
  4. [Effectiveness of Introducing Sememes] The improvements of 1.9% and 5.5% are relative percentages, while Table 5 shows absolute BLEU differences (0.29 and 0.80); please state that these are relative improvements to avoid apparent inconsistency.
  5. [Conclusion] The conclusion refers to 'IWSLT17 ZH-ZH' (should be ZH-EN) and to a '1B model' (elsewhere the paper says Phi3-3B or Phi3-8B); please correct these references.
  6. [Figure 5] The caption of Figure 5 does not explain the axes, units, or the meaning of the red/blue/black lines; please add a full caption.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: the Phi3 ARC 'test results' are evaluated on data drawn from the training set; the translation core remains externally benchmarked and non-circular.

  1. fitted input called prediction [Compression on Phi3, paragraph beginning 'Regarding training...']
    "only token embeddings that appeared in the fine-tuning training set can be learned during fine-tuning. This paper preliminarily demonstrates the feasibility of embedding SEE in large models, and thus selected some data from the training set as a test set."

    The Phi3 ARC-c/ARC-e numbers are introduced as 'test results' immediately after the paper states that the test set was selected from the fine-tuning training set. Because the model's parameters were fitted on that training set, the reported 0.9%/0.2% degradation is an in-sample measure, not an out-of-sample prediction. The claim that SEE stabilizes performance on a 3B model therefore uses the training data as its own test data, so this particular evaluation is circular by construction. The translation experiments remain external benchmarks and are not affected by this step.

full rationale

Score is 6, not 0, because one evaluation is circular: the Phi3 ARC experiment uses a test set drawn from the fine-tuning training set, so its 'test results' are in-sample. The translation core (WMT17, IWSLT17) is not circular: the embedding is a concrete low-rank reconstruction (Eq. 4) from external HowNet sememes and morphemes, and it is compared against external baselines without reusing fitted values as evidence. The initialization note that 'embedding calculations ... are completed during the model initialization' is a serious correctness risk for the claimed parameter savings if the dense table is materialized, but that is an implementation-validity problem rather than a circular derivation. The fixed r x o decomposition and missing HowNet coverage for some BPE tokens are applicability limitations, not circular steps.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends mainly on HowNet external knowledge and on three tuned hyperparameters (r, o, m). No new physical or conceptual entities are introduced; "entanglement" is used as a mathematical metaphor for tensor products, not a new object.

free parameters (4)
  • rank r = 5
    Hyperparameter chosen by sensitivity analysis in the Performance analysis section; rank controls the number of semantic/structural senses summed in Eq. 3.
  • order o = 3
    Hyperparameter chosen by sensitivity analysis; sets the number of tensor factors per sense and the low-dimensional vector dimension d^(1/o).
  • m, number of low-dimensional vector sets per basic unit = 18, 9, 4, 2 for 10x, 20x, 40x, 80x on MT; 10 for Phi3
    m is the main knob controlling the compression ratio in the parameter formula (|S|+|M|) * d^(1/o) * m.
  • distillation loss weights alpha, beta, gamma = not reported
    These weights in Eq. 9 are required to reproduce the training and are not given in the paper.
assumptions (4)
  • standard math Tensor product of vectors produces a vector in the product space, and sums of such products can represent word embeddings (Eq. 1 to Eq. 4).
    Unproved background from linear algebra used throughout the method.
  • domain assumption HowNet sememe annotations are a reliable semantic decomposition for the target vocabulary.
    The method relies on expert sememes as the rank structure, invoked in the Sememe section and Table 1; no evidence is given that annotations cover BPE subword tokens.
  • ad hoc to paper Every word has exactly r-1 senses each containing o sememes plus one morpheme group.
    Stated as "we define a word as having r x o basic units" in Methodology, but words have variable numbers of senses in HowNet; no fallback is specified.
  • ad hoc to paper Test data selected from the training set is acceptable for evaluating the compressed Phi3 model.
    The Phi3 section says "selected some data from the training set as a test set", which is an evaluation assumption that breaks standard generalization testing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression." pith.science (2026). https://pith.science/paper/JPAWXZT3

@misc{pith2026241212204,
  author       = {Pith},
  title        = {Pith review of: SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JPAWXZT3}},
  note         = {Machine review of arXiv:2412.12204}
}
read the original abstract

Transformer-based large language models exhibit groundbreaking capabilities, but their storage and computational costs are prohibitively high, limiting their application in resource-constrained scenarios. An effective approach is to eliminate redundant model parameters and computational costs while incorporating efficient expert-derived knowledge structures to achieve a balance between compression and performance. Therefore, we propose the \textit{Sememe Entanglement Encoding (SEE)} algorithm. Guided by expert prior knowledge, the model is compressed through the low-rank approximation idea. In Entanglement Embedding, basic semantic units such as sememes are represented as low-dimensional vectors, and then reconstructed into high-dimensional word embeddings through the combination of generalized quantum entanglement. We adapt the Sememe Entanglement Encoding algorithm to transformer-based models of different magnitudes. Experimental results indicate that our approach achieves stable performance while compressing model parameters and computational costs.

Figures

Figures reproduced from arXiv: 2412.12204 by the authors.

Figure 1
Figure 1. The diagram of Word2Ket and MorphTE namely morpheme groups, as the last rank index. From a compression perspective, the number of sememes in HowNet is only around 2000, and the number of mor￾phemes is much smaller than the normal word vocabu￾lary. Each sememe and morpheme are represented by low￾dimensional vectors. Compared to the original embedding layer parameters |V | × d, where |V | can reach around 800K, and d … view at source ↗
Figure 2
Figure 2. The Overall Method 3 for the entangled embedding. This algorithm can learn the commonalities and differences between different words from a linguistic structural perspective, while further reducing the size of the vocabulary. However, both of these algorithms inadequately consider the interaction of learning fine-grained semantic meanings, making it challenging to enhance the model’s ability to understand metaphors.… view at source ↗
Figure 4
Figure 4. Example of Sememe Decomposition annotated concepts represented by several hundred thousand Chinese and English words with these sememe (Qi et al. 2019). Therefore, compared to MorphTE, our vocabulary size has increased by only two thousand. With the achieved rank flexibility and We typically set the value of m to be small, we can achieve higher degrees of compression. Finally, we introduce abstract knowledge structu… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Hyperparameter Analysis distillation loss. In terms of experiments, we validate our method on the WMT17 ZH-EN and IWSLT17 ZH-ZH trans￾lation datasets based on the Transformer architecture. Our approach achieves substantial compression while maintain￾ing performance. Th…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 8 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    A.; Awan, A

    Abdin, M.; Jacobs, S. A.; Awan, A. A.; Aneja, J.; Awadallah, A.; Awadalla, H.; Bach, N.; Bahree, A.; Bakhtiari, A.; Behl, H.; et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    Cettolo, M.; Federico, M.; Bentivogli, L.; Niehues, J.; St \"u ker, S.; Sudoh, K.; Yoshino, K.; and Federmann, C. 2017. Overview of the iwslt 2017 evaluation campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation, 2--14

    work page 2017

  6. [6]

    Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  7. [7]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  8. [8]

    Gan, G.; Zhang, P.; Li, S.; Lu, X.; and Wang, B. 2022. Morphte: Injecting morphology in tensorized embeddings. Advances in Neural Information Processing Systems, 35: 33186--33200

    work page 2022

Show all 22 references
  1. [9]

    Li, S.; Zhang, P.; Gan, G.; Lv, X.; Wang, B.; Wei, J.; and Jiang, X. 2022. Hypoformer: Hybrid decomposition transformer for edge-friendly neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 7056--7068

    2022

  2. [10]

    Ma, X.; Zhang, P.; Zhang, S.; Duan, N.; Hou, Y.; Zhou, M.; and Song, D. 2019. A tensorized transformer for language modeling. Advances in neural information processing systems, 32

    2019

  3. [11]

    Mnih, A.; and Salakhutdinov, R. R. 2007. Probabilistic matrix factorization. Advances in neural information processing systems, 20

    2007

  4. [12]

    Myrvold, W. C. 2011. Nonseparability, classical, and quantum. The British Journal for the Philosophy of Science

    2011

  5. [13]

    Oseledets, I. V. 2011. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5): 2295--2317

    2011

  6. [14]

    Panahi, A.; Saeedi, S.; and Arodz, T. 2019. word2ket: Space-efficient word embeddings inspired by quantum entanglement. arXiv preprint arXiv:1911.04975

    2019 arXiv

  7. [15]

    Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311--318

    2002

  8. [16]

    Qi, F.; Yang, C.; Liu, Z.; Dong, Q.; Sun, M.; and Dong, Z. 2019. Openhownet: An open sememe-based lexical knowledge base. arXiv preprint arXiv:1901.09957

    2019 arXiv

  9. [17]

    Sennrich, R.; Birch, A.; Currey, A.; Germann, U.; Haddow, B.; Heafield, K.; Barone, A. V. M.; and Williams, P. 2017. The University of Edinburgh's neural MT systems for WMT17. arXiv preprint arXiv:1708.00726

    2017 arXiv

  10. [18]

    Sennrich, R.; Haddow, B.; and Birch, A. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909

    2015 arXiv

  11. [19]

    Szalay, S.; Pfeffer, M.; Murg, V.; Barcza, G.; Verstraete, F.; Schneider, R.; and Legeza, \"O . 2015. Tensor product methods and entanglement optimization for ab initio quantum chemistry. International Journal of Quantum Chemistry, 115(19): 1342--1391

    2015

  12. [20]

    Thakker, U.; Beu, J.; Gope, D.; Dasika, G.; and Mattina, M. 2020. Rank and run-time aware compression of NLP applications. arXiv preprint arXiv:2010.03193

    2020 arXiv

  13. [21]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  14. [22]

    Wu, Z.; Liu, Z.; Lin, J.; Lin, Y.; and Han, S. 2020. Lite transformer with long-short range attention. arXiv preprint arXiv:2004.11886

    2020 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.