Pith. sign in

REVIEW 14 cited by

Representation Degeneration Problem in Training Natural Language Generation Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1907.12009 v1 pith:34ZZCEDE submitted 2019-07-28 cs.CL

Jun Gao , Di He , Xu Tan , Tao Qin , Liwei Wang , Tie-Yan Liu This is my paper · ORCID
classification cs.CL
keywords problemlanguagerepresentationtrainingdegenerationgenerationnaturalembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Token similarity in Post-Norm decoders is amplified by causal attention at initialization, and RMSNorm backward contraction prevents gradients from repairing the resulting collapse.

  2. SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation

    cs.IR 2026-07 conditional novelty 6.0 of 10

    SpecFormer is a spectral-aware Transformer that flattens the singular-value spectrum of embeddings to prevent embedding/attention collapse, outperforming baselines on CTR benchmarks and scaling with layer depth.

  3. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  4. NITP: Next Implicit Token Prediction for LLM Pre-training

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    NITP augments standard next-token prediction with implicit semantic prediction in representation space using shallow-layer self-supervision, reporting consistent downstream gains on 0.5B-9B models including 5.7% on MM...

  5. Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning

    cs.CL 2026-01 conditional novelty 6.0 of 10

    PALU shows that unlearning only needs local intervention—the first few tokens of the sensitive span and the top-k logits—not full-sequence, full-vocabulary suppression.

  6. Accurate and Efficient Multivariate Time Series Forecasting via Offline Clustering

    cs.LG 2025-05 conditional novelty 6.0 of 10

    FOCUS forecasts multivariate time series by learning segment prototypes offline and attending to these prototypes online, achieving linear complexity and claimed state-of-the-art accuracy on seven benchmark datasets.

  7. Better Embeddings with Coupled Adam

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Coupled Adam, which gives every embedding the same effective learning rate by averaging Adam's second moment over the vocabulary, largely removes the mean-shift that causes anisotropic embeddings and improves embeddin...

  8. Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation

    cs.IR 2024-12 conditional novelty 6.0 of 10

    SMORE improves multimodal recommendation by denoising and fusing visual and textual features with FFT-based spectral filters and graph learning.

  9. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

  10. ChaosProbe: A Neurochaotic Lens on Frozen Transformer Input-Embedding Spaces

    cs.LG 2026-08 conditional novelty 5.0 of 10

    In a four-model proof of concept, deterministic neurochaotic fingerprints of frozen transformer embedding tables place GPT-2/DistilGPT2 and BERT/RoBERTa as mutual nearest neighbors under Pearson, Spearman, and cosine ...

  11. SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Using multiple LLM-generated summaries of the same abstract as positive pairs trains scientific text embeddings that beat citation-trained baselines on retrieval and clustering, while the new benchmark shares its trai...

  12. Low-Perplexity LLM-Generated Sequences and Where To Find Them

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Only about 40% of low-perplexity 6-token spans generated by Pythia-6.9B can be exactly matched to The Pile, and the authors categorize matched and unmatched spans into four classes.

  13. Modality Alignment with Multi-scale Bilateral Attention for Multimodal Recommendation

    cs.IR 2025-09 conditional novelty 4.0 of 10

    MambaRec improves multimodal recommendation accuracy on Baby, Sports, and Clothing datasets through local dilated-attention alignment and global MMD/contrastive alignment.

  14. 2-Tier SimCSE: Elevating BERT for Robust Sentence Embeddings

    cs.CL 2025-01 conditional novelty 3.0 of 10

    Stacking unsupervised SimCSE before supervised SimCSE on minBERT yields a dev STS correlation of 0.811, a small gain over supervised SimCSE alone (0.806), but the evidence is single-run.

Pith tools