Pith. sign in

REVIEW 4 cited by

Lessons on Parameter Sharing across Layers in Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2104.06022 v4 pith:SMGHUSWE submitted 2021-04-13 cs.CL cs.LG

Sho Takase , Shun Kiyono This is my paper · ORCID
classification cs.CLcs.LG
keywords parameterproposedstrategiestransformerscomputationalcyclelayerlayers
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We propose a parameter sharing method for Transformers (Vaswani et al., 2017). The proposed approach relaxes a widely used technique, which shares parameters for one layer with all layers such as Universal Transformers (Dehghani et al., 2019), to increase the efficiency in the computational time. We propose three strategies: Sequence, Cycle, and Cycle (rev) to assign parameters to each layer. Experimental results show that the proposed strategies are efficient in the parameter size and computational time. Moreover, we indicate that the proposed strategies are also effective in the configuration where we use many training data such as the recent WMT competition.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 24 citations worldwide. Full citation record

  1. Scaling Latent Reasoning via Looped Language Models

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.

  2. FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A block-pruning and model-extension method for LLMs that replaces pruned blocks with weight-shared blocks plus low-rank adapters, reporting state-of-the-art recovery on several benchmarks.

  3. Reproducing Recurrent Transformers: The CoTFormer

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A reproduction of CoTFormer confirms its perplexity results, finds its adaptive-compute claims fragile, and shows looped computation benefits p-hop retrieval but not inductive counting.

  4. Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition

    cs.LG 2024-11 conditional novelty 5.0 of 10

    A shared low-rank basis plus sparse per-layer projections compresses transformer MLPs by up to 75% with small accuracy loss.

Pith tools