REVIEW 1 cited by
Transformer tricks: Precomputing the first layer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This micro-paper describes a trick to speed up inference of transformers with RoPE (such as LLaMA, Mistral, PaLM, and Gemma). For these models, a large portion of the first transformer layer can be precomputed, which results in slightly lower latency and lower cost-per-token. Because this trick optimizes only one layer, the relative savings depend on the total number of layers. For example, the maximum savings for a model with only 4 layers (such as Whisper tiny) is limited to 25%, while a 32-layer model is limited to 3% savings. See https://github.com/OpenMachine-ai/transformer-tricks for code and more transformer tricks.
Forward citations
Cited by 1 Pith paper
-
APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
A bit-sliced Tensor Core scheme with a bipolar integer encoding and adaptive kernel autotuning reports 1.65-3.99x speedups for ultra-low-bit GPTQ-quantized LLM inference on three NVIDIA GPUs.
Discussion (0). Sign in to comment.