REVIEW 5 cited by
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Deploying large language models (LLMs) of several billion parameters can be impractical in most industrial use cases due to constraints such as cost, latency limitations, and hardware accessibility. Knowledge distillation (KD) offers a solution by compressing knowledge from resource-intensive large models to smaller ones. Various strategies exist, some relying on the text generated by the teacher model and optionally utilizing his logits to enhance learning. However, these methods based on logits often require both teacher and student models to share the same tokenizer, limiting their applicability across different LLM families. In this paper, we introduce Universal Logit Distillation (ULD) loss, grounded in optimal transport, to address this limitation. Our experimental results demonstrate the effectiveness of ULD loss in enabling distillation across models with different architectures and tokenizers, paving the way to a more widespread use of distillation techniques.
Forward citations
Cited by 5 Pith papers
-
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
A federated fine-tuning framework that compresses foundation models on clients via SVD, aggregates adapters within groups and full-rank reconstructions across groups, then distills the result back into the full server model.
-
The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models
Response-based knowledge distillation makes small models less likely to pick stereotypes on unambiguous questions but destroys which specific ambiguous questions they abstain on, a per-item harm aggregate bias metrics hide.
-
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.
-
Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit
OMP sparse coding of donor token embeddings, with coefficients transferred to the base embedding space, preserves LLM performance after tokenizer replacement better than published zero-shot baselines, though simple he...
-
Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models
An affine map trained on The Pile transfers steering vectors from Gemma-2B to Gemma-9B and reproduces much of the large model's native steering behavior.
Discussion (0). Sign in to comment.