Contribution Weights combine attention, value magnitude, and directional alignment to measure token influence more faithfully than attention alone, and show attention sinks actively suppress information via a convex sink-rate to output-norm relationship.
Dynamic temperature knowledge distillation.arXiv preprint arXiv:2404.12711
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
years
2026 3verdicts
UNVERDICTED 3representative citing papers
CIST uses per-sample adaptive temperatures for both teacher and student in knowledge distillation to ensure consistent entropy in soft labels and reports gains on vision and language tasks.
TS-OPSD internalizes temperature via on-policy self-distillation to reheat entropy-collapsed RL policies in LLMs, providing stronger initialization for further training than continued RL or rollout temperature adjustment.
citing papers explorer
-
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
Contribution Weights combine attention, value magnitude, and directional alignment to measure token influence more faithfully than attention alone, and show attention sinks actively suppress information via a convex sink-rate to output-norm relationship.
-
Consistently Informative Soft-Label Temperature for Knowledge Distillation
CIST uses per-sample adaptive temperatures for both teacher and student in knowledge distillation to ensure consistent entropy in soft labels and reports gains on vision and language tasks.
-
Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning
TS-OPSD internalizes temperature via on-policy self-distillation to reheat entropy-collapsed RL policies in LLMs, providing stronger initialization for further training than continued RL or rollout temperature adjustment.