Architectural details, especially QK-normalization, GQA, sliding windows, and 4K pretraining context, compound to degrade downstream long-context performance in a controlled 26-model pool.
Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Architectural details, especially QK-normalization, GQA, sliding windows, and 4K pretraining context, compound to degrade downstream long-context performance in a controlled 26-model pool.