L1 Augmented Attention as an Improved Vector Similarity Metric

Kurt Godden

REVIEW 4 major objections 6 minor 7 references

L1 Augmented Attention as an Improved Vector Similarity Metric

T0 review · 4 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read L1-augmented attention reduces transformer perplexity by up to 14.5 percent.

desk verdict A plausible attention tweak with a clear write-up, but the 14.5% headline is a best-of-N test-selected number from single runs, so the real effect size is unquantified. read the letter →

arxiv 2607.18027 v1 pith:MY5IOVFV submitted 2026-07-20 cs.LG cs.CL

Kurt Godden This is my paper · ORCID

classification cs.LGcs.CL

keywords attentionsimilaritymetricL1normtransformerlanguagemodelingperplexitykernelmethodsvectorgeometry

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the scaled dot product is an incomplete similarity metric for transformer attention because it hides the separate contributions of vector direction and magnitude. It proposes subtracting a learned, head-specific L1 distance between queries and keys from the dot-product score, so the attention score rewards directional alignment while explicitly penalizing coordinate-wise deviations. On WikiText-2 with a compact two-block transformer, the modified attention reportedly lowers test perplexity by up to 14.5% compared with the original baseline and beats an RBF-L2 kernel. The paper also shows that projecting queries and keys into low-dimensional subspaces both cuts the cost of the L1 computation and, with learned per-head weights, can improve perplexity further. The wider point is that enriching attention with L1 geometry is a direct way to improve similarity computation without abandoning the transformer's parallel structure.

What carries the argument

The central object is the L1-augmented attention score, softmax(QK^T/sqrt(d_k) - lambda_h * L1(Q,K)), where L1(Q,K) is the matrix of per-coordinate absolute differences between queries and keys and lambda_h is a learned, head-specific weight initialized to zero. The L1 term supplies the magnitude and coordinate-wise information that the dot product masks, while the dot product preserves directional signal; the learned lambda lets each head decide how much geometric penalty to apply. To make the penalty affordable, the model projects Q and K into low-dimensional subspaces using a shared projection matrix with disjoint per-head slices, and those projections specialize during training to preser

What would settle it

Run multiple seeds with a fixed projection dimension (for example, 8) using the same architecture and WikiText-2 setup, and compare seed-averaged test perplexity against an equally tuned baseline; if the average advantage is not reliably positive, the central claim is refuted. A second check: pre-register one projection dimension before seeing test results and see whether the gain persists.

Watch

Extended reading notes

Core claim

The central claim is that attention scores should combine two complementary geometries: the dot product, which is sensitive to directional alignment, and an L1 penalty, which directly measures coordinate-wise distance. Concretely, the paper studies scores of the form softmax(QK^T/sqrt(d_k) - lambda_h * ||q_i - k_j||_1), with per-head learned lambda initialized at zero and queries and keys projected to low dimensions. On a small transformer trained on WikiText-2, this L1-augmented attention achieved the lowest perplexity (159.9) among the tested variants, a 14.5% reduction from the original transformer baseline (187.1), and outperformed an RBF-L2 kernel (178.1). The paper further reports that

Load-bearing premise

The 14.5% reduction is load-bearing but rests on selecting the best projection dimensionality from test-set perplexity across many variants; the paper reports that the best dimension changed with the seed and reports no error bars or seed averages.

Editorial extensions

If this is right

If the result holds, any transformer-based language model can replace its similarity score with this hybrid without changing the rest of the architecture.
Because the L1 term is embarrassingly parallel, the perplexity gain need not cost extra wall-clock time when computed alongside the dot product.
Learned per-head lambda provides an interpretable knob: heads can choose between direction-focused and geometry-focused similarity, which may aid analysis of attention.
The projection-dimension sweet spot suggests that L1-specialized low-dimensional query and key embeddings are themselves a useful learned representation, analogous to learned relative-position biases.
The hybrid L18/RBF result indicates a resource-accuracy tradeoff: using L1 only where norms matter most recovers about half the gain at about half the cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The geometric argument for why dot product fails as a similarity metric is dataset-independent; if correct, the same L1 augmentation should transfer to encoder tasks such as classification or retrieval, though the paper only tests autoregressive language modeling.
The reported best projection dimension changed with the random seed, so a practical deployment may need per-run validation or ensembling rather than a single fixed dimension; the paper acknowledges this but does not quantify it.
The concluding Laplacian-kernel formulation, exp(-lambda * ||u(q)-v(k)||_1), suggests a path toward linear-complexity attention that preserves the L1 geometry, but that extension is not implemented here.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The idea is simple and worth a look: subtract a learned, head-specific L1 penalty between queries and keys from the scaled dot-product, with a low-dimensional projection to keep the L1 term cheap. I don't see this exact combination in the cited work, and the motivation is legitimate — dot product does conflate direction and magnitude. The toy examples make the point well, and the norm-variance/lambda analysis is a nice descriptive addition.

What the paper does well is that it is honest about scope: it says upfront this is a preliminary report, and the limitations section admits a single dataset and a tiny model. The implementation is described clearly enough to reproduce with moderate effort, though no code is provided.

The soft spots are real, and the stress-test note is right on target. The 14.5% headline is the best of nine projection dimensions, chosen after looking at test perplexity. The paper itself says a different seed made L114 the best, so the number is an upper bound from a multiple-comparisons procedure, not an estimate of the method's expected gain. Every row in Table 1 is a single run, so we have no variance information at all. The baseline is also a reduced architecture (two blocks, three heads, dim 192), not the original Vaswani setup, so the comparison is against a modified reference point. These are fixable issues, but they mean the central claim is unquantified as it stands.

One minor point: the paper's own example in Section 6 shows a failure case where the L1-augmented score gives equal values for a perfect match and a less perfect one. The authors mention it but don't discuss how often that happens or whether it matters. It's a small caveat, not a fatal one.

I'd send this to peer review. The idea is new enough, the writing is clear, and the empirical problems are the kind a referee can point to and the author can address with more runs and proper model selection. As is, I wouldn't cite it as evidence of a 14.5% gain, but I'd cite it as a proposed attention variant if I needed to reference the idea.

For a reading group, I'd bring it in maybe — the methodology lesson is useful.

Referee Report

4 major / 6 minor

Summary. The paper proposes L1-augmented attention, defined in Eq. (6) as softmax(QK^T/sqrt(d_k) - lambda_h * ||q_i - k_j||_1), with a learned per-head weight lambda_h and low-dimensional projections of queries and keys before computing the L1 distance. The motivation is that the dot product conflates directional alignment with vector magnitude, while the L1 term explicitly penalizes coordinate-wise deviations. On WikiText-2 with a compact two-block, three-head transformer (d_model=192), the authors report that their best projected L1 model (L18) achieves a 14.5% perplexity reduction over the Vaswani baseline (Table 1), and also outperforms an RBF-L2 kernel. They additionally analyze per-head lambda values and norm variance, claiming distinct geometric roles in lower and upper blocks. The paper closes with a limitations section acknowledging the single-dataset, small-model scope and a note in Section 1 describing the work as a preliminary report.

Significance. The idea is simple and the geometric intuition is appealing: augmenting a directional dot-product score with a coordinate-wise L1 penalty could plausibly help attention allocate weight more precisely. The toy examples in Section 3 illustrate the failure modes of the dot product and how the L1 term fixes them. If the reported improvement survives seed averaging and proper model selection, the contribution would be a useful, low-cost modification to attention. The per-head lambda analysis is also a nice interpretability direction. However, the current quantitative evidence is a single-run Table 1 with test-set-based selection of the projection dimension, and no code is provided. Therefore the significance is conditional and cannot be evaluated from the manuscript as it stands.

major comments (4)

[Section 5, Table 1] The headline 14.5% reduction (L18, test perplexity 159.9 vs 187.1) is selected as the best among nine projection dimensionalities, with the same held-out test set used for selection. The text immediately states that with a different seed, L114 performed best, and all rows appear to be single runs with no error bars, seed averages, or validation-based model selection. This is a multiple-comparisons selection-bias problem: the reported gain is the maximum over configurations, not an unbiased estimate of expected improvement. Please report seed-averaged results with variance, select d_proj on validation data, and state the number of seeds used.
[Section 4] The 'Vaswani baseline' is not the original transformer from Vaswani et al. (2017), but a reduced architecture with two blocks, three heads, and d_model=192. The internal comparison may be fair, but the claim in the abstract and conclusions about improving similarity in 'modern language models' goes beyond what a compact single-dataset experiment can support. The baseline should be described accurately as a compact transformer, and the claims should either be narrowed or validated on a standard-scale setup. In addition, the training hyperparameters, tokenizer configuration, and model-selection protocol are not reported.
[Sections 4-5] The paper states that 'the code was seeded' and that models were trained deterministically, but no code, configuration files, or data-processing scripts are provided. Given the acknowledged seed sensitivity and the unexplained L116 timing anomaly, an independent check of Table 1 is impossible. Please release code and reproducibility artifacts; otherwise the central empirical claim cannot be verified by readers.
[Section 6 and Section 1 note] The paper itself states in Section 1 that it is a preliminary report and in Section 6 that L1 attention was tested on only one dataset, one model type, and a very small model. This honest caveat is in tension with the abstract's conclusion that L1 augmentation 'provides a principled and effective improvement to similarity computation in modern language models.' The load-bearing empirical claim needs either additional datasets and model scales, or a carefully conditional phrasing that limits the claim to the studied setup.

minor comments (6)

[Section 3] The notation around Eq. (6) is ambiguous: lambda_h is indexed by head, but Q and K are full matrices, and the L1 matrix is defined only implicitly. Please clarify that each head computes the L1 norm on its own slice.
[Section 5] Table 1 lacks error bars and the number of runs; even the 'reproduced' L116 anomaly is not quantified. The phrase 'significant reduction' in the text is used without any statistical test.
[Section 5, Table 2] The Pearson correlations between norm variance and lambda are reported without sample size, number of epochs, or confidence intervals. Please provide these details or describe them as exploratory.
[Section 4] The description of the hybrid L18/RBF model says it uses '8 dimensional projections in the lower heads' and 'the Kim RBF-L2 decrement in the upper heads with no projections,' but the earlier discussion refers to 'lower block' and 'upper block.' Please make the terminology consistent.
[Section 2] The related-work discussion of Guo et al. and Kobayashi et al. is useful, but the connection to the proposed method could be sharper. The claim that L1 'preserves proportional differences' is stated without formal justification; please clarify what 'proportional' means here.
[Section 1] Minor typos and formatting issues: equation (2) is not numbered, and the vector examples in Section 1 are rendered as inline arrays that are hard to read. The footnote-like note 'This is a preliminary report' should be integrated into a proper limitations or reproducibility section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline claim is a measured held-out perplexity comparison, not a value derived from the method's own equations; the reported seed-sensitivity is selection bias, not circular reasoning.

full rationale

The paper's load-bearing claim is an empirical perplexity comparison on WikiText-2 among L1-augmented attention, the Vaswani baseline, and an RBF-L2 kernel. Equation (6) defines the L1-augmented score using queries, keys, and learned per-head lambdas and projections; these parameters are trained by standard language-modeling loss, and the reported perplexities are held-out test measurements rather than quantities reconstructed from the fitted parameters. The Section 3 toy examples are illustrative demonstrations of the proposed metric's intended behavior, not predictions derived from fitted values, so they do not constitute a circular derivation. No load-bearing step is justified by a self-citation, a uniqueness theorem, or an ansatz imported from the author's prior work. The paper itself notes the main methodological caveat: selecting the best projection dimensionality from test perplexity and observing that a different seed made L114 best (Section 5, Table 1) is a model-selection and reproducibility concern. That is selection bias or effect-size uncertainty, not circularity, because the test perplexity is not an input to the definition or training of the method. Therefore the central claim retains independent empirical content and the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the choice of L1 distance as a meaningful dissimilarity measure, on the standard transformer framework, on the interpretability of lambda-norm correlations, and on the assumption that the reduced baseline architecture is a fair comparison. No physically invented entities are introduced.

free parameters (3)

lambda_h (per-head L1 weight) = trained per head; values not reported numerically (lower block larger, upper block smaller)
Scales the L1 penalty in Eq. (6); central to the method and learned from data.
L1 projection matrices = learned, unconstrained; matrices not shipped
Project queries/keys from d_k=64 to d_proj before L1 computation (Section 4, Section 5); their learned specialization is credited for part of the gain.
projection dimensionality d_proj = 8 for best reported model (L18); 14 for another seed
Hyperparameter selected by comparing test perplexity across L164, L116, L114, L112, L110, L18, L16 (Table 1). Using the test set for selection inflates the reported improvement.

assumptions (5)

domain assumption Scaled dot-product attention on learned query/key embeddings is a valid starting point for similarity in language models.
Used throughout Section 1 and Section 3; the paper modifies rather than replaces the standard attention equation.
domain assumption L1 norm of coordinate-wise differences is a meaningful dissimilarity metric for embeddings.
Motivating examples in Section 1 and the penalty in Eq. (6) depend on this geometric choice; no independent validation is provided.
ad hoc to paper Projection layers can learn to preserve useful L1 structure after dimensionality reduction.
The claim in Section 5 that projections compensate for information loss by learning to specialize is an untested modeling assumption introduced by this paper.
domain assumption Correlations between norm variance and lambda indicate functional specialization rather than coincidental co-variation.
Section 5 interprets Table 2's high correlations as evidence that lower layers rely on L1 geometry; this causal reading is not experimentally established.
standard math Gradient-based joint training of attention weights, lambda, and projections effectively optimizes the objective.
Relies on standard backpropagation for neural networks; not demonstrated in this paper beyond the reported runs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of L1 Augmented Attention as an Improved Vector Similarity Metric." pith.science (2026). https://pith.science/paper/MY5IOVFV

@misc{pith2026260718027,
  author       = {Pith},
  title        = {Pith review of: L1 Augmented Attention as an Improved Vector Similarity Metric},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MY5IOVFV}},
  note         = {Machine review of arXiv:2607.18027}
}

read the original abstract

Scaled dot product attention conflates directional alignment and vector magnitude, limiting its effectiveness as a similarity metric in Transformer models. We introduce L1 augmented attention, a simple and computationally parallelizable modification that subtracts a learned, head specific L1 distance between queries and keys from the dot product score. This hybrid similarity captures complementary geometric information. Dot product rewards directional alignment, while L1 penalizes coordinate deviations. To reduce the cost of L1 computation, we project queries and keys into low dimensional subspaces whose parameters specialize to preserve informative L1 structure. Evaluated on WikiText 2 using a compact transformer, L1 augmented attention achieves up to a 14.5% reduction in perplexity over the original transformer baseline and outperforms an RBF L2 kernel. Analysis of norm variance and learned L1 weights reveals distinct geometric roles across layers and strong head level specialization. These results demonstrate that enriching attention with L1 geometry provides a principled and effective improvement to similarity computation in modern language models, with practical benefits for both accuracy and parallel efficiency.

Figures

Figures reproduced from arXiv: 2607.18027 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 1 linked inside Pith

[2]

Lipschitz constraints can endow models with provable robustness against adversarial perturbations …, and guaranteed generalisation bounds

Related Work A wide variety of modiﬁcations to the attention equation have been proposed to address different issues, but few explicitly address the conﬂation of magnitude and direction. For example, Shaw et al. (2018) modiﬁed attention to incorporate relative distance between a query and its associated keys. Huang et al. (2019) made subsequent revisions ...

2018
[3]

As we will see, this explicitly unmasks the otherwise hidden norm information and provides access to a richer geometry when calculating vector similarity

L1 Augmented Attention To address the limitations of dot product as a vector similarity metric, we propose augmenting the scaled dot product with an L1 decrement: (6) , where H refers to the attention heads to which the weighting parameter is indexed, and is the matrix of pairwise L1 distances between queries and keys. As we will see, this explicitly unma...
[4]

Implementation The Vaswani attention equation is implemented by Keras/TensorFlow in the MultiHeadAttention class. We can modify the equation by subclassing that parent as ModiﬁedMultiHeadAttention, copying the parent code for equation (1), and augmenting it to calculate the -weighted L1 term of equation (6). We then subtract the L1 norm from the scaled do...

2016
[5]

A striking ﬁnding in empirical work on word embeddings is that there is a sweet spot for the dimensionality of word vectors: neither too small, nor too large

Modeling Results A baseline was ﬁrst established that used the existing Keras class MultiHeadAttention which implements the Vaswani attention equation in (1), using 64 dimensional query, key and value vectors for each attention head. This was followed by an implementation of equation (5) for the RBF-L2 equation softmax scores as described in Kim et. al. (...

2021
[6]

While section 1 presents some limitations of dot product, L1 Augmented Attention is also imperfect

Limitations The present study is not the last word on a vector similarity metric. While section 1 presents some limitations of dot product, L1 Augmented Attention is also imperfect. For example, consider these vectors : λ λ λ λ λ Q⋅K⊤ Head 0 0.9668 0.6412 Head 1 0.9232 0.7860 Head 2 0.9579 0.5079 Table 2: Pearson correlations between norm variance and λ v...
[7]

Another line of research to be explored is related to the choice of feature map for linear attention

Conclusions We have presented clear evidence that a weighted L1 norm decrement to the scaled attention score results in a signiﬁcant reduction of perplexity in a language model. This reduction comes at a computational cost, which motivated projections of the queries and keys to smaller dimensions in the attention heads. In addition to large reductions in ...

2020
[8]

Word Embeddings: Explaining their Properties

References Arora, S. Word Embeddings: Explaining their Properties. (2016). URL http:// www.oﬀconvex.org/2016/02/14/word-embeddings-2/ Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. (2019). What does BERT look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NL...

arXiv 2016

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [2]

Lipschitz constraints can endow models with provable robustness against adversarial perturbations …, and guaranteed generalisation bounds

Related Work A wide variety of modiﬁcations to the attention equation have been proposed to address different issues, but few explicitly address the conﬂation of magnitude and direction. For example, Shaw et al. (2018) modiﬁed attention to incorporate relative distance between a query and its associated keys. Huang et al. (2019) made subsequent revisions ...

2018

[2] [3]

As we will see, this explicitly unmasks the otherwise hidden norm information and provides access to a richer geometry when calculating vector similarity

L1 Augmented Attention To address the limitations of dot product as a vector similarity metric, we propose augmenting the scaled dot product with an L1 decrement: (6) , where H refers to the attention heads to which the weighting parameter is indexed, and is the matrix of pairwise L1 distances between queries and keys. As we will see, this explicitly unma...

[3] [4]

Implementation The Vaswani attention equation is implemented by Keras/TensorFlow in the MultiHeadAttention class. We can modify the equation by subclassing that parent as ModiﬁedMultiHeadAttention, copying the parent code for equation (1), and augmenting it to calculate the -weighted L1 term of equation (6). We then subtract the L1 norm from the scaled do...

2016

[4] [5]

A striking ﬁnding in empirical work on word embeddings is that there is a sweet spot for the dimensionality of word vectors: neither too small, nor too large

Modeling Results A baseline was ﬁrst established that used the existing Keras class MultiHeadAttention which implements the Vaswani attention equation in (1), using 64 dimensional query, key and value vectors for each attention head. This was followed by an implementation of equation (5) for the RBF-L2 equation softmax scores as described in Kim et. al. (...

2021

[5] [6]

While section 1 presents some limitations of dot product, L1 Augmented Attention is also imperfect

Limitations The present study is not the last word on a vector similarity metric. While section 1 presents some limitations of dot product, L1 Augmented Attention is also imperfect. For example, consider these vectors : λ λ λ λ λ Q⋅K⊤ Head 0 0.9668 0.6412 Head 1 0.9232 0.7860 Head 2 0.9579 0.5079 Table 2: Pearson correlations between norm variance and λ v...

[6] [7]

Another line of research to be explored is related to the choice of feature map for linear attention

Conclusions We have presented clear evidence that a weighted L1 norm decrement to the scaled attention score results in a signiﬁcant reduction of perplexity in a language model. This reduction comes at a computational cost, which motivated projections of the queries and keys to smaller dimensions in the attention heads. In addition to large reductions in ...

2020

[7] [8]

Word Embeddings: Explaining their Properties

References Arora, S. Word Embeddings: Explaining their Properties. (2016). URL http:// www.oﬀconvex.org/2016/02/14/word-embeddings-2/ Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. (2019). What does BERT look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NL...

arXiv 2016