{"id":"4d0fb13f-ca9d-43bc-ad83-d9e324cd6903","arxiv_id":"2607.18027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Subtracting a learned, per-head L1 distance between projected queries and keys from the scaled dot-product attention score reduced test perplexity by up to 14.5% on a small WikiText-2 language model.","lead":"This paper adds a learned, per-head L1 distance penalty to the scaled dot product in transformer attention and reports up to 14.5% lower perplexity on WikiText-2 with a small model. The result is preliminary: no code is released, only one dataset is used, and the best projection size was selected after inspecting test-set results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-of-9 projection selection and seed sensitivity make the 14.5% gain unquantified; need seed-averaged, validation-selected comparison.","rationale":"The reader's weakest assumption is correct and is the most load-bearing concern. Section 5 explicitly admits: 'different seeding of the training data impacted which dimensionality obtained the lowest overall perplexity. On one run with different seeding, the L114 model performed best.' This directly confirms that the L18 result is seed-specific. Since Table 1 reports only test perplexity for a single run of each configuration, and the paper appears to have selected the best among nine projection dimensions based on that test set, the 14.5% figure is an upper bound from a multiple-comparisons procedure, not a stable effect size. The manuscript's own 'preliminary report' note and Section 6 limitations (single dataset, small model) are scope limitations rather than internal inconsistencies. There is independent support for the direction of the effect: the L1 Baseline without projections improves perplexity by 9.6% over Vaswani, so the L1 term itself has a plausible effect. This is why the verdict should remain CONDITIONAL rather than REJECT: the method may work, but the headline magnitude is not yet reliable. The proposed test—seed averaging and validation-based selection—would settle whether the improvement is robust and how large it actually is.","tokens_in":10496,"tokens_out":6998,"duration_ms":82243,"concrete_test":"Train all Table 1 models (Vaswani, RBF-L2, L1 Baseline, L164, L116, L114, L112, L110, L18, L16, L18/RBF) under 5 different seeds using the same training setup. For each model, report mean and standard deviation of test perplexity. Select the projection dimensionality using the validation set (not test) for each seed; then compute the test perplexity of that selected model and compare to a Vaswani baseline selected similarly. If the mean improvement of the validation-selected L1 model over baseline is less than, say, 5 percentage points or its 95% CI includes zero, the headline 14.5% is not reproducible. Also report the frequency with which each projection dimension is selected across seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on a single test-set number: Table 1 reports L18 at 159.9 perplexity, a 14.5% reduction over Vaswani. But this number was selected as the best among nine projection dimensionalities (L6–L64 and L1 Baseline) on the same test set, and the paper explicitly states that with a different seed L114 was best. No error bars, seed averages, or validation-based model selection are reported. Since the test-set perplexity is used to pick the architecture, the reported gain is an upper bound of a multiple-comparisons procedure, not an estimate of the method's expected improvement. All runs appear to be single-seed; the 'up to' in the abstract cannot be interpreted as a robust effect size. This is load-bearing because the '14.5%' is the headline quantitative contribution; if the true expected improvement after averaging seeds or selecting on validation is, say, 5–8%, the claim as stated is technically true but misleading, and the conclusion that L1 augmentation 'significantly' reduces perplexity is unsupported in magnitude.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes L1-augmented attention, defined in Eq. (6) as softmax(QK^T/sqrt(d_k) - lambda_h * ||q_i - k_j||_1), with a learned per-head weight lambda_h and low-dimensional projections of queries and keys before computing the L1 distance. The motivation is that the dot product conflates directional alignment with vector magnitude, while the L1 term explicitly penalizes coordinate-wise deviations. On WikiText-2 with a compact two-block, three-head transformer (d_model=192), the authors report that their best projected L1 model (L18) achieves a 14.5% perplexity reduction over the Vaswani baseline (Table 1), and also outperforms an RBF-L2 kernel. They additionally analyze per-head lambda values and norm variance, claiming distinct geometric roles in lower and upper blocks. The paper closes with a limitations section acknowledging the single-dataset, small-model scope and a note in Section 1 describing the work as a preliminary report.","tokens_in":10772,"tokens_out":3568,"duration_ms":41825,"significance":"The idea is simple and the geometric intuition is appealing: augmenting a directional dot-product score with a coordinate-wise L1 penalty could plausibly help attention allocate weight more precisely. The toy examples in Section 3 illustrate the failure modes of the dot product and how the L1 term fixes them. If the reported improvement survives seed averaging and proper model selection, the contribution would be a useful, low-cost modification to attention. The per-head lambda analysis is also a nice interpretability direction. However, the current quantitative evidence is a single-run Table 1 with test-set-based selection of the projection dimension, and no code is provided. Therefore the significance is conditional and cannot be evaluated from the manuscript as it stands.","major_comments":[{"comment":"The headline 14.5% reduction (L18, test perplexity 159.9 vs 187.1) is selected as the best among nine projection dimensionalities, with the same held-out test set used for selection. The text immediately states that with a different seed, L114 performed best, and all rows appear to be single runs with no error bars, seed averages, or validation-based model selection. This is a multiple-comparisons selection-bias problem: the reported gain is the maximum over configurations, not an unbiased estimate of expected improvement. Please report seed-averaged results with variance, select d_proj on validation data, and state the number of seeds used.","section":"Section 5, Table 1"},{"comment":"The 'Vaswani baseline' is not the original transformer from Vaswani et al. (2017), but a reduced architecture with two blocks, three heads, and d_model=192. The internal comparison may be fair, but the claim in the abstract and conclusions about improving similarity in 'modern language models' goes beyond what a compact single-dataset experiment can support. The baseline should be described accurately as a compact transformer, and the claims should either be narrowed or validated on a standard-scale setup. In addition, the training hyperparameters, tokenizer configuration, and model-selection protocol are not reported.","section":"Section 4"},{"comment":"The paper states that 'the code was seeded' and that models were trained deterministically, but no code, configuration files, or data-processing scripts are provided. Given the acknowledged seed sensitivity and the unexplained L116 timing anomaly, an independent check of Table 1 is impossible. Please release code and reproducibility artifacts; otherwise the central empirical claim cannot be verified by readers.","section":"Sections 4-5"},{"comment":"The paper itself states in Section 1 that it is a preliminary report and in Section 6 that L1 attention was tested on only one dataset, one model type, and a very small model. This honest caveat is in tension with the abstract's conclusion that L1 augmentation 'provides a principled and effective improvement to similarity computation in modern language models.' The load-bearing empirical claim needs either additional datasets and model scales, or a carefully conditional phrasing that limits the claim to the studied setup.","section":"Section 6 and Section 1 note"}],"minor_comments":[{"comment":"The notation around Eq. (6) is ambiguous: lambda_h is indexed by head, but Q and K are full matrices, and the L1 matrix is defined only implicitly. Please clarify that each head computes the L1 norm on its own slice.","section":"Section 3"},{"comment":"Table 1 lacks error bars and the number of runs; even the 'reproduced' L116 anomaly is not quantified. The phrase 'significant reduction' in the text is used without any statistical test.","section":"Section 5"},{"comment":"The Pearson correlations between norm variance and lambda are reported without sample size, number of epochs, or confidence intervals. Please provide these details or describe them as exploratory.","section":"Section 5, Table 2"},{"comment":"The description of the hybrid L18/RBF model says it uses '8 dimensional projections in the lower heads' and 'the Kim RBF-L2 decrement in the upper heads with no projections,' but the earlier discussion refers to 'lower block' and 'upper block.' Please make the terminology consistent.","section":"Section 4"},{"comment":"The related-work discussion of Guo et al. and Kobayashi et al. is useful, but the connection to the proposed method could be sharper. The claim that L1 'preserves proportional differences' is stated without formal justification; please clarify what 'proportional' means here.","section":"Section 2"},{"comment":"Minor typos and formatting issues: equation (2) is not numbered, and the vector examples in Section 1 are rendered as inline arrays that are hard to read. The footnote-like note 'This is a preliminary report' should be integrated into a proper limitations or reproducibility section.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"This reads like an early technical report rather than a finished journal submission. The core idea is plausible and worth testing, but the central quantitative claim is currently unquantified: single runs, test-set-based selection of d_proj, seed sensitivity acknowledged in the text, and no code. I would not be comfortable accepting without a proper multi-seed evaluation with validation-based selection and release of code. If the effect is real, this could become a solid short paper; if not, the current abstract significantly overstates the evidence. The reduced architecture is acceptable if labeled accurately, but the 'Vaswani baseline' phrase is misleading and should be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The idea is simple and worth a look: subtract a learned, head-specific L1 penalty between queries and keys from the scaled dot-product, with a low-dimensional projection to keep the L1 term cheap. I don't see this exact combination in the cited work, and the motivation is legitimate — dot product does conflate direction and magnitude. The toy examples make the point well, and the norm-variance/lambda analysis is a nice descriptive addition.\n\nWhat the paper does well is that it is honest about scope: it says upfront this is a preliminary report, and the limitations section admits a single dataset and a tiny model. The implementation is described clearly enough to reproduce with moderate effort, though no code is provided.\n\nThe soft spots are real, and the stress-test note is right on target. The 14.5% headline is the best of nine projection dimensions, chosen after looking at test perplexity. The paper itself says a different seed made L114 the best, so the number is an upper bound from a multiple-comparisons procedure, not an estimate of the method's expected gain. Every row in Table 1 is a single run, so we have no variance information at all. The baseline is also a reduced architecture (two blocks, three heads, dim 192), not the original Vaswani setup, so the comparison is against a modified reference point. These are fixable issues, but they mean the central claim is unquantified as it stands.\n\nOne minor point: the paper's own example in Section 6 shows a failure case where the L1-augmented score gives equal values for a perfect match and a less perfect one. The authors mention it but don't discuss how often that happens or whether it matters. It's a small caveat, not a fatal one.\n\nI'd send this to peer review. The idea is new enough, the writing is clear, and the empirical problems are the kind a referee can point to and the author can address with more runs and proper model selection. As is, I wouldn't cite it as evidence of a 14.5% gain, but I'd cite it as a proposed attention variant if I needed to reference the idea.\n\nFor a reading group, I'd bring it in maybe — the methodology lesson is useful.","headline":"A plausible attention tweak with a clear write-up, but the 14.5% headline is a best-of-N test-selected number from single runs, so the real effect size is unquantified.","tokens_in":11214,"tokens_out":2772,"would_cite":false,"duration_ms":30493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"L1-augmented attention reduces transformer perplexity by up to 14.5 percent.","keywords":["attention","similarity metric","L1 norm","transformer","language modeling","perplexity","kernel methods","vector geometry"],"falsifier":"Run multiple seeds with a fixed projection dimension (for example, 8) using the same architecture and WikiText-2 setup, and compare seed-averaged test perplexity against an equally tuned baseline; if the average advantage is not reliably positive, the central claim is refuted. A second check: pre-register one projection dimension before seeing test results and see whether the gain persists.","tokens_in":10373,"feed_emoji":"📉","tokens_out":3908,"duration_ms":41697,"temperature":0.7,"pith_summary":"This paper tries to show that the scaled dot product is an incomplete similarity metric for transformer attention because it hides the separate contributions of vector direction and magnitude. It proposes subtracting a learned, head-specific L1 distance between queries and keys from the dot-product score, so the attention score rewards directional alignment while explicitly penalizing coordinate-wise deviations. On WikiText-2 with a compact two-block transformer, the modified attention reportedly lowers test perplexity by up to 14.5% compared with the original baseline and beats an RBF-L2 kernel. The paper also shows that projecting queries and keys into low-dimensional subspaces both cuts the cost of the L1 computation and, with learned per-head weights, can improve perplexity further. The wider point is that enriching attention with L1 geometry is a direct way to improve similarity computation without abandoning the transformer's parallel structure.","feed_headline":"L1-augmented attention cuts perplexity by 14.5%","feed_subtitle":"Adding a learned per-head L1 penalty to dot-product scores beat the baseline and RBF-L2 on WikiText-2.","key_machinery":"The central object is the L1-augmented attention score, softmax(QK^T/sqrt(d_k) - lambda_h * L1(Q,K)), where L1(Q,K) is the matrix of per-coordinate absolute differences between queries and keys and lambda_h is a learned, head-specific weight initialized to zero. The L1 term supplies the magnitude and coordinate-wise information that the dot product masks, while the dot product preserves directional signal; the learned lambda lets each head decide how much geometric penalty to apply. To make the penalty affordable, the model projects Q and K into low-dimensional subspaces using a shared projection matrix with disjoint per-head slices, and those projections specialize during training to preser","core_discovery":"The central claim is that attention scores should combine two complementary geometries: the dot product, which is sensitive to directional alignment, and an L1 penalty, which directly measures coordinate-wise distance. Concretely, the paper studies scores of the form softmax(QK^T/sqrt(d_k) - lambda_h * ||q_i - k_j||_1), with per-head learned lambda initialized at zero and queries and keys projected to low dimensions. On a small transformer trained on WikiText-2, this L1-augmented attention achieved the lowest perplexity (159.9) among the tested variants, a 14.5% reduction from the original transformer baseline (187.1), and outperformed an RBF-L2 kernel (178.1). The paper further reports that","pith_inferences":["The geometric argument for why dot product fails as a similarity metric is dataset-independent; if correct, the same L1 augmentation should transfer to encoder tasks such as classification or retrieval, though the paper only tests autoregressive language modeling.","The reported best projection dimension changed with the random seed, so a practical deployment may need per-run validation or ensembling rather than a single fixed dimension; the paper acknowledges this but does not quantify it.","The concluding Laplacian-kernel formulation, exp(-lambda * ||u(q)-v(k)||_1), suggests a path toward linear-complexity attention that preserves the L1 geometry, but that extension is not implemented here."],"forward_implications":["If the result holds, any transformer-based language model can replace its similarity score with this hybrid without changing the rest of the architecture.","Because the L1 term is embarrassingly parallel, the perplexity gain need not cost extra wall-clock time when computed alongside the dot product.","Learned per-head lambda provides an interpretable knob: heads can choose between direction-focused and geometry-focused similarity, which may aid analysis of attention.","The projection-dimension sweet spot suggests that L1-specialized low-dimensional query and key embeddings are themselves a useful learned representation, analogous to learned relative-position biases.","The hybrid L18/RBF result indicates a resource-accuracy tradeoff: using L1 only where norms matter most recovers about half the gain at about half the cost."],"fun_headline_variants":["Adding L1 distance to attention scores cuts perplexity 14.5%","L1-augmented attention beats dot product and RBF-L2","Per-head L1 penalty improves attention geometry","Hybrid similarity: dot product plus L1 distance wins","Attention upgrade: L1 penalty drops perplexity 14.5%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 14.5% reduction is load-bearing but rests on selecting the best projection dimensionality from test-set perplexity across many variants; the paper reports that the best dimension changed with the seed and reports no error bars or seed averages.","fun_headline_variants_meta":{"raw":{"variants":["Adding L1 distance to attention scores cuts perplexity 14.5%","L1-augmented attention beats dot product and RBF-L2","Per-head L1 penalty improves attention geometry","Hybrid similarity: dot product plus L1 distance wins","Attention upgrade: L1 penalty drops perplexity 14.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2384,"prompt_tokens":732,"completion_tokens":1652,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":1564}},"tokens_in":476,"tokens_out":1652,"duration_ms":16819,"temperature":1.0,"reasoning_tokens":1564,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:18:00.577983+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run multiple seeds with a fixed projection dimension (for example, 8) using the same architecture and WikiText-2 setup, and compare seed-averaged test perplexity against an equally tuned baseline; if the average advantage is not reliably positive, the central claim is refuted. A second check: pre-register one projection dimension before seeing test results and see whether the gain persists.","supporting_citations":[],"review_version":1}