{"id":"8d1e527d-a66d-4717-bcc4-3d2c2b1c399e","arxiv_id":"2608.00418","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Initializing a learnable attention mask with task structure, rather than the query-key projections, keeps the prior alive through training and changes Transformer extrapolation on Boolean and arithmetic tasks.","lead":"This paper shows that where you put a structural prior in a Transformer matters: encoding it directly into the attention mask survives training, while encoding it through the query-key projections gets erased. On logic puzzles and arithmetic, mask-based priors sharply improve extrapolation and low-data learning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Boolean extrapolation gains rest on exact, effectively frozen oracle masks; persistence and adaptability under imperfect priors are never tested, so the central mechanism is not established.","rationale":"The reader's weakest assumption was the oracle-prior limitation. I agree that this is the main practical caveat, but the more precise load-bearing issue is that the paper never demonstrates that the learnable mask is actually learning or adapting in the regime where it succeeds. The +-10 log-sigma parameterization saturates so strongly that the mask is nearly fixed, making the Boolean result a clean demonstration that a correctly specified hard constraint can change extrapolation, rather than a demonstration that additive learnable biases persist better than QK initializations. This matters because the paper's stated contribution is specifically the persistence of a learnable additive bias. Without misspecified-mask experiments, the central claim is underdetermined: the QK baselines wash out, but the mask baseline may simply be incapable of washing out because its parameters are pinned by the saturated transform. The proposed test would settle this cleanly by measuring whether the mask parameterization both persists and adapts when the prior is wrong. Because the reader's verdict was already CONDITIONAL, and the concern does not overturn the empirical comparison but sharpens its interpretation, I leave the verdict unchanged rather than escalating it.","tokens_in":19125,"tokens_out":11453,"duration_ms":122992,"concrete_test":"Run the 2Parity GOTU experiment with three mask initializations at the same +/-10 scale: the correct mask (pairs tokens 0 and 1), a deliberately misspecified mask (pairs tokens 0 and 2), and a neutral all-zero mask. Track test extrapolation accuracy and, every 10 epochs, the Frobenius distance between the current attention-logit bias log-sigma(M) and the initial bias, for at least 5 seeds. If the misspecified mask remains close to its initial pattern and extrapolation stays near the default 75%, then the Boolean result is explained by freezing an exact oracle prior, not by adaptable persistence; if the misspecified mask drifts away and accuracy recovers, the learnable-mask claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the parameterization of the prior, additive learnable mask versus query-key projections, determines whether the prior persists through optimization. The decisive evidence is the Boolean GOTU comparison in Table 1 and Figure 5. But those masks are hand-designed from the exact ground-truth function decomposition (Section E), and they are initialized at open=10, close=-10 through log-sigma. Because log-sigma saturates, the effective bias for open positions is near 0 and for closed positions is about -10, with sigma(-m) values of 4.5e-5 and 0.99995 respectively. The mask update for a closed position is proportional to its already tiny attention probability, so the learned mask is effectively frozen throughout training. Thus the near-perfect extrapolation may simply reflect that an exactly correct hard constraint is baked in at initialization and cannot be forgotten, rather than showing that a learnable additive bias is a better substrate for persistent priors. The all-zero mask ablation (Table 4) shows that the structure matters, but it does not distinguish 'this parameterization preserves the prior' from 'this parameterization encodes the exact answer and then freezes.' The paper's own Table 5 weakens the learnability story further: on CIFAR100 a fixed mask matches or beats the learnable mask, and the main textual evidence for washout is qualitative attention-map visualization in Figure 5. Theorem 1 and Corollary 1 do not close this gap: they bound derivatives of softmax probabilities with respect to q, k, and m, not the actual parameter-update magnitudes under Adam, and the comparison ignores the shared downstream loss gradient. The Limitations section explicitly concedes the oracle assumption, but the paper never tests how the mechanism behaves when the prior is imperfect, which is exactly the regime where learnability and persistence need to be separated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to initialize a learnable additive attention-logit bias (a 'mask') from task structure, arguing that this parameterization preserves the injected inductive bias through training, whereas structured initialization that encodes the same prior through query-key projections is overwritten. On Boolean generalization-on-the-unseen (GOTU) tasks, the proposed mask initialization yields near-perfect test accuracy (e.g., 100% on 2Parity vs 75% for baseline and QK-based methods), improves low-data arithmetic, and matches or slightly improves vision and language benchmarks. The authors provide a gradient-level argument (Theorem 1 and Corollary 1), ablations (all-zero mask, fixed mask), and visualizations of attention patterns.","tokens_in":19391,"tokens_out":7589,"duration_ms":67714,"significance":"If the central claim were fully established, the work would offer a simple and practically useful mechanism for encoding persistent inductive bias in Transformers, complementing QK-based structured initialization. The paper's framing of the GOTU setting as a test of inductive bias is appropriate, and the all-zero mask ablation (Table 4) is a useful control showing that the gains are not merely from extra parameters. However, the reported evidence is currently limited by the oracle construction of Boolean masks, single-seed runs, and a theoretical argument that does not directly establish 'slower updates'; the paper's own limitations passage acknowledges the oracle assumption.","major_comments":[{"comment":"The Boolean masks are initialized with open=10 and close=-10 through logσ(M), and are constructed from the exact ground-truth function decomposition of the target function, including its behavior on the unseen domain (Section E). Because logσ(10) is approximately 0 and logσ(-10) is approximately -10, closed positions receive essentially zero probability after softmax; consequently, the gradient ∂p_i/∂m_j in Eq. (4) is vanishingly small for open entries (σ(-10)≈4.5e-5) and for closed entries (p_j≈0). The mask is therefore effectively frozen at the oracle pattern, so the near-perfect extrapolation in Table 1 reflects a correctly specified hard constraint baked in at initialization, not a learnable prior that survives optimization. The comparison with QK-based init is also confounded because QK methods must represent the pattern through a low-rank product and are not given a near-hard, exactly-oracle constraint of the same effective scale. To support the persistence claim, the paper needs experiments with imperfect, coarsely specified, or lower-contrast masks, and measurements of how quickly each parameterization deviates from the initial attention pattern.","section":"§4.1, Eq. (4), Table 7, Section E"},{"comment":"The gradient bounds in Eqs. (5)–(7) do not establish that mask-based priors update slower than QK-based priors. The bounds compare per-parameter gradient norms in different parameter spaces: the QK bounds scale with B_k and B_q, which can be small or large depending on normalization, while the mask bound is at most 1 and is largest (≈1) precisely for closed positions when p_i is large; the smallness of the open-mask gradient comes from the specific choice m=10, an initialization-scale effect rather than a property of the additive parameterization. The statement that 'mask-based priors will update slower' is a dynamical claim that requires measuring the actual drift of attention patterns over training (e.g., Frobenius distance from the initial pattern as a function of step) for both parameterizations at matched effective initialization. Please either add such measurements or soften the conclusion.","section":"§6.1, Theorem 1, Corollary 1"},{"comment":"All experiments are reported from a single seed (seed 0 for Boolean and most vision tasks, seed 42 for ImageNet100, and no seed specified for arithmetic in Table 8). The differences between methods, especially the small gaps on cyclic tasks (87.45 vs 87.48) and the near-chance 75% accuracy on 2Parity, cannot be assessed for significance without multiple independent runs. At minimum, the central Boolean and arithmetic comparisons should be repeated over at least three seeds and reported as mean ± standard deviation.","section":"Tables 7–11 and All Boolean/Arithmetic Results"},{"comment":"Mask Init on ImageNet100 is reported as 85.92 in Table 3 but as 85.82 (+2.44) in Table 5; the implied improvement over Default (83.38) is 2.54 under the Table 3 value, not 2.44. Please correct the inconsistency and verify the underlying run.","section":"Table 3 vs Table 5"}],"minor_comments":[{"comment":"Please clarify whether the reported open/close initialization values (e.g., 10 and -10) are the raw mask parameters M or the effective logσ(M) values, since the text describes M as being transformed by logσ.","section":"§4.1"},{"comment":"The related-work paragraph contains a malformed citation ('Wang et al., Zhong and Andreas, 2024'); the citation and reference list should be cleaned up.","section":"§2.1"},{"comment":"The arithmetic setup table omits the seed and some optimizer hyperparameters (e.g., AdamW betas), which should be added for reproducibility.","section":"Appendix G.2"},{"comment":"The vision benchmarks in Table 3 report no variance; adding at least two additional seeds would strengthen the claim that Mask Init 'consistently' outperforms baselines.","section":"§5.3"},{"comment":"The bottom-left subfigure label appears garbled ('(a) Addition10010010'); please correct the caption and the subfigure labels.","section":"Figure 2"},{"comment":"The TinyStories test perplexity difference (10.48 vs 10.49) is within noise; the text should avoid claiming a 'consistent' improvement based on this difference alone.","section":"Appendix F, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clearly written and the question is well motivated, but the central empirical claim depends on oracle masks that are effectively frozen at initialization. I recommend major revision rather than rejection because the core idea is testable and the missing experiments (imperfect priors, multi-seed runs, drift measurements) are feasible within the paper's scope. The lack of code release is also a reproducibility concern for the central Boolean results, since the mask construction details are essential to reproduce the exact attention patterns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper has a real, large-effect empirical result: on the Boolean GOTU tasks from Abbe et al., a structural prior encoded as an additive attention-logit bias changes the learned extrapolation rule (2Parity goes from 75% to 100%, and the other three Boolean tasks move similarly), while the same prior encoded through QK projections leaves the Transformer stuck at its default minimum-degree behavior. Second, the paper does not actually establish the mechanism it claims: it shows persistence, but not that a learnable additive bias is what produces it.\n\nThe controlled comparison is the main strength. Both initialization schemes start from the same target attention pattern; the QK methods match it at initialization and lose it during training, while the mask keeps it. That is a fair and informative experiment, and the all-zero mask ablation rules out a trivial 'extra parameters help' explanation. The arithmetic gains are dramatic: 77% sequence accuracy on 3-digit addition at 0.4% training data, versus 3% for the best baseline. The authors also deserve credit for candor: the Limitations section explicitly concedes the oracle assumption, and the Section 6.1 analysis is framed as intuition, not proof.\n\nThe soft spots, in proportion. The most serious: the Boolean masks are hand-designed from the exact ground-truth function decomposition and initialized at ±10 through log-sigma, which saturates. Closed positions start with attention probability near 4.5e-5, so their mask gradients are proportionally tiny, and open positions are suppressed by the sigma(-m) factor. The mask is effectively frozen, so the near-perfect extrapolation may just be a correct hard constraint baked in at initialization, rather than a parameterization that resists repurposing. Theorem 1 and Corollary 1 are correct as far as they go, but they bound derivatives of attention probabilities, not actual update magnitudes under Adam, and they ignore the shared downstream loss gradient; the 'updates slower' conclusion is a heuristic. The vision results add a caution: a fixed mask beats the learnable mask on CIFAR100. Minor issues: single seed throughout, no code release, and ImageNet100 Mask Init appears as 85.92 in Table 3 but 85.82 in Table 5. The citation pattern is fine, and the overlap with Duan et al. is acknowledged.\n\nThis is for anyone working on inductive-bias steering, structured initialization, or the GOTU line. The persistence finding is worth citing; the mechanism is underdetermined. It deserves a serious referee—send it to review, but the referee should push for code, multiple seeds, a less saturated mask initialization, and robustness to imperfect priors, since that is where learnability and persistence can actually be separated.","headline":"Clean, large-effect evidence that where a structural prior lives in a Transformer determines whether it survives training; the why is underdetermined.","tokens_in":20007,"tokens_out":9395,"would_cite":true,"duration_ms":78934,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Mask-based priors fix Transformer extrapolation where QK init fails","keywords":["attention masks","structured initialization","inductive bias","generalization on the unseen","Boolean extrapolation","minimum-degree interpolator","low-data arithmetic","transformers"],"falsifier":"Run 2Parity with a deliberately misspecified mask, placing the same 2Parity pattern on a non-interacting token pair such as x1–x2 instead of x0–x1 while keeping all training settings fixed. If extrapolation accuracy stays near 100 percent, the persistence mechanism rather than the task-aligned pattern is doing the work; if it collapses to the roughly 75 percent minimum-degree level, the oracle interaction prior is the active ingredient.","tokens_in":18915,"feed_emoji":"🎯","tokens_out":11716,"duration_ms":91492,"temperature":0.7,"pith_summary":"Transformers often fit the observed domain on Boolean extrapolation tasks yet generalize according to a simpler minimum-degree interpolator rather than the true target function. This paper asks whether that systematically wrong extrapolation can be corrected by injecting the target's interaction structure into attention, and answers that it depends on where the prior lives. Encoded indirectly through query-key projections, the same structural prior is rapidly overwritten during training and the wrong extrapolation rule persists. Encoded directly as a learnable additive attention-logit bias, the prior survives optimization and produces near-perfect extrapolation on the Boolean tasks studied. The same mechanism improves low-data arithmetic and stays competitive on vision and language benchmarks, suggesting attention masks are a simple substrate for persistent inductive bias.","feed_headline":"Mask-based priors fix Transformer extrapolation where QK init fails","feed_subtitle":"Encoded as an attention-logit bias, the same prior survives training and changes the extrapolation rule.","key_machinery":"The carrying object is a learnable additive attention mask $M\\in\\mathbb{R}^{N\\times N}$, introduced into attention as $\\mathrm{softmax}(XW_QW_K^\\top X^\\top/\\sqrt{d}+\\log\\sigma(M))$, where $\\sigma$ is the sigmoid. The mask is a finite, learnable attention-logit bias initialized from task-level interaction structure: entries for desired token interactions are opened and undesired ones closed, and because $\\log\\sigma(M)\\leq 0$ the bias can only down-weight interactions, with softmax renormalizing afterward. Unlike causal or padding masks, it is not a hard architectural constraint but a persistent, adaptable inductive bias. The argument turns on where the prior sits: structure encoded through $QK^\\top$ is entangled with query and key feature scales, so attention gradients that grow with query and key norms can overwrite it, while the additive-mask parameterization keeps the structural bias outside the content-dependent similarity score, with gradient $\\partial p_i/\\partial m_j=p_i(\\delta_{ij}-p_j)\\sigma(-m_j)$ bounded independently of feature norms. This decoupling is what lets the prior persist through optimization and change the extrapolation rule.","core_discovery":"On Boolean Generalization-on-the-Unseen tasks, the central discovery is that parameterization, not just pattern, determines whether an injected structural prior changes what a Transformer learns. Fitting the same interaction pattern into the query and key projections so that $\\mathrm{softmax}(QK^\\top/\\sqrt{d})$ approximates the desired attention structure matches that structure at initialization, but by the end of training the pattern is washed out: 2Parity accuracy stays at 75 percent, matching the minimum-degree interpolator, and Cyclic3 and 3-Bit Majority stay near 87 percent. Encoding the identical pattern as a finite, learnable additive mask, $\\mathrm{softmax}(XW_QW_K^\\top X^\\top/\\sqrt{d}+\\log\\sigma(M))$, changes the learned rule: 2Parity and 3-Bit Majority reach 100 percent accuracy, Cyclic3 reaches 99.98 percent, and losses drop by orders of magnitude. A gradient analysis explains why: derivatives of attention probabilities with respect to query and key scale with the query and key norms, so QK-encoded structure is easily distorted, whereas the mask gradient is bounded by $p_i\\sigma(-m_j)\\leq p_i$ and decouples structural bias from content similarity. The same mask mechanism lifts low-data arithmetic sequence accuracy from 5.01 to 69.49 percent on 3-by-3 multiplication at a 3 percent training fraction and from 1.19 to 77.49 percent on addition at 0.4 percent.","pith_inferences":["Inference: the paper's mechanism predicts a testable scaling pattern — the more a prior can be stated as a sparse interaction graph, the larger the expected gain from mask initialization over QK initialization; on dense, content-dependent tasks the advantage should shrink, which matches the modest NLP improvements reported.","Inference: the gradient argument implies that normalizing query-key feature scales, for example with spectral norm or LayerNorm on the QK logits, should make QK-initialized priors more persistent; if such an intervention closes the gap with mask initialization, the gradient-entanglement explanation would be confirmed rather than some other difference between parameterizations.","Inference: since the paper's own ablation shows an all-zero mask matches vanilla performance, the practical value of the method depends on obtaining interaction priors from data or pretrained models; learning masks from data is the natural next step that the paper leaves open."],"forward_implications":["On the four Boolean GOTU tasks, mask initialization reaches near-perfect extrapolation, with 100 percent accuracy on 2Parity and 3-Bit Majority and 99.98 percent on Cyclic3, while vanilla and both QK-initialized baselines remain at the minimum-degree plateau.","QK-based structured initialization can reproduce the desired attention pattern at initialization but does not reliably change the learned extrapolation rule because the prior is overwritten during training, as shown for both SVD-based and directly optimized QK priors.","Mask-based priors substantially improve low-data arithmetic: on 3-by-3 digit multiplication at a 3 percent training fraction, sequence accuracy rises from 27.82 percent for the best structured baseline to 69.49 percent, and on addition at 0.4 percent from 3.39 to 77.49 percent.","On CIFAR10, CIFAR100, and ImageNet100 with a ViT-T model, mask initialization achieves the best top-1 accuracy among default, mimetic, and impulse initialization, indicating the persistence mechanism transfers to practical vision.","Because gradients of attention probabilities with respect to mask parameters are bounded by $p_i$ and independent of query and key norms, the injected structure updates more slowly and remains identifiable in the attention logits after training, as the paper visualizes for Head 0 at Layer 0."],"supporting_citations":[{"why":"Defines the generalization-on-the-unseen setting, the minimum-degree interpolator, and the four Boolean tasks whose extrapolation failure the paper targets.","marker":"[Abbe et al., 2024]"},{"why":"Supplies the Structured (SVD) query-key initialization baseline and the impulse-filter spatial prior used in the vision experiments.","marker":"[Zheng et al., 2025]"},{"why":"Supplies the Structured (Opt) direct query-key optimization baseline that the paper compares against.","marker":"[Zheng et al., 2024]"},{"why":"Provides the mimetic initialization baseline that initializes attention near an identity pattern.","marker":"[Trockman and Kolter, 2023]"},{"why":"Defines self-attention and the additive attention-mask interface that the proposed log-sigmoid mask modifies.","marker":"[Vaswani et al., 2017]"},{"why":"Shows attention bias can act as an inductive bias for arithmetic, motivating the mask-based restriction of token interactions.","marker":"[Duan et al., 2024]"},{"why":"Grounds the arithmetic task design, showing that Transformers can perform arithmetic with the right embeddings and data setup.","marker":"[McLeish et al., 2024]"}],"fun_headline_variants":["Mask-based priors persist where QK init washes out","Parameterization decides if a prior sticks in Transformers","Attention-logit bias beats QK scores for inductive bias","Mask init hits 100% on Boolean extrapolation, QK stays stuck","Prior persistence: mask wins, QK gets overwritten"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the oracle prior assumption: for every Boolean task the interaction mask is hand-designed from the exact ground-truth function decomposition, including its behavior on the unseen domain, so if a meaningful interaction prior is not known in advance the near-perfect extrapolation and arithmetic gains are not guaranteed to survive.","fun_headline_variants_meta":{"raw":{"variants":["Mask-based priors persist where QK init washes out","Parameterization decides if a prior sticks in Transformers","Attention-logit bias beats QK scores for inductive bias","Mask init hits 100% on Boolean extrapolation, QK stays stuck","Prior persistence: mask wins, QK gets overwritten"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000703,"raw_usage":{"total_tokens":3259,"prompt_tokens":1123,"completion_tokens":2136,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":739,"completion_tokens_details":{"reasoning_tokens":2062}},"tokens_in":739,"tokens_out":2136,"duration_ms":13209,"temperature":1.0,"reasoning_tokens":2062,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:20:11.316537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run 2Parity with a deliberately misspecified mask, placing the same 2Parity pattern on a non-interacting token pair such as x1–x2 instead of x0–x1 while keeping all training settings fixed. If extrapolation accuracy stays near 100 percent, the persistence mechanism rather than the task-aligned pattern is doing the work; if it collapses to the roughly 75 percent minimum-degree level, the oracle interaction prior is the active ingredient.","supporting_citations":[],"review_version":2}