{"id":"d8f41f7d-b384-415a-b1b7-3b079737fe2c","arxiv_id":"2505.09466","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SaPE2 adds a 2D content-dependent position bias to vision transformer attention and reports top-1 accuracy gains on CIFAR-10 and CIFAR-100 over several position-encoding baselines.","lead":"The authors propose SaPE2, a position encoding for vision transformers that creates position information from image content rather than from fixed spatial coordinates. A reader might care because it reports accuracy gains on image classification while claiming better translation robustness, although the evidence is limited to two small datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (10) and Appendix D implement a boundary-relative suffix count, not a relative position; Table 1's gains may not test the claimed mechanism.","rationale":"The reader's weakest assumption identifies the same issue: the position value is a boundary-relative suffix count, not a relative position. I agree. The included code confirms the equation, so this is not a typo. This concern is load-bearing because it goes to the identity of the method: if Eq. (10) does not measure relative position, the conceptual claims (translation equivariance, resolution generalization, semantic relative position encoding) are unsupported even if the accuracy numbers reproduce. However, the empirical result could still be valid as a content-dependent bias, so the appropriate verdict remains CONDITIONAL rather than REJECT: the paper must either correct the definition and claims or show that the corrected relative-position version preserves the gains. No change to the reader's verdict.","tokens_in":12145,"tokens_out":6861,"duration_ms":71115,"concrete_test":"Modify SaPE_unit.forward to compute pos as the gated count between the query index i and key index m along the row/column, e.g., for i <= m, pos[..., i, m] = (cumsum(gates, dim=-1)[..., i, m] - cumsum(gates, dim=-1)[..., i, i-1]), with the symmetric expression for m < i, and keep everything else identical. Retrain SaPE2+APE on CIFAR-10 with the same hyperparameters and compare top-1 to Table 1 (93.98) and to CoPE+APE (92.54). If accuracy stays near 93.98, the boundary-suffix construction is not essential; if it drops toward or below the CoPE baseline, the reported gain depends on the unadvertised suffix-sum implementation rather than on semantic relative positions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is that the position value p^x_im defined in Eq. (10) and implemented in Appendix D is not a relative position between patches i and m. In the code, pos = gates.flip(-1).cumsum(dim=-1).flip(-1) computes, for each query row, p_im = sum_{j=m}^{W} sigma(q_i^T k_j), i.e., the gated mass from the target patch to the row end. This is a boundary-relative count: for a fixed query i, p_im - p_i,m+1 equals the gate at m, and two targets at different distances from i can receive identical encodings whenever their suffix gate sums coincide. The query index i enters only through the gates, not through any interval between i and m, so the encoding does not answer the question stated in the text, 'what is the relative position of the m-th patch from the i-th patch's perspective?' The subsequent bias b^x_in = ||SaPE2_x[i] - SaPE2_x[n]||_2 compares two whole-row boundary-relative vectors, so it is not a function of relative displacement either. The y-axis uses the identical suffix construction. Because the paper's central contribution is a semantic-aware relative position encoding, and its arguments for translation equivariance and resolution generalization depend on relative positions, the construction as written does not support the stated mechanism. The empirical gain in Table 1 may still be real, but it would be evidence for the suffix-count boundary bias, not for the claimed relative-position method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SaPE2, a 2D semantic-aware position encoding for vision transformers. The method decomposes the 2D position bias into two 1D encodings along the x- and y-axes. For each axis, gate values are computed as sigmoid(q_i^T k_j) between patches in the same row or column, and a position value is obtained by summing these gates; the position value is converted to a continuous embedding by interpolating learnable integer position embeddings, and the final attention bias is the Euclidean distance between per-axis position vectors of two patches. Experiments on CIFAR-10 and CIFAR-100 with a ViT-Small backbone report that SaPE2+APE outperforms the compared baselines. The authors claim that the method improves translation equivariance, generalization across resolutions, and aggregation of semantically similar but spatially distant patches.","tokens_in":12460,"tokens_out":5420,"duration_ms":58479,"significance":"If the mechanism were as described, content-dependent 2D position biases would be a useful extension of contextual position encoding to vision. The paper includes a complexity analysis, a core implementation listing in Appendix D, and a case-study visualization, which are helpful for assessing the method. However, the central implementation does not match the stated relative-position mechanism, so the current empirical gains in Table 1, even if reproducible, do not yet support the paper's main claim. The idea may be salvageable after reimplementation of the position computation, but as written the contribution and the experimental evidence need substantial revision.","major_comments":[{"comment":"The quantity p^x_im is not a relative position between patches i and m. As defined, p^x_im = Σ_{j: y_i=y_j=y_m, x_m ≤ x_j} σ(q_i^T k_j), and the code line pos = gates.flip(-1).cumsum(dim=-1).flip(-1) computes exactly this suffix sum to the row end. For a fixed query row, p^x_im depends on the target column m only through the sum of gate values from column m to the row boundary, and it contains no term that counts or measures the interval between x_i and x_m. Two targets at different distances from i can produce identical position values whenever their suffix gate sums coincide, so the construction does not answer the question stated in the text, 'what is the relative position of the m-th patch from the i-th patch's perspective?' Consequently SaPE2_x[i] and the bias b^x_in in Eq. (15) are boundary-relative semantic quantities, not relative displacements. This is a load-bearing mismatch between the claimed mechanism and the implemented one; it affects the y-axis in the same way. Please replace the suffix sum by a sum over the interval between x_i and x_m (in Eq. (10), Eq. (12), and Appendix D) and rerun the experiments, or revise the conceptual claims throughout to describe a boundary-relative gated position encoding rather than a relative-position encoding.","section":"Section 3, Eq. (10) and Appendix D"},{"comment":"The claims of improved translation equivariance and resolution generalization are not supported by the implemented construction. Because p^x_im is anchored at the row end, translating the image content changes the suffix sums both at object boundaries and at the image boundary; the Euclidean distance between two suffix-count vectors is therefore not a translation-invariant or resolution-invariant quantity. The abstract and introduction promise 'generalization across varying image resolutions and scales' and 'improved translation equivariance,' but no experiment directly tests either property. Please provide targeted evaluations, such as shifted-crop classification and training-at-one-resolution-then-testing-at-another, or remove these claims from the contribution list.","section":"Section 1, Section 6, and Eq. (15)"},{"comment":"The gate values that define the position are computed from the same q_i^T k_j dot products that the positional bias subsequently modifies. This self-referential design makes it difficult to attribute the observed accuracy gain to positional information rather than to an additional content-dependent attention pathway that partly duplicates the original attention computation. This concern applies to CoPE as well, but since the paper proposes SaPE2 as a distinct method, please include an ablation that decorrelates the positional signal from the attention signal, for example by computing the gates with stop-gradient on q and k or by using a separate small network for the gates, and report whether the Table 1 gain survives.","section":"Section 3, Eq. (9)"},{"comment":"The headline performance comparison is based on single training runs with no reported number of seeds, no standard deviations, and no error bars. On CIFAR-100 the difference between SaPE2+APE and CoPE+APE is only 0.13 percentage points, which is within the typical run-to-run variation for this type of training setup. Please report mean and standard deviation over at least three independent seeds, and state the full training configuration (optimizer hyperparameters, learning-rate schedule, weight decay, augmentation, and number of attention heads/layers) so that the comparison is reproducible.","section":"Section 5.2, Table 1"}],"minor_comments":[{"comment":"The notation m1, m2, ..., mW is introduced only after the equation; please define that m_r denotes the patch in the same row with x-coordinate r, i.e., x_{m_r} = r.","section":"Section 3, Eq. (12)"},{"comment":"In the SaPE.forward code, the x-axis result is reshaped as (B, H, heads, W, H), which is only valid when W = H. The experiments use 32x32 inputs with patch size 4, so W = H = 8 and the issue is hidden, but the implementation does not support the claimed general 2D case. Please reshape to (B, H, heads, W, W) or otherwise correct the rectangular-image handling.","section":"Appendix D"},{"comment":"There is a typo in the dataset name: 'CIFAR1100' should be 'CIFAR100'.","section":"Section 5.2"},{"comment":"The figure uses inconsistent labels: 'SaPE2_y' appears alongside 'RePE2_y', and the notation b^x_in is written with a superscript x while the text defines b^x_in; please unify the symbols.","section":"Figure 2"},{"comment":"The attention score formula uses the index m for the key patch while Eq. (15) defines the bias for patch n; using two different indices in the same derivation is confusing and should be made consistent.","section":"Section 3, Eq. (16)"}],"recommendation":"major_revision","confidential_remarks":"The suffix-sum issue in Eq. (10) is the central problem: the implemented method is a boundary-relative gated position encoding, not a relative-position encoding as claimed. If the authors change the implementation to an interval sum, the method becomes a fairly direct 2D extension of CoPE, so the novelty assessment may need to be revisited; if they instead keep the current implementation, the paper's main claims about relative position, translation equivariance, and resolution generalization must be substantially weakened. The single-run results in Table 1 would need to be strengthened regardless of the outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes SaPE2, a 2D extension of CoPE for vision. It applies gated counting independently along rows and columns, interpolates learnable embeddings, and uses Euclidean distance between the resulting axis vectors as an attention bias. That combination is genuinely new relative to CoPE and 2D RoPE, and the CIFAR results (93.98 on CIFAR-10, 72.23 on CIFAR-100 with APE) are strong if they hold. What the paper does well: it gives a clean description of the intended mechanism, provides core code in the appendix, and includes a straightforward complexity analysis. The visualization in Figure 4 does show that the learned bias groups semantically similar regions.\n\nThe catch is that the mechanism is not what the text says it is. Eq. (10) defines p^x_im as a sum of gates from the target patch to the row end, with no dependence on the query index over the interval between i and m. The code in Appendix D confirms this: pos = gates.flip(-1).cumsum(dim=-1).flip(-1) computes a suffix sum to the boundary, not a sum between query and key. So the claim that this answers 'what is the relative position of the m-th patch from the i-th patch's perspective' is wrong on its own terms. The resulting bias is still a legitimate content-dependent boundary-relative quantity, and it might even work well, but the paper's central interpretive claim—semantic-aware relative position encoding—is not supported by the equations or code. This looks fixable (use a masked cumsum over the interval), but as written it is an internal contradiction.\n\nThe empirical support is also thin. Single runs, no error bars, and only CIFAR-10/100. The abstract's claims about translation equivariance and resolution generalization are never tested. A 2D CoPE baseline is missing; 'CoPE+APE' is 1D, so the comparison doesn't isolate the 2D decomposition. The self-referential gate design (using the same q-k dot product that defines attention) is not itself a fatal flaw—CoPE works the same way—but it should be discussed more honestly.\n\nIf the authors correct the position definition and add variance and a 2D CoPE baseline, the paper could be a reasonable contribution to semantic position encoding for transformers. In its current form, I would not cite it, but I would send it to review precisely because the flaw is precise and the fix is testable. It deserves a serious referee, not a desk rejection.\n\nBest.","headline":"A plausible semantic-aware PE idea undermined by a mismatch between the claimed relative-position formula and the implemented suffix-sum code.","tokens_in":13007,"tokens_out":4049,"would_cite":false,"duration_ms":38601,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing coordinate-based position encodings with content-gated semantic position counts improves vision transformer accuracy, reaching 93.98% top-1 on CIFAR-10.","keywords":["position encoding","vision transformer","semantic-aware attention","relative position encoding","contextual position encoding","2D position decomposition","image classification","attention bias"],"falsifier":"Train the same ViT-Small configuration with Eq. (10) replaced by a true pairwise relative position, e.g., $p^x_{im}=\\sum_{j=i}^{m} g_{ij}$ for $m\\ge i$ with a symmetric form for $m<i$, keeping everything else fixed. If accuracy does not drop, the suffix-sum definition is not contributing as a relative position; if accuracy drops, the boundary-relative count itself carries the gain. Additionally, with all gates set to 1, Eq. (10) reduces to $p^x_{im}=W-x_m+1$, independent of the query patch $i$, so checking whether the bias changes when $i$ moves while $m$ stays fixed directly tests whether a relative position is being measured.","tokens_in":11922,"feed_emoji":"🖼️","tokens_out":6684,"duration_ms":59511,"temperature":0.7,"pith_summary":"The paper sets out to show that the position encodings vision transformers inherit from language models—absolute coordinates or fixed relative offsets—ignore what patches actually contain, and that this hurts accuracy on images with repeated textures, scattered objects, or shifted subjects. It proposes SaPE2, a 2D position encoding that measures a patch's position by summing content-dependent gate values along its row and column, converts those counts into learned embeddings, and turns the distance between embedding vectors into an attention bias. On CIFAR-10 and CIFAR-100 with a ViT-Small backbone, the method reports 93.98% and 72.23% top-1 accuracy, the best among the compared position encodings including 2D RoPE and CoPE+APE. If this holds, position encoding can be made sensitive to perceptual similarity rather than fixed grid coordinates, with direct consequences for translation robustness and resolution scaling.","feed_headline":"Content-aware encoding lifts ViT to 93.98% on CIFAR-10","feed_subtitle":"SaPE2 swaps fixed grid positions for gate-summed semantic counts, adding 6.57 accuracy points over plain ViT.","key_machinery":"The load-bearing object is the gated suffix-position sum along each image axis. In Eq. (10), the position of patch $m$ relative to patch $i$ along the x-axis is defined as $p^x_{im}=\\sum_{j\\in I_{im}} g_{ij}$ with $I_{im}=\\{j \\mid y_i=y_j=y_m,\\ x_m\\le x_j\\}$, where $g_{ij}=\\sigma(q_i^\\top k_j)$; a continuous position value is then mapped through interpolation of learnable integer embeddings, and the attention bias is the Euclidean distance between the resulting axis-wise position vectors. This construction is what carries the semantic-awareness claim: instead of fixed coordinates, the gate values decide how much each trailing patch counts toward the position, so visually related content can cluster in position space. The same mechanism is applied independently to the y-axis, and the two biases are added into the attention logits.","core_discovery":"On the paper's own terms, the central discovery is that semantic-aware, content-dependent position representations improve vision transformer accuracy over coordinate-only encodings. SaPE2 decomposes the image into x- and y-axes; for each axis it computes a gate value $\\sigma(q_i^\\top k_j)$ for patch pairs sharing that coordinate, defines a position value $p^x_{im}$ as the sum of gates over the suffix from patch $m$ to the row end, interpolates learnable integer position embeddings at that continuous value, and forms an axis-specific position vector by stacking the query's dot products with those embeddings. The Euclidean distance between two patches' axis vectors provides the relative position bias added into attention, and adding the absolute position encoding (APE) on top gives the best results: 93.98% top-1 on CIFAR-10 and 72.23% on CIFAR-100. The authors interpret the attention maps as evidence that patches with similar content receive similar position biases, so the model groups foreground and background consistently.","pith_inferences":["A direct test implied but not run by the paper: replace the suffix-sum definition with a true positional sum between the query and target patches; if the accuracy gain persists, the boundary-relative count is not the active ingredient, and if it disappears, the paper's interpretation of Eq. (10) as a relative position needs revision.","Because $p^x_{im}$ does not depend on the distance between $i$ and $m$ except through the query's gate values, the encoding may be better described as learning a semantic 'distance to the row or column edge' per query rather than a pairwise relative position; this reframing could change how the method is extended to variable image widths.","The reported gains are on 32x32 images; the paper's stated goals of resolution and scale generalization would be testable on ImageNet at multiple input sizes, which the paper does not include.","If the gate-based position is recomputed per query, the position bias is asymmetric and content-dependent, so the method may also serve as a learned attention-prior mechanism independent of whether it is ultimately a position encoding."],"forward_implications":["On both CIFAR-10 and CIFAR-100, SaPE2+APE beats the vanilla ViT APE baseline (93.98 vs 87.41 and 72.23 vs 66.54), so content-based position bias is a direct accuracy lever under this setup.","Applying SaPE2 to the key side (SaPE2(K)+APE) outperforms the query side (93.98% vs 90.17% on CIFAR-10), indicating that enriching key representations with semantic position information is the more effective integration point.","Because the bias is computed as a Euclidean distance between axis-wise position vectors, arbitrary patch pairs across different rows and columns can receive a position bias, not just patches on the same row or column.","The complexity analysis places the bottleneck at O(N^2) pairwise bias storage and computation per axis, so scaling the method to larger images requires reducing that quadratic cost.","The qualitative attention visualization shows smaller SaPE2 distances among patches of the same semantic region (sky, subject, ground), supporting the claim that the encoding groups perceptually similar patches."],"supporting_citations":[{"why":"Defines the ViT architecture and the learnable absolute position encoding baseline that all comparisons modify.","marker":"[2]"},{"why":"Supplies the contextual position encoding mechanism: gate values summed over positions and interpolation between integer embeddings.","marker":"[11]"},{"why":"Provides the 2D RoPE baseline the paper compares against and the axial 2D extension of rotary position embedding.","marker":"[12]"},{"why":"Establishes relative position representations as additive biases in self-attention, the mechanism SaPE2 extends to semantic content.","marker":"[8]"},{"why":"Provides the transformer attention formulation and sinusoidal absolute position encoding that are the starting point of the work.","marker":"[1]"},{"why":"Introduces rotary position embedding as the 1D relative-position baseline that 2D RoPE is compared with.","marker":"[10]"}],"fun_headline_variants":["Semantic-aware position encoding boosts ViT to 93.98% on CIFAR-10","Content-driven positions lift ViT accuracy to 93.98% on CIFAR-10","SaPE2: semantic position encoding adds 6.57 points to ViT","ViT reaches 93.98% on CIFAR-10 with semantic-aware positions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the suffix sum in Eq. (10)—the accumulated gates from the target patch to the end of its row or column—is a valid measure of the relative position between two patches; if that boundary-relative count is not a true relative position, the semantic-aware mechanism the paper describes is not what the experiments actually measure.","fun_headline_variants_meta":{"raw":{"variants":["Semantic-aware position encoding boosts ViT to 93.98% on CIFAR-10","Content-driven positions lift ViT accuracy to 93.98% on CIFAR-10","SaPE2: semantic position encoding adds 6.57 points to ViT","ViT reaches 93.98% on CIFAR-10 with semantic-aware positions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000341,"raw_usage":{"total_tokens":1889,"prompt_tokens":966,"completion_tokens":923,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":839}},"tokens_in":582,"tokens_out":923,"duration_ms":7183,"temperature":1.0,"reasoning_tokens":839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:31:12.908699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same ViT-Small configuration with Eq. (10) replaced by a true pairwise relative position, e.g., $p^x_{im}=\\sum_{j=i}^{m} g_{ij}$ for $m\\ge i$ with a symmetric form for $m<i$, keeping everything else fixed. If accuracy does not drop, the suffix-sum definition is not contributing as a relative position; if accuracy drops, the boundary-relative count itself carries the gain. Additionally, with all gates set to 1, Eq. (10) reduces to $p^x_{im}=W-x_m+1$, independent of the query patch $i$, so checking whether the bias changes when $i$ moves while $m$ stays fixed directly tests whether a relative position is being measured.","supporting_citations":[{"cited_title":"Rotary position embedding for vision transformer","cited_arxiv_id":null,"evidence_quote":"Provides the 2D RoPE baseline the paper compares against and the axial 2D extension of rotary position embedding."},{"cited_title":"Gomez, Łukasz Kaiser, and Illia Polosukhin","cited_arxiv_id":null,"evidence_quote":"Provides the transformer attention formulation and sinusoidal absolute position encoding that are the starting point of the work."}],"review_version":1}